Skip to content

fix(e2e): repair test bugs, trim browser matrix, add failure alert#383

Merged
JesperDramsch merged 2 commits into
mainfrom
fix/e2e-trim-and-fix
Jun 27, 2026
Merged

fix(e2e): repair test bugs, trim browser matrix, add failure alert#383
JesperDramsch merged 2 commits into
mainfrom
fix/e2e-trim-and-fix

Conversation

@JesperDramsch

Copy link
Copy Markdown
Owner

Per the decision to fix + trim + keep non-blocking, this revives the Playwright E2E suite that had been red on every nightly run for ~6 months and silently ignored.

The failures were test bugs, not site bugs

1. Over-strict timezone assertion (countdown-timers.spec.js)

expect(timezone).toMatch(/^([A-Z][a-z]+\/[A-Z][a-z]+|UTC[+-]\d+)$/);

The template emits real IANA zones like America/Los_Angeles / America/New_York (underscores, multi-segment) and the UTC-12 AoE default — all rejected by that regex, so the test failed deterministically on real data. Replaced with Intl.DateTimeFormat-based validation that also accepts the UTC[±N] pseudo-zone. Unit-checked against real IANA zones, UTC offsets, and garbage.

2. Mobile multiselect click timeouts (conference-filters.spec.js)

bootstrap-multiselect option labels resolve but aren't "actionable" on the Pixel 5 viewport, timing out at 10s. Use click({ force: true }) for those 3 option clicks (the test.skip guards for missing options are unchanged).

Maintenance changes

  • Trim CI matrix 5 → 2 browsers: chromium + mobile-chrome. firefox/webkit/mobile-safari produced most of the historical cross-browser flakiness for little signal on a JS-light Jekyll site. They remain available via workflow_dispatch and the local Playwright config.
  • Add notify-failure job: opens/updates a tracking issue when an unattended run (nightly cron or push to main) fails — so a red suite can't rot unnoticed for months again.
  • Stays non-blocking: no pull_request trigger; data/conference PRs are never gated on it.

Verification

  • playwright test --list parses the config and all 5 specs cleanly (90 chromium tests).
  • Timezone validation logic unit-tested.
  • ⚠️ The full browser run requires the Jekyll server + browser binaries — best validated by triggering the workflow (workflow_dispatch) once merged, or via the next nightly.

Follow-up (separate, optional)

The specs are written defensively with many test.skip() guards, so a "pass" can mean "selector not found, skipped." Worth tightening assertions later if the suite proves valuable.

…alert

The nightly E2E suite had been red for ~6 months and was silently ignored.
The failures were test bugs, not site bugs:

- countdown-timers: the data-timezone assertion used an over-strict regex
  (/^([A-Z][a-z]+\/[A-Z][a-z]+|UTC[+-]\d+)$/) that rejected real IANA zones
  emitted by the template — America/Los_Angeles, America/New_York (underscores,
  multi-segment) and bare UTC. Replace it with Intl.DateTimeFormat-based
  validation that also accepts the site's "UTC-12" (Anywhere-on-Earth) pseudo-zone.

- conference-filters: bootstrap-multiselect option clicks timed out on the
  Pixel 5 (mobile-chrome) viewport because the elements resolve but aren't
  "actionable". Use click({ force: true }) for those option clicks.

Also, per maintenance review:
- Trim the CI browser matrix from 5 to chromium + mobile-chrome (firefox,
  webkit and mobile-safari produced most of the historical cross-browser
  flakiness for little signal on a JS-light Jekyll site). Other browsers
  remain available via workflow_dispatch and local playwright config.
- Add a notify-failure job that opens/updates a tracking issue when the
  unattended (cron/push) run fails, so a red suite can't rot unnoticed again.

The suite stays non-blocking (no pull_request gate on data PRs).
Validated: playwright test --list parses config + all specs; timezone
validation unit-checked against real IANA/UTC/garbage inputs.
@JesperDramsch JesperDramsch merged commit a7222b4 into main Jun 27, 2026
9 of 10 checks passed
@JesperDramsch JesperDramsch deleted the fix/e2e-trim-and-fix branch June 27, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant