Flaky VideoBlock JS tests

[kdmccormick]

Several VideoBlock karma tests seem to be flakey.

By my estimate, one of any of these fails around 10% of the time.

1) player state was changed
     Video HTML5Video non-hls encoding on non-Touch devices  events: [play]
     Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. in http://localhost:9876/absolute/home/runner/work/xblocks-core/xblocks-core/node_modules/jasmine-core/lib/jasmine-core/jasmine.js?da99c5b057693d025fad3d7685e1590600ca376d (line 3938)
QueueRunner.prototype.run/attemptAsync/timeoutId<@http://localhost:9876/absolute/home/runner/work/xblocks-core/xblocks-core/node_modules/jasmine-core/lib/jasmine-core/jasmine.js?da99c5b057693d025fad3d7685e1590600ca376d:3938:23
2) callback was called
     Video HTML5Video non-hls encoding on non-Touch devices  events: [play]
     Expected [ 3, 1 ] to contain -1.
../tests/js/html5_video_spec.js/</</describeInfo</</</</<@http://localhost:9876/base/public/js/commons.js?70e765de13f56a983143d5c15cc43f401e110fe8:65273:72
Deferred/then/</</<@http://localhost:9876/base/assets/js/src/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3366:33
fire@http://localhost:9876/base/assets/js/src/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3187:31
fireWith@http://localhost:9876/base/assets/js/src/vendor/jquery.js?26e6b63b81813d8ad942c90d369df2673602b812:3317:7     Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. in http://localhost:9876/absolute/home/runner/work/xblocks-core/xblocks-core/node_modules/jasmine-core/lib/jasmine-core/jasmine.js?da99c5b057693d025fad3d7685e1590600ca376d (line 3938)
QueueRunner.prototype.run/attemptAsync/timeoutId<@http://localhost:9876/absolute/home/runner/work/xblocks-core/xblocks-core/node_modules/jasmine-core/lib/jasmine-core/jasmine.js?da99c5b057693d025fad3d7685e1590600ca376d:3938:23
3) callback was called
     Video HTML5Video hls encoding on non-Touch devices  events: [play]
     Error: Timeout - Async callback was not invoked within timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL. in http://localhost:9876/absolute/home/runner/work/xblocks-core/xblocks-core/node_modules/jasmine-core/lib/jasmine-core/jasmine.js?da99c5b057693d025fad3d7685e1590600ca376d (line 3938)
QueueRunner.prototype.run/attemptAsync/timeoutId<@http://localhost:9876/absolute/home/runner/work/xblocks-core/xblocks-core/node_modules/jasmine-core/lib/jasmine-core/jasmine.js?da99c5b057693d025fad3d7685e1590600ca376d:3938:23

[kdmccormick]

@openedx/axim-aximprovements No action needed right now, but I just wanted to share that I've been noticing that there is flakiness in a few of the VideoBlock JS tests. Curious if any of you guys hit this when developing, too. We can consider fixing or removing them in the future if they prove to be disruptive.

[kdmccormick]

summary from Claude Opus. Claude is pretty confident about this, but I'm not exactly sure.

---

Flaky video JS tests: waitUntil timeouts blame the wrong spec

Symptom

Occasional failures in the video-xblock Karma/Jasmine suite
(src/xblock_video), most often on specs named player state was changed
under Video HTML5Video … events: [play] | [pause] | …, reported as:

Failed: Timeout has expired

Characteristics that make it confusing:

• The failing spec name varies run to run ([play] one time, [pause]
  the next, sometimes an hls vs non-hls variant).
• It sometimes lands on a synchronous spec that has no waitUntil and
  therefore cannot itself time out.
• It fails more frequently in CI than on a local machine with a real,
  focused browser.

Root cause

The shared helper tests/spec_helpers/jasmine-waituntil.js has two properties
that combine badly:

1. It polls with the real, unmocked setTimeout, so its timer keeps
   running across spec boundaries.
2. On timeout it calls the global fail(), which fails whichever spec is
   currently executing.

So when a waitUntil started in spec A never satisfies its condition, its
timer keeps ticking after A finishes and, ~jasmine.DEFAULT_TIMEOUT_INTERVAL
later, calls fail() during some unrelated later spec B. B gets blamed — even
if B is synchronous. The reported spec name is essentially "whoever was running
when the leaked timer expired," which is why it wanders.

The trigger is that many of these specs wait on real, timing-dependent
HTML5 media events — especially reaching STATUS.PLAYING, which needs the
browser's real playing event on a <video> actually decoding the fixture.
Headless CI (no autoplay/user-gesture, no window focus) makes those events fire
late or never, so the waits time out and the leaked fail() fires. A local run
with a real focused browser reaches those states more reliably, hence the lower
local failure rate. Anything that perturbs event-loop/focus timing (e.g. a
browser modal) shifts the failure rate too.

Suggested fixes (in preference order)

1. Harden the helper. Track pending waitUntil timers and clear them in a
   global afterEach so no timer survives its spec. Keep failing the spec on a
   genuinely unmet condition — but attribute it to the spec that owns the wait,
   not a bystander. This is a small change to shared infra and removes the
   cross-spec mislabeling that makes the flake so hard to read.

2. Make individual specs deterministic. Where a spec asserts a state, have
   its waitUntil poll for that state rather than a looser condition, and/or
   fire synthetic media events instead of relying on real playback. Example
   already applied: the [pause] "player state was changed" spec now waits for
   === STATUS.PAUSED (the asserted state) instead of !== STATUS.PLAYING
   (which BUFFERING also satisfies), closing that particular race.

Fix (1) addresses the mechanism; fix (2) reduces how often it's triggered.

Reproducing / debugging

• --grep does not filter on this toolchain (karma-jasmine 0.3.8 +
  jasmine-core 2.6.4 ignore the adapter's specFilter). Use Jasmine focus
  (fit / fdescribe) to narrow to a spec, and/or comment out spec imports in
  assets/karma.webpack.entry.js to shrink the bundle.
• Because failures are mislabeled, isolating a single spec can hide the flake
  (the real offender is a different, earlier spec). Run a representative
  subset repeatedly and watch whether the named failing spec changes — that
  variance is itself the tell for cross-spec contamination.