Skip to content

[core] event-log-race-repro cannot classify a corruption: errorMessage is empty on world-vercel #3418

Description

@VaguelySerious

event-log-race-repro records outcome: CORRUPTED_EVENT_LOG for a failing run but not which divergence it hit, so the job cannot tell a fresh regression from the known residual.

Why the field is empty

The harness reads the message from the run record:

errorMessage: failure.error?.message,

On world-vercel world.runs.get() returns the plaintext errorCode with error rehydrated as a bare Error carrying no message, so errorMessage is undefined for every corrupted run. Sample: run 31421949094, 7/14 corrupted, errorMessage absent on all 7 entries of the uploaded artifact.

The ReplayDivergenceError text ("step event step_created for belongs to X, but the current step consumer is Y", plus the diverged/recovery-replay counts) exists only in the deployment's runtime logs.

Why it matters

Triage of this job depends on the signature, not the count. A 14-run job cannot separate two branches on rate alone: re-running one job on an identical commit and deployment has moved the count 4 -> 7. What distinguishes a real regression from the standing residual is the shape of the divergence, e.g. correlation numbers going backwards mid-log (two concurrent replays numbering from different log lengths) versus a specific consumer inversion. Without the message, every failing run is an unclassifiable tally mark, and any fix PR measured on this job is unfalsifiable.

Suggested fix

The e2e lanes already solved this: .github/scripts/fetch-e2e-runtime-logs.mjs pulls historical runtime logs from GET https://vercel.com/api/logs/request-logs, which accepts search=<workflowRunId>. The repro job does not call it.

Wire it in after the repro step, scoped to the failing runIds from the results file (never a bulk dump: the shared deployments produce thousands of rows per 10 minutes), and either fold the first divergence line into each result entry or upload it alongside the artifact. Then the sticky comment can group failures by signature rather than only counting them.

Second, smaller option that does not depend on log access: have the workflow itself surface the divergence detail in a way that survives the run record, so runs.get() carries it.

Context: #3406 is a fix whose local world-postgres measurement (8/18 -> 0/18) cannot be checked against this job for exactly this reason.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions