Fix deadline callback data encoding in migration 0094 (scheduler CrashLoopBackOff)

[seanmuth]

Closes #69980.

What's the problem

Migration 0094_3_2_0_replace_deadline_inline_callback_with_fkey moves inline deadline callbacks into the callback table, whose data column is ExtendedJSON — read back at runtime via BaseSerialization.deserialize, which requires every nested dict to be wrapped as {"__type": "dict", "__var": {...}}.

Both upgrade paths built callback.data without recursively extended-serializing the callback kwargs:

• Postgres (_upgrade_postgresql): json_build_object wrapped only the top level, embedding the old callback's kwargs jsonb raw.
• MySQL/SQLite (_upgrade_mysql_sqlite): wrote the callback dict via a plain JSON column with no extended wrapping at all.

So any callback whose kwargs contain a nested dict (e.g. metric tags) is stored unreadable. When the scheduler loads it in the deadline-processing loop, BaseSerialization.deserialize recurses into the outer dict node, hits the bare nested object, runs encoded_var[Encoding.VAR], and raises KeyError(<Encoding.VAR: '__var'>) — crashing the scheduler on every loop (CrashLoopBackOff), before it heartbeats, with no OOM and no other logged exception.

What's in this PR

• Both upgrade paths now extended-serialize the callback data, matching BaseSerialization.serialize:
  • Postgres: a recursive, session-local pg_temp.encode_extended(jsonb) function used inside the existing batch CTE (keeps the one-round-trip design).
  • MySQL/SQLite: a _serialize_extended() helper applied to the callback dict.
• Both downgrade paths decode kwargs back to raw (pg_temp.decode_extended / _deserialize_extended), so an upgrade→downgrade round-trip is preserved. Both decoders are lenient, so they also correctly downgrade data written by the pre-fix version of this migration.
• Tests (test_0094_deadline_callback_migration.py): existing assertions now verify callback.data round-trips through the real BaseSerialization; added a nested-kwargs/tags regression test and an encode/decode helper round-trip test.

Already-affected deployments

This hotfix protects deployments that have not yet run 0094. Deployments already upgraded across 0094 with nested-dict callback kwargs have latent corrupt callback.data rows (a scheduler crash that fires when such a deadline becomes due); those rows must be repaired out of band — e.g. re-encode the affected rows:

-- find affected rows
SELECT count(*) FROM callback c
WHERE jsonb_path_exists(c.data::jsonb, '$.**.__var.* ? (@.type()=="object" && !exists(@.__type))');

---

🤖 Opened with Claude Code (model: Claude Opus 4.8, claude-opus-4-8[1m]).