Skip to content

Fix dag parsing persistence when bundle cache is missing after processor restart#69972

Open
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/dags-api-empty-after-dag-processor-restart-69249
Open

Fix dag parsing persistence when bundle cache is missing after processor restart#69972
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:fix/dags-api-empty-after-dag-processor-restart-69249

Conversation

@GayathriSrividya

@GayathriSrividya GayathriSrividya commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

closes #69249

What this PR changes

  • make dag parsing persistence robust when the in-memory bundle version cache has no entry for a bundle
  • use a safe lookup in DagFileProcessorManager.handle_parsing_result so persistence still proceeds with bundle_version=None
  • add a regression test that verifies parsing results are persisted when the bundle version cache is missing

Why this fixes the issue

After a dag-processor restart, parsing can proceed before _bundle_versions is populated for a bundle. The previous direct index access raised KeyError, which short-circuited persistence and prevented normal DAG metadata updates. Falling back to None keeps persistence moving and allows stale DAG state to be refreshed instead of remaining stuck.

Tests

  • /opt/homebrew/bin/ruff format airflow-core/src/airflow/dag_processing/manager.py airflow-core/tests/unit/dag_processing/test_manager.py
  • /opt/homebrew/bin/ruff check --fix airflow-core/src/airflow/dag_processing/manager.py airflow-core/tests/unit/dag_processing/test_manager.py
  • uv run --project airflow-core pytest airflow-core/tests/unit/dag_processing/test_manager.py -k "bundle_version_cache_is_missing or updates_stats_after_successful_persist or throttles_retry_when_first_persist_fails" -xvs (fails during collection in this environment with ModuleNotFoundError: airflow.providers.hashicorp before running tests)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GET /api/v2/dags returns empty list after dag-processor restart despite CLI showing DAGs correctly

1 participant