Fix DAG-level on_failure_callback not firing by Sathvik-Chowdary-Veerapaneni · Pull Request #63692 · apache/airflow

Sathvik-Chowdary-Veerapaneni · 2026-03-16T04:31:22Z

Fixed DAG-level on_failure_callback not firing.

When the scheduler builds a DagCallbackRequest, DagRunContext needs to read DagRun ORM relationship data. In the reported failure path, SQLAlchemy can raise more than DetachedInstanceError, which prevented the callback request from being produced.

Changes:

Catch SQLAlchemyError while building DagRunContext and reload the DagRun from the DB before collecting context.
Keep the callback request using full server-provided context.
Keep scheduler callback logging at debug.
Avoid extra DAG processor bundle checks; callback fetching is already scoped by bundle.

Tests:

airflow-core/tests/unit/models/test_dagrun.py::TestDagRun::test_dagrun_update_state_with_handle_callback_failure
airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_fetch_callbacks_ignores_other_bundles

closes #63374

boring-cyborg · 2026-03-16T04:31:27Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Previously only DetachedInstanceError was caught when accessing consumed_asset_events on ORM DagRun objects. Other SQLAlchemy exceptions (e.g. InvalidRequestError) crashed the scheduler. closes: apache#63374

DagRunContext creation could crash when ORM relationship access failed, preventing the callback from being produced entirely. The callback is now sent with minimal context on failure.

uranusjr · 2026-04-15T06:54:44Z

Is this related?

uranusjr · 2026-04-15T06:55:09Z

Do we want this in info? Wondering if debug is enough.

uranusjr · 2026-04-15T06:55:40Z

+            except Exception:
+                self.log.exception(
+                    "Failed to build DagRunContext for dag_id=%s run_id=%s; "
+                    "sending callback with minimal context",
+                    self.dag_id,
+                    self.run_id,
+                )
+                context_from_server = None


When does this ever happen?

Even if it happens, this is bad: running callback without the full context is worse than the callback failing. Also, the exception is broad here

ephraimbuddy · 2026-04-15T07:43:10Z

+                        "this DAG processor (serving bundles: %s). Skipping.",
+                        getattr(req, "dag_id", "unknown"),
+                        req.bundle_name,
+                        bundle_names,


Any callback fetched at this point is bound to have bundle_name in bundle_names. Debugging here?

ephraimbuddy · 2026-04-15T07:45:35Z

+            except Exception:
+                self.log.exception(
+                    "Failed to build DagRunContext for dag_id=%s run_id=%s; "
+                    "sending callback with minimal context",
+                    self.dag_id,
+                    self.run_id,
+                )
+                context_from_server = None


Even if it happens, this is bad: running callback without the full context is worse than the callback failing. Also, the exception is broad here

vatsrahul1001 · 2026-05-12T10:57:47Z

@Sathvik-Chowdary-Veerapaneni can you address open comments?

- Removed duplicate bundle-name guard after scoped callback fetch

- Updated scheduler callback request log from info to debug

- Removed broad fallback around DagRunContext creation

- Restored test name and comments for existing bundle filtering behavior

- Removed regression test for dropped minimal-context callback fallback

Sathvik-Chowdary-Veerapaneni · 2026-05-13T16:59:15Z

Thanks for the reminder. I pushed updates addressing the open review comments by narrowing the PR back to the core DAG callback fix:

Removed the extra bundle-name guard in DagFileProcessorManager; fetched callbacks are already scoped by the bundle query.
Changed the scheduler callback request log from info to debug.
Removed the broad fallback around DagRunContext; callbacks now require the full server context instead of being sent with partial/minimal context.
Removed the fallback-specific regression test.

I also updated the PR description to remove the stale fallback/logging notes.

Verified locally in a clean worktree:

airflow-core/tests/unit/models/test_dagrun.py::TestDagRun::test_dagrun_update_state_with_handle_callback_failure
airflow-core/tests/unit/dag_processing/test_manager.py::TestDagFileProcessorManager::test_fetch_callbacks_ignores_other_bundles

Both passed.

Sathvik-Chowdary-Veerapaneni requested review from XD-DENG, ashb, ephraimbuddy and jedcunningham as code owners March 16, 2026 04:31

boring-cyborg Bot added area:DAG-processing area:Scheduler including HA (high availability) scheduler labels Mar 16, 2026

uranusjr approved these changes Mar 16, 2026

View reviewed changes

Sathvik-Chowdary-Veerapaneni added 7 commits March 27, 2026 18:31

Broadened exception handling in DagRunContext validator

9f7626f

Previously only DetachedInstanceError was caught when accessing consumed_asset_events on ORM DagRun objects. Other SQLAlchemy exceptions (e.g. InvalidRequestError) crashed the scheduler. closes: apache#63374

Made produce_dag_callback resilient to DagRunContext failures

2eaa1f9

DagRunContext creation could crash when ORM relationship access failed, preventing the callback from being produced entirely. The callback is now sent with minimal context on failure.

Added warning log for callbacks with mismatched bundle_name

057b126

Added info log when DAG callback is sent to processor

1ece90c

Added tests for DAG callback resilience and bundle mismatch warning

a2d4cf0

Added newsfragment for apache#63374

bbcec76

Renamed newsfragment to match PR number

dfd11a3

Sathvik-Chowdary-Veerapaneni force-pushed the dag-failure-callback-not-firing-63374 branch from f49ef66 to dfd11a3 Compare March 27, 2026 22:32

eladkal added this to the Airflow 3.2.0 milestone Mar 27, 2026

eladkal added the type:bug-fix Changelog: Bug Fixes label Mar 27, 2026

eladkal requested a review from vatsrahul1001 March 27, 2026 22:33

uranusjr reviewed Mar 30, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/callbacks/callback_requests.py Outdated

vatsrahul1001 modified the milestones: Airflow 3.2.0, Airflow 3.2.1 Mar 30, 2026

potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Apr 2, 2026

Narrow DagRunContext exception catch to SQLAlchemyError

9a0183e

eladkal added the backport-to-v3-2-test Mark PR with this label to backport to v3-2-test branch label Apr 3, 2026

jscheffl reviewed Apr 3, 2026

View reviewed changes

Comment thread airflow-core/newsfragments/63692.bugfix.rst Outdated

Remove newsfragment per reviewer request

e790b9b

uranusjr reviewed Apr 15, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/dag_processing/manager.py

Copy link
Copy Markdown

Member

uranusjr Apr 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this related?

uranusjr reviewed Apr 15, 2026

View reviewed changes

ephraimbuddy reviewed Apr 15, 2026

View reviewed changes

vatsrahul1001 modified the milestones: Airflow 3.2.1, Airflow 3.2.2 Apr 15, 2026

Sathvik-Chowdary-Veerapaneni added 5 commits May 13, 2026 12:56

Removed redundant callback bundle guard

d427ae6

- Removed duplicate bundle-name guard after scoped callback fetch

Changed callback request log level

fd432e3

- Updated scheduler callback request log from info to debug

Removed partial DAG callback context fallback

29274fd

- Removed broad fallback around DagRunContext creation

Updated callback bundle filter test wording

3afca9d

- Restored test name and comments for existing bundle filtering behavior

Removed partial context fallback test

20480f7

- Removed regression test for dropped minimal-context callback fallback

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix DAG-level on_failure_callback not firing#63692

Fix DAG-level on_failure_callback not firing#63692
Sathvik-Chowdary-Veerapaneni wants to merge 14 commits into
apache:mainfrom
Sathvik-Chowdary-Veerapaneni:dag-failure-callback-not-firing-63374

Sathvik-Chowdary-Veerapaneni commented Mar 16, 2026 •

edited

Loading

Uh oh!

boring-cyborg Bot commented Mar 16, 2026

Uh oh!

Uh oh!

Uh oh!

uranusjr Apr 15, 2026

Uh oh!

uranusjr Apr 15, 2026

Uh oh!

uranusjr Apr 15, 2026

Uh oh!

ephraimbuddy Apr 15, 2026

Uh oh!

ephraimbuddy Apr 15, 2026

Uh oh!

ephraimbuddy Apr 15, 2026

Uh oh!

vatsrahul1001 commented May 12, 2026

Uh oh!

Sathvik-Chowdary-Veerapaneni commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

Sathvik-Chowdary-Veerapaneni commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boring-cyborg Bot commented Mar 16, 2026

Uh oh!

Uh oh!

Uh oh!

uranusjr Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

uranusjr Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

uranusjr Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

ephraimbuddy Apr 15, 2026

Choose a reason for hiding this comment

Uh oh!

vatsrahul1001 commented May 12, 2026

Uh oh!

Sathvik-Chowdary-Veerapaneni commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Sathvik-Chowdary-Veerapaneni commented Mar 16, 2026 •

edited

Loading