Fix #60763: Set bundle_name non-nullable via migration for legacy DAGs upgraded from 2.x

[kalluripradeep]

Fixes #60763

What changed

Replaced the query-based fix with a proper database migration, based on feedback from @SameerMesiah97, @uranusjr, and @Nataneljpwd.

Problem

When upgrading from Airflow 2.x to 3.0.x, DAGs retain bundle_name = NULL. The deactivation logic in deactivate_stale_dags() queries by bundle_name.in_(bundle_names), which excludes NULL values — so removed legacy DAGs are never deactivated.

Solution

New migration 0101 that:

1. Sets bundle_name = 'dags-folder' for all rows where bundle_name IS NULL
2. Makes bundle_name non-nullable going forward

[SameerMesiah97]

It’s great that you identified the issue but I am not sure if this is the right fix.

[SameerMesiah97]

You narrowed down the root cause (missing bundle name for 2.x) but I am uncomfortable with widening the query like this. This would be treating bundle-less DAGs as if they belong to any/all bundles, which works fine in most scenarios. But what about situations where bundle resolution fails? Would it not be masking the bug? I understand that it would still be a silent error here but non-deactivation of stale bundle-less DAGs would at least give us a breadcrumb to follow.

I think it might be better to apply the same migration to 2.x upgrades (i.e. set NULL to ‘dags-folder’ for bundle_name) as 3.0.x to 3.1.

[kalluripradeep]

Thanks for the thoughtful feedback, @SameerMesiah97!

You raise a valid concern about potentially masking bundle resolution failures. I can see two approaches:

Option 1: Migration-based (your suggestion)
Set NULL bundle_name → 'dags-folder' during upgrade, similar to migration 0082 in 3.1.0. This would be cleaner long-term.

Option 2: Query-based (current approach)
Include NULL in deactivation query. Simpler for 3.0.x users who don't have migration 0082 yet.

I'm happy to implement Option 1 if you think that's the better path forward. Would this require:

1. A new migration for 3.0.x branch?
2. Or should we document this as a known issue with workaround until users upgrade to 3.1+?

Let me know which direction you prefer and I'll update the PR accordingly!

[SameerMesiah97]

In my opinion, option 1 would be better in the long-run. If we go that route, it would be better to create a new migration for 3.0.x.

But we should let a maintainer/committer weigh in first.

[kalluripradeep]

In my opinion, option 1 would be better in the long-run. If we go that route, it would be better to create a new migration for 3.0.x.

But we should let a maintainer/committer weigh in first.

@jedcunningham @ephraimbuddy @potiuk Any Leads?

[uranusjr]

There’s one potential advantage to keeping null and change the runtime query; the dags-folder bundle name can be changed, and setting all null to that string could be confusing if the user changed that bundle name prior to migration.

However, this is admittedly a very edgy edge case, and even if it happens, it’s only a one-time issue that can be fixed either by a new bundle parse (for up-to-date dags) and a manual database update (for stale ones, which shouldn’t matter either in the first place).

I don’t have a strong opinion either way, but if you’re going with option 1 (using migrations), please also change the database schema so the bundle_name column is non-nullable to reflect it.

[kalluripradeep]

There’s one potential advantage to keeping null and change the runtime query; the dags-folder bundle name can be changed, and setting all null to that string could be confusing if the user changed that bundle name prior to migration.

However, this is admittedly a very edgy edge case, and even if it happens, it’s only a one-time issue that can be fixed either by a new bundle parse (for up-to-date dags) and a manual database update (for stale ones, which shouldn’t matter either in the first place).

I don’t have a strong opinion either way, but if you’re going with option 1 (using migrations), please also change the database schema so the bundle_name column is non-nullable to reflect it.

Thanks for weighing in, @uranusjr!

I appreciate both perspectives. Given that:

1. @SameerMesiah97 prefers Option 1 (migration) for long-term cleanliness
2. You suggest making bundle_name non-nullable if we go that route
3. The edge case you mentioned is manageable

I'll proceed with Option 1: Create a migration for 3.0.x to set NULL → 'dags-folder', and update the schema to make bundle_name non-nullable.

I'll update the PR with the migration approach. Should this migration target the 3.0.x branch specifically, or main with a backport label?

[kalluripradeep]

@uranusjr any leads please

[Nataneljpwd]

I am not sure that this approach is correct, maybe instead it should be a db migrations script rather than a change in the query?

[kalluripradeep]

Updated the PR based on feedback from @SameerMesiah97, @uranusjr, and @Nataneljpwd.

Replaced the query change with migration 0101 which sets bundle_name = 'dags-folder' for legacy NULL rows and makes the column non-nullable — same approach as migration 0082 in 3.1.0.

The original manager.py query is untouched.

@uranusjr — targeting main for now. Please let me know if this should instead target the v3-0 branch directly, or if a backport label is preferred.

[Nataneljpwd]

Looks good, but this will only fix an upgrade from 2.x to 3.2, I wonder if this can be backported

[kalluripradeep]

@Nataneljpwd — v3-1-test already has migration 0082_3_1_0_make_bundle_name_not_nullable.py which covers this fix for 3.1.x users. This PR targets main (3.2.x) for users upgrading directly from 2.x to 3.2 who would miss that migration.

[Nataneljpwd]

@Nataneljpwd — v3-1-test already has migration 0082_3_1_0_make_bundle_name_not_nullable.py which covers this fix for 3.1.x users. This PR targets main (3.2.x) for users upgrading directly from 2.x to 3.2 who would miss that migration.

As long as it is made clear, great!

[Nataneljpwd]

Looks good to me!
Thank you for the fix!