Filter unreleased phantom versions from registry build

[kaxil]

Summary

dev/registry/extract_metadata.py took the top entry of provider.yaml's versions: list as a provider's "latest" version with no verification that a real release tag exists. Two concrete failure modes shipped phantom pointers to the live registry:

• providers/celery/provider.yaml lists 3.19.0 at the top, but only providers-celery/3.19.0rc1 and rc2 tags exist — no final. The registry would advertise celery 3.19.0 as latest while pip install apache-airflow-providers-celery resolves to 3.18.0.
• providers/akeyless/ is brand-new in-tree with versions: [1.0.0] but no providers-akeyless/* tag. The provider would appear on the registry with broken outbound links to its non-existent PyPI release, GitHub tag, and docs page.

Design rationale

One-shot tag load: load_release_tags() runs git tag --list 'providers-*' once and returns a set, so per-provider lookups are O(1). Wave tags (providers/<YYYY-MM-DD>) don't match the glob (different prefix); only per-provider release tags are loaded.

Walk newest-first: find_latest_released_version walks versions: in document order and returns the first entry with a matching tag. All current provider.yaml files are newest-first.

Filter the list, not just the latest pointer: filters both Provider.version (singular, the latest) AND Provider.versions (the list). The latter is read by extract_versions.py's backfill, which would otherwise try to git show from non-existent phantom tags.

Skip vs. fallback: when no entry in versions: has a tag (truly unreleased provider), the provider is skipped from providers.json rather than emitted with version="0.0.0". The old 0.0.0 fallback shipped a registry page with broken outbound links — strictly worse than the page not existing.

--allow-unreleased opt-out for staging: maintainers preview newly-bumped versions on staging before tagging. The new flag bypasses the filter for staging dispatches and local dev. Default (no flag) = filter, which is correct for live. The registry-build.yml workflow auto-sets the flag when destination=staging.

Workflow change: registry-build.yml's checkout sets fetch-tags: true. The default fetch-depth: 1 checkout has no tags, which would silently return an empty set from load_release_tags() and trigger the same versions[0] fallback the rest of this PR is removing. registry-backfill.yml's primary checkout already uses fetch-depth: 0, so tags are present there.

Behaviour change

For destination=live (default):

• Previously: provider with no tagged version → emitted with version="0.0.0" and broken links.
• Now: provider with no tagged version → skipped from the registry entirely, with a per-provider "skipping" log line and an end-of-run summary.

For destination=staging:

• No behaviour change — workflow passes --allow-unreleased, filter is bypassed, all providers including unreleased ones are emitted as before. Maintainers can preview on staging before tagging.

For breeze registry extract-data invoked locally without --allow-unreleased: same as live. Pass --allow-unreleased for local previewing.

Inspecting current main HEAD: exactly two providers fall into the new "skip" category for live — akeyless (brand-new, never released) and any provider where the latest pointer falls back (celery 3.19.0 → 3.18.0).

Gotchas

• Run on a machine without git installed → load_release_tags() returns an empty set, falls back to the previous behaviour with a one-line warning. Acceptable for local dev; production runs through CI which always has git.
• A bogus tag (e.g. providers-amazon/99.99.0 exists but provider.yaml doesn't exist at that commit) would still pass the existence check. Defending against that requires a git show validation per tag and is left as follow-up — current tag hygiene is good enough that no real provider has this issue today.

Known follow-ups (not in scope here)

• extract_parameters.py runs against provider.yaml directly and doesn't yet honour the skip signal — an unreleased provider's modules can still leak into modules.json. Tracked separately.

[potiuk]

Hmm. We should likely release celery ? possibly even automatically ?

[github-actions]

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 38d8d41 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

[vatsrahul1001]

Manual backport #66902