Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run dag-processor separate from scheduler in local dev #1788

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

BasPH
Copy link
Contributor

@BasPH BasPH commented Jan 27, 2025

Description

In advance of Airflow 3, where the DAG processor will always run as a separate process, I suggest running the DAG processor separately from the scheduler in the local dev setup.

Noteworthy details:

  • I didn't add a way to configure this on/off, since the option to run the DAG processor separately was added in Airflow 2.3.0. Since the oldest supported Airflow that Astro supports is Airflow 2.7, I assume the option to separate the DAG processor is available to all users.
  • New commands:
    • astro dev logs -d/--dag-processor
    • astro dev bash -d/--dag-processor

Go is a bit out of my comfort zone and I'm unfamiliar with the repository setup, so I'd appreciate a thorough review.

🎟 Issue(s)

Related #XXX

🧪 Functional Testing

List the functional testing steps to confirm this feature or fix.

📸 Screenshots

Add screenshots to illustrate the validity of these changes.

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

@BasPH BasPH marked this pull request as draft January 28, 2025 12:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant