Airflow can not open path in zipped DAG folder #59310
Description
Apache Airflow version
3.1.4
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
The bug in the following issue was never solved:
Jinja2 templating does not work and one can not change the template_searchpath such that the (for example) sql file is recognized.
There is just a workaround using the open_maybe_zipped utils function and using it to open the file manually before giving the path to the Operator, like this:
from airflow.utils.file import open_maybe_zipped
open_maybe_zipped("/home/airflow/gcs/dags/test.zip/test_sql/test.sql").read()`
What you think should happen instead?
No response
How to reproduce
In /home/airflow/gcs/dags/test.zip/test_sql/test.sql:
SELECT column_a FROM test
In /home/airflow/gcs/dags/test.zip/test.py:
from airflow import DAG
from airflow.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperator
from airflow.utils.file import open_maybe_zipped
# Define DAG
with DAG(
dag_id="test_import_path_dag",
schedule_interval=None,
default_args = {"retries": 0}
) as dag:
test = BigQueryInsertJobOperator(
task_id="test_task",
configuration={
"query": {
"query": "test_sql/test.sql" , # PATH HERE IS NOT FOUND WHEN DAGS ARE ZIPPED
"useLegacySql": False,
"destinationTable": {
"projectId": "test",
"datasetId": "test",
"tableId": "test",
},
}
},
)
test
Operating System
Linux
Versions of Apache Airflow Providers
No response
Deployment
Google Cloud Composer
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct