Fix S3ToGCSOperator deferrable mode to return list of copied files#63533
Fix S3ToGCSOperator deferrable mode to return list of copied files#63533yuseok89 wants to merge 3 commits intoapache:mainfrom
Conversation
SameerMesiah97
left a comment
There was a problem hiding this comment.
Looks good. Just one nit if you want to implement it.
providers/google/src/airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated
Show resolved
Hide resolved
|
@yuseok89 This PR has a few issues that need to be addressed before it can be reviewed — please see our Pull Request quality criteria. Issues found:
What to do next:
There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack. |
c762874 to
eeb1839
Compare
a54e0ac to
5ed2630
Compare
|
@potiuk |
This PR addresses functionality originally proposed in #49768. Since that PR has been stalled, and #11323 is effectively blocked by it, I've proceeded with this implementation in the hope of unblocking progress.
If the original author of #49768 returns and continues his work, I can close this PR so that his proceeds instead. If not, would it be acceptable to merge this PR so that deferrable S3→GCS transfers can return the list of copied files and downstream tasks can consume them via XCom?
S3ToGCSOperatorin deferrable mode now returns the list of copied file paths (same as non-deferrable mode), so downstream tasks can consume them via XCom. This fixes the previous inconsistency wheredeferrable=Truedid not return any value.Verified in testing: both deferrable=True and deferrable=False return file names, not URIs. Returning URIs instead could be addressed in a separate PR if desired.
Screenshots
Deferrable state
XCom
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.