Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Oct 27, 2025

What changes were proposed in this pull request?

This PR aims to update Spark Connect-generated Swift source code with Apache Spark 4.1.0-preview3 RC1.

Why are the changes needed?

There are many changes between Apache Spark 4.1.0-preview2 and preview3.

To use the latest bug fixes and new messages to develop for new features of 4.1.0-preview3.

$ git clone -b v4.1.0-preview3 https://github.com/apache/spark.git
$ cd spark/sql/connect/common/src/main/protobuf/
$ protoc --swift_out=. spark/connect/*.proto
$ protoc --grpc-swift_out=. spark/connect/*.proto

// Remove empty GRPC files
$ cd spark/connect
$ grep 'This file contained no services' * | awk -F: '{print $1}' | xargs rm

Does this PR introduce any user-facing change?

Pass the CIs.

How was this patch tested?

Pass the CIs. I manually tested with Apache Spark 4.1.0-preview3 (with the two SDP ignored tests).

$ swift test --no-parallel
...
✔ Test run with 203 tests in 21 suites passed after 19.088 seconds.

### Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun dongjoon-hyun marked this pull request as draft October 27, 2025 20:18
viirya
viirya previously approved these changes Oct 27, 2025
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Oct 27, 2025

Thank you, @viirya . There are many breaking changes during Spark Data Pipeline development. For that, I matched the compilation only for now because SDP will be updated again before the official 4.1.0. I'll revisit soon to catch up the developed feature semantics.

@dongjoon-hyun dongjoon-hyun marked this pull request as ready for review October 27, 2025 20:41
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Oct 27, 2025

This is manually tested with Apache Spark 4.1.0-preview3 (with the two SDP ignored tests).

$ swift test --no-parallel
...
✔ Test run with 203 tests in 21 suites passed after 19.088 seconds.

@dongjoon-hyun
Copy link
Member Author

All tests passed. Could you review this, @viirya and @peter-toth ?

@dongjoon-hyun
Copy link
Member Author

Thank you so much, @viirya . Merged to main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants