[ZEPPELIN-6717] Install locked Python dependencies in Linux CI

[miinhho]

What is this PR for?

Linux CI jobs repeatedly resolve the same Python 3.9 Conda dependencies during environment setup. This PR makes those jobs install from explicit Linux x86-64 Conda specifications instead, and pins the PyFlink packages installed with pip.

It also separates Maven, npm, Conda, and pip package caches by dependency type. The Livy job now caches the Spark distribution at the path used by its download script. Lock update and validation instructions are documented in testing/locks/README.md.

This targets dependency setup time.

What type of PR is it?

Improvement

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-6717

How should this be tested?

• Run the affected Linux CI jobs and confirm they install from the committed Conda specifications without dependency resolution.
• Confirm both PyFlink jobs install their pinned requirements and pass.
• Run CI again to check package-cache reuse and the Livy Spark distribution cache hit.
• Compare environment-setup durations separately from test durations.

Questions

• License files: No.
• Breaking changes for older versions: No product-runtime change. The explicit Conda specifications are Linux x86-64-specific.
• Documentation: Lock maintenance instructions are included.

[miinhho]

While working on this PR, I noticed that core.yml repeats several setup steps across jobs. I put together a follow-up commit that uses YAML anchors to reduce that duplication without changing the expanded job configuration.

I plan to propose it as a separate PR after this one is merged, so this PR can stay focused on dependency setup time.

[voidmatcha]

I confirmed that this PR substantially improves dependency setup time. As a small follow-up, I put together an example change that includes the Spark and Hadoop versions in the cache key and restore prefix, so a future version change cannot keep hitting the previous cache key. Please take a look.

[voidmatcha]

LGTM 👍

[voidmatcha]

Merged into master (33f1fff).

[pan3793]

I confirmed that this PR substantially improves dependency setup time.

I wonder how fast it gains. TBH, I feel this will become stale over time

[pan3793]

and one advantage of not using lock file, is that we can observe the library's upstream breaking changes quickly

[voidmatcha]

@pan3793 Thanks for raising this. Comparing the same 15 CI jobs before and after the merge, median environment setup time dropped from 7m00s to 1m02s, about an 85% reduction. This includes the added cache restore and PyFlink installation steps. These are historical CI observations, not a controlled benchmark.

I agree with the concerns about stale locks and delayed detection of upstream breaking changes. @miinhho, could you look into a follow-up approach, such as scheduled compatibility checks with freshly resolved dependencies and a lock refresh process?

[miinhho]

Thanks for sharing the timing comparison. Building on your suggestion, perhaps we could combine periodic lock refreshes with compatibility checks: resolve fresh dependencies and validate them before updating the locks. That could help keep the locks current and catch upstream issues while retaining the setup-time savings.

What do you think? If that sounds good, I’ll look into this approach for a follow-up.

[voidmatcha]

Sounds good, please go ahead with that approach. Resolving fresh dependencies and validating them before refreshing the locks should address both the staleness and the delayed upstream-breakage detection.

[miinhho]

Thanks! I opened ZEPPELIN-6739 to track this follow-up and will start working on it.

[pan3793]

another issue found: developers may use macOS with Apple Silicon, while the README instruction says it should use Ubuntu 26.04 with x86_64 processor to generate the conda lock file, I managed to use Docker to simulate x86_64 platform and run it in an Ubuntu container when working on #5465 to achieve this.

[voidmatcha]

Thank you for looking at this so carefully. I overlooked the Apple Silicon workflow when reviewing #5492. The README currently documents lock generation on Ubuntu 24.04 x86-64, but not how to do it from an Apple Silicon Mac.

@miinhho, when updating the maintenance documentation for ZEPPELIN-6739, could you consider including an example of generating and validating the locks in a Docker linux/amd64 environment? This is not a requirement for the scheduled refresh itself, so if it does not fit that issue, we can track it as a small documentation follow-up.

[miinhho]

@voidmatcha Agreed, documenting how to generate and validate the locks using Docker linux/amd64 would address the immediate gap for contributors.

Longer term, I think we can simplify the process so contributors do not need to prepare a Docker environment just to generate locks. I looked at pandas’ pixi.toml: it defines the test environments and target platforms, while Pixi generates the committed pixi.lock. Pixi resolves those declared platforms without first installing the target environments.

For Zeppelin, we could define our Python test environments in the same way, allowing contributors to update the definitions and run pixi lock locally instead of creating an Ubuntu x86-64 environment and manually exporting the package lists.

This would also help with the periodic refresh work in ZEPPELIN-6739. pandas has a weekly workflow that runs pixi update and opens a lock-update PR. We could use the same approach for scheduled refreshes, with CI validating the generated changes while regular runs continue installing locked dependencies.