Open
Conversation
> >The build was failing due to a dependency conflict between tensorflow-metadata and protobuf. This commit updates the protobuf version in the WORKSPACE file to the version required by tensorflow-metadata, which resolves the build failure.
> >Updated the version constraints for TFX-related dependencies in nightly_test_constraints.txt to align with the requirements of the TFX nightly build. This resolves the dependency conflicts previously encountered during package installation.
> >Updated the TensorFlow dependency constraint in tfx/dependencies.py from '>=2.16.0,<2.17' to '>=2.16.0,<2.18'. This change allows TFX to be built with TensorFlow 2.17.1, resolving the previous dependency conflict.
> >Updated the TensorFlow version in test_constraints.txt from 2.16.2 to 2.17.1. This resolves a dependency conflict caused by different TensorFlow versions being specified in test_constraints.txt and nightly_test_constraints.txt.
….txt > >Reverted the TFX-related dependency constraints in nightly_test_constraints.txt back to the 1.17.x versions. This is to resolve a build failure caused by an inconsistency between the TFX package's default dependencies and the previously updated nightly constraints.
> >Updated the tensorflow-data-validation version constraint in test_constraints.txt from '>=1.17.1' to '>=1.17.0'. This resolves a dependency conflict with the TFX package's default dependencies, allowing the build to succeed.
> >Pinned the tensorflow-text dependency to version 2.17.0 in both nightly_test_constraints.txt and test_constraints.txt. This explicitly sets the version to resolve persistent dependency conflicts.
> >Pinned the tensorflow-text dependency to version 2.17.0 in nightly_test_constraints.txt, test_constraints.txt, and tfx/tools/docker/requirements.txt. This explicitly sets the version to resolve persistent dependency conflicts across various build and test environments.
> >Updated the pinned version of protobuf from 3.20.3 to 4.25.6 in nightly_test_constraints.txt, test_constraints.txt, and tfx/tools/docker/requirements.txt. This resolves a significant dependency conflict between ml-metadata, apache-beam, and other google-cloud libraries.
> >Pinned the tensorboard dependency to version 2.17.1 in nightly_test_constraints.txt, test_constraints.txt, and tfx/tools/docker/requirements.txt. This explicitly sets the version to resolve persistent dependency conflicts with TensorFlow 2.17.1.
> >Downgraded the pinned version of apache-beam from 2.59.0 to 2.50.0 in all constraint files. This resolves a dependency conflict with tensorflow-data-validation, which requires an earlier version of apache-beam.
> >Pinned the numpy dependency to version 1.24.4 in all constraint files. This resolves a dependency conflict between tfx-dev, scikit-learn, and apache-beam, ensuring compatibility across all packages.
> >Downgraded the pinned version of objsize from 0.7.0 to 0.6.1 in all constraint files. This resolves a dependency conflict with apache-beam, which requires an earlier version of objsize.
> >Pinned the protobuf dependency to version 4.23.4 in all constraint files. This resolves a dependency conflict with apache-beam, which requires protobuf<4.24.0.
> >Pinned the google-auth-httplib2 dependency to version 0.1.1 in all constraint files. This resolves a dependency conflict with apache-beam[gcp], which requires an earlier version of google-auth-httplib2.
> >Upgraded the pinned version of tensorflow-decision-forests from 1.9.2 to 1.10.1 in all constraint files. This resolves a dependency conflict with tensorflow, as version 1.10.1 is compatible with TensorFlow 2.17.1.
- Pin setuptools to 70.0.0 in CI tests - Downgrade Bazel from 7.2.1 to 6.5.0 in wheel builds
Add http_archive rules for: - rules_apple 0.34.1 - apple_support 1.11.1 - rules_swift 3eeeb53c - grpc 1.46.3 These are needed to support gRPC compilation on macOS.
Replace deprecated py_proto_library with custom rule implementation. Changes: - Define custom ProtoDescriptorInfo provider - Implement _py_proto_library_impl for proto compilation - Add support for gRPC Python stub generation - Create individual file aliases for backward compatibility - Update distribution_validator_proto dependency reference
- Pass workspace root to Bazel proto generation via --define - Set TFX_ROOT environment variable for build context - Add patch file support to tfx_github_archive rule - Apply tensorflow_metadata proto v0 patch
Change nightly version specs from dev releases to stable releases for libraries which do not have nightly release available: - ml-metadata: >=1.18.0.dev -> >=1.17.0 - tensorflow-data-validation: >=1.18.0.dev -> >=1.17.0 - tensorflow-model-analysis: >=0.49.0.dev -> >=0.48.0 - tensorflow-transform: >=1.18.0.dev -> >=1.17.0 - tfx-bsl: >=1.18.0.dev -> >=1.17.1 - struct2tensor: >=0.49.0.dev -> >=0.48.0 Fix trailing whitespace in tflite_support section.
Add recursive normalization function to clear field options and json_name for nested message types to handle environment differences in proto descriptors.
Replace string-based assertEqual comparisons with assertProtoEquals for proper proto message comparison. Parse expected artifacts and artifact types using text_format.Parse for clearer test code. Fixes test brittleness from string representation changes.
Refactor Proto Build System and Update Dependencies
…ency conflicts - Pin setuptools==70 across all pyproject.toml files to prevent version backtracking - Create patches/tfx.patch to remove TFX ecosystem packages from pip resolution - Pre-download 5 specific wheels (tfdv, tfx-bsl, tfma, tft, tf-cloud) in build script - Add build_constraints.txt and use PIP_CONSTRAINT in Docker stages - Implement automatic patch application/cleanup in build_docker_image.sh Fixes Docker builds failing with "resolution-too-deep" and missing manylinux_2_39 distributions on gcr.io/deeplearning-platform-release/tf2-cpu.2-17-1.py310
Fix Docker Build Dependency Resolution Issues
…e-built wheels Build tensorflow-data-validation and tfx-bsl from source within the Docker image rather than downloading pre-built manylinux_2_39 wheels. This is necessary because repairing wheels to manylinux_2_17 compatibility fails due to ABI incompatibilities between the pre-built wheel binaries and the Docker container's runtime environment. Changes: - Install Miniforge3 and create conda environment with Python 3.10 - Set up conda-based GCC 13 toolchain with compatible binutils - Add build scripts for TFDV (v1.17.0) and tfx-bsl (v1.17.1) - Apply patches to fix zetasql dependency issues - Install Bazel 6.5.0 for building from source - Remove pre-built wheel downloads from build_docker_image.sh The conda environment ensures consistent compiler toolchain (GCC 13, binutils 2.40) to avoid linker errors and ABI mismatches. Both packages are built with USE_BAZEL_VERSION=6.5.0 and output wheels directly into the Docker build context. This approach provides better compatibility and eliminates the need for wheel repair utilities like auditwheel, which cannot resolve fundamental ABI differences between build environments.
Build TFDV and tfx-bsl from source in Docker to avoid ABI mismatches
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.