|
1 | | -# Hermetic Python |
| 1 | +## Managing hermetic Python |
2 | 2 |
|
3 | | -Hermetic Python allows not to rely on system-installed Python, and |
4 | | -system-installed Python packages. \ |
5 | | -Instead, an independent Python toolchain is registered, ensuring the right |
6 | | -dependencies are always used. \ |
7 | | -See https://github.com/bazelbuild/rules_python/ for more details. |
| 3 | +To make sure that TensorFlow's build is reproducible, behaves uniformly across |
| 4 | +supported platforms (Linux, Windows, MacOS) and is properly isolated from |
| 5 | +specifics of a local system, we rely on hermetic Python (see |
| 6 | +[rules_python](https://github.com/bazelbuild/rules_python)) for all build |
| 7 | +and test commands executed via Bazel. This means that your system Python |
| 8 | +installation will be ignored during the build and Python interpreter itself |
| 9 | +as well as all the Python dependencies will be managed by bazel directly. |
8 | 10 |
|
9 | | -### Specifying the Python version |
| 11 | +### Specifying Python version |
10 | 12 |
|
11 | | -Note: Only a number of minor Python versions are supported at any given time. |
| 13 | +The hermetic Python version is controlled by `HERMETIC_PYTHON_VERSION` |
| 14 | +environment variable, which could be setin one of the following ways: |
12 | 15 |
|
13 | | -By default, the lowest supported version is used. |
14 | | - |
15 | | -To set a different version, use the `TF_PYTHON_VERSION` environment variable, |
16 | | -e.g. |
17 | | - |
18 | | -``` |
19 | | -export TF_PYTHON_VERSION=3.11 |
20 | 16 | ``` |
| 17 | +# Either add an entry to your `.bazelrc` file |
| 18 | +build --repo_env=HERMETIC_PYTHON_VERSION=3.12 |
21 | 19 |
|
22 | | -To specify the version via a Bazel command argument, use the following: |
| 20 | +# OR pass it directly to your specific build command |
| 21 | +bazel build <target> --repo_env=HERMETIC_PYTHON_VERSION=3.12 |
23 | 22 |
|
| 23 | +# OR set the environment variable globally in your shell: |
| 24 | +export HERMETIC_PYTHON_VERSION=3.12 |
24 | 25 | ``` |
25 | | ---repo_env=TF_PYTHON_VERSION=3.11 |
26 | | -``` |
27 | | - |
28 | | -## Requirements updater |
29 | 26 |
|
30 | | -Requirements updater is a standalone tool, intended to simplify process of |
31 | | -updating requirements for multiple minor versions of Python. |
| 27 | +You may run builds and tests against different versions of Python sequentially |
| 28 | +on the same machine by simply switching the value of `HERMETIC_PYTHON_VERSION` |
| 29 | +between the runs. All the python-agnostic parts of the build cache from the |
| 30 | +previous build will be preserved and reused for the subsequent builds. |
32 | 31 |
|
33 | | -It takes in a file with a set of dependencies, and produces a more detailed |
34 | | -requirements file for each version, with hashes specified for each |
35 | | -dependency required, as well as their sub-dependencies. |
| 32 | +### Specifying Python dependencies |
36 | 33 |
|
37 | | -### How to update/add requirements |
| 34 | +During bazel build all TensorFlow's Python dependencies are pinned to their |
| 35 | +specific versions. This is necessary to ensure reproducibility of the build. |
| 36 | +The pinned versions of the full transitive closure of TensorFlow's dependencies |
| 37 | +together with their corresponding hashes are specified in |
| 38 | +`requirements_lock_<python version>.txt` files (e.g. |
| 39 | +`requirements_lock_3_12.txt` for `Python 3.12`). |
38 | 40 |
|
39 | | -By default, the name of the base requirements file is `requirements.in`, but it |
40 | | -can be set using the `REQUIREMENTS_FILE_NAME` variable. \ |
41 | | -For example: |
| 41 | +To update the lock files, make sure |
| 42 | +`ci/official/requirements_updater/requirements.in` contains the desired direct |
| 43 | +dependencies list and then execute the following command (which will call |
| 44 | +[pip-compile](https://pypi.org/project/pip-tools/) under the hood): |
42 | 45 |
|
43 | 46 | ``` |
44 | | -export REQUIREMENTS_FILE_NAME=my_requirements.in |
| 47 | +bazel run //ci/official/requirements_updater:requirements.update --repo_env=HERMETIC_PYTHON_VERSION=3.12 |
45 | 48 | ``` |
46 | 49 |
|
47 | | -To specify the file via a Bazel command argument, use the following: |
| 50 | +where `3.12` is the `Python` version you wish to update. |
| 51 | + |
| 52 | +Note, since it is still `pip` and `pip-compile` tools used under the hood, so |
| 53 | +most of the command line arguments and features supported by those tools will be |
| 54 | +acknowledged by the Bazel requirements updater command as well. For example, if |
| 55 | +you wish the updater to consider pre-release versions simply pass `--pre` |
| 56 | +argument to the bazel command: |
48 | 57 |
|
49 | 58 | ``` |
50 | | ---repo_env=REQUIREMENTS_FILE_NAME=my_requirements.in |
| 59 | +bazel run //ci/official/requirements_updater:requirements.update --repo_env=HERMETIC_PYTHON_VERSION=3.12 -- --pre |
51 | 60 | ``` |
52 | 61 |
|
53 | | -### How to run the updater |
| 62 | +If you need to upgrade all of the packages in requirements lock file, just pass |
| 63 | +the `--upgrade` parameter: |
54 | 64 |
|
55 | 65 | ``` |
56 | | -bash updater.sh |
| 66 | +bazel run //ci/official/requirements_updater:requirements.update --repo_env=HERMETIC_PYTHON_VERSION=3.12 -- --upgrade |
57 | 67 | ``` |
58 | 68 |
|
59 | | -## How to add a new Python version |
60 | | - |
61 | | -Note: Updating the |
62 | | -[rules-python](https://github.com/bazelbuild/rules_python/releases) version may |
63 | | -be required before going through the steps below. This is due to the new Python |
64 | | -versions becoming available through `rules-python`. \ |
65 | | -See |
66 | | -[here](https://github.com/tensorflow/tensorflow/commit/f91457f258fdd78f693044a57efa63a38335d1de), |
67 | | -and |
68 | | -[here](https://github.com/tensorflow/tensorflow/commit/052445e04ce20fd747657e0198a1bcec2b6dff5b), |
69 | | -for an example. |
70 | | - |
71 | | -See |
72 | | -[this commit](https://github.com/tensorflow/tensorflow/commit/5f7f05a80aac9b01325a78ec3fcff0dbedb1cc23) |
73 | | -as a rough example of the steps below. |
74 | | - |
75 | | -All the files referenced below are located in the same directory as this README, |
76 | | -unless indicated otherwise. |
77 | | - |
78 | | -1) Add the new version to the `VERSIONS` variable inside |
79 | | - `tensorflow/tools/toolchains/python/python_repo.bzl`. \ |
80 | | - While this isn't necessary for running the updater, it is required for |
81 | | - actually using the new version with Tensorflow. |
82 | | - |
83 | | -2) In the `WORKSPACE` file, add the new version to the `python_versions` |
84 | | - parameter of the `python_register_multi_toolchains` function. |
85 | | - |
86 | | -3) In the `BUILD.bazel` file, add a load statement for the new version, e.g. |
87 | | - |
88 | | - ``` |
89 | | - load("@python//3.11:defs.bzl", |
90 | | - compile_pip_requirements_3_11 = "compile_pip_requirements") |
91 | | - ``` |
92 | | - |
93 | | - Add a new entry for the loaded `compile_pip_requirements`, e.g. |
94 | | - |
95 | | - ``` |
96 | | - compile_pip_requirements_3_11( |
97 | | - name = "requirements_3_11", |
98 | | - extra_args = ["--allow-unsafe"], |
99 | | - requirements_in = "requirements.in", |
100 | | - requirements_txt = "requirements_lock_3_11.txt", |
101 | | - ) |
102 | | - ``` |
103 | | - |
104 | | - ``` |
105 | | - compile_pip_requirements_3_11( |
106 | | - name = "requirements_3_11_release", |
107 | | - extra_args = [ |
108 | | - "--allow-unsafe", |
109 | | - "-P keras-nightly", |
110 | | - "-P tb-nightly", |
111 | | - ], |
112 | | - requirements_in = "requirements.in", |
113 | | - requirements_txt = "requirements_lock_3_11.txt", |
114 | | - ) |
115 | | - ``` |
116 | | - |
117 | | -4) Add the version to `SUPPORTED_VERSIONS` in `updater.sh`, and |
118 | | - `release_updater.sh` |
119 | | - |
120 | | -5) Run the `updater.sh` shell script. \ |
121 | | - If the base requirements file hasn't yet been updated to account for the new |
122 | | - Python version, which will require different versions for at least some |
123 | | - dependencies, it will need to be updated now, for the script to run |
124 | | - successfully. |
125 | | - |
126 | | -6) A new `requirements_lock_3_11.txt` file should appear under the root of the |
127 | | - `tensorflow` directory. |
| 69 | +For the full set of supported parameters please check |
| 70 | +[pip-compile](https://pip-tools.readthedocs.io/en/latest/cli/pip-compile/) |
| 71 | +documentation |
0 commit comments