Error when `pip.parse.experimental_index_url` is a private index

[dougthor42]

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

No. From what I can tell, it's always been present since experimental_index_url was added.

Description

Something wonky goes on when using a private index as the default pip index url. It seems like the credential helper is not working, but that may be a red herring.

🔬 Minimal Reproduction

This is fine:

pip.parse(
    experimental_extra_index_urls = ["https://[REDACTED]/simple"],
    experimental_index_url = "https://pypi.org/simple",
    hub_name = "pypi",
    python_version = "3.11",
    requirements_lock = "//:requirements.txt",
)

This fails:

pip.parse(
    experimental_extra_index_urls = ["https://pypi.org/simple"],
    experimental_index_url = "https://[REDACTED]/simple",
    hub_name = "pypi",
    python_version = "3.11",
    requirements_lock = "//:requirements.txt",
)

🔥 Exception or Error

ERROR: Traceback (most recent call last):                                                                              
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 472, column 52, in _pip_impl                                                                                                                                                                                                                                                                                                
                is_hub_reproducible = _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides, hub_group_map, simpleapi_cache)                                                                                                                                                                                                                                                                                                                                                     
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 187, column 50, in _create_whl_repos                                                                                                                                                                                                                                                                                        
                requirements_by_platform = parse_requirements(                                                                                                                                                                                 
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 348, column 37, in parse_requirements                                                                                                                                                                                                                                                                               
                whls, sdist = _add_dists(                                                                              
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 452, column 24, in _add_dists                                                                                                                                                                                                                                                                                       
                logger.warn("Could not find a whl or an sdist with sha256={}".format(sha256))                                                                                                                                                  
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 78, column 39, in lambda                                                                                                                                                                                                                                                                                                    
                warn = lambda message_cb: _log(0, "WARNING", message_cb),                                                                                                                                                                      
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 72, column 71, in _log                                                                                                                                                                                                                                                                                                      
                print("\nrules_python: {}: ".format(level.upper()), message_cb())  # buildifier: disable=print                                                                                                                                 
Error: 'string' object is not callable                                                                                 
ERROR: error evaluating module extension pip in @@rules_python~//python/extensions:pip.bzl                                                                                                                                                     
INFO: Elapsed time: 8.798s                                                                                             
INFO: 0 processes.                                                                                                     
ERROR: Build did NOT complete successfully                                                                             
Loading: 0 packages loaded                                                                                             
    currently loading:                                                                                                 
    Fetching module extension pip in @@rules_python~//python/extensions:pip.bzl; starting 8s

🌍 Your Environment

Operating System:

gLinux

Output of bazel version:

$ bazel --version
bazel 7.2.0rc1

We're using 7.2.0rc1 because of bazelbuild/bazel#22664 which is actually a bug with jupyter/testpath that I haven't made a pip.override for yet.

Rules_python version:

Confirmed with:

0.33.1
0.33.2

Anything else relevant?

• We use a credential helper to access the private index.
• A different error happens with the latest rules_python commit at the time of writing (084b877):

  ERROR: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~'
  ERROR: /usr/local/google/home/dthor/dev/pyle/BUILD.bazel:81:24: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' and referenced by '//:gazelle_python_manifest.test'
  ERROR: Analysis of target '//:gazelle_python_manifest.test' failed; build aborted: Analysis failed
  INFO: Elapsed time: 9.454s
  INFO: 0 processes.
  ERROR: Build did NOT complete successfully
  

  • This error happens for both "public pypi is main index" and "private index is main index" cases.

[aignas]

@dougthor42, could you see if main works differently? #2017 should help with debugging at least.

[dougthor42]

could you see if main works differently?

Sorry I didn't put that front-and-center - it's hidden at the bottom of "Anything else relevant". When running on main, both cases result in:

ERROR: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~'
ERROR: /usr/local/google/home/dthor/dev/pyle/BUILD.bazel:81:24: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' and referenced by '//:gazelle_python_manifest.test'
ERROR: Analysis of target '//:gazelle_python_manifest.test' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.454s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

[aignas]

This does look like the gazelle plugin got removed during testing with main. It does not make sense that overriding rules_python would cause this. Unless you overrode both, gazelle and rules_python and we have a much bigger issue with the plugin. I am slightly confused. On the original topic though, the behaviour is different if your private index has some packages that have the same name as the public index, but the sha256 values in the private index are not the ones that your lock file is requesting. This is my only idea why you would see any difference in behaviour.

…

On 4 July 2024 08:27:40 GMT+09:00, Douglas Thor ***@***.***> wrote: > could you see if main works differently? Sorry I didn't put that front-and-center - it's hidden at the bottom of "Anything else relevant". When running on `main`, _both_ cases result in: ``` ERROR: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as ***@***.***_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' ERROR: /usr/local/google/home/dthor/dev/pyle/BUILD.bazel:81:24: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as ***@***.***_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' and referenced by '//:gazelle_python_manifest.test' ERROR: Analysis of target '//:gazelle_python_manifest.test' failed; build aborted: Analysis failed INFO: Elapsed time: 9.454s INFO: 0 processes. ERROR: Build did NOT complete successfully ``` -- Reply to this email directly or view it on GitHub: #2033 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[dougthor42]

I did play it fast and loose with the git_override when I was testing main. It's possible I f'd something up haha. We've got a long weekend coming up so you might not hear back until July 8th/9th.

[aignas]

Thanks, see you next week then. :)

…

On 4 July 2024 08:51:43 GMT+09:00, Douglas Thor ***@***.***> wrote: I _did_ play it fast and loose with the `git_override` when I was testing `main`. It's possible I f'd something up haha. We've got a long weekend coming up so you might not hear back until July 8th/9th. -- Reply to this email directly or view it on GitHub: #2033 (comment) You are receiving this because you commented. Message ID: ***@***.***>

[dougthor42]

OK here's more info:

TL;DR: It looks like commit b4b52fc (#1917) is the culprit, and it was fixed in 11133b3 (#2017).

It looks like (a) my git_override was mucked up - I'll have to check that later - and that (b) I didn't actually check main when I thought that I did. Sorry about that!

The rest of this comment is kept for my own sanity and reference. Feel free to ignore.

Procedure

To test, I chose a semi-random target and would run tests:

bazel test //src/pyle_xc/layout/scripts/drc:run_drc_test --cache_test_results=no

I cloned rules_python and used local_path_override to point my project to it. I then checked 0.33.1 through 0.33.2 but the issue persisted (meaning it wasn't already fixed in anything that's released).

I also went back to 0.32.0, when experimental_(extra_)?index_urls? were added and found that things worked. I also checked main (7bba79d at the time of writing) and that did work.

So I git bisected backwards to find the issue and forward to find the fix.

Lastly, because I only realized at the end that I forgot to bazel clean --expunge, I did so and then tested the commit before the break, the breaking commit, the commit before the fix, and the fixing commit.

Results Table:

rules_python version	experimental _index_url	experimental _extra_index_urls	Result
0.32.0	pypi	private	OK
0.32.0	private	pypi	OK
b4b52fc	pypi	private	OK
b4b52fc	private	pypi	Error: 'string' object is not callable
0.33.1 (our current version)	pypi	private	OK
0.33.1 (our current version)	private	pypi	Error: 'string' object is not callable
0.33.2	pypi	private	OK
0.33.2	private	pypi	Error: 'string' object is not callable
7bba79d (main)	pypi	private	OK
7bba79d (main)	private	pypi	OK

How I patched:

# in MODULE.bazel
local_path_override(
    module_name = "rules_python",
    path = "/usr/local/google/home/dthor/dev/rules_python",
)
local_path_override(
    module_name = "rules_python_gazelle_plugin",
    path = "/usr/local/google/home/dthor/dev/rules_python/gazelle",
)

`Error: 'string' object is not callable'

ERROR: Traceback (most recent call last):
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 472, column 52, in _pip_impl
                is_hub_reproducible = _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides, hub_group_map, simpleapi_cache)
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 187, column 50, in _create_whl_repos
                requirements_by_platform = parse_requirements(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 348, column 37, in parse_requirements
                whls, sdist = _add_dists(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 452, column 24, in _add_dists
                logger.warn("Could not find a whl or an sdist with sha256={}".format(sha256))
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 78, column 39, in lambda
                warn = lambda message_cb: _log(0, "WARNING", message_cb),
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 72, column 71, in _log
                print("\nrules_python: {}: ".format(level.upper()), message_cb())  # buildifier: disable=print
Error: 'string' object is not callable

[dougthor42]

(a) my git_override was mucked up - I'll have to check that later

It was correct, but #2057 was preventing it from working correctly. 🙃

[aignas]

Thanks for the thorough investigation and the new ticket!