Skip to content

refactor: order generated packages in npm_translate_lock #2176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

jbedard
Copy link
Member

@jbedard jbedard commented Apr 20, 2025

To reduce make the generated bzl more deterministic.

Changes are visible to end-users: no

Test plan

  • Covered by existing test cases

@jbedard jbedard requested a review from dzbarsky April 20, 2025 00:00
Copy link

aspect-workflows bot commented Apr 20, 2025

Test

2 test targets passed

Targets
//npm/private:_test_gendocs_0_0 [k8-fastbuild] 31ms
//npm/private:_test_gendocs_0_1 [k8-fastbuild] 37ms

Total test execution time was 68ms. 223 tests (99.1%) were fully cached saving 33s.


Test

e2e/bzlmod

All tests were cache hits

5 tests (100.0%) were fully cached saving 502ms.


Test

e2e/gyp_no_install_script

All tests were cache hits

2 tests (100.0%) were fully cached saving 151ms.


Test

e2e/js_image_oci

All tests were cache hits

1 test (100.0%) was fully cached saving 2s.


Test

e2e/npm_link_package

All tests were cache hits

3 tests (100.0%) were fully cached saving 448ms.


Test

e2e/npm_link_package-esm

All tests were cache hits

3 tests (100.0%) were fully cached saving 533ms.


Test

e2e/npm_translate_lock

All tests were cache hits

1 test (100.0%) was fully cached saving 30ms.


Test

e2e/npm_translate_lock_empty

All tests were cache hits

1 test (100.0%) was fully cached saving 30ms.


Test

e2e/npm_translate_lock_multi

All tests were cache hits

2 tests (100.0%) were fully cached saving 268ms.


Test

e2e/npm_translate_lock_partial_clone

All tests were cache hits

1 test (100.0%) was fully cached saving 31ms.


Test

e2e/npm_translate_lock_replace_packages

All tests were cache hits

3 tests (100.0%) were fully cached saving 603ms.


Test

e2e/npm_translate_lock_subdir_patch

All tests were cache hits

1 test (100.0%) was fully cached saving 98ms.


Test

e2e/npm_translate_package_lock

All tests were cache hits

1 test (100.0%) was fully cached saving 31ms.


Test

e2e/npm_translate_yarn_lock

All tests were cache hits

1 test (100.0%) was fully cached saving 31ms.


Test

e2e/package_json_module

All tests were cache hits

1 test (100.0%) was fully cached saving 253ms.


Test

e2e/pnpm_lockfiles

4 test targets passed

Targets
//v54:repos_0_test [k8-fastbuild]              38ms
//v60:repos_0_test [k8-fastbuild]              31ms
//v61:repos_0_test [k8-fastbuild]              35ms
//v90:repos_0_test [k8-fastbuild]              36ms

Total test execution time was 140ms. 38 tests (90.5%) were fully cached saving 2s.


Test

e2e/pnpm_workspace

All tests were cache hits

10 tests (100.0%) were fully cached saving 2s.


Test

e2e/pnpm_workspace_rerooted

All tests were cache hits

12 tests (100.0%) were fully cached saving 2s.


Test

e2e/repo_mapping

All tests were cache hits

3 tests (100.0%) were fully cached saving 404ms.


Test

e2e/rules_foo

All tests were cache hits

2 tests (100.0%) were fully cached saving 187ms.


Test

e2e/runfiles

All tests were cache hits

1 test (100.0%) was fully cached saving 173ms.


Test

e2e/vendored_node

All tests were cache hits

1 test (100.0%) was fully cached saving 99ms.


Buildifier      Format

@jbedard jbedard force-pushed the pnpm-links-sorted branch from 2f47809 to fbac5b7 Compare April 20, 2025 00:03
@@ -98,7 +98,7 @@ def generate_repository_files(rctx, pnpm_lock_label, importers, packages, patche

npm_imports = helpers.get_npm_imports(importers, packages, patched_dependencies, only_built_dependencies, root_package, rctx.name, rctx.attr, rctx.attr.lifecycle_hooks, rctx.attr.lifecycle_hooks_execution_requirements, rctx.attr.lifecycle_hooks_use_default_shell_env, npm_registries, default_registry, npm_auth)

link_packages = [helpers.link_package(root_package, import_path) for import_path in importers.keys()]
link_packages = sorted([helpers.link_package(root_package, import_path) for import_path in importers.keys()])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

starlark spec says "Iteration yields the dictionary's keys in the order in which they were inserted; updating the value associated with an existing key does not affect the iteration order." so not sure this is necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This did change the generated code, however maybe the importers should be sorted elsewhere? Or maybe it is deterministic (lockfile order) and there is no reason for this one. I might revert this line...

Copy link
Collaborator

@dzbarsky dzbarsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not sure of the full flow so its possible the non-determinism (was it observed or theoretical?) came from earlier. see comment below, .keys() and iteration should be deterministic

@jbedard jbedard force-pushed the pnpm-links-sorted branch from fbac5b7 to 179d672 Compare April 20, 2025 01:15
@jbedard jbedard marked this pull request as draft April 20, 2025 01:18
@jbedard
Copy link
Member Author

jbedard commented Apr 20, 2025

I'm mainly trying to ensure that refactoring all the pnpm logic doesn't unnecessary change the order of these fields. Maybe it's my other branch breaking the deterministic order and this isn't the root cause of it so I'm going to convert this to a draft for now...

@jbedard
Copy link
Member Author

jbedard commented Apr 24, 2025

I think you're right that we can depend on the insertion order. Most of the things that changed order in #2177 are due to other reasons so I'll close this for now.

@jbedard jbedard closed this Apr 24, 2025
@jbedard jbedard deleted the pnpm-links-sorted branch April 24, 2025 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants