request: Optimize melos bootstrap by Skipping Unchanged Packages Based on Git Commit Hashes

[mahmuttaskiran]

Is there an existing feature request for this?

• I have searched the existing issues.

Command

melos bootstrap --cache

Description

First of all, this may be a stupid idea. I just want to get your feedback on it.

Problem Statement
In large Flutter monorepos containing a significant number of local packages (e.g, around 300) that depend on each other via path references, running melos bootstrap can be time-consuming. Even with caching mechanisms for third-party dependencies in the .pub_cache folder, the process involves running flutter pub get for each local package, regardless of whether it has changed since the last bootstrap. This results in unnecessary processing time and slows down development workflows.

Proposed Solution
Introduce an optimization in the melos boostrap process to skip running flutter pub get for packages that havn't changed since the last bootstrap. This can be achieved by:

1. Tracking Package Commit Hashes

• Create a JSON file, let's say package_hashes.json that stores a mapping of each package to its latest Git commit hash.
• Since all the local packages are part of the same Git repo, we can obtain the latest commit hash for each package directory.

2. Modifying the Bootstrap Process:

• During melos bootstrap, for each package:
  • Obtain the current commit hash
  • Compare it with the stored hash in package_hashes.json
  • If the hash hasn't changed, skip running flutter pub get for that package
  • if the hash has changed, proceed with flutter pub get
• After bootstraping, update package_hashes.json with the latest commit hashes.

Implementation Details
1. Getting the Latest Comit Hash for a Package
Get the last commit for a path that the package is located: git log -n 1 --pretty=format:%H -- path/to/package. We may need to expand this solution to catch un-commited changes as well.
2. Storing Commit Hashes
The package_hashes.json file would have a structure like:

{
 "packages/package_a": "abcd1234efgh5678ijkl9012mnop3456qrst7890",
 "packages/package_b": "1234abcd5678efgh9012ijkl3456mnop7890qrst",
 ...
}

This file will be updated whenever melos bootstrap is called if the feature enabled. A question arises here: What will happen when we run flutter pub get manually for a package.

root/
├── packages/
│   ├── package_a/
│   │   ├── lib/
│   │   ├── pubspec.yaml
│   │   └── ...
│   ├── package_b/
│   │   ├── lib/
│   │   ├── pubspec.yaml
│   │   └── ...
│   └── ...
├── melos.yaml
└── package_hashes.json

Potential Limitations and Some Whys in my Mind
Dependency Changes: If a package depends on another local package that has changed, even if the dependent package's code hasn't changed, will it still require flutter pub get?

Uncommited Changes
Uncommited changes won't be detected using commit hashes. Can computing checksum of uncommited changes computed along with the last commit changes help here?

Synchronization Issues
Running melos bootstrap will make sure to sync the packages_hashes.json but there can be other situations that needs to run synchronization. For example, running flutter pub get manually.

Melos's Diff Option
Do we really need this feature while we have a diff option for bootstrapping?

Reasoning

I believe this feature will be valuable because subsequent melos bootstrap commands currently take about 3 minutes to execute in our monorepo.

melos bootstrap  773.86s user 338.77s system 616% cpu 3:00.48 total

Additional context and comments

No response

[spydon]

After we have implemented the new workspaces feature from Dart in Melos you will no longer have to run melos bootstrap. :)
You can follow this issue to get updates on the progress: #747.

[spydon]

Thanks for a well-written issue though!

[mahmuttaskiran]

Thanks for the reference, it seems cool :)