Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI Tool for Build System #1985

Merged
merged 181 commits into from
Aug 9, 2024
Merged

Conversation

AmmarAbouZor
Copy link
Member

@AmmarAbouZor AmmarAbouZor commented Feb 17, 2024

This PR closes the issue #1966.
This PR continue the great work in the PR #1967.

At the end of this drafts we should have a working CLI tool to build, lint, clean and test the project to replace the current solution using rake

Here is the road map for this PR:

  • Support build, lint, clean and test commands
  • Providing an option to report all the build logs to stdout or to the given file
  • Test the tool across platforms
  • Find a better way to get the root repository path. (Using Git?)
  • Complete missing test commands
  • Install the command as a cargo sub command
  • Shell-completion
  • Add check for development dependencies as a command and this check must always run before all other commands too
  • Rebuild only the changed parts from the last run (using Checksum?)
  • Keep track on the running task to prevent duplicated and parallel runs to the same task
  • Add Run command that builds the necessary parts then runs the app
  • Replace the async trait Manager with the enum Target
  • Provide a command to print the needed environment tools to build the app besides checking if they are installed
  • Provide a command to print a print_dots to visualize the dependencies graph
  • Add the Update target (Build, Clean...)
  • Resolve tasks tree upfront and show all of them in progress bar, keeping the execution running concurrently
  • Apply new changes after switching to yarn 4

DmitryAstafyev and others added 27 commits January 22, 2024 10:29
- Display is more suitable when printing infos for the users
- Use rust built-in path methods
- Change method signature to return Path instead of String
- Function is rewritten in a more idiomatic way without having to
  allocate memory and keeping track of the length manually
- Use Path built-in methods to set the path of the build command
- Module for wasm management have been created with the needed
  implementation to integrate in the system.
- Some Todos are left to be clarified later.
- Method to run the tests has been implemented to use when running the
  tests is built in the cli app
- TODOs has been removed after clarification
- Set Color flag to always in build command
- Test Cli Command defined and implemented
- Methods for tests in Manager trait has been defined and implemented
- Test commands in wasm module added
- Spawn Options has been added to provide more control over how spawn call
  should behave.
- It's used to suppress sending messages in wasm npm test command
  because it caused weird behavior in the progress bars because when too
  many lines are sent at the same time
- Rewrite jobs alignment to keep the braces and the symbols in the same
  horizontal alignment among all lines
- Small refactoring to get the length or max
- Change job numbers placeholder from zero to spaces in tracker
- Job duration is moved to the prefix part of the bars.
- Total time is appended to the results.
The states of jobs has been growing and getting too complex for a tuple
The padding of finishing time can change on job ends which led to
miss-alignment in the bars in some cases
Some build commands send big part of their infos to stderr than stdout,
which gives the app a little more responsive looking if we update the
bars on stderr too
- Bounded channels with 1 capacity is replaced with oneshot channels.
- Unused Clone is removed from Tick enum because the oneshot channels
  aren't clone
- Native Async trait can't be applied in near feature
@DmitryAstafyev
Copy link
Collaborator

Originally created PR with useful discussion

- The trait `RemoveDuplicates` is removed in favor of using the dedup
  method from the standard library
- Boolean is more representative of the release flag since it has two
  states only
@AmmarAbouZor
Copy link
Member Author

AmmarAbouZor commented Jul 24, 2024

@DmitryAstafyev @marcmo

  • I've created integration tests for all provided commands on the CLI tool + Tests for the checksum implementation for detect code changes and for the dependencies between build targets.
  • I've also fixed unit-tests from wasm-bindens that weren't working on Windows and added Clippy fixes for Windows.
  • The integration tests are tested on Linux and Windows only
  • Integration Tests has been added to the README of the Build CLI Tool in the contributing section

- Use the CLI tool with GitHub Actions for Pull Requests along side with
  the current ruby actions for comparison
- Install latest rust manually instead of the deprecated GitHub Action
- Use Cargo cache action for speed up installing the build CLI tool
- Dependencies are installed manually for now but this can improved
@AmmarAbouZor
Copy link
Member Author

@DmitryAstafyev @marcmo
I've integrated the CLI tool in a separate GitHub Actions for reference now.
These steps are just for testing for now and would need more improvements once we want to replace the current ones with them

- Install command will always run in develop mode initially and then the
  command will rerun in production mode if it's active after the build
  is done.
- This problem was solved by setting the production flag on false always
  for install job definition which lead to problem by calculating the
  checksums because production flag on install was always false
  leading to loading the checksums on development always even on
  production builds
- The issue is solved by assigning the production flag in the internal
  command since we are calling it anyway with the production flag in the
  reinstall process
- Production and development use the same artifacts which will lead to false
  positives when the artifacts are modified via another build but the checksum of
  source files still the same.
- To solve this problem we will reset the checksums of the opposite build production
  type when build is involved in the current process
Move the new CI checks using the CLI tool to their own GitHub action
that will run manually only since those jobs are still work on progress
and not needed to run along side with the current CI checks using rake
commands
* Use Enum instead of boolean for the checksum comparison of the given
  target to improve code readability and avoid potential bugs
* Fix Typo & Improve documentations
* Remove the unnecessary build steps from rust targets for running tests
* Remove the build steps from TS targets that don't have tests and
  aren't dependencies of other TS targets that have tests.
* Add assertion for that case if we add tests to those targets in the
  future.
* Refactor the function to filter out the not included jobs by moving
  the additional filter to a closure that will run after the standard
  filter which allowed us to add the assertion mentioned above
* Replace `matches!()` macro and catch-all place holder in match
  statements for targets and job types with explicit enum types to
  enforce handling new types to targets and job enums.
* The build system has grown to be complicated, therefore it's useful to
  get assistant from the compiler when new targets or job types are
  added
@AmmarAbouZor
Copy link
Member Author

AmmarAbouZor commented Aug 6, 2024

Hi @marcmo,
As we had spoken yesterday, the following changes have been added:

  • mathces!() macros and catch-all place holders in match statements have been replaced with explicit types, this will keep the compiler assistance powerful when we add more targets or job types in the future
  • Boolean in Checksum comparison map has been replaced with an enum to enforce logic correctness with rust type system
  • While making the changes I've found out that we have been running build steps for all test targets which is needed for typescript and WASM targets only. This has been fixed as well.
  • I added notes to the code as comments to emphasize that match arms must be written implicitly without any wild card matching, because I couldn't find an option to enable this ling only by clippy

* Remove async from the declarations of functions that don't have await
  inside of them
* Use the value itself instead of a reference for target enums
  because their size is only 1 Kilobyte, which make it more
  efficient to copy it instead of copying a reference (usize) to it.
* This applies to JobDefinition and DevTool enums as well
* Add comments in code to emphasize that explicit match arms with
  targets and job types should be used for keep the compiler assistance
  when adding new targets or job types.
* Use reference for a string instead of moving the whole string since
  it's not consumed inside the function.
Copy link
Member

@marcmo marcmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very impressive PR Ammar! Good job!!
it's huge so we should try to merge it ASAP and possibly create issues on top of it

* Remove redundant test
* Use assigning to _ to directly drop file after creating it
* Use one function to create process commands with yarn as cmd and the
  given arguments and use it in code when needed
* Remove TODO comment from ts testing file and improve the relating
  comment for it in `wrapper.rs file
@AmmarAbouZor
Copy link
Member Author

@marcmo Requested changes are implemented/answered. The PR can be reviewed again 👍

@AmmarAbouZor AmmarAbouZor requested a review from marcmo August 7, 2024 10:55
Tokio Commands has the option to kill the underline process when the
command is dropped. This will provide more grateful cancellation and
shut down functionality
Use { .. } for wild card matching for JobType where production value
should be ignored.
@AmmarAbouZor AmmarAbouZor merged commit 2168eb5 into esrlabs:master Aug 9, 2024
4 checks passed
@AmmarAbouZor AmmarAbouZor deleted the build-cli-tool branch August 9, 2024 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants