review prototype #12
review prototype #12
Conversation
* Update README * add setup files * add tests * add scaffolding * add CI checks * Move test requirements outside hidden folder * Cleanup formatting * Update requirements * Fix pip install * simplify pip install * Report code coverage * Tag the upload with environment variables to delineate python versions * Only push latest python vertion to codeco
* Stage subverb template from colcon metadata * Update subverb template * Stage ListMetadataSubverb as template * Update subverb template * Create capture task submodule * Stage PythonStageTask as template from colcon/colcon-core#428 * Refactor snapshot capture task * Stage StageVerb as template from colcon/colcon-core#428 * Stage refactor * Rename file * Stage working task * Stage template from dirhash * Stage WIP on git task * Change to markdown * Fix linter errors
* Stage from colcon_package_selection as template * Refactor event handler to store lockfiles * Rename event handler * Refactor tasks and helper functions * Stage slection logic * simplify slection logic * Fix linter * Overload comparison for SnapshotLockfile * Fix event handler * Regester snapshot package_selection entry point * Fix linters
* Add package augmentation extensions and use them to guid in task capture extension selection * Fix linter * Pass lockfile via metadat rather than returncode between capture task and store lockfile event handler * Allow the default vcs_type to be overridden using other package augmentation extensions with a diffrent default vcs_type and higher priortiy * Cleanup comments
from colcon_snapshot to colcon_cache
from colcon_snapshot to colcon_cache
Rename extension to colcon_cache
* Do not set locfile if return code is not passing even if the job is a cache task * Stage arg reduction * Fix return codes to ints * Use verb as hint for mapping refrence * Fix if logic so a missing refrence lockfile isn't ignored becuase of missing verb lockfile * clarify help string * Add package selection for lock change status * Stage PackageSelectionExtensionPoint as template * Remove unused code * Rename temp file * Add verb handler extensions * Use verb handler extensions * Update if guards * Add warning for missing verb handler extentions * Fix spelling typo * Fix linter errors
* Set python path * Stage all args from dirhash CLI * Remove unneeded args * Namespace dirhash args * Simplify import * Workarround to display help description as colcon help display seems to ignore add_argument_group * Rename default package augmentation to dirhash * Update default package augmentation to dirhash * Map dirhash keyargs * Update job option * Fix excape characters * Add reset and ratchet modifiers for dirhash * Fix linters ignore dirhash for E501 for now
* Use revision reference and diff for git task * Fix linters * Fix argparse dest for kargs namespace filtering * Use diff-index instead of diff as it's faster and includes file modes
* Refactor CacheLockfile class * Add consistency checks for lock_type * Fix CacheLockfile load usage * Simplify * Revert usage of git diff-index as output does not change with respect to the changing of uncommited tracked files * Fix linters
* Set lockfile upon task completion as event handler is too late when caching dependencies * Pass recursive dependencies with build base paths * Add lockfile tracking of recursive dependencies
* Fix linter for line length * Add example usage
* Simplify VerbHandlerExtensionPoint class * Add tests for package_augmentation * Stage test template from colcon-core/test/test_build_python.py * Ignore deprecation warnings in 3rd party modules such as in scantree * Checking WIP test need to fix arg parsing * Add init file to test folder * Update requirements * Skip WIP test for now * Fix augmention for non git packages * Extend tests * Drop 3.5 from CI * Simplify style * Add test for cache lockfiles * Extend test for cache lockfiles * Remove dead code
Rename capture verb to lock
Codecov Report
@@ Coverage Diff @@
## review #12 +/- ##
=========================================
Coverage ? 27.65%
=========================================
Files ? 19
Lines ? 564
Branches ? 79
=========================================
Hits ? 156
Misses ? 408
Partials ? 0 Continue to review full report at Codecov.
|
|
FYI I'll be doing a show an tell about this during tomorrow's Tooling Working Group meeting. CC'ing colcon contributors and perhaps interested parties in case you'd like to drop by: |
|
BTW, @SteveMacenski @nuclearsandwich, this is the extension I was envisioning to be able to cache entire colcon workspaces across CI runs, so CI Jobs could intelligently build and test only the packages that have changed from the target PR branch and/or previous run/commit. This assumes the CI itself already properly keys the cache of the workspace appropriately, i.e. using the branch, arch, underlay state, etc, like we already do for Nav2's repo CI. |
|
@ruffsl I'm interested. Please invite me to the meeting. |
@mikepurvis , Calendar invites for ROS working groups should be publicly accessable from the ROS events calendar: https://docs.ros.org/en/rolling/Governance.html#upcoming-ros-events The agenda for the Tooling WG is also viewable here: https://docs.google.com/document/d/1UXWaEkT6ji0Z0uTtJJSni5Y6fiopM9qOOzv4ldeNaAM/edit?usp=drivesdk |
|
Thanks for the ping. I'll do my best to join the show&tell tomorrow. |
* Rename cache selection arguments to use terms void and valid * Add argument to denote respective cache verbs * Simplify respective cache verb choices
|
Here is an example of colcon-cache in action for caching builds and tests in CI: ruffsl/navigation2#6 |
* Rephrase arg from changed to modified * Prefix external entrypoints with cache * Raname extention classes * Rename files * Rename class * Rename modified arg * Rename valid arg * Add key arg * Set missing kind using verb_handler_extension
Rephrase package selection extensions
as yaml safe load is too slow
A placeholder PR to review the current extension prototype of colcon-cache.
You can basically use the lock subverb to cache the current state of the packages in the workspace into lockfiles; either with dirhash, git, etc, depending upon what revision control system is detected using package augmentation extensions. Then when using build or test verbs, event handlers will update cache lock files for successful packages. The lock files also track dependency changes by default, making it simple to ensure upstream changes break downstream package caches. The package selection extensions enable users to filter packages by how the package source has changed, or if the current verb's cache for selected packages remain valid. See the Readme for example usage.
Preliminary feedback on extension architecture or suggestions for unit/system testing are welcome.