generated from threeal/action-starter
-
Notifications
You must be signed in to change notification settings - Fork 3
use sticky disk grpc port from the env if possible #54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
aayushshah15
wants to merge
25
commits into
useblacksmith:main
from
useblacksmith:dynamic-sticky-disk-port
Closed
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
6d82cf9
*: initial scaffolding for the stickydisk action
adityamaru 925fd98
Merge pull request #1 from useblacksmith/initial-commit
adityamaru 1e9b1c3
README: update use cases in README
adityamaru 86672c0
README: update README
adityamaru f9a0feb
Merge pull request #3 from useblacksmith/update-readme
adityamaru 133fc0b
README: add blacksmith logo (#4)
aayushshah15 d29289d
main: maintain runner perms when mounting a stickydisk
adityamaru 719a784
Merge pull request #5 from useblacksmith/fix-perms
adityamaru 60fc5d5
update README (#6)
aayushshah15 ea2e5f8
README: update README with use cases and arch diagram
adityamaru 3d6bf16
.github: fix typo
adityamaru 16ff158
Merge pull request #7 from useblacksmith/add-readme
adityamaru 6d5bf41
README: fix nits
adityamaru c8fc92b
Merge pull request #8 from useblacksmith/fix-nits
adityamaru af6814d
Update README.md
adityamaru b26d421
.github: clarify in first sentence
adityamaru a978eb7
Update README.md
adityamaru 7464a4c
Update README.md
adijp d197c9d
Merge pull request #9 from useblacksmith/adijp-patch-1
adijp a85354c
src: add sync before umount
adityamaru 9c61fba
Merge pull request #10 from useblacksmith/sd-fixes
adityamaru 5175c4c
src: increase timeout to the same as build-push-action
adityamaru abd1f46
Merge pull request #11 from useblacksmith/timeout-fails
adityamaru eb09d7f
src: explicitly flush journal before umounting (#12)
aayushshah15 617cc46
use the env var for sticky disk grpc port
aayushshah15 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| name: Basic Sticky Disk Test | ||
| on: | ||
| workflow_dispatch: | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| test-sticky-disk: | ||
| name: Test Sticky Disk | ||
| runs-on: blacksmith | ||
| strategy: | ||
| matrix: | ||
| iteration: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] | ||
| fail-fast: false | ||
| steps: | ||
| - name: Create directory called | ||
| run: sudo mkdir -p ./shouldseethis | ||
|
|
||
| - name: Mount First Sticky Disk | ||
| uses: useblacksmith/stickydisk@main | ||
| with: | ||
| key: foo | ||
| path: ./shouldseethis | ||
|
|
||
| - name: List first directory if exists | ||
| run: | | ||
| if [ -d "./shouldseethis" ]; then | ||
| ls -la ./shouldseethis | ||
| fi | ||
|
|
||
| - name: Write test file to first disk | ||
| run: sudo sh -c 'echo "Hello from first sticky disk" > ./shouldseethis/test.txt' | ||
|
|
||
| - name: Mount Second Sticky Disk | ||
| uses: useblacksmith/stickydisk@main | ||
| with: | ||
| key: bar | ||
| path: ./seconddisk | ||
|
|
||
| - name: List second directory if exists | ||
| run: | | ||
| if [ -d "./seconddisk" ]; then | ||
| ls -la ./seconddisk | ||
| fi | ||
|
|
||
| - name: Write test file to second disk | ||
| run: sudo sh -c 'echo "Hello from second sticky disk" > ./seconddisk/test.txt' | ||
|
|
||
| - name: Mount Third Sticky Disk | ||
| uses: useblacksmith/stickydisk@main | ||
| with: | ||
| key: baz | ||
| path: ./thirddisk | ||
|
|
||
| - name: List third directory if exists | ||
| run: | | ||
| if [ -d "./thirddisk" ]; then | ||
| ls -la ./thirddisk | ||
| fi | ||
|
|
||
| - name: Write test file to third disk | ||
| run: sudo sh -c 'echo "Hello from third sticky disk" > ./thirddisk/test.txt' | ||
|
|
||
| - name: Mount First Sticky Disk Again | ||
| uses: useblacksmith/stickydisk@main | ||
| with: | ||
| key: foo | ||
| path: ./shouldseethis_again | ||
|
|
||
| - name: List first directory again if exists | ||
| run: | | ||
| if [ -d "./shouldseethis_again" ]; then | ||
| ls -la ./shouldseethis_again | ||
| fi | ||
|
|
||
| - name: Write test file to first disk again | ||
| run: sudo sh -c 'echo "Hello again from first sticky disk" > ./shouldseethis_again/test2.txt' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| name: Bump Tag | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| bump-tag: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: write | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| ref: main | ||
|
|
||
| - name: Bump v1 tag | ||
| run: | | ||
| git config --global user.name "github-actions[bot]" | ||
| git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com" | ||
| git tag -f v1 | ||
| git push origin v1 -f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,76 +1,93 @@ | ||
| # Blacksmith Cache Delete Action | ||
| <p align="center"> | ||
| <picture> | ||
| <!-- Dark mode --> | ||
| <source media="(prefers-color-scheme: dark)" srcset="./Blacksmith_Logo-White-Large.png" width="300"> | ||
| <!-- Light mode --> | ||
| <source media="(prefers-color-scheme: light)" srcset="./Blacksmith_Logo-Black-Large.png" width="300"> | ||
| <img alt="Blacksmith Logo" src="./Blacksmith_Logo-Black-Large.png" width="300"> | ||
| </picture> | ||
| </p> | ||
|
|
||
| A GitHub Action to delete caches from Blacksmith's cache storage. This action allows you to delete either a specific cache version or all versions of a cache key. This action only works Blacksmith runners. | ||
| A GitHub Action that helps persist state written to disk across jobs. This action can serve as a superior alternative to the [Actions cache](https://github.com/useblacksmith/cache), especially when the cache artifacts are extremely large. Each sticky disk is hot-loaded into the runner and mounted at the specified path. | ||
| The sticky disk is formatted as an ext4 filesystem. | ||
|
|
||
| ## Usage | ||
| # Architecture | ||
|
|
||
| ```yaml | ||
| - name: Delete Cache | ||
| uses: useblacksmith/cache-delete@v1 | ||
| with: | ||
| key: Linux-composer-ecf6e2e236589e4d34ba89662b6bc2afe8e15237cd19a13df9dc0cb599ff4826 | ||
| version: v213asda2cf # Optional: specific version to delete | ||
| ``` | ||
|
|
||
| ## Inputs | ||
| <p align="center"> | ||
| <picture> | ||
| <!-- Dark mode --> | ||
| <source media="(prefers-color-scheme: dark)" srcset="./arch-dark-mode.png" width="1000"> | ||
| <!-- Light mode --> | ||
| <source media="(prefers-color-scheme: light)" srcset="./arch-light.png" width="1000"> | ||
| <img alt="Blacksmith Logo" src="./arch-light.png" width="1000"> | ||
| </picture> | ||
| </p> | ||
|
|
||
| | Input | Description | Required | Default | | ||
| | --------- | --------------------------------------- | -------- | ------- | | ||
| | `key` | The cache key to delete | No\* | - | | ||
| | `version` | Specific version of the cache to delete | No | - | | ||
| | `prefix` | Treat key as a prefix for bulk deletion | No | false | | ||
| Blacksmith stores sticky disk artifacts in a secure, highly performant Ceph cluster, running on local NVMe drives. Our runners proxy their requests through our Storage Agents to interact with the Ceph cluster. Each sticky disk is uniquely identified by a key. When a GitHub Action job requests a sticky disk, the last committed snapshot will be cloned and mounted into the runner at the specified path. Once the job completes, the sticky disk will be unmounted and committed for future invocations. At the moment, customers can use up to 5 sticky disks in a single GitHub Action job. | ||
|
|
||
| \* Required unless `prefix` is true, in which case it can be empty to match all cache keys | ||
| # Use Cases | ||
|
|
||
| ## Examples | ||
|
|
||
| ### Delete All Versions of a Cache | ||
|
|
||
| ```yaml | ||
| - name: Delete All Cache Versions | ||
| uses: useblacksmith/cache-delete@v1 | ||
| with: | ||
| key: npm-cache | ||
| ``` | ||
| ## NPM Package Caching | ||
|
|
||
| ### Delete a Specific Cache Version | ||
| Node.js projects can have extensive dependency trees, leading to large `node_modules` directories. Sticky disks provide persistent, high-performance storage for your NPM packages. | ||
|
|
||
| ```yaml | ||
| - name: Delete Specific Cache Version | ||
| uses: useblacksmith/cache-delete@v1 | ||
| with: | ||
| key: npm-cache | ||
| version: v1.0 | ||
| jobs: | ||
| build: | ||
| runs-on: blacksmith | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Node.js | ||
| uses: useblacksmith/setup-node@v5 | ||
| with: | ||
| node-version: '18.x' | ||
|
|
||
| - name: Mount NPM Cache | ||
| uses: useblacksmith/stickydisk@v1 | ||
| with: | ||
| key: ${{ github.repository }}-npm-cache | ||
| path: ~/.npm | ||
|
|
||
| - name: Mount node_modules | ||
| uses: useblacksmith/stickydisk@v1 | ||
| with: | ||
| key: ${{ github.repository }}-node-modules | ||
| path: ./node_modules | ||
|
|
||
| - name: Install Dependencies | ||
| run: npm ci | ||
|
|
||
| - name: Build | ||
| run: npm run build | ||
| ``` | ||
|
|
||
| ### Delete All Caches with a Prefix | ||
|
|
||
| ```yaml | ||
| - name: Delete All npm Caches | ||
| uses: useblacksmith/cache-delete@v1 | ||
| with: | ||
| key: npm- | ||
| prefix: true | ||
| ``` | ||
| ## Bazel Build Caching | ||
|
|
||
| ### Delete All Caches | ||
| Bazel's remote cache can significantly improve build times, but uploading and downloading cached artifacts can still be a bottleneck. Using sticky disks with Blacksmith runners provides near-instant access to your Bazel caches as they are bind mounted into your runners on demand. Our [`useblacksmith/setup-bazel@v2`](https://github.com/useblacksmith/setup-bazel) action is a zero-confg way to use sticky disks to store the disk, repository, and external cache. | ||
|
|
||
| ```yaml | ||
| - name: Delete All Caches | ||
| uses: useblacksmith/cache-delete@v1 | ||
| with: | ||
| key: "" | ||
| prefix: true | ||
| jobs: | ||
| build: | ||
| runs-on: blacksmith | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup Bazel | ||
| uses: useblacksmith/setup-bazel@v2 | ||
| with: | ||
| version: '6.x' | ||
|
|
||
| - name: Build | ||
| run: | | ||
| bazel build //... | ||
| ``` | ||
|
|
||
| ## Error Handling | ||
|
|
||
| The action will: | ||
|
|
||
| - Fail if the cache deletion request fails (non-404 error) | ||
| - Log a message if the cache is not found (404) | ||
| - Successfully complete if the cache is deleted | ||
| ### Cache Performance Comparison | ||
|
|
||
| ## License | ||
| | Caching Solution | Cache Size | Average Download Speed | Time to Access | | ||
| |-----------------|------------|----------------|----------------| | ||
| | GitHub Actions Cache | 6GB | 90 MB/s | ~1m6s | | ||
| | Blacksmith Cache | 6GB | 400 MB/s | ~15s | | ||
| | Sticky Disks | 6GB | N/A | 3 seconds | | ||
|
|
||
| This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,20 +1,17 @@ | ||
| name: "Delete Blacksmith Cache" | ||
| author: Aayush Shah | ||
| description: "Deletes a cache or specific cache version from Blacksmith" | ||
| name: "Blacksmith Sticky Disk" | ||
| author: Aditya Maru | ||
| description: "Creates a sticky disk on Blacksmith" | ||
| branding: | ||
| icon: folder-plus | ||
| color: black | ||
| inputs: | ||
| key: | ||
| description: "The cache key to delete" | ||
| required: false | ||
| version: | ||
| description: "Specific version of the cache to delete (optional)" | ||
| required: false | ||
| prefix: | ||
| description: "Treat key as a prefix for bulk deletion" | ||
| required: false | ||
| default: "false" | ||
| description: "A unique key to identify the sticky disk" | ||
| required: true | ||
| path: | ||
| description: "The path at which to mount the sticky disk" | ||
| required: true | ||
| runs: | ||
| using: "node20" | ||
| main: "dist/index.js" | ||
| post: "dist/post/index.js" |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo detected: 'zero-confg' should be 'zero-config'.