Skip to content

Commit 0aee085

Browse files
BioPhotonmatejchalk
BioPhoton
and
matejchalk
authored
feat: add caching options to cli (#1059)
This PR depends on: - #1058 - #1057 This PR includes: - docs on [`--cache` options and caching](https://github.com/code-pushup/cli/pull/1059/files#diff-5357afeca9b96eccbc161aa37719ecf7668ac0ebae529e0695f2a9da5cf3b144) - docs on [how to cache with Nx](https://github.com/code-pushup/cli/pull/1059/files#diff-15986190ef9581ab59bcd5483b2c09e7fd0bd439d6f6cddbc94b0b1de094ee51) - docs on how to use with Turborepo - logic to respect cache options - e2e tests testing the full cache flow (inc `--skipReports`) Suggested followup PR would be to implement Nx caching. This is partially started here: #1035. > [!note] > Potential changes arrive after #1058 is merged Closes #1048 --------- Co-authored-by: Matěj Chalk <[email protected]>
1 parent 49adbf3 commit 0aee085

File tree

19 files changed

+543
-55
lines changed

19 files changed

+543
-55
lines changed

‎e2e/cli-e2e/tests/__snapshots__/help.e2e.test.ts.snap‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,14 @@ Global Options:
3636
-p, --onlyPlugins List of plugins to run. If not set all plugins are run.
3737
[array] [default: []]
3838
39+
Cache Options:
40+
--cache Cache runner outputs (both read and write)
41+
[boolean]
42+
--cache.read Read runner-output.json from file system
43+
[boolean]
44+
--cache.write Write runner-output.json to file system
45+
[boolean]
46+
3947
Persist Options:
4048
--persist.outputDir Directory for the produced reports
4149
[string]

‎e2e/cli-e2e/tests/collect.e2e.test.ts‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,13 @@ import {
77
TEST_OUTPUT_DIR,
88
teardownTestFolder,
99
} from '@code-pushup/test-utils';
10-
import { executeProcess, fileExists, readTextFile } from '@code-pushup/utils';
10+
import {
11+
executeProcess,
12+
fileExists,
13+
readJsonFile,
14+
readTextFile,
15+
} from '@code-pushup/utils';
16+
import { dummyPluginSlug } from '../mocks/fixtures/dummy-setup/dummy.plugin';
1117

1218
describe('CLI collect', () => {
1319
const dummyPluginTitle = 'Dummy Plugin';
@@ -61,6 +67,28 @@ describe('CLI collect', () => {
6167
expect(md).toContain(dummyAuditTitle);
6268
});
6369

70+
it('should write runner outputs if --cache is given', async () => {
71+
const { code } = await executeProcess({
72+
command: 'npx',
73+
args: ['@code-pushup/cli', '--no-progress', 'collect', '--cache'],
74+
cwd: dummyDir,
75+
});
76+
77+
expect(code).toBe(0);
78+
79+
await expect(
80+
readJsonFile(
81+
path.join(dummyOutputDir, dummyPluginSlug, 'runner-output.json'),
82+
),
83+
).resolves.toStrictEqual([
84+
{
85+
slug: 'dummy-audit',
86+
score: 0.3,
87+
value: 3,
88+
},
89+
]);
90+
});
91+
6492
it('should not create reports if --persist.skipReports is given', async () => {
6593
const { code } = await executeProcess({
6694
command: 'npx',

‎packages/cli/README.md‎

Lines changed: 43 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -207,18 +207,40 @@ Each example is fully tested to demonstrate best practices for plugin testing as
207207

208208
### Common Command Options
209209

210-
| Option | Type | Default | Description |
211-
| --------------------------- | -------------------- | -------- | --------------------------------------------------------------------------- |
212-
| **`--persist.outputDir`** | `string` | n/a | Directory for the produced reports. |
213-
| **`--persist.filename`** | `string` | `report` | Filename for the produced reports without extension. |
214-
| **`--persist.format`** | `('json' \| 'md')[]` | `json` | Format(s) of the report file. |
215-
| **`--persist.skipReports`** | `boolean` | `false` | Skip generating report files. (useful in combination with caching) |
216-
| **`--upload.organization`** | `string` | n/a | Organization slug from portal. |
217-
| **`--upload.project`** | `string` | n/a | Project slug from portal. |
218-
| **`--upload.server`** | `string` | n/a | URL to your portal server. |
219-
| **`--upload.apiKey`** | `string` | n/a | API key for the portal server. |
220-
| **`--onlyPlugins`** | `string[]` | `[]` | Only run the specified plugins. Applicable to all commands except `upload`. |
221-
| **`--skipPlugins`** | `string[]` | `[]` | Skip the specified plugins. Applicable to all commands except `upload`. |
210+
#### Global Options
211+
212+
| Option | Type | Default | Description |
213+
| ---------------------- | ---------- | ------- | ------------------------------------------------------------------------------ |
214+
| **`--onlyPlugins`** | `string[]` | `[]` | Only run the specified plugins. Applicable to all commands except `upload`. |
215+
| **`--skipPlugins`** | `string[]` | `[]` | Skip the specified plugins. Applicable to all commands except `upload`. |
216+
| **`--onlyCategories`** | `string[]` | `[]` | Only run the specified categories. Applicable to all commands except `upload`. |
217+
| **`--skipCategories`** | `string[]` | `[]` | Skip the specified categories. Applicable to all commands except `upload`. |
218+
219+
#### Cache Options
220+
221+
| Option | Type | Default | Description |
222+
| ------------------- | --------- | ------- | --------------------------------------------------------------- |
223+
| **`--cache`** | `boolean` | `false` | Cache runner outputs (both read and write). |
224+
| **`--cache.read`** | `boolean` | `false` | If plugin audit outputs should be read from file system cache. |
225+
| **`--cache.write`** | `boolean` | `false` | If plugin audit outputs should be written to file system cache. |
226+
227+
#### Persist Options
228+
229+
| Option | Type | Default | Description |
230+
| --------------------------- | -------------------- | -------- | ------------------------------------------------------------------ |
231+
| **`--persist.outputDir`** | `string` | n/a | Directory for the produced reports. |
232+
| **`--persist.filename`** | `string` | `report` | Filename for the produced reports without extension. |
233+
| **`--persist.format`** | `('json' \| 'md')[]` | `json` | Format(s) of the report file. |
234+
| **`--persist.skipReports`** | `boolean` | `false` | Skip generating report files. (useful in combination with caching) |
235+
236+
#### Upload Options
237+
238+
| Option | Type | Default | Description |
239+
| --------------------------- | -------- | ------- | ------------------------------ |
240+
| **`--upload.organization`** | `string` | n/a | Organization slug from portal. |
241+
| **`--upload.project`** | `string` | n/a | Project slug from portal. |
242+
| **`--upload.server`** | `string` | n/a | URL to your portal server. |
243+
| **`--upload.apiKey`** | `string` | n/a | API key for the portal server. |
222244

223245
> [!NOTE]
224246
> All common options, except `--onlyPlugins` and `--skipPlugins`, can be specified in the configuration file as well.
@@ -327,3 +349,12 @@ In addition to the [Common Command Options](#common-command-options), the follow
327349
| Option | Required | Type | Description |
328350
| ------------- | :------: | ---------- | --------------------------------- |
329351
| **`--files`** | yes | `string[]` | List of `report-diff.json` paths. |
352+
353+
## Caching
354+
355+
The CLI supports caching to speed up subsequent runs and is compatible with Nx and Turborepo.
356+
357+
Depending on your strategy, you can cache the generated reports files or plugin runner output.
358+
For fine-grained caching, we suggest caching plugin runner output.
359+
360+
The detailed example for [Nx caching](./docs/nx-caching.md) and [Turborepo caching](./docs/turbo-caching.md) is available in the docs.

‎packages/cli/docs/nx-caching.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Caching Example Nx
2+
3+
To cache plugin runner output, you can use the `--cache.write` and `--cache.read` options in combination with `--onlyPlugins` and `--persist.skipReports` command options.
4+
5+
## `{projectRoot}/code-pushup.config.ts`
6+
7+
```ts
8+
import coveragePlugin from '@code-pushup/coverage-plugin';
9+
import jsPackagesPlugin from '@code-pushup/js-packages-plugin';
10+
import type { CoreConfig } from '@code-pushup/models';
11+
12+
export default {
13+
plugins: [
14+
await coveragePlugin({
15+
reports: ['coverage/lcov.info'],
16+
}),
17+
await jsPackagesPlugin(),
18+
],
19+
upload: {
20+
server: 'https://api.code-pushup.example.com/graphql',
21+
organization: 'my-org',
22+
project: 'lib-a',
23+
apiKey: process.env.CP_API_KEY,
24+
},
25+
} satisfies CoreConfig;
26+
```
27+
28+
## `{projectRoot}/project.json`
29+
30+
```json
31+
{
32+
"name": "lib-a",
33+
"targets": {
34+
"int-test": {
35+
"cache": true,
36+
"outputs": ["{options.coverage.reportsDirectory}"],
37+
"executor": "@nx/vite:test",
38+
"options": {
39+
"configFile": "packages/lib-a/vitest.int.config.ts",
40+
"coverage.reportsDirectory": "{projectRoot}/coverage/int-test"
41+
}
42+
},
43+
"unit-test": {
44+
"cache": true,
45+
"outputs": ["{options.coverage.reportsDirectory}"],
46+
"executor": "@nx/vite:test",
47+
"options": {
48+
"configFile": "packages/lib-a/vitest.unit.config.ts",
49+
"coverage.reportsDirectory": "{projectRoot}/coverage/unit-test"
50+
}
51+
},
52+
"code-pushup-coverage": {
53+
"cache": true,
54+
"outputs": ["{projectRoot}/.code-pushup/coverage"],
55+
"executor": "nx:run-commands",
56+
"options": {
57+
"command": "npx @code-pushup/cli collect",
58+
"args": ["--config={projectRoot}/code-pushup.config.ts", "--cache.write=true", "--persist.skipReports=true", "--persist.outputDir={projectRoot}/.code-pushup", "--upload.project={projectName}"]
59+
},
60+
"dependsOn": ["unit-test", "int-test"]
61+
},
62+
"code-pushup": {
63+
"cache": true,
64+
"outputs": ["{projectRoot}/.code-pushup"],
65+
"executor": "nx:run-commands",
66+
"options": {
67+
"command": "npx @code-pushup/cli",
68+
"args": ["--config={projectRoot}/code-pushup.config.ts", "--cache.read=true", "--persist.outputDir={projectRoot}/.code-pushup", "--upload.project={projectName}"]
69+
},
70+
"dependsOn": ["code-pushup-coverage"]
71+
}
72+
}
73+
}
74+
```
75+
76+
## Nx Task Graph
77+
78+
This configuration creates the following task dependency graph:
79+
80+
**Legend:**
81+
82+
- 🐳 = Cached target
83+
84+
```mermaid
85+
graph TD
86+
A[lib-a:code-pushup 🐳] --> B[lib-a:code-pushup-coverage 🐳]
87+
B --> C[lib-a:unit-test 🐳]
88+
B --> D[lib-a:int-test 🐳]
89+
```
90+
91+
## Command Line Example
92+
93+
```bash
94+
# Run all affected project plugins `coverage` and cache the output if configured
95+
nx affected --target=code-pushup-coverage
96+
97+
# Run all affected projects with plugins `coverage` and `js-packages` and upload the report to the portal
98+
nx affected --target=code-pushup
99+
```
100+
101+
This approach has the following benefits:
102+
103+
1. **Parallel Execution**: Plugins can run in parallel
104+
2. **Fine-grained Caching**: Code level cache invalidation enables usage of [affected](https://nx.dev/recipes/affected-tasks) command
105+
3. **Dependency Management**: Leverage Nx task dependencies and its caching strategy
106+
4. **Clear Separation**: Each plugin has its own target for better debugging and maintainability

‎packages/cli/docs/turbo-caching.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Caching Example Turborepo
2+
3+
To cache plugin runner output with Turborepo, wire Code Pushup into your turbo.json pipeline and pass Code Pushup flags (`--cache.write`, `--cache.read`, `--onlyPlugins`, `--persist.skipReports`) through task scripts. Turborepo will cache task outputs declared in outputs, and you can target affected packages with `--filter=[origin/main]`.
4+
5+
## `{projectRoot}/code-pushup.config.ts`
6+
7+
```ts
8+
import coveragePlugin from '@code-pushup/coverage-plugin';
9+
import jsPackagesPlugin from '@code-pushup/js-packages-plugin';
10+
import type { CoreConfig } from '@code-pushup/models';
11+
12+
export default {
13+
plugins: [
14+
await coveragePlugin({
15+
reports: ['coverage/lcov.info'],
16+
}),
17+
await jsPackagesPlugin(),
18+
],
19+
upload: {
20+
server: 'https://api.code-pushup.example.com/graphql',
21+
organization: 'my-org',
22+
project: 'lib-a',
23+
apiKey: process.env.CP_API_KEY,
24+
},
25+
} satisfies CoreConfig;
26+
```
27+
28+
## Root `turbo.json`
29+
30+
```json
31+
{
32+
"$schema": "https://turbo.build/schema.json",
33+
"tasks": {
34+
"unit-test": {
35+
"outputs": ["coverage/unit-test/**"]
36+
},
37+
"int-test": {
38+
"outputs": ["coverage/int-test/**"]
39+
},
40+
"code-pushup-coverage": {
41+
"dependsOn": ["unit-test", "int-test"],
42+
"outputs": [".code-pushup/coverage/**"]
43+
},
44+
"code-pushup": {
45+
"dependsOn": ["code-pushup-coverage"],
46+
"outputs": [".code-pushup/**"]
47+
}
48+
}
49+
}
50+
```
51+
52+
## `packages/lib-a/package.json`
53+
54+
```json
55+
{
56+
"name": "lib-a",
57+
"scripts": {
58+
"unit-test": "vitest --config packages/lib-a/vitest.unit.config.ts --coverage",
59+
"int-test": "vitest --config packages/lib-a/vitest.int.config.ts --coverage",
60+
"code-pushup-coverage": "code-pushup collect --config packages/lib-a/code-pushup.config.ts --cache.write --persist.skipReports --persist.outputDir packages/lib-a/.code-pushup --onlyPlugins=coverage",
61+
"code-pushup": "code-pushup autorun --config packages/lib-a/code-pushup.config.ts --cache.read --persist.outputDir packages/lib-a/.code-pushup"
62+
}
63+
}
64+
```
65+
66+
> **Note:** `--cache.write` is used on the collect step to persist each plugin's audit-outputs.json; `--cache.read` is used on the autorun step to reuse those outputs.
67+
68+
## Turborepo Task Graph
69+
70+
This configuration creates the following task dependency graph:
71+
72+
**Legend:**
73+
74+
- ⚡ = Cached target (via outputs)
75+
76+
```mermaid
77+
graph TD
78+
A[lib-a:code-pushup ⚡] --> B[lib-a:code-pushup-coverage]
79+
B --> C[lib-a:unit-test ⚡]
80+
B --> D[lib-a:int-test ⚡]
81+
```
82+
83+
## Command Line Examples
84+
85+
```bash
86+
# Run all affected project plugins `coverage` and cache the output if configured
87+
turbo run code-pushup-coverage --filter=[origin/main]
88+
89+
# Run all affected projects with plugins `coverage` and `js-packages` and upload the report to the portal
90+
turbo run code-pushup --filter=[origin/main]
91+
```
92+
93+
This approach has the following benefits:
94+
95+
1. **Parallel Execution**: Plugins can run in parallel
96+
2. **Finegrained Caching**: Code level cache invalidation enables usage of affected packages filtering
97+
3. **Dependency Management**: Leverage Turborepo task dependencies and its caching strategy
98+
4. **Clear Separation**: Each plugin has its own target for better debugging and maintainability

‎packages/cli/src/lib/implementation/core-config.middleware.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
import { autoloadRc, readRcByPath } from '@code-pushup/core';
22
import {
3+
type CacheConfig,
4+
type CacheConfigObject,
35
type CoreConfig,
46
DEFAULT_PERSIST_FILENAME,
57
DEFAULT_PERSIST_FORMAT,
@@ -41,6 +43,7 @@ export async function coreConfigMiddleware<
4143
tsconfig,
4244
persist: cliPersist,
4345
upload: cliUpload,
46+
cache: cliCache,
4447
...remainingCliOptions
4548
} = processArgs;
4649
// Search for possible configuration file extensions if path is not given
@@ -59,8 +62,10 @@ export async function coreConfigMiddleware<
5962
...rcUpload,
6063
...cliUpload,
6164
});
65+
6266
return {
6367
...(config != null && { config }),
68+
cache: normalizeCache(cliCache),
6469
persist: buildPersistConfig(cliPersist, rcPersist),
6570
...(upload != null && { upload }),
6671
...remainingRcConfig,
@@ -79,5 +84,15 @@ export const normalizeBooleanWithNegation = <T extends string>(
7984
? false
8085
: ((rcOptions?.[propertyName] as boolean) ?? true);
8186

87+
export const normalizeCache = (cache?: CacheConfig): CacheConfigObject => {
88+
if (cache == null) {
89+
return { write: false, read: false };
90+
}
91+
if (typeof cache === 'boolean') {
92+
return { write: cache, read: cache };
93+
}
94+
return { write: cache.write ?? false, read: cache.read ?? false };
95+
};
96+
8297
export const normalizeFormats = (formats?: string[]): Format[] =>
8398
(formats ?? []).flatMap(format => format.split(',') as Format[]);

0 commit comments

Comments
 (0)