ci: add ccache/sccache compilation caching to speed up CI builds#360
Open
Cuiyus wants to merge 7 commits intoalibaba:mainfrom
Open
ci: add ccache/sccache compilation caching to speed up CI builds#360Cuiyus wants to merge 7 commits intoalibaba:mainfrom
Cuiyus wants to merge 7 commits intoalibaba:mainfrom
Conversation
- Use hendrikmuhs/ccache-action@v1.2 for Linux/macOS/iOS/Android/clang-tidy (auto-installs ccache, manages cache, sets env vars, shows stats) - Use mozilla-actions/sccache-action@v0.0.9 for Windows (MSVC compatible) - Add CMAKE_C/CXX_COMPILER_LAUNCHER to all CMake build steps - Exclude wheel build and nightly coverage workflows per decision
- Replace CMAKE_GENERATOR='Unix Makefiles' with 'Ninja' in pip build - Replace 'make unittest -j' with 'cmake --build --target unittest --parallel' - Add '-G Ninja' to C++ and C example cmake configure steps - Replace 'make -j' with 'cmake --build --parallel' for examples - Aligns with Windows and Android workflows which already use Ninja
- Use CMake ProcessorCount module to detect available CPU cores
- Add -j ${NPROC} to ctest command for parallel test execution
- Add --timeout 300 to prevent individual tests from hanging CI
- Fallback to NPROC=1 when ProcessorCount returns 0
- iOS target unchanged (build-only, no test execution)
This reverts commit d196dac.
Windows (05-windows-build.yml): - Remove mozilla-actions/sccache-action: sccache incompatible with MSVC /FS flag - Remove SCCACHE_GHA_ENABLED env var - Remove CMAKE_C/CXX_COMPILER_LAUNCHER=sccache from build steps - Remove 'Show sccache statistics' step - MSVC /FS (global PDB concurrency flag) causes fatal C1041 when used with sccache MacOS & Linux (03-macos-linux-build.yml): - Fix cmake.define values: remove extra quotes around 'ccache' and 'ON' - Bare values required: cmake.define.FOO=bar not cmake.define.FOO="bar"
| uses: actions/cache@v5 | ||
| with: | ||
| path: build_host | ||
| key: ${{ runner.os }}-host-protoc-${{ hashFiles('thirdparty/protobuf/**', 'CMakeLists.txt') }} |
Collaborator
There was a problem hiding this comment.
这里是不是也可以优化下？包括下面的if [ ! -f "build_host/bin/protoc" ]
Collaborator
Author
There was a problem hiding this comment.
这里暂不需要优化。因为protoc 的编译有两层cache用来加速。第一层的是action@cache（目录级cache），而protoc 这种thirdparty 修改频率很低，一般不会cache miss，若出现miss 会采用编译器出现的ccache. （如果缓存紧张的话，可以去掉 protoc的ccache）
| key: ios-${{ matrix.platform }} | ||
|
|
||
| - name: Cache host protoc build | ||
| uses: actions/cache@v5 |
Collaborator
Author
There was a problem hiding this comment.
不需要。目前多平台（除windows）的ci都是加了两层cache加速。整个cache用量目前在2G左右。
Collaborator
There was a problem hiding this comment.
macOS和linux的是一层cache吧？两层指的是？
Collaborator
There was a problem hiding this comment.
两层cache加速的效果应该不大吧？另外这里的cache size也会随着action的运行累加，比如此时此刻
Collaborator
There was a problem hiding this comment.
主要是ios的cache比较大（～890MB+350MB），这里用两层感觉有点浪费
| - name: Setup ccache | ||
| uses: hendrikmuhs/ccache-action@v1.2 | ||
| with: | ||
| key: ${{ inputs.platform }}-${{ inputs.os }} |
Collaborator
There was a problem hiding this comment.
把compiler也加到key上吧，不然也容易cache miss吧？
| key: ios-${{ matrix.platform }} | ||
|
|
||
| - name: Cache host protoc build | ||
| uses: actions/cache@v5 |
Collaborator
There was a problem hiding this comment.
macOS和linux的是一层cache吧？两层指的是？
| key: ios-${{ matrix.platform }} | ||
|
|
||
| - name: Cache host protoc build | ||
| uses: actions/cache@v5 |
Collaborator
There was a problem hiding this comment.
两层cache加速的效果应该不大吧？另外这里的cache size也会随着action的运行累加，比如此时此刻
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.