cc_wrapper: Spill arguments back to a response file #430
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… into a response file
fmeum
reviewed
Jan 6, 2025
| if [[ ${!i} == @* && -r "${i:1}" ]]; then | ||
| # Create a temporary file that we'll spill sanitized options to since they | ||
| # were originally read from a response file. | ||
| temp_file=$(mktemp) |
There was a problem hiding this comment.
Can we do this only if the arguments have a length that gets somewhat close to the limit? Otherwise we are just doing needless file I/O.
| done <"${!i:1}" | ||
| mv "${temp_file}" "${!i:1}" |
There was a problem hiding this comment.
This replaces an input file, which is something actions should avoid. Could we rewrite the argument to point to the temporary file instead?
| done <"${!i:1}" | ||
| mv "${temp_file}" "${!i:1}" | ||
| else |
There was a problem hiding this comment.
The temporary file should be deleted after the compiler returns.
|
Hello! What is the status of this PR, it would be great to see it merged! We actually ran into a similar but different issue: golang (which is using our hermetic llvm toolchain for linking) will in certain situations produce response files where all arguments are quoted: https://github.com/golang/go/blob/ecc06f0db79193a4fe16138148c7eb26d9af96f1/src/cmd/link/internal/ld/lib.go#L2123-L2124 This does not work with the current sanitization logic: clang supports quotes in the response file but not when "unrolled" into regular args on the command line, leading to errors like |
dfreese
added a commit
to dfreese/toolchains_llvm
that referenced
this pull request
Apr 9, 2025
The cc_wrapper.sh scripts inspects response files and sanitizes each line within them. How the logic is structured currently causes all of these arguments to the be directly used later in the script. This can run into argument overflows. It can also run into problems where quoting in response files and from args is treated differently. This addresses some of the comments in bazel-contrib#430, namely cleaning up files and not overwriting the input files. I have not tried to apply a threshold here as that is best left up to the tooling that previously decided to use response files, which can cause difficult to debug changes in behavior. Fixes bazel-contrib#421
dfreese
added a commit
to dfreese/toolchains_llvm
that referenced
this pull request
Apr 9, 2025
The cc_wrapper.sh scripts inspects response files and sanitizes each line within them. How the logic is structured currently causes all of these arguments to the be directly used later in the script. This can run into argument overflows. It can also run into problems where quoting in response files and from args is treated differently. This addresses some of the comments in bazel-contrib#430, namely cleaning up files and not overwriting the input files. I have not tried to apply a threshold here as that is best left up to the tooling that previously decided to use response files, which can cause difficult to debug changes in behavior. Fixes bazel-contrib#421
dfreese
added a commit
to dfreese/toolchains_llvm
that referenced
this pull request
Apr 9, 2025
The cc_wrapper.sh scripts inspects response files and sanitizes each line within them. How the logic is structured currently causes all of these arguments to the be directly used later in the script. This can run into argument overflows. It can also run into problems where quoting in response files and from args is treated differently. This addresses some of the comments in bazel-contrib#430, namely cleaning up files and not overwriting the input files. I have not tried to apply a threshold here as that is best left up to the tooling that previously decided to use response files, which can cause difficult to debug changes in behavior. This conflicts with bazel-contrib#479, but is trying to address the same underlying problem of quoted arguments in response files as the result of golang's link actions. Fixes bazel-contrib#421
fmeum
pushed a commit
that referenced
this pull request
Apr 10, 2025
The cc_wrapper.sh scripts inspects response files and sanitizes each line within them. How the logic is structured currently causes all of these arguments to the be directly used later in the script. This can run into argument overflows. It can also run into problems where quoting in response files and from args is treated differently. This addresses some of the comments in #430, namely cleaning up files and not overwriting the input files. I have not tried to apply a threshold here as that is best left up to the tooling that previously decided to use response files, which can cause difficult to debug changes in behavior. This conflicts with #479, but is trying to address the same underlying problem of quoted arguments in response files as the result of golang's link actions. Fixes #421
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #421
This PR tweaks the
*cc_wraper.shscripts to spill command line arguments from response files, back into a file after sanitizing. Previously they were expanded into thecmdvariable which would then get specified on the command line, possibly leading to an overflow.