strip prefix of temporary file names when it exceeds filesystem name length limit #145005
Conversation
rustbot
added
S-waiting-on-review
This comment has been minimized.
This comment has been minimized.
|
This method is also used for generating output files in various places. Mangling the name of output files would break cargo and other build tools that pre-generate the expected filenames based on |
|
@bjorn3 would it be more appropriate to put the filename stripping in temp_path_ext_for_cgu ? those files if I understand correctly are temporary and unlinked after the rustc invocation. |
tardyp
force-pushed
the
lto_big_filesize
branch
2 times, most recently
from
4d6eb1e to
3239eeb
Compare
|
r? bjorn3 feel free to reassign |
| filename[..stripped_len].hash(&mut hasher); | ||
| let hash = hasher.finish::<Hash64>(); | ||
|
|
||
| path.set_file_name(format!("{:x}-{}", hash, &filename[stripped_len..])); |
There was a problem hiding this comment.
The start of the filename has more useful information, but we do absolutely need to preserve file extensions (plural: eg .rcgu.o) for correctness and it isn't trivial to distinguish between file extensions and normal dot separated filename components. So I guess the current implementation is good enough.
There was a problem hiding this comment.
This long filename may cause issues on windows where the total path length may not be more than 254 characters unless you use \\?\ prefixed paths, which many programs don't do (including possibly git). Maybe a run-make test which writes the file dynamically would be better? Libstd internally does whatever is necessary to support long paths on Windows.
There was a problem hiding this comment.
Thanks for this thorough review. I updated the test to use run-make.
tardyp
force-pushed
the
lto_big_filesize
branch
from
3239eeb to
7d98d96
Compare
tardyp
force-pushed
the
lto_big_filesize
branch
2 times, most recently
from
4e616be to
63fd877
Compare
This comment has been minimized.
This comment has been minimized.
…length limit When doing lto, rustc generates filenames that are concatenating many information. In the case of this testcase, it is concatenating crate name and rust file name, plus some hash, and the extension. In some other cases it will concatenate even more information reducing the maximum effective crate name to about 110 chars on linux filesystems where filename max length is 255 This commit is ensuring that the temporary file names are limited in size, while still reasonabily ensuring the unicity (with hashing of the stripped part)
tardyp
force-pushed
the
lto_big_filesize
branch
from
63fd877 to
bc14ad3
Compare
When doing lto, rustc generates filenames that are concatenating many information.
In the case of this testcase, it is concatenating crate name and rust file name, plus some hash, and the extension. In some other cases it will concatenate even more information reducing the maximum effective crate name to about 110 chars on linux filesystems where filename max length is 255
This commit is ensuring that the temporary file names are limited in size, while still reasonably ensuring the unicity (with hashing of the stripped part)
Fix: #49914