Skip to content

Commit e28dd9c

Browse files
committed
rdrand: Avoid inlining unrolled retry loops.
The rdrand implementation contains three calls to rdrand(): 1. One in the main loop, for full words of output. 2. One after the main loop, for the potential partial word of output. 3. One inside the self-test loop. In the first two cases, each call is unrolled into: ``` rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> rdrand <register> jb <success> ``` In the third case, the self-test loop, the same unrolling happens, but then the self-test loop is also unrolled, so the result is a sequence of 160 instructions. With this change, each call to `rdrand()` now looks like this: ``` rdrand <register> jb <success> call retry test rax, rax jne ... jmp ... ``` The loop in `retry()` still gets unrolled though. Since rdrand will basically never fail, the `jb <success>` in each call is going to be predicted as succeeding, so the number of instructions doesn't change. But, instruction cache pressure should be reduced.
1 parent b2bca0f commit e28dd9c

File tree

1 file changed

+20
-10
lines changed

1 file changed

+20
-10
lines changed

src/rdrand.rs

+20-10
Original file line numberDiff line numberDiff line change
@@ -14,20 +14,30 @@ cfg_if! {
1414
}
1515
}
1616

17-
// Recommendation from "Intel® Digital Random Number Generator (DRNG) Software
18-
// Implementation Guide" - Section 5.2.1 and "Intel® 64 and IA-32 Architectures
19-
// Software Developer’s Manual" - Volume 1 - Section 7.3.17.1.
20-
const RETRY_LIMIT: usize = 10;
21-
2217
#[target_feature(enable = "rdrand")]
2318
unsafe fn rdrand() -> Option<Word> {
24-
for _ in 0..RETRY_LIMIT {
25-
let mut val = 0;
26-
if rdrand_step(&mut val) == 1 {
27-
return Some(val);
19+
#[cold]
20+
unsafe fn retry() -> Option<Word> {
21+
// Recommendation from "Intel® Digital Random Number Generator (DRNG) Software
22+
// Implementation Guide" - Section 5.2.1 and "Intel® 64 and IA-32 Architectures
23+
// Software Developer’s Manual" - Volume 1 - Section 7.3.17.1.
24+
25+
// Start at 1 because the caller already tried once.
26+
for _ in 1..10 {
27+
let mut val = 0;
28+
if rdrand_step(&mut val) == 1 {
29+
return Some(val);
30+
}
2831
}
32+
None
33+
}
34+
35+
let mut val = 0;
36+
if rdrand_step(&mut val) == 1 {
37+
Some(val)
38+
} else {
39+
retry()
2940
}
30-
None
3141
}
3242

3343
// "rdrand" target feature requires "+rdrand" flag, see https://github.com/rust-lang/rust/issues/49653.

0 commit comments

Comments
 (0)