forked from tylov-fork/PractRand
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathTests_results.txt
648 lines (611 loc) · 41.3 KB
/
Tests_results.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
*****************************************************************************
0. Contents of this file
*****************************************************************************
0. Contents of this file
1. Summary of test results
2. Full results
3. Details of the results format
4. Tests that were not included
This file documents the results of the various test suites on various RNGs.
The test suites used include PractRand, TestU01, a not-yet-releasd version
of RaBiGeTe, and Dieharder. The RNGs include over over 100 RNGs I selected
from a wide variety of categories. Some were RNGs actually in use, some
were obscure RNGs, some were RNGs I made up just to have a wider variety.
*****************************************************************************
1. Summary of results
*****************************************************************************
number of RNGs bias was found in VS test battery VS time
10 secs 1 min 5 mins 30 mins 1 hour 4 hours 8 hours 1 day 4 days
PractRand std 43 65 83 104 112 130 136+ 142+ 147+
TestU01 *Crush 20 -- -- 58 -- 68 -- -- --
RaBiGeTe ext 19 26 31 40 45 -- -- -- --
gjrand mcp 52 67 90 -- 102 -- 124 -- 134+
Dieharder (custom) 10 14 27 -- -- -- -- -- --
Dieharder (-a) -- -- -- -- 17 -- -- -- --
Times are very rough figures based upon modern desktop x86 CPUs, but using only one core (no multithreading).
(times were fairly similar on a variety of such CPUs)
All results are based upon a set of 190 RNGs, listed below.
Some categories of RNGs were omitted - candidate RNGs were not counted,
nor were "Transforms applied to various low quality RNGs".
total RNGs per category
category ALL recom. simple nonLCG LCGish cbuf indirect
number 190 19 50 19 33 29 40
broken down by category
recom. simple non-LCG LCGish cbuff indirect
-- (10 seconds)
PractRand std 512 MB 0 14 5 7 11 6
TestU01 SmallCrush 0 8 3 7 2 0
RaBiGeTe ext 2 Mb 0 6 4 7 2 0
gjrand mcp --small 0 14 7 8 15 8
Dieharder (custom 10sec) 0 4 4 1 2 0
-- (1 minute)
PractRand std 4 GB 0 21 5 13 15 11
RaBiGeTe ext 16 Mb 0 7 4 11 4 0
gjrand mcp --standard 0 18 10 12 16 11
Dieharder (custom 1min) 0 6 5 2 2 0
-- (5 minutes)
PractRand std 16 GB 0 27 6 14 20 16
RaBiGeTe ext 64 Mb 0 9 4 14 4 0
gjrand mcp --big 0 24 11 15 22 18
Dieharder (custom 5min) 0 8 6 2 8 5
-- (30 minutes)
PractRand std 128 GB 0 35 9 19 21 20
TestU01 Crush 1 22 8 16 6 5
RaBiGeTe ext 256 Mb 0 9 5 19 6 1
-- (1 hour)
PractRand std 256 GB 1 35 15 20 22 24
RaBiGeTe ext 512 Mb 0 9 8 21 8 2
gjrand mcp --huge 1 32 15 16 23 19
Dieharder -a 0 7 2 4 2 2
-- (4 hours)
PractRand std 1 TB 1 38 19 24 24 29
TestU01 BigCrush 1 24 13 18 8 7
-- (8 hours)
PractRand std 2 TB 1 40 20 24+ 25 31
gjrand mcp --tera 1 36 16 22 27 27
-- (1 day)
PractRand std 4 TB 1 42 20 24+ 26 34
-- (4 days)
PractRand std 16 TB 1 46 22 24+ 26 34
gjrand mcp --ten-tera 1 39 19 23 27 31+
test suites vs features
name PractRand TestU01 RaBiGeTe gjrand NIST STS Dieharder
unlimited test length YES no no YES? NO no
interim results YES no no no no no
multithreading YES no YES YES? no no
scriptable interface YES YES no YES no? YES
minimal false positives YES YES ~almost YES NO NO
cross-platform YES ~almost no ? ? NO
useful & concise output YES YES NO YES-ish NO ~almost
PractRand std:
The standard test battery of PractRand.
The only test suite to allow functionally unlimited test lengths.
update: gjrand allows such, though requires custom battery setup past 10 TB
The only test suite to offer interim results.
caveats:
Cheats slightly on all RNGs by looking at the metadata - it treats 8 bit RNGs slightly differently than 32 bit RNGs, etc
Requires more bits to find bias than any other test suite (of those listed).
May be cheating slightly on the LCGs - it often runs out the periods on those instead of finding bias the normal way.
Multithreading is supported, but max speedup tends to be limited to about 3x.
TestU01 *Crush:
Meaning SmallCrush, Crush, and BigCrush from TestU01.
The only test suite with a real academics in the field behind it.
The only test to guarantee (on default settings anyway) that all subtest results are 100% independant.
Although I think that only applies to subtest results with distinct numbers.
caveats:
Does not support multithreading.
More sensitive to RNG speed than the other test suites (of those listed) at default settings.
RaBiGeTe ext:
The "extended" preset of RaBiGeTe, but with DFT disabled.
(DFT wasn't adding anything to the results and was preventing longer tests from working)
Can find bias in many RNGs using fewer bits than the other test suites.
Supports multithreading better than the other test suites (of those listed).
caveats:
Tended to crash a bit.
I was using a pre-release version - the released versions at the time had some issues.
I had to ignore the authors instructions and make up my own to get clear & correct results.
The recent versions seem to be GUI-only, making it harder to automate.
The "30 minute" results actually took about 40 minutes... I was kinda generous there.
Dieharder (-a):
Dieharder used with the -a command line option.
(the only predefined set of tests in Dieharder, so far as I can tell)
Very few biases found
only a few tests showed significant ability to detect bias:
dab_monobit2 (the best test in Dieharder by far)
dab_dct (the next best test, I think)
dab_filltree
dab_filltree2
diehard_opso
diehard_oqso
dab_bytedistrib
and many tests had problems:
Test name Problems
marsaglia_tsang_gcd FALSE POSITIVES, very slow
sts_serial FALSE POSITIVES
rgb_lagged_sum FALSE POSITIVES
diehard_dna very slow
diehard_sums broken for metatests
diehard_runs broken for metatests
rgb_bitdist very slow
rgb_minimum_distance very slow
rgb_kstest broken for metatests
("very slow" refers to as parameterized in the "-a" set of tests)
takes about 40 minutes on my computer on a fast RNG
Dieharder (custom):
Used some shell scripts to run Dieharder with a custom arrangement of tests
Used several sets of tests & parameters for various time increments:
tests: monobit2 dct filltree2 filltree bytedistrib
10 second: 100000000 100000 -- -- --
1 minute: 500000000 500000 5000000 15000000 51200000
5 minutes: 2000000000 3000000 48000000 48000000 512000000
(all of those test names have the prefix "dab_" omitted)
Unfortunately, Dieharder does not permit test lengths longer than about
2 billion, so dab_monobit2 cannot increase beyond what's used in the 5
minute version there. Increasing the rest of the tests lengths produces
very little return on the investment, so there's not much point to a run
longer than 5 minutes.
As an example, the command lines I use for the 10 second test on the RNG
named ranrot32 is:
./RNG_output ranrot32 name
./RNG_output ranrot32 123456789000000 | dieharder -g 200 -D 504 -D 16384 -d dab_monobit2 -t 100000000
./RNG_output ranrot32 123456789000000 | dieharder -g 200 -D 504 -D 16384 -d dab_dct -t 100000
NIST STS (Statistical Test Suite)
I've made a few modifications to improve performance and allow it to
work on larger samples than it could normally.
It's producing false positives for me on the non-overlapping template
test, when using larger sample sizes and larger numbers of samples. At
128 samples the largest sample size I can safely use on that test is 8
megabits.
It seems to strongly encourage the use of large numbers of small
samples - its summary report will actually avoid listing individual
test p-values even if only one sample was run, instead showing only
number of test p-values within the normal range and a meta-test p-value.
Possibly it doesn't trust its basic p-value quality enough to show them,
or possibly it's just a brain-damaged interface.
It is extremely inefficient. It likes to unpack binary data to 1 bit
per byte, among other inefficiencies. It is also extremely weak, with
very poor efficiency, poor sensivity, and poor or very poor breadth.
gjrand mcp / pmcp
When limited to 10 seconds, I use mcp --small, for 1 minute I use
mcp --standard, for 5 minutes mcp --big, for half an hour... there's no
good option for that one, for 1 hour I use mcgp --huge, for 4 hours...
there's no good option for that, for 8 hours I use --tera (I added the
8 hour slot primarily for gjrand) and there's also --ten-tera but that
takes half a week and --tiny which takes ~3 seconds, neither of which
match any of my time slots well. (he used factors of 10 between each
test level, where I used factors of 2). I think it supports custom
test lengths, but since there were sufficient variety in the named
test lengths I stuck to those. pmcp is functionally multithreaded out
of the box, I think. The author recommends using the overall p-value
for primary evaluation, but with my current skills with grep it's
easier for me to use the "grade (XX) failure (SERIOUSLY BAD)" messages
instead, which work about as well. Plus there is the extra information
content that way of the number of tests failed.
This seems to be overall the best PRNG test suite out there aside from
PractRand. In fact, it's better than many versions of PractRand, and
part of the reason PractRand has improved is because of gjrand - the
idea used for PractRand's 'mod3n' test was based upon the mod3 test in
gjrand.
In terms of interface, IMHO it's not as good as PractRand or TestU01,
but better than anything else. The final summary is extremely readable
but missing the identity of which tests it failed (a relativeely high
priority datum I think). The individual test results are too verbose,
printing out numbers that have no meaning to those who don't know how
the tests work internally. The documentation fails to cover a few too
many areas.
The "slightly bad" result evaluation shows false positives occasionally,
but the "SERIOUSLY BAD" never has so far, though I suspect it will
sooner or later based upon some of the p-values I've seen it flunk.
Overall it's an awesome set of tests with a reasonable interface. My
biggest complaint is that it silently fails, producing garbage results,
if multiple instances of the test suite are run simultaneously.
From its output, I'm currently counting "grade 10 failures" and up as
failure, and grade 4 to grade 9 failures as suspicious. I run gjrand
mcp or pmcp once for each test length from --small to --tera, in
order, and if there are no failures or suspicious results I put a -
in the gjrand results column (from left to right, matching the
test length). If there are no failures, but there are suspicious
results, then I put a ~ in the results column. If there are
failures, I count them and put the number in the results column (in
hexadecimal, so that I can record numbers beyond 9 without throwing
off the formating). Thus a gjrand result listed as ---~3 means that
mcp or pmcp reported nothing suspicious at --small, --standard,
or --big, zero failing results but one or more suspicious results at
--huge, and three failing results at --tera. If there were no
failures at --tera, and I have spare CPU time, I then run it at
--ten-tera to add a final result.
*****************************************************************************
2. Full results
*****************************************************************************
recommended RNGs
RNG Quality PractRand TestU01 RaBiGeTe gjrand others
rating standard *Crush Extended mcp
hc256 5 > 1 TB pass pass ----- -
trivium 5 > 4 TB pass pass ----- -
efiix64x384 5 > 1 TB pass? pass? ----- -
efiix32x384 5 > 2 TB pass pass ----- -
efiix16x384 5 > 1 TB pass pass ----- -
efiix8x384 5 > 2 TB pass pass ----- -
isaac64x256 5 > 1 TB pass? pass? ----- -
isaac32x256 5 > 4 TB pass pass ----- -
chacha(8+) 5 > 4 TB pass? pass? ----- -
salsa(8+) 5+ > 4 TB pass? pass? ----- -
arbee 4 > 1 TB pass? pass? ----- -
xsm64 4 > 16 TB pass? pass? ----- -
xsm32 3 > 8 TB pass pass ------ -
jsf64 3 > 1 TB pass? pass? ----- -
jsf32 3 > 1 TB pass pass ------ -
sfc64 3 > 1 TB pass? pass? ----- -
sfc32 3 > 1 TB pass pass ------ -
sfc16 2 512 TB pass pass ------ -
rarns16 3 ?
mt19937 2 256 GB 0/2/2 pass ---111 -
(usually "pass?" means that a lower quality variant passed so I didn't fully test the higher quality variant)
other RNGs - simple (49)
RNG Quality PractRand TestU01 RaBiGeTe gjrand others
subscores standard *Crush Extended*1 mcp
* xorshift32 0/0/0/0 64 KB 5/53/? 64 Kb 579CC DHA0, STS
* xorshift64 0/4/2/0 64 KB 1/25/? 128 Kb 55556 DHA0, STS
xorshift32of128 0/5/3/1 512 KB 0/6/? 1 Mb 11111 -
xoroshiro128plus 0/5/3/1 32 MB pass pass --111 -
* xorwow32of96 0/5/3/1 2 MB 0/3/5 64 Mb 11344 -
xorwow32x6 1/5/4/1 16 MB 0/1/2 pass -1112 -
xsalta16x3 2/1/1/1 1 MB 0/1/3 pass 11111 -
xsaltb16x3 2/1/1/1 2 GB pass pass --135 -
* xsaltc16x3 1/1/1/1 2 GB 0/1/1 pass --124 -
* sapparot 0/2/2/1 16 MB 0/0/1 64 Mb 22345 -
* sap16of48 1/1/1/1 512 KB 0/3/2 1 Mb 11124 -
sap32of96 4/3/3/1 2 MB pass 4 Mb 11112 -
* flea32x1 0/4/3/2 1 MB pass pass 22245 -
jsf16 4/2/2/2 16 TB pass pass ------ -
* sfc_v1_16 1/1/1/1 8 GB 1/7/? pass -1124 DHA1, STS
* sfc_v1_32 1/3/2/1 8 TB 0/2/7 pass ---12 -
* sfc_v2_16 0/1/1/1 2 GB 0/1/0 pass ---23 DH2
. sfc_v2_32 2/3/3/2 4 TB 0/0/1 pass ----1 -
sfc_v3_16 2/1/1/1 512 GB pass pass -----~ -
sfc_v3_32 5/3/3/2 > 16 TB pass pass ------ -
* simpleA 1/3/3/1 2 GB pass pass 12334 DH2
* simpleB 0/1/1/0 128 MB 1/6/2 pass ---33 DH1
simpleC 0/1/1/1 16 GB 1/6/2 pass ---~1 -
* simpleD 2/3/3/2 128 GB pass pass -1233 -
* simpleE 3/3/3/2 1 TB pass pass -~113 -
* simpleF 2/2/2/1 16 GB pass pass ---12 -
* simpleG 1/3/3/2 128 GB 0/18/? pass --255 DHA
trivium_weakenedA 4/4/3/3 > 16 TB pass pass ------ -
trivium_weakenedB 3/3/3/3 1 TB pass pass ----4 -
* mo_Lesr32 0/0/0/0 64 KB 7/?/? 256 Kb 888AD DHA0, STS1
* mo_ResrRers32 4/2/2/2 2 TB pass pass ----~1 -
* mo_Rers32of64 0/2/2/0 8 MB 0/10/? 16 Kb 44455 DHA0
* mo_Resdra32of64 1/2/2/1 4 GB 0/2/6 pass -~-5C -
* murmlacish 3/3/3/2 4 TB 0/1/0 pass ----2 -
gjishA ?/?/?/? 32 GB pass pass ---12 -
gjishB ?/?/?/? > 16 TB pass pass ------ -
gjishC ?/?/?/? 64 GB 0/1/4 pass --~11 DHA
gjishD ?/?/?/? 128 GB pass pass --~12 -
ara16 2/1/1/1 1 GB 1/3/2 pass 11111 -
ara32 4/3/3/1 2 GB pass pass 11111 -
arx16 1/1/1/1 16 GB 1/3/2 pass -1122 -
arx32 4/3/3/2 2 TB pass pass ------ -
hara16 2/1/1/2 64 GB pass pass ~----- -
harx16 2/1/1/2 128 GB pass pass ----~2 -
learx16 2/1/1/2 128 GB pass pass --1112 -
hlearx16 1/1/1/2 16 GB pass pass ---~~2 -
alearx16 4/1/1/3 16 TB pass pass ------ -
arac16 1/2/2/2 8 GB 0/1/1 pass ------ -
arxc16 5/2/2/3 > 16 TB pass pass ------ -
*** 10 sec 14 (512 MB) 8 (SC) 6 (2 Mb) 14 (small)
*** 1 minute 21 (4 GB) -- 7 (16 Mb) 18 (standard)
*** 5 minutes 27 (16 GB) -- 9 (64 Mb) 24 (big)
*** 30 minutes 35 (128 GB) 22 (C) 9 (256 Mb) --
*** 1 hour 35 (256 GB) -- 9 (512 Mb) 32 (huge)
*** 4 hours 38 (1 TB) 24 (BC) -- --
*** 8 hours 40 (2 TB) -- -- 36 (tera)
*** 1 day 42 (4 TB) -- -- --
*** 4 days 45 (16 TB) -- -- 39 (ten-tera)
other RNGs - non-LCG multiplication based
RNG Quality PractRand TestU01 RaBiGeTe gjrand others
subscores standard *Crush Extended*1 mcp
garthy16 0/1/1/1 64 MB 0/15/? 256 Mb 128CC DH0
garthy32 2/3/3/1 1 TB pass pass --133 -
binarymult16 0/3/3/1 512 KB 1/31/2 1 Mb 2248D DHA0
binarymult32 2/4/3/2 2 MB pass 1 Mb 11224 DH1
rxmult16 0/2/2/1 128 GB ~pass pass ------ -
multish3x32 1/3/3/2 16 GB 0/0/1 pass 11111 -
multish4x16 1/2/2/1 512 KB 0/2/1 1 Mb 11111 -
mwrca16 0/0/0/0 16 MB 1/4/10 128 Mb ~23AC DH2
mwrca32 4/2/2/1 8 TB pass pass -----1 -
mwrcc16 0/1/1/1 128 MB 0/2/7 32 Mb 12447 DH0
mwrcc32 2/3/3/2 64 GB 0/0/1 16 Mb 13345 -
mwrcca16 0/1/1/1 512 MB pass pass ---~4 -
mwrcca32 4/3/3/2 > 32 TB pass pass -----1 -
old_mwlac16 2/2/2/2 256 GB pass pass ------ -
mwlac_varA ?/1/1/? 32 GB 0/1/1 pass -111~4 -
mwlac_varB ?/1/1/? 1 TB 0/2/2 pass -----4 -
mwlac_varC ?/1/1/? 64 GB ~pass pass -----2 -
mwlac_varD ?/1/1/? 16 TB 1/3/4 pass 11111 -
mwlac_varE ?/1/1/? 2 TB pass pass ------ -
mo_Cmfr32 0/0/0/0 64 KB 6/58/? 64 Kb 58ABD DHA0, STS
mo_Cmr32of64 3/2/2/2 512 GB 0/0/1 pass -1111 -
mulcr16 ?/1/1/2 512 GB ~pass pass -11--4 -
mulcr32 ?/3/3/? > 32 TB pass ~pass ------ -
mmr16 ?/1/1/1 32 GB 0/1/1 pass -----3 -
mmr32 ?/3/3/3 > 16 TB pass pass 11111 -
*** 10 sec 8 (512 MB) 4 (SC) 4 (2 Mb) 10 (small)
*** 1 minute 8 (4 GB) -- 5 (16 Mb) 14 (standard)
*** 5 minutes 9 (16 GB) -- 6 (64 Mb) 15 (big)
*** 30 minutes 14 (128 GB) 10 (C) 8 (256 Mb) --
*** 1 hour 15 (256 GB) -- 8 (512 Mb) ~15 (huge)
*** 4 hours 19 (1 TB) 13 (BC) -- --
*** 8 hours 20 (2 TB) -- -- ~16 (tera)
*** 1 day 20 (4 TB) -- -- --
*** 4 days 22 (16 TB) -- -- 21 (ten-tera)
other RNGs - LCG-based RNGs
RNG Quality PractRand TestU01 RaBiGeTe gjrand others
subscores standard *Crush Extended*1 mcp
lcg(16,32) 0/0/0/0 512 KB 6/48/?? 32 Kb 7ACBB DHA1,STS
lcg(16,40) 0/1/1/0 32 MB 1/29/?? 128 Kb 3559B -
lcg(16,48) 0/1/1/1 32 MB 1/7/?? 512 Kb 23334 -
lcg(16,56) 0/2/1/1 128 MB 1/6/?? 2 Mb 11223 -
lcg(16,64) 0/4/2/2 1 GB 0/3/4 4 Mb 11112 -
lcg(16,72) 1/4/2/2 4 GB pass 8 Mb -1111 -
lcg(16,80) 2/5/2/2 32 GB pass? 16 Mb --111 -
lcg(16,96) 4/5/3/2 1 TB pass? 256 Mb ------ -
lcg(16,112) 4/5/3/2 > 1 TB pass? pass -----_ -
xlcg(16,32) 0/0/0/0 512 KB 10/?/? 2 Mb 59BBB DHA0,STS
xlcg(16,40) 0/1/1/0 64 MB 1/11/? 128 Mb -238B -
xlcg(16,48) 1/1/1/1 8 GB 0/2/3 pass ---24 -
xlcg(16,56) 2/2/1/1 64 GB pass pass? ----2 -
xlcg(16,64) 3/4/2/2 128 GB pass pass? ----1 -
xlcg(16,72) 3/4/2/2 256 GB pass? pass? ------ -
xlcg(16,80) ?/5/2/2 > 1 TB pass? pass? ______ -
clcg(16,64) 1/3/2/1 1 GB 0/4/? 64 Mb -1323 -
clcg(16,68) 1/4/2/1 32 GB 0/2/3 128 Mb --1-2 -
clcg(16,72) 1/4/2/2 256 GB 0/1/1 256 Mb ---11 -
clcg(16,76) 1/4/2/2 1 TB pass 512 Mb ----1 -
clcg(16,80) 4/4/2/2 > 1 TB pass 512 Mb ----1 -
clcg(16,84) ?/4/2/2 > 16 TB? pass? pass ------ -
cxlcg(16,64) 0/3/2/1 1 GB 0/1/2 64 Mb --122 -
cxlcg(16,68) 2/4/2/1 64 GB pass 256 Mb ----2 -
cxlcg(16,72) 4/4/2/2 1 TB pass pass -----1 -
bblcg(32,160,32) ?/5/3/0 1 MB 0/21/?? 2 Mb 44555 DHA
bblcg(32,192,32) ?/5/4/1 2 GB 0/11/?? 4 Mb -1144 DHA
bblcg(32,224,32) ?/5/4/1 1 TB 0/2/5 64 Mb ------ -
? bblcg(32,256,32) ?/5/5/2 > 1 TB 0/0/1 ~pass -----_ -
? bblcg(32,288,32) ?/5/5/2 > 16 TB? pass pass -----_ -
pcg32_norot 1/4/2/2 2 GB 0/1/2 8 Mb ----1 -
? pcg32 5/4/2/3 > 4 TB pass pass ------ -
? cmrg32of192 5/5/4/3 > 1 TB pass pass ------ -
? xsh_lcg_bad 4/5/5/2 > 1 TB pass 32 Mb ------ -
*** 10 sec 7 (512 MB) 7 (SC) 7 (2 Mb) 8 (small)
*** 1 minute 13 (4 GB) -- 12 (16 Mb) 12 (standard)
*** 5 minutes 14 (16 GB) -- 16 (64 Mb) 15 (big)
*** 30 minutes 19 (128 GB) 17 (C) 21 (256 Mb) --
*** 1 hour 20 (256 GB) -- 23 (512 Mb) 16 (huge)
*** 4 hours 24 (1 TB) 18 (BC) -- --
*** 8 hours 24+(2 TB) -- -- 22 (tera)
*** 1 day 24+(4 TB) -- -- --
*** 4 days 24+(16 TB) -- -- 23 (ten-tera)
notes on the meaning of *lcg names:
lcg(X,Y) means an LCG with modulus 2^Y, discarding (Y-X) low bits to produce X bit output
clcg(X,Y) means lcg(X,Y-32) combined with a 32 bit non-power-of-2-modulus LCG
xlcg(X,Y) is similar to lcg(X,Y), but xor is used in place of addition
(i.e. equivalent to suppresing the carries on the addition in binary)
cxlcg(X,Y) means xlcg(X,Y-32) combined with a 32 bit non-power-of-2-modulus LCG
bigbadlcg(X,Y,Z) aka bblcg(X,Y,Z) means an LCG with modulus 2^Y, discarding (Y-X) low
bits to produce X bit output, with a low-quality multiplier of (1 + 2^Z)
other RNGs - cyclic buffer based (or other static structures)
RNG Quality PractRand TestU01 RaBiGeTe gjrand others
subscores standard *Crush Extended*1 mcp
mm32 0/5/5/0 2 MB 1/8/12 pass 45678 DH2
mm32_awc 0/5/5/1 256 MB 0/8/11 pass? 34567 DH2
mm16of32 0/5/5/0 1 GB 0/0/2 pass? 44555 DH2
mm16of32_awc 0/5/5/1 2 GB 0/0/1 pass? 44555 DH2
mwc4691 1/5/5/1 2 GB pass pass -1111 -
cbuf_accum 0/5/5/2 128 MB pass pass 14444 -
cbuf_accum_big 0/5/5/2 8 GB ~pass pass? --214 -
cbuf_2accum_small 0/5/4/2 512 GB pass pass --~22 -
cbuf_2accum 0/5/5/2 64 TB pass pass? ------ -
dual_cbuf_small 0/5/5/2 4 MB pass 16 Mb 12333 -
dual_cbuf 0/5/5/3 128 MB pass? pass --112 -
dual_cbufa_small 0/5/5/2 4 MB pass 16 Mb 11112 -
dual_cbuf_accum 0/5/5/2 128 MB pass? pass --1112 -
fibmul16of32 1/5/5/1 16 GB pass pass --124 -
fibmul32of64 3/5/5/2 512 GB pass pass ----1 -
fibmulmix16 ?/5/5/2 2 GB pass pass 11345 -
fibmulmix32 ?/5/5/3 64 GB pass? pass? 1111 -
ranrot32small ?/5/5/2 256 MB 0/1/0 256 Mb 44445 DHA0
ranrot32 2/5/5/2 2 GB pass pass 12222 -
ranrot32big 3/5/5/2 16 GB pass? pass? ----2 -
ranrot3tap32small ?/5/5/2 512 MB pass pass 22444 -
ranrot3tap32 3/5/5/3 256 GB pass pass --222 -
ranrot3tap32big 4/5/5/3 4 TB pass? pass? --~-2 -
ranrot32hetsmall 4/5/5/3 8 GB pass pass 11234 -
ranrot32het 4/5/5/3 2 TB pass pass? -~123 -
ranrot32hetbig 4/5/5/3 64 TB pass? pass? ----1 -
mt19937_unhashed 2/5/5/1 32 GB 0/2/2 pass 11122 -
salsa(3 rounds) 0/5/5/0 4 KB 13/??/?? 16 Kb BBBCC DHA0, STS
chacha(3 rounds) 0/5/5/1 32 MB 0/6/28 2 Mb 36999 DH2, STS
salsa(4 rounds) 1/5/5/2 16 GB ~pass 256 Mb 11257 DH2
chacha(4 rounds) 5/5/5/3 > 16 TB pass pass ------ -
*** 10 sec 11 (512 MB) 2 (SC) 2 (2 Mb) 17 (small)
*** 1 minute 16 (4 GB) -- 4 (16 Mb) 18 (standard)
*** 5 minutes 21 (16 GB) -- 4 (64 MB) 24 (big)
*** 30 minutes 23 (128 GB) 6 (C) 6 (256 Mb) --
*** 1 hour 24 (256 GB) -- 6 (512 Mb) 25 (huge)
*** 4 hours 26 (1 TB) 8 (BC) -- --
*** 8 hours 27 (2 TB) -- -- 29 (tera)
*** 1 day 28 (4 TB) -- -- --
*** 4 days 28 (16 TB) -- -- 29 (ten-tera)
other RNGs - indirection based
RNG Quality PractRand TestU01 RaBiGeTe gjrand others
subscores standard *Crush Extended*1 mcp
ibaa8x2 0/0/0/1 512 MB 0/8/19 512 Mb -~8CC DH2
ibaa8x4 1/1/1/2 8 GB pass pass --~12 -
ibaa8x8 4/3/2/3 1 TB pass pass -----1 -
ibaa16x2 2/2/2/2 2 GB pass pass -1123 -
ibaa16x4 4/3/3/2 256 GB pass? pass ----1 -
ibaa16x8 5/3/3/3 > 16 TB pass? pass? ------ -
ibaa32x2 4/3/3/2 2 GB pass pass 11111 -
ibaa32x4 4/3/3/2 256 GB pass? pass? --111 -
ibaa32x8 4/5/5/3 > 16 TB? pass? pass? ----1 -
rc4_weakenedA 1/5/5/2 8 GB 0/1/1 pass 12226 -
rc4_weakenedB 2/5/5/2 256 GB pass pass ---~1 -
rc4_weakenedC 1/5/5/2 16 MB 0/27/? 128 Mb 15579 DHA
rc4_weakenedD 1/5/5/2 64 GB pass pass ----~4 -
rc4 4/5/5/3 1 TB pass pass -----2 -
isaac16x4 0/4/3/2 4 MB 0/0/1 pass ---14 -
isaac16x8 0/5/4/3 1 GB pass pass? -----1 -
isaac16x16 4/5/5/4 2 TB pass? pass? ------ -
isaac32x4 0/5/4/2 8 MB pass pass 11112 -
isaac32x8 0/5/5/3 256 MB pass pass? --111 -
isaac32x16 4/5/5/4 4 TB pass? pass? -----1 -
genindA(5) 0/5/5/1 128 MB 0/0/1 pass 25567 -
genindA(7) 3/5/5/2 256 GB pass? pass? 12334 -
genindA(9) 4/5/5/3 64 TB pass? pass? ------ -
genindB(1) 0/0/2/0 1 GB 0/52/? pass ~~CCC DHA2
genindB(2) 1/1/3/1 16 GB pass pass? ---4C -
genindB(3) 1/3/4/2 512 GB pass? pass? ----2 -
genindB(4) 4/4/5/2 4 TB pass? pass? -----2 -
genindC(2) 1/3/3/1 2 GB 0/2/2 pass -3444 DH2
genindC(3) 2/4/3/2 64 GB pass pass? 11344 -
genindC(4) 4/5/5/2 4 TB pass? pass? ----1 -
genindC(5) 4/5/5/3 > 16 TB pass? pass? -----1 -
genindD(6) 1/5/5/2 32 GB pass pass -1123 DH2
genindD(9) 3/5/5/3 2 TB pass? pass? ~~-~1 -
genindE(1) 2/1/3/1 8 GB pass pass ---~3 -
genindE(2) 3/2/3/2 512 GB pass? pass? -~--1 -
genindE(3) 4/5/5/3 64 TB pass? pass? -~---- -
genindF(2) 1/2/3/2 16 GB *pass pass --247 DH2
genindF(3) 2/5/5/2 128 GB pass pass? --~24 -
genindF(4) 3/5/5/3 1 TB pass? pass? ----~3 -
genindF(5) 4/5/5/3 32 TB pass? pass? -----~ -
*** 10 sec 6 (512 MB) 0 (SC) 0 (2 Mb) 7 (small)
*** 1 minute 11 (4 GB) -- 0 (16 Mb) 10 (standard)
*** 5 minutes 16 (16 GB) -- 0 (64 Mb) 15 (big)
*** 30 minutes 20 (128 GB) 5 (C) 1 (256 Mb) --
*** 1 hour 24 (256 GB) -- 2 (512 Mb) 19 (huge)
*** 4 hours 29 (1 TB) 7 (BC) -- --
*** 8 hours 31 (2 TB) -- -- 27 (tera)
*** 1 day 34 (4 TB) -- -- --
*** 4 days 34 (16 TB) -- -- 34 (ten-tera)
note: I wrote a few generic indirection-based PRNGs so there's more variety here.
They're generally pretty mediocre. To keep quality low enough to find flaws
in I mostly kept their accumulator / mixing pool down to a single word, and
limited their word size to 16 bits. To determine the size of their indirection
arrays I gave them each a parameter that is the log2 of that size, and started
it at the lowest value to fail at least 3 different test batteries (usually
PractRand and gjrand make up two of those three, and the third is either
TestU01 or Dieharder). If that resulted in a size too small for indirection
to occur then the PRNG was discarded. From there I increased the
log2-of-array-size parameter by 50% of the lowest value of that parameter,
rounded down to a minimum of 1, and kept increasing it by that amount until it
looked likely to pass all test batteries at the next size.
*****************************************************************************
3. Details of the results format
*****************************************************************************
columns:
RNG
The name of the RNG algorithm used;
The name used is the name it is called in PractRand, not necessarily
its real full name - for instance, hc256 instead of HC-256.
PractRand standard
The PractRand standard tests are those returned by
PractRand::Tests::Batteries::get_standard_tests().
quirks:
test set is (slightly) dependent upon RNG metadata
speed on a fast RNG on my computer (in single-threaded mode):
1 GB: ~15 seconds
1 TB: ~4 hours
format:
The length of sequence required for the PractRand standard tests
to detect bias. If multiple seeds were tested, and bias was
found in them at different sequence lengths, then the median
result is used. If bias was not found then the length of the
longest sequence tested is recorded, with a ">" prefixed in
front of it.
Bias is considered to have been found when the standard tool
rates the result as "FAIL" (with or without exclamation marks).
(p-value threshold vary with circumstances but are typically
near 1e-10 on the low side and 1 - 1e-10 on the high side)
The length is recorded as a number followed by KB, MB, GB, or
TB. Those stand for kilobyte, megabyte, gigabyte, or terabyte
respectively.
All lengths are rounded up to the nearest power of 2.
TestU01 *Crush
This refers to the batteries "SmallCrush", "Crush", and "BigCrush"
in TestU01.
quirks:
does not accept a sequence length parameter
works on RNGs outputing 31 bits per call:
on 32 bit RNGs it works but disregards the lowest bit?!?
on 8/16 bit RNGs, I concatonate multiple outputs together to produce a 32 bit output
on 64 bit RNGs... I am currently discarding the upper 32 bits to produce a 32 bit output
might, in the future, change that to returning half and buffering half for the next call
does not print any results until all are ready
speed on a fast RNG on my computer:
SmallCrush: ~8 seconds
Crush: ~30 minutes
BigCrush: ~4 hours
format:
If all 3 are passed then the result is recorded as "pass".
If all 3 are passed, but only barely, the result is recorded as "~pass".
If all 3 are passed, but it is noticed that bit-reversed versions of the PRNG fail one or more,
the result is recorded as "*pass" (note that this case is not checked systematically)
Otherwise, the result is recorded as X/Y/Z, where X is the
number of tests failed in SmallCrush, Y is the number failed in
Crush, and Z is the number failed in BigCrush.
Failure is defined as a p-value <= 1.0e-10 or >= (1 - 1.0e-10)
RaBiGeTe extended
This refers to the preset testing parameters named "extended", but see the quirks section.
quirks:
In order to get 512 Mb samples to run on extended I had to disable the DFT test
I used a more recent version of RaBiGeTe that has not been publicly released at this time.
I used RaBiGeTe in a slightly non-standard way, testing single long samples instead of many short samples.
In my testing this method worked much better than the recommended way.
It crashed several times during testing.
speed on a fast RNG on my computer:
16 Mb: just over 1 minute
512 Mb: a little under 40 minutes
format:
The same format was used as for PractRand standard, with three exceptions.
A result was considered a failure if any p-value was <= 1e-10, or if 5 or
more p-values were <= 0.000001 or >= 0.999999, or if any p-values were
omitted from the results. I never consider a single p-value close to one
a failure in RaBiGeTe even if it is exactly 1.0.
The second exception is that lengths are listed in bits instead of bytes.
The third exception is that if an RNG passed 512 megabits of testing then it was listed as "pass".
... because RaBiGeTe was unable to test sequences longer than 512 Mb on the settings I used.
If a result is listed as "?" that means that that test has not been run
on that RNG. If a result has a normal value followed by a "?" that
means that either that result is strongly implied by tests on nearly
equivalent RNGs, or that the test was run and is believed to have
produced that result but the record-keeping involved was incomplete and
I can't be sure without re-running the test.
*****************************************************************************
4. Tests that were not included
*****************************************************************************
tests that were not included:
PractRand expanded standard
This test battery was not used because:
A. It significantly underperforms PractRand standard on fast RNGs.
B. If I listed its test results here I'd have to update them
every time I added a new test to it.
TestU01 Rabbit
This test battery was not used because:
A. It consistently crashes if used on a long sequence
B. Its time to complete a test does not increase linearly with
sequence length - in some cases it actually decreases with
length. This appears to be partially caused by the test
set being a function of the testing length.
C. Its results do not seem to improve in power with sequence
length consistently enough.
TestU01 Pseudo-diehard
This test battery was not used because its quality is too low.
NIST RNG tests
This test battery was not used because:
A. Severe difficulty with longer test runs due to not only
keeping the whole sequence in memory, but also
inflating the sequence to store only one bit per byte.
B. Poor quality relative to speed, poor maximum quality
given the short maximum sample lengths.
*****************************************************************************