Dieharder Goldstandard Reference Values

Dieharder (http://www.phy.duke.edu/~rgb/General/dieharder.php) is a test suite for random number generators developed by R.G. Brown, D. Eddelbuettel, D. Bauer and others as open source and released under the GPL license.

One issue that inevitably crops up whenever you want to test a new Pseudo Random Number Generator (PRNG) is the question of quality: How good is the randomness of the PRNG? While it is very difficult if not impossible to rigorously prove the quality of a PRNG with mathematical means (although this would be very much appreciated) one can use statistical tests (as those provided by Dieharder) on the PRNG and compare the results to other PRNGs. To do this, one has to have a notion of what accepted "goldstandard" PRNGs achieve in each test. Unfortunately, when I started looking for these reference values, they were still missing. So I decided to spend some computation time of my university desktop pc (thanks Univesity of Bristol!) to run Dieharder destructively on the AES in Output Feedback Mode PRNG provided by Dieharder.

This page hardly presents any results of my own. It presents the results of Dieharder which has been developed by R.G. Brown et al. as noted above. All I've done is letting the program run for a couple of weeks and write this page so that other people don't have to do all the effort over and over again.

A few cautionary notes:

  • I am not an expert in PRNG testing. Please take everything I write here cautiously and feel free to alert me if you believe that this site contains errors. (I wouldn't be surprised if there weren't at least minor errors.)
  • Statistical tests always fail with some probability; if they are run indefinitely one would expect certain failure.
  • To decide, whether a test or a PRNG failed, one has to see whether the PRNG failed a test earlier (or more often/regular) than accepted goldstandard PRNGs.
  • AES has been thoroughly researched and the results further down shouldn't raise any doubts about its security. Of course, everyone is invited to investigate AES further based on those results but these results rather comment on the quality of the test than on the quality of AES.

Results
The test of AES was invoked using:
   ./dieharder -a -S 1 -g 205 -k 2 -Y 2 \
      > report_dieharder_205_AES_OFB-1_k2_Y2.txt \
      2>&1

  • Version: 3.31.0
  • -a: Run all tests with default options
  • -S 1: Seed = 1; The program ignored this parameter and used a random seed = 1939101936 instead
  • -g 205: generator 205, i.e. AES_OFB
  • -k 2: calculations accurate up to machine precision
  • -Y 2: Xtrategy = test to destruction

For Threefish and mt19937 a script was used to run one test after another. As far as results for these generators are available, they have been added to the table. The full results can be downloaded here: AES, Threefish, mt19937. A brief overview is given below:

test_name ntup tsamples psamples p-value Assessment Comparison: psamples/p-value for Threefish Comparison: psamples/p-value for mt19937
diehard_birthdays 0 100 70900 0.00000098 FAILED 98300/0.00000099 63300/0.00000095
diehard_operm5 0 1000000 100000 0.61458795 PASSED 100000/0.51650055 100000/0.78084462
diehard_rank_32x32 0 40000 100000 0.44714624 PASSED 100000/0.34086065 100000/0.37098925
diehard_rank_6x8 0 100000 100000 0.00545488 PASSED 100000/0.06448326 100000/0.03060202
diehard_bitstream 0 2097152 100000 0.80038777 PASSED 100000/0.90000067 100000/0.81656048
diehard_opso 0 2097152 100000 0.38789443 PASSED 100000/0.13998456 100000/0.21886882
diehard_oqso 0 2097152 100000 0.01138907 PASSED 100000/0.00252519 100000/0.01686032
diehard_dna 0 2097152 5200 0.00000094 FAILED 5400/0.00000094 4600/0.00000099
diehard_count_1s_str 0 256000 100000 0.00993817 PASSED 100000/0.05100613 100000/0.03027278
diehard_count_1s_byt 0 256000 100000 0.00648072 PASSED 100000/0.00057207 100000/0.04740271
diehard_parking_lot 0 12000 12800 0.00000099 FAILED 18800/0.00000060 22200/0.00000093
diehard_2dsphere 2 8000 100000 0.87986578 PASSED 100000/0.62853683 100000/0.17408518
diehard_3dsphere 3 4000 100000 0.32167897 PASSED 100000/0.99485902 100000/0.14567653
diehard_squeeze 0 100000 100000 0.01283570 PASSED 100000/0.51678600 100000/0.10679120
diehard_sums 0 100 700 0.00000060 FAILED 1300/0.00000059 1100/0.00000049
diehard_runs 0 100000 16700 0.00000092 FAILED 19100/0.00000099 35800/0.00000091
diehard_craps 0 200000 100000 0.30141395 PASSED 100000/0.71807874 100000/0.12183268
marsaglia_tsang_gcd 0 10000000 12300 0.00000092 FAILED 24500/0.00000089 21200/0.00000055
sts_monobit 1 100000 100000 0.71599336 PASSED 100000/0.40375682 100000/0.51651080
sts_runs 2 100000 100000 0.09495699 PASSED 100000/0.04156468 100000/0.30099063
sts_serial 1 100000 100000 0.56516109 PASSED 100000/0.99337503 100000|0.82514017
sts_serial 2 100000 100000 0.36582639 PASSED 100000/0.88591962 100000|0.39184630
sts_serial 3 100000 100000 0.69589183 PASSED 100000/0.72490571 100000|0.24943275
sts_serial 3 100000 100000 0.59979201 PASSED 100000/0.90627245 100000|0.82126508
sts_serial 4 100000 100000 0.38110949 PASSED 100000/0.20353619 100000|0.49788351
sts_serial 4 100000 100000 0.75954703 PASSED 100000/0.33796210 100000|0.50418965
sts_serial 5 100000 100000 0.11607107 PASSED 100000/0.53697418 100000|0.60645437
sts_serial 5 100000 100000 0.13453141 PASSED 100000/0.85212859 100000|0.32245589
sts_serial 6 100000 100000 0.57327060 PASSED 100000/0.52314382 100000|0.59154197
sts_serial 6 100000 100000 0.43786665 PASSED 100000/0.36292000 100000|0.64183029
sts_serial 7 100000 100000 0.59187740 PASSED 100000/0.97767584 100000|0.39081130
sts_serial 7 100000 100000 0.80834235 PASSED 100000/0.31627474 100000|0.75436348
sts_serial 8 100000 100000 0.88563574 PASSED 100000/0.85682445 100000|0.46720454
sts_serial 8 100000 100000 0.36247609 PASSED 100000/0.51140364 100000|0.53479737
sts_serial 9 100000 100000 0.88518457 PASSED 100000/0.07746903 100000|0.91514892
sts_serial 9 100000 100000 0.60236339 PASSED 100000/0.11589292 100000|0.31768462
sts_serial 10 100000 100000 0.39632022 PASSED 100000/0.14755926 100000|0.84210437
sts_serial 10 100000 100000 0.97670536 PASSED 100000/0.01189872 100000|0.38814409
sts_serial 11 100000 100000 0.55077412 PASSED 100000/0.53061459 100000|0.43575790
sts_serial 11 100000 100000 0.78379064 PASSED 100000/0.76267505 100000|0.41640359
sts_serial 12 100000 100000 0.26861331 PASSED 100000/0.53880531 100000|0.18025512
sts_serial 12 100000 100000 0.47122842 PASSED 100000/0.99054259 100000|0.21155732
sts_serial 13 100000 100000 0.29405651 PASSED 100000/0.76679694 100000|0.09715719
sts_serial 13 100000 100000 0.14729023 PASSED 100000/0.52081311 100000|0.69027960
sts_serial 14 100000 100000 0.55473799 PASSED 100000/0.08759404 100000|0.87956134
sts_serial 14 100000 100000 0.26361467 PASSED 100000/0.59788860 100000|0.05784208
sts_serial 15 100000 100000 0.60072257 PASSED 100000/0.61179224 100000|0.30265685
sts_serial 15 100000 100000 0.19461093 PASSED 100000/0.37987593 100000|0.29764078
sts_serial 16 100000 100000 0.40353109 PASSED 100000/0.12528466 100000|0.80556510
sts_serial 16 100000 100000 0.63363466 PASSED 100000/0.14839045 100000|0.17906999
rgb_bitdist 1 100000 100000 0.71299018 PASSED
rgb_bitdist 2 100000 100000 0.82642105 PASSED
rgb_bitdist 3 100000 100000 0.06861896 PASSED
rgb_bitdist 4 100000 100000 0.64790849 PASSED
rgb_bitdist 5 100000 100000 0.27438362 PASSED
rgb_bitdist 6 100000 100000 0.31212000 PASSED
rgb_bitdist 7 100000 100000 0.53367909 PASSED
rgb_bitdist 8 100000 100000 0.49169214 PASSED
rgb_bitdist 9 100000 100000 0.51180971 PASSED
rgb_bitdist 10 100000 100000 0.32340691 PASSED
rgb_bitdist 11 100000 100000 0.00001399 WEAK
rgb_bitdist 12 100000 85400 0.00000099 FAILED
rgb_minimum_distance 2 10000 100000 0.57129044 PASSED
rgb_minimum_distance 3 10000 100000 0.99176454 PASSED
rgb_minimum_distance 4 10000 100000 0.10725057 PASSED
rgb_minimum_distance 5 10000 67200 0.00000099 FAILED
rgb_permutations 2 100000 100000 0.09749864 PASSED
rgb_permutations 3 100000 100000 0.94280777 PASSED
rgb_permutations 4 100000 100000 0.70544607 PASSED
rgb_permutations 5 100000 100000 0.65510453 PASSED 100000/0.29119227 100000/0.47199357
rgb_lagged_sum 0 1000000 100000 0.93655760 PASSED 100000/0.67558969 100000/0.24115259
rgb_lagged_sum 1 1000000 100000 0.71434042 PASSED
rgb_lagged_sum 2 1000000 100000 0.50431680 PASSED
rgb_lagged_sum 3 1000000 100000 0.05946470 PASSED
rgb_lagged_sum 4 1000000 100000 0.09233652 PASSED
rgb_lagged_sum 5 1000000 100000 0.78429507 PASSED
rgb_lagged_sum 6 1000000 100000 0.70431509 PASSED
rgb_lagged_sum 7 1000000 100000 0.84959226 PASSED
rgb_lagged_sum 8 1000000 100000 0.23160937 PASSED
rgb_lagged_sum 9 1000000 100000 0.62978194 PASSED
rgb_lagged_sum 10 1000000 100000 0.80398889 PASSED
rgb_lagged_sum 11 1000000 100000 0.51451670 PASSED
rgb_lagged_sum 12 1000000 100000 0.04198232 PASSED
rgb_lagged_sum 13 1000000 100000 0.49844099 PASSED
rgb_lagged_sum 14 1000000 100000 0.23883388 PASSED
rgb_lagged_sum 15 1000000 100000 0.58890320 PASSED
rgb_lagged_sum 16 1000000 100000 0.33541037 PASSED
rgb_lagged_sum 17 1000000 100000 0.00866121 PASSED
rgb_lagged_sum 18 1000000 100000 0.21696271 PASSED
rgb_lagged_sum 19 1000000 100000 0.52035392 PASSED
rgb_lagged_sum 20 1000000 100000 0.90284911 PASSED
rgb_lagged_sum 21 1000000 100000 0.62080074 PASSED
rgb_lagged_sum 22 1000000 100000 0.50470000 PASSED
rgb_lagged_sum 23 1000000 100000 0.30885183 PASSED
rgb_lagged_sum 24 1000000 100000 0.56839772 PASSED
rgb_lagged_sum 25 1000000 100000 0.46229428 PASSED
rgb_lagged_sum 26 1000000 100000 0.00231344 WEAK
rgb_lagged_sum 27 1000000 100000 0.85672672 PASSED
rgb_lagged_sum 28 1000000 100000 0.53200641 PASSED
rgb_lagged_sum 29 1000000 100000 0.15289233 PASSED
rgb_lagged_sum 30 1000000 100000 0.75899614 PASSED
rgb_lagged_sum 31 1000000 100000 0.11741910 PASSED
rgb_lagged_sum 32 1000000 100000 0.67451804 PASSED
rgb_kstest_test 0 10000 61100 0.00000091 FAILED 69200/0.00000089 99300/0.00000092
dab_bytedistrib 0 51200000 100000 0.97693343 PASSED 100000/0.73031233 100000/0.58626499
dab_dct 256 50000 100000 0.17400145 PASSED 100000/0.14731270 100000/0.24133917
dab_filltree 32 15000000 100000 0.43722842 PASSED 100000/0.29776867 100000/0.83860736
dab_filltree 32 15000000 100000 0.39516879 PASSED 100000/0.76524376 100000/0.32081520
dab_filltree2 0 5000000 100000 0.01584526 PASSED 100000/0.63271678 100000/0.13847552
dab_filltree2 1 5000000 100000 0.39737549 PASSED 100000/0.38677116 100000/0.58485250
dab_monobit2 12 65000000 901 0.00000004 FAILED 501/0.00000057 901/0.00000023