Dieharder (http://www.phy.duke.edu/~rgb/General/dieharder.php) is a test suite for random number generators developed by R.G. Brown, D. Eddelbuettel, D. Bauer and others as open source and released under the GPL license.
One issue that inevitably crops up whenever you want to test a new Pseudo Random Number Generator (PRNG) is the question of quality: How good is the randomness of the PRNG? While it is very difficult if not impossible to rigorously prove the quality of a PRNG with mathematical means (although this would be very much appreciated) one can use statistical tests (as those provided by Dieharder) on the PRNG and compare the results to other PRNGs. To do this, one has to have a notion of what accepted "goldstandard" PRNGs achieve in each test. Unfortunately, when I started looking for these reference values, they were still missing. So I decided to spend some computation time of my university desktop pc (thanks Univesity of Bristol!) to run Dieharder destructively on the AES in Output Feedback Mode PRNG provided by Dieharder.
This page hardly presents any results of my own. It presents the results of Dieharder which has been developed by R.G. Brown et al. as noted above. All I've done is letting the program run for a couple of weeks and write this page so that other people don't have to do all the effort over and over again.
A few cautionary notes:
Results
The test of AES was invoked using:
./dieharder -a -S 1 -g 205 -k 2 -Y 2 \
> report_dieharder_205_AES_OFB-1_k2_Y2.txt \
2>&1
For Threefish and mt19937 a script was used to run one test after another. As far as results for these generators are available, they have been added to the table. The full results can be downloaded here: AES, Threefish, mt19937. A brief overview is given below:
test_name | ntup | tsamples | psamples | p-value | Assessment | Comparison: psamples/p-value for Threefish | Comparison: psamples/p-value for mt19937 |
---|---|---|---|---|---|---|---|
diehard_birthdays | 0 | 100 | 70900 | 0.00000098 | FAILED | 98300/0.00000099 | 63300/0.00000095 |
diehard_operm5 | 0 | 1000000 | 100000 | 0.61458795 | PASSED | 100000/0.51650055 | 100000/0.78084462 |
diehard_rank_32x32 | 0 | 40000 | 100000 | 0.44714624 | PASSED | 100000/0.34086065 | 100000/0.37098925 |
diehard_rank_6x8 | 0 | 100000 | 100000 | 0.00545488 | PASSED | 100000/0.06448326 | 100000/0.03060202 |
diehard_bitstream | 0 | 2097152 | 100000 | 0.80038777 | PASSED | 100000/0.90000067 | 100000/0.81656048 |
diehard_opso | 0 | 2097152 | 100000 | 0.38789443 | PASSED | 100000/0.13998456 | 100000/0.21886882 |
diehard_oqso | 0 | 2097152 | 100000 | 0.01138907 | PASSED | 100000/0.00252519 | 100000/0.01686032 |
diehard_dna | 0 | 2097152 | 5200 | 0.00000094 | FAILED | 5400/0.00000094 | 4600/0.00000099 |
diehard_count_1s_str | 0 | 256000 | 100000 | 0.00993817 | PASSED | 100000/0.05100613 | 100000/0.03027278 |
diehard_count_1s_byt | 0 | 256000 | 100000 | 0.00648072 | PASSED | 100000/0.00057207 | 100000/0.04740271 |
diehard_parking_lot | 0 | 12000 | 12800 | 0.00000099 | FAILED | 18800/0.00000060 | 22200/0.00000093 |
diehard_2dsphere | 2 | 8000 | 100000 | 0.87986578 | PASSED | 100000/0.62853683 | 100000/0.17408518 |
diehard_3dsphere | 3 | 4000 | 100000 | 0.32167897 | PASSED | 100000/0.99485902 | 100000/0.14567653 |
diehard_squeeze | 0 | 100000 | 100000 | 0.01283570 | PASSED | 100000/0.51678600 | 100000/0.10679120 |
diehard_sums | 0 | 100 | 700 | 0.00000060 | FAILED | 1300/0.00000059 | 1100/0.00000049 |
diehard_runs | 0 | 100000 | 16700 | 0.00000092 | FAILED | 19100/0.00000099 | 35800/0.00000091 |
diehard_craps | 0 | 200000 | 100000 | 0.30141395 | PASSED | 100000/0.71807874 | 100000/0.12183268 |
marsaglia_tsang_gcd | 0 | 10000000 | 12300 | 0.00000092 | FAILED | 24500/0.00000089 | 21200/0.00000055 |
sts_monobit | 1 | 100000 | 100000 | 0.71599336 | PASSED | 100000/0.40375682 | 100000/0.51651080 |
sts_runs | 2 | 100000 | 100000 | 0.09495699 | PASSED | 100000/0.04156468 | 100000/0.30099063 |
sts_serial | 1 | 100000 | 100000 | 0.56516109 | PASSED | 100000/0.99337503 | 100000|0.82514017 |
sts_serial | 2 | 100000 | 100000 | 0.36582639 | PASSED | 100000/0.88591962 | 100000|0.39184630 |
sts_serial | 3 | 100000 | 100000 | 0.69589183 | PASSED | 100000/0.72490571 | 100000|0.24943275 |
sts_serial | 3 | 100000 | 100000 | 0.59979201 | PASSED | 100000/0.90627245 | 100000|0.82126508 |
sts_serial | 4 | 100000 | 100000 | 0.38110949 | PASSED | 100000/0.20353619 | 100000|0.49788351 |
sts_serial | 4 | 100000 | 100000 | 0.75954703 | PASSED | 100000/0.33796210 | 100000|0.50418965 |
sts_serial | 5 | 100000 | 100000 | 0.11607107 | PASSED | 100000/0.53697418 | 100000|0.60645437 |
sts_serial | 5 | 100000 | 100000 | 0.13453141 | PASSED | 100000/0.85212859 | 100000|0.32245589 |
sts_serial | 6 | 100000 | 100000 | 0.57327060 | PASSED | 100000/0.52314382 | 100000|0.59154197 |
sts_serial | 6 | 100000 | 100000 | 0.43786665 | PASSED | 100000/0.36292000 | 100000|0.64183029 |
sts_serial | 7 | 100000 | 100000 | 0.59187740 | PASSED | 100000/0.97767584 | 100000|0.39081130 |
sts_serial | 7 | 100000 | 100000 | 0.80834235 | PASSED | 100000/0.31627474 | 100000|0.75436348 |
sts_serial | 8 | 100000 | 100000 | 0.88563574 | PASSED | 100000/0.85682445 | 100000|0.46720454 |
sts_serial | 8 | 100000 | 100000 | 0.36247609 | PASSED | 100000/0.51140364 | 100000|0.53479737 |
sts_serial | 9 | 100000 | 100000 | 0.88518457 | PASSED | 100000/0.07746903 | 100000|0.91514892 |
sts_serial | 9 | 100000 | 100000 | 0.60236339 | PASSED | 100000/0.11589292 | 100000|0.31768462 |
sts_serial | 10 | 100000 | 100000 | 0.39632022 | PASSED | 100000/0.14755926 | 100000|0.84210437 |
sts_serial | 10 | 100000 | 100000 | 0.97670536 | PASSED | 100000/0.01189872 | 100000|0.38814409 |
sts_serial | 11 | 100000 | 100000 | 0.55077412 | PASSED | 100000/0.53061459 | 100000|0.43575790 |
sts_serial | 11 | 100000 | 100000 | 0.78379064 | PASSED | 100000/0.76267505 | 100000|0.41640359 |
sts_serial | 12 | 100000 | 100000 | 0.26861331 | PASSED | 100000/0.53880531 | 100000|0.18025512 |
sts_serial | 12 | 100000 | 100000 | 0.47122842 | PASSED | 100000/0.99054259 | 100000|0.21155732 |
sts_serial | 13 | 100000 | 100000 | 0.29405651 | PASSED | 100000/0.76679694 | 100000|0.09715719 |
sts_serial | 13 | 100000 | 100000 | 0.14729023 | PASSED | 100000/0.52081311 | 100000|0.69027960 |
sts_serial | 14 | 100000 | 100000 | 0.55473799 | PASSED | 100000/0.08759404 | 100000|0.87956134 |
sts_serial | 14 | 100000 | 100000 | 0.26361467 | PASSED | 100000/0.59788860 | 100000|0.05784208 |
sts_serial | 15 | 100000 | 100000 | 0.60072257 | PASSED | 100000/0.61179224 | 100000|0.30265685 |
sts_serial | 15 | 100000 | 100000 | 0.19461093 | PASSED | 100000/0.37987593 | 100000|0.29764078 |
sts_serial | 16 | 100000 | 100000 | 0.40353109 | PASSED | 100000/0.12528466 | 100000|0.80556510 |
sts_serial | 16 | 100000 | 100000 | 0.63363466 | PASSED | 100000/0.14839045 | 100000|0.17906999 |
rgb_bitdist | 1 | 100000 | 100000 | 0.71299018 | PASSED | ||
rgb_bitdist | 2 | 100000 | 100000 | 0.82642105 | PASSED | ||
rgb_bitdist | 3 | 100000 | 100000 | 0.06861896 | PASSED | ||
rgb_bitdist | 4 | 100000 | 100000 | 0.64790849 | PASSED | ||
rgb_bitdist | 5 | 100000 | 100000 | 0.27438362 | PASSED | ||
rgb_bitdist | 6 | 100000 | 100000 | 0.31212000 | PASSED | ||
rgb_bitdist | 7 | 100000 | 100000 | 0.53367909 | PASSED | ||
rgb_bitdist | 8 | 100000 | 100000 | 0.49169214 | PASSED | ||
rgb_bitdist | 9 | 100000 | 100000 | 0.51180971 | PASSED | ||
rgb_bitdist | 10 | 100000 | 100000 | 0.32340691 | PASSED | ||
rgb_bitdist | 11 | 100000 | 100000 | 0.00001399 | WEAK | ||
rgb_bitdist | 12 | 100000 | 85400 | 0.00000099 | FAILED | ||
rgb_minimum_distance | 2 | 10000 | 100000 | 0.57129044 | PASSED | ||
rgb_minimum_distance | 3 | 10000 | 100000 | 0.99176454 | PASSED | ||
rgb_minimum_distance | 4 | 10000 | 100000 | 0.10725057 | PASSED | ||
rgb_minimum_distance | 5 | 10000 | 67200 | 0.00000099 | FAILED | ||
rgb_permutations | 2 | 100000 | 100000 | 0.09749864 | PASSED | ||
rgb_permutations | 3 | 100000 | 100000 | 0.94280777 | PASSED | ||
rgb_permutations | 4 | 100000 | 100000 | 0.70544607 | PASSED | ||
rgb_permutations | 5 | 100000 | 100000 | 0.65510453 | PASSED | 100000/0.29119227 | 100000/0.47199357 |
rgb_lagged_sum | 0 | 1000000 | 100000 | 0.93655760 | PASSED | 100000/0.67558969 | 100000/0.24115259 |
rgb_lagged_sum | 1 | 1000000 | 100000 | 0.71434042 | PASSED | ||
rgb_lagged_sum | 2 | 1000000 | 100000 | 0.50431680 | PASSED | ||
rgb_lagged_sum | 3 | 1000000 | 100000 | 0.05946470 | PASSED | ||
rgb_lagged_sum | 4 | 1000000 | 100000 | 0.09233652 | PASSED | ||
rgb_lagged_sum | 5 | 1000000 | 100000 | 0.78429507 | PASSED | ||
rgb_lagged_sum | 6 | 1000000 | 100000 | 0.70431509 | PASSED | ||
rgb_lagged_sum | 7 | 1000000 | 100000 | 0.84959226 | PASSED | ||
rgb_lagged_sum | 8 | 1000000 | 100000 | 0.23160937 | PASSED | ||
rgb_lagged_sum | 9 | 1000000 | 100000 | 0.62978194 | PASSED | ||
rgb_lagged_sum | 10 | 1000000 | 100000 | 0.80398889 | PASSED | ||
rgb_lagged_sum | 11 | 1000000 | 100000 | 0.51451670 | PASSED | ||
rgb_lagged_sum | 12 | 1000000 | 100000 | 0.04198232 | PASSED | ||
rgb_lagged_sum | 13 | 1000000 | 100000 | 0.49844099 | PASSED | ||
rgb_lagged_sum | 14 | 1000000 | 100000 | 0.23883388 | PASSED | ||
rgb_lagged_sum | 15 | 1000000 | 100000 | 0.58890320 | PASSED | ||
rgb_lagged_sum | 16 | 1000000 | 100000 | 0.33541037 | PASSED | ||
rgb_lagged_sum | 17 | 1000000 | 100000 | 0.00866121 | PASSED | ||
rgb_lagged_sum | 18 | 1000000 | 100000 | 0.21696271 | PASSED | ||
rgb_lagged_sum | 19 | 1000000 | 100000 | 0.52035392 | PASSED | ||
rgb_lagged_sum | 20 | 1000000 | 100000 | 0.90284911 | PASSED | ||
rgb_lagged_sum | 21 | 1000000 | 100000 | 0.62080074 | PASSED | ||
rgb_lagged_sum | 22 | 1000000 | 100000 | 0.50470000 | PASSED | ||
rgb_lagged_sum | 23 | 1000000 | 100000 | 0.30885183 | PASSED | ||
rgb_lagged_sum | 24 | 1000000 | 100000 | 0.56839772 | PASSED | ||
rgb_lagged_sum | 25 | 1000000 | 100000 | 0.46229428 | PASSED | ||
rgb_lagged_sum | 26 | 1000000 | 100000 | 0.00231344 | WEAK | ||
rgb_lagged_sum | 27 | 1000000 | 100000 | 0.85672672 | PASSED | ||
rgb_lagged_sum | 28 | 1000000 | 100000 | 0.53200641 | PASSED | ||
rgb_lagged_sum | 29 | 1000000 | 100000 | 0.15289233 | PASSED | ||
rgb_lagged_sum | 30 | 1000000 | 100000 | 0.75899614 | PASSED | ||
rgb_lagged_sum | 31 | 1000000 | 100000 | 0.11741910 | PASSED | ||
rgb_lagged_sum | 32 | 1000000 | 100000 | 0.67451804 | PASSED | ||
rgb_kstest_test | 0 | 10000 | 61100 | 0.00000091 | FAILED | 69200/0.00000089 | 99300/0.00000092 |
dab_bytedistrib | 0 | 51200000 | 100000 | 0.97693343 | PASSED | 100000/0.73031233 | 100000/0.58626499 |
dab_dct | 256 | 50000 | 100000 | 0.17400145 | PASSED | 100000/0.14731270 | 100000/0.24133917 |
dab_filltree | 32 | 15000000 | 100000 | 0.43722842 | PASSED | 100000/0.29776867 | 100000/0.83860736 |
dab_filltree | 32 | 15000000 | 100000 | 0.39516879 | PASSED | 100000/0.76524376 | 100000/0.32081520 |
dab_filltree2 | 0 | 5000000 | 100000 | 0.01584526 | PASSED | 100000/0.63271678 | 100000/0.13847552 |
dab_filltree2 | 1 | 5000000 | 100000 | 0.39737549 | PASSED | 100000/0.38677116 | 100000/0.58485250 |
dab_monobit2 | 12 | 65000000 | 901 | 0.00000004 | FAILED | 501/0.00000057 | 901/0.00000023 |