Add minimum performance goals, document them

Author: will-keen

README.md

@@ -15,7 +15,9 @@ These goals should define the future development of the library.
     - It does not need to exactly replicate the syntax and oddities of SystemVerilog, but it must adhere to the principle of declariative randomization.
   - Repeatable, i.e. works deterministically with the same seed.
   - Fast. Or at least as fast as can be expected from a Python library rather than say C/C++.
-    - Originally, the creation of the library was motivated by [`pyvsc`](https://github.com/fvutils/pyvsc). `pyvsc` is feature-rich and user-friendly for those with an SV background, but not fast enough for production usage.
+    - Originally, the creation of the library was motivated by [`pyvsc`](https://github.com/fvutils/pyvsc).
+    - `pyvsc` is feature-rich and user-friendly for those with an SV background, but we found it was not fast enough for production usage.
+    - This library aims to achieve at least a 10x speedup over `pyvsc`. See the `benchmarks/` directory or run `python -m benchmarks` for testing.
 
 ## Motivation
 Why bother with a Python library for this? Why not just use procedural randomization?

benchmarks/__main__.py

@@ -75,6 +75,10 @@ def test_basic(self):
         randobjs = {'vsc': vsc_basic(), 'cr': cr_basic(Random(0))}
         def check(results):
             self.assertGreater(results['cr'][1], results['vsc'][1])
+            # This testcase is typically 40-50x faster, which may vary depending
+            # on machine. Ensure it doesn't fall below 30x.
+            speedup = results['cr'][1] / results['vsc'][1]
+            self.assertGreater(speedup, 30, "Performance has degraded!")
         self.run_benchmark(randobjs, 100, check)
 
     def test_in(self):
@@ -88,6 +92,10 @@ def test_in(self):
         }
         def check(results):
             self.assertGreater(results['cr'][1], results['vsc'][1])
+            # This testcase is typically 13-15x faster, which may vary depending
+            # on machine. Ensure it doesn't fall below 10x.
+            speedup = results['cr'][1] / results['vsc'][1]
+            self.assertGreater(speedup, 10, "Performance has degraded!")
         self.run_benchmark(randobjs, 100, check)
 
     def test_ldinstr(self):
@@ -100,6 +108,10 @@ def test_ldinstr(self):
         }
         def check(results):
             self.assertGreater(results['cr'][1], results['vsc'][1])
+            # This testcase is typically 13-15x faster, which may vary depending
+            # on machine. Ensure it doesn't fall below 10x.
+            speedup = results['cr'][1] / results['vsc'][1]
+            self.assertGreater(speedup, 10, "Performance has degraded!")
         self.run_benchmark(randobjs, 100, check)