You will be the first person to learn and experience the 'automated testing' aspects outside of the I3 group.
My impression of the regression tests is that knowledge of them is distributed sparingly inside the I3 group as well. IOW the tests are obviously being neglected.
Could you elaborate more on the part 'not 100% deterministic'? The 'non-deterministic' part does not sound good.
I've noticed that sometimes when I start tests/run_regression_tests.sh some of the tests fail, and when I start it again these same tests report success.
The regression test suite creates a dummy test network of one or more clients. Each client executes the commands in one of the .log files in the test-specific subdirectory. Commands and results are logged to an output file which is later compared to the original. The comparison skips some things that are necessarily different in each run, like timestamps.
The tricky part is that some operations require random input. This is supposed to be overcome by using deterministic random numbers via deterministic_openssl_rand.[ch]pp.
Another tricky part is timing. The client starts a lot of threads that are running in parallel under the hood. Sometimes some of the threads do things in a different order. Background threads can also be faster (or slower) than the actual test execution. The test suite does nothing to enforce a specific execution order, so there is certainly potential for non-determinism.