PTS forked and testnet

[cube]

Just taken a look at it.  Seems a number of refactored codes.  Does the rebase affect qt-wallet?

[pc]

I'm not sure. Might be clever to rebase qt + web_wallet on v0.4.23.1 as well.

But don't do it yet, test #3 still isn't working. :-(

[pc]

OK, testnet #3 is up and running: https://github.com/pmconrad/PTS/tree/pts-test-3
That tag has changed a couple of times, so make sure you have the correct revision 7e2bb1a6be9c145e1fdc4cd282574fa00a1445c0 . I know this is bad practice. Still learning git.

After debugging this I'm wondering why the previous testnets were working at all... oh well. :-/

[cube]

I'm not sure. Might be clever to rebase qt + web_wallet on v0.4.23.1 as well.

But don't do it yet, test #3 still isn't working. :-(

I am building the GUI wallet and so I need those codes working too.  I see you have removed those specific bts 'hard-coded' changes that are not meant for generic DAC. Well done.  Please continue the thorough check effort.

[cube]

Trying the latest PC repo:

Linking CXX executable wallet_tests
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libboost_unit_test_framework.a(unit_test_main.o): In function `main':
(.text.startup+0xf): undefined reference to `init_unit_test_suite(int, char**)'
collect2: error: ld returned 1 exit status
make[2]: *** [tests/wallet_tests] Error 1
make[1]: *** [tests/CMakeFiles/wallet_tests.dir/all] Error 2
make: *** [all] Error 2

Do you encounter this compilation error?

Is test #3  working?

[pc]

Trying the latest PC repo:

Linking CXX executable wallet_tests
/usr/lib/gcc/i686-linux-gnu/4.8/../../../i386-linux-gnu/libboost_unit_test_framework.a(unit_test_main.o): In function `main':
(.text.startup+0xf): undefined reference to `init_unit_test_suite(int, char**)'
collect2: error: ld returned 1 exit status
make[2]: *** [tests/wallet_tests] Error 1
make[1]: *** [tests/CMakeFiles/wallet_tests.dir/all] Error 2
make: *** [all] Error 2

Do you encounter this compilation error?

No. I'm dynamically linking the boost libraries which requires the ALTERNATIVE_INIT_API. According to the boost documentation, the alternative API should work with static linking too, but apparently it doesn't. I've pushed a fix, please try again.

Is test #3  working?

I'm not even sure how that is supposed to work. When I start wallet_tests without arguments I get a pts client prompt that reacts to my input. Not exactly my understanding of an automated test.

What I have done is run all the individual regression tests using the run_regression_tests.sh script. Some of them seem to be not 100% deterministic, probably timing issues in combination with PRNGs.

[cube]

I'm not even sure how that is supposed to work. When I start wallet_tests without arguments I get a pts client prompt that reacts to my input. Not exactly my understanding of an automated test.

What I have done is run all the individual regression tests using the run_regression_tests.sh script. Some of them seem to be not 100% deterministic, probably timing issues in combination with PRNGs.

Thanks for fixing the linking problem.  I think for now we will stick with static link to make it easy for others to participate in the testing.

You will be the first person to learn and experience the 'automated testing' aspects outside of the I3 group.  The lessons learnt would be useful and could be recorded down for other volunteers to pick it up. 

Could you elaborate more on the part 'not 100% deterministic'?  The 'non-deterministic' part does not sound good.

[pc]

You will be the first person to learn and experience the 'automated testing' aspects outside of the I3 group.

My impression of the regression tests is that knowledge of them is distributed sparingly inside the I3 group as well. IOW the tests are obviously being neglected.

Could you elaborate more on the part 'not 100% deterministic'?  The 'non-deterministic' part does not sound good.

I've noticed that sometimes when I start tests/run_regression_tests.sh some of the tests fail, and when I start it again these same tests report success.

The regression test suite creates a dummy test network of one or more clients. Each client executes the commands in one of the .log files in the test-specific subdirectory. Commands and results are logged to an output file which is later compared to the original. The comparison skips some things that are necessarily different in each run, like timestamps.

The tricky part is that some operations require random input. This is supposed to be overcome by using deterministic random numbers via deterministic_openssl_rand.[ch]pp.
Another tricky part is timing. The client starts a lot of threads that are running in parallel under the hood. Sometimes some of the threads do things in a different order. Background threads can also be faster (or slower) than the actual test execution. The test suite does nothing to enforce a specific execution order, so there is certainly potential for non-determinism.

[cube]

I went into the regression tests folder and saw some logs. I am not sure if the logs were produced by your tests.  One of the logs - blockchain_get_account.log shows

"(wallet closed) >>> blockchain_get_account delegate0" and it shows a positive result:

ID    NAME (* next in line)           APPROVAL       PRODUCED MISSED   RELIABILITY   PAY RATE PAY BALANCE         LAST BLOCK  VERSION
==========================================================================================================================================
1     delegate0                       0.00000000 %   0        0        N/A           100 %    0.00000000 PTS      NONE

while

"(wallet closed) >>> blockchain_get_account deleg" shows a negative result:

"No account found.
(wallet closed) >>> quit"

Taking the above log as an example, the regression test for this specific command "blockchain_get_account" has passed.   Could you let us know what are the tests that passed and those that failed?

[pc]

I went into the regression tests folder and saw some logs. I am not sure if the logs were produced by your tests.  One of the logs - blockchain_get_account.log shows

"(wallet closed) >>> blockchain_get_account delegate0" and it shows a positive result:

ID    NAME (* next in line)           APPROVAL       PRODUCED MISSED   RELIABILITY   PAY RATE PAY BALANCE         LAST BLOCK  VERSION
==========================================================================================================================================
1     delegate0                       0.00000000 %   0        0        N/A           100 %    0.00000000 PTS      NONE

while

"(wallet closed) >>> blockchain_get_account deleg" shows a negative result:

"No account found.
(wallet closed) >>> quit"

Taking the above log as an example, the regression test for this specific command "blockchain_get_account" has passed.   Could you let us know what are the tests that passed and those that failed?

These logs *are* the tests. The wallet_tests program parses the logfiles and looks for a CLI prompt (">>>"), then takes the rest of that line as a command to execute.
While executing these commands, wallet_tests writes the commands plus the client output to another logfile. After all commands have been executed, the original logfile and the new logfile are compared. The test is successful if both files are equal (apart from timestamps etc.).

So in the above case, the test runner executes the command "blockchain_get_account delegate0". Following that, the logfile contains the expected output of that command, i. e. the account description of delegate0. Then, the runner executes "blockchain_get_account deleg" and expects the response "No account found.", because that account doesn't exist.

[cube]

I went into the regression tests folder and saw some logs. I am not sure if the logs were produced by your tests.  One of the logs - blockchain_get_account.log shows

"(wallet closed) >>> blockchain_get_account delegate0" and it shows a positive result:

ID    NAME (* next in line)           APPROVAL       PRODUCED MISSED   RELIABILITY   PAY RATE PAY BALANCE         LAST BLOCK  VERSION
==========================================================================================================================================
1     delegate0                       0.00000000 %   0        0        N/A           100 %    0.00000000 PTS      NONE

while

"(wallet closed) >>> blockchain_get_account deleg" shows a negative result:

"No account found.
(wallet closed) >>> quit"

Taking the above log as an example, the regression test for this specific command "blockchain_get_account" has passed.   Could you let us know what are the tests that passed and those that failed?

These logs *are* the tests. The wallet_tests program parses the logfiles and looks for a CLI prompt (">>>"), then takes the rest of that line as a command to execute.
While executing these commands, wallet_tests writes the commands plus the client output to another logfile. After all commands have been executed, the original logfile and the new logfile are compared. The test is successful if both files are equal (apart from timestamps etc.).

So in the above case, the test runner executes the command "blockchain_get_account delegate0". Following that, the logfile contains the expected output of that command, i. e. the account description of delegate0. Then, the runner executes "blockchain_get_account deleg" and expects the response "No account found.", because that account doesn't exist.

I see. The tests are using 'scraped command output' to form the input for automated tests.  Strange way of doing automated testing.  Where are the post-test generated logs?  Are they having the same file names which overwrite the 'tests'?  I do not see 'delegate0' in the blockchain. So 'delegate0' should have turned out 'negative' ie 'No account found'.

[pc]

I see. The tests are using 'scraped command output' to form the input for automated tests.  Strange way of doing automated testing.

Yes. I hadn't seen something like that before.

Where are the post-test generated logs?  Are they having the same file names which overwrite the 'tests'?  I do not see 'delegate0' in the blockchain. So 'delegate0' should have turned out 'negative' ie 'No account found'.

The generated logs are in <BASEDIR>/../regression_tests_output/regression_tests/<TEST>/client/console.log . The regression tests use their own genesis block (in tests/test_genesis.json), this also contains "delegate0".."delegate100" and initial balances for each delegate.