Fix several issues in CI/tests #5

mrc0mmand · 2016-12-04T20:39:32Z

During the review of test results I found out that several test are missing rlGetTestState command at their ends, which results in false-positives (as the default script return code is zero). This should be fixed in commit 75e6128. I also added a simple sanity check, which ensures that all non-library test scripts (runtest.sh) contain the mentioned rlGetTestState. If a script with missing rlGetTestState is encountered, the job is immediately aborted (31dcda2).
After extending the nss test suite, we managed to hit both Travis' limits - job length, which is 50 minutes, and log size, which is 4 MB. Commit ce96c52 implements a simple solution, which allows specifying an environment by an extended bash file glob expression (extglob). I already tested it on the aforementioned nss test suite and it works pretty well.
Last issue concerns the gnutls/softhsm-integration test, which has wrong metadata in its Makefile. This test is unsupported on RHEL/CentOS 6 and has an already fixed bug on RHEL/CentOS 7.2 - in this case we have to simply wait for a docker image for CentOS 7.3. I disabled this test on both RHEL/CentOS 6 and RHEL/CentOS 7 and will create an issue to re-enable it on CentOS 7 once the appropriate docker image is released.

Check if all tests have rlGetTestState command to prevent false-positives

Centos 6 - unsupported Centos 7 - there is a bug, which was fixed in RHEL/Centos 7.3 => re-enable this test when 7.3 docker image is available

mrc0mmand · 2016-12-04T22:33:47Z

The Travis' fail is caused by the Beakerlib bug mentioned in #7 (rlIsRHEL doesn't work on CentOS). I guess we'll have to wait with merging this PR as well, until it's fixed.

tomato42 · 2016-12-05T19:00:18Z

scripts/test-runner.sh

+# $2 - glob pattern
+function test_name_relevancy() {
+    # See: http://wiki.bash-hackers.org/syntax/pattern#extended_pattern_language
+    shopt -s extglob


if we want to preserve the environment, it should first query the state of the setting and then set it, and then restore it, otherwise it will disable it if it is run on a system with it enabled

That definitely makes sense, thanks.

mrc0mmand · 2016-12-19T09:27:45Z

In a few recent runs I noticed a strange fail on Fedora:

:: [  BEGIN   ] :: Running 'rlWait -s 9 18359'
/usr/share/beakerlib/synchronisation.sh: line 89: 18359 Killed                  expect openssl-server.expect openssl s_server -key 1024dsa-server/key.pem -cert 1024dsa-server/cert.pem -CAfile <(cat $(x509Cert ca) ${C_SUBCA[$j]}) -cipher DHE-DSS-AES128-SHA > server.log 2> server.err
:: [   FAIL   ] :: Command 'rlWait -s 9 18359' (Expected 0,143, got 137)

As its occurrence is completely random, it's slightly annoying, because it breaks test results. Re-running the offending job usually helps.

tomato42 · 2017-01-04T17:40:21Z

for that failure, it would be best to tweak the openssl-server.expect script. Which test case was this in?

mrc0mmand · 2017-01-23T09:41:32Z

Unfortunately, this does happen (or happened at least once) in all test cases which use expect scripts. I spent almost entire weekend trying to find the root cause without any success. For some strange reason it gets stuck after the end of a *-client.expect script. You can find a log with expect debug mode enabled here. Even though it successfully matches the "Server: Generic Web Server" expect section, which should be followed by close and exit 0, it just stops responding instead. close should send an EOF to the spawned process (according to the manpage) which should kill it, but maybe it doesn't...

tomato42 · 2017-01-23T13:28:04Z

EOF doesn't generally kill the process. It will kill openssl process only if it is actually waiting for input at that very moment. I wonder if making it waif for text that shows up later wouldn't help...

mrc0mmand · 2017-01-28T19:02:23Z

After another few hours of experimenting and discussing this issue with @ep69 I fixed the problematic client expect scripts. In some cases your idea was enough, in other cases I had to use a sleep - waiting for a following text is kind of hard as expect -re ".+" waits for any following text, which does not mean everything which follows. In these cases we'd have to wait for a specific last line from that particular output or something like that, which may not be possible in all cases. Sleep is a simple and (so far) working solution - hopefully it'll stay this way.

tomato42 · 2017-01-30T16:18:35Z

looking at the travis failures, looks like we're missing SIGTERM trap in the openssl-server.expect (see: https://stackoverflow.com/questions/23836136/expect-interrupt-program-ctrlc)

tomato42 · 2017-01-30T16:43:43Z

It still tries to run SHA384 PRF tests with RHEL 6 NSS...

mrc0mmand · 2017-01-30T16:50:54Z

It still tries to run SHA384 PRF tests with RHEL 6 NSS...

Yes, this is fixed by #8.

tomato42 · 2017-02-09T18:19:44Z

Yes, this is fixed by #8.

and why it can't be fixed by #5 ?

mrc0mmand · 2017-02-12T17:19:50Z

It's a part of the Beakerlib fix, as it adds an alias rlIsRHEL for rlIsCentOS. Without this alias, the condition if ! rlIsRHEL 6; then... for SHA384 PRF is equal to true even on RHEL 6.

tomato42 · 2017-02-13T15:14:56Z

then lets combine it

mrc0mmand · 2017-02-20T14:34:00Z

Fixed conflicts and merged all changes into #8.

mrc0mmand added 4 commits December 4, 2016 21:21

test-setup.sh: Add test sanity check

31dcda2

Check if all tests have rlGetTestState command to prevent false-positives

renego&resumption: Add missing rlGetTestState

75e6128

Split test environments using extended globs

ce96c52

softhsm-integration: Disable on Centos 6 and 7

96cd72d

Centos 6 - unsupported Centos 7 - there is a bug, which was fixed in RHEL/Centos 7.3 => re-enable this test when 7.3 docker image is available

tomato42 requested changes Dec 5, 2016

View reviewed changes

test-runner.sh: Preserve the environment

7f66a98

tomato42 approved these changes Jan 4, 2017

View reviewed changes

Send an EOF when it's actually expected

607d686

mrc0mmand mentioned this pull request Jan 30, 2017

gnutls/renegotiation-with-OpenSSL: Test extension #10

Open

mrc0mmand mentioned this pull request Jan 30, 2017

gnutls/renegotiation-with-NSS: Test extension #9

Merged

mrc0mmand mentioned this pull request Jan 30, 2017

Beakerlib fix #8

Merged

mrc0mmand closed this Feb 20, 2017

mrc0mmand deleted the missing-rlGetTestState branch February 25, 2017 21:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix several issues in CI/tests #5

Fix several issues in CI/tests #5

Uh oh!

mrc0mmand commented Dec 4, 2016 •

edited

Loading

Uh oh!

mrc0mmand commented Dec 4, 2016

Uh oh!

tomato42 Dec 5, 2016

Uh oh!

mrc0mmand Dec 18, 2016

Uh oh!

mrc0mmand commented Dec 19, 2016

Uh oh!

tomato42 commented Jan 4, 2017

Uh oh!

mrc0mmand commented Jan 23, 2017

Uh oh!

tomato42 commented Jan 23, 2017

Uh oh!

mrc0mmand commented Jan 28, 2017

Uh oh!

tomato42 commented Jan 30, 2017

Uh oh!

tomato42 commented Jan 30, 2017

Uh oh!

mrc0mmand commented Jan 30, 2017 •

edited

Loading

Uh oh!

tomato42 commented Feb 9, 2017

Uh oh!

mrc0mmand commented Feb 12, 2017 •

edited

Loading

Uh oh!

tomato42 commented Feb 13, 2017

Uh oh!

mrc0mmand commented Feb 20, 2017

Uh oh!

Uh oh!

Fix several issues in CI/tests #5

Fix several issues in CI/tests #5

Uh oh!

Conversation

mrc0mmand commented Dec 4, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrc0mmand commented Dec 4, 2016

Uh oh!

tomato42 Dec 5, 2016

Choose a reason for hiding this comment

Uh oh!

mrc0mmand Dec 18, 2016

Choose a reason for hiding this comment

Uh oh!

mrc0mmand commented Dec 19, 2016

Uh oh!

tomato42 commented Jan 4, 2017

Uh oh!

mrc0mmand commented Jan 23, 2017

Uh oh!

tomato42 commented Jan 23, 2017

Uh oh!

mrc0mmand commented Jan 28, 2017

Uh oh!

tomato42 commented Jan 30, 2017

Uh oh!

tomato42 commented Jan 30, 2017

Uh oh!

mrc0mmand commented Jan 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomato42 commented Feb 9, 2017

Uh oh!

mrc0mmand commented Feb 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomato42 commented Feb 13, 2017

Uh oh!

mrc0mmand commented Feb 20, 2017

Uh oh!

Uh oh!

mrc0mmand commented Dec 4, 2016 •

edited

Loading

mrc0mmand commented Jan 30, 2017 •

edited

Loading

mrc0mmand commented Feb 12, 2017 •

edited

Loading