Previously .txt files with full backtrace may be identified as a
crashed test:
I:Core dumps were found for the following system tests:
I: core.19948-backtrace.txt
I: shutdown
Now .txt files are removed from the list.
Change 'run.sh.in' to match the core matching pattern in
'testsummary.sh'.
(cherry picked from commit c2dcd95966)
Each system test can be marked as failed not only due to some tested
component(s) not behaving as expected, but also because of core dumps,
assertion failures, and/or ThreadSanitizer reports being found among its
artifacts. Make the system test summary list the tests which exhibit
such atypical symptoms to more clearly present the nature of problems
found.
Some clean.sh scripts contain overly broad file deletion wildcards which
cause the test.output file (used by the system test framework for
collecting output) in a given system test's directory to be erroneously
removed immediately after the test is started (due to setup.sh scripts
calling clean.sh at the beginning). This prevents the test's output
from being placed in bin/tests/system/systests.output at the end of a
test suite run and thus can lead to test failures being ignored. Fix by
storing each test's output in a test.output.<test-name> file in
bin/tests/system/, which prevents clean.sh scripts from removing it (as
they should only ever affect files contained in a given system test's
directory).
At the end of each system test suite run, the system test framework
collects all existing test.output files from system test subdirectories
and produces bin/tests/system/systests.output from those files.
However, it does not check whether a test.output file was found for
every executed test. Thus, if the test.output file is accidentally
deleted by the system test itself (e.g. due to an overly broad file
removal wildcard present in clean.sh), its output will not be included
in bin/tests/system/systests.output. Since the result of each system
test suite run is determined by bin/tests/system/testsummary.sh, which
only operates on the contents of bin/tests/system/systests.output, this
can lead to test failures being ignored. Fix by ensuring the number of
test results found in bin/tests/system/systests.output is equal to the
number of tests run and triggering a system test suite failure in case
of a discrepancy between these two values.
The current regular expression used for extracting system test results
from systests.output, "^R:", is anchored at the start of a line, which
prevents colored system test output from being properly processed. As
just "R:" would arguably be too general, extend the pattern a bit to
ensure it will only match lines containing system test results.
Prevent runall.sh and "make test" from even attempting to run system
tests when "ifconfig.sh up" has not been run beforehand. This ensures
the user is not flooded with error messages in such a case.
- add CHANGES note
- update copyrights and license headers
- add -j to the make commands in .gitlab-ci.yml to take
advantage of parallelization in the gitlab CI process
When running all the system tests, output from a test is sent to a
test.output file in the test directory. These are combined in to
systests.output when the run finishes.
Miscellaneous tidying up of run management. The most significant
change is that "runall.sh" now runs _all_ the tests, even the
ones that can run in parallel. runsequential.sh is the script
to run tests that have not been converted to parallel running.
