Remove code fork for legacy runner, reorganize imports and move a
pylint-silencing snippet to the top of the file. The rest of the code
was just unindented.
(cherry picked from commit d16e5fd5e5)
In order to python system tests, pytest (runner) has to be used
directly. This makes it possible to simplify the pytest runner and make
its behavior simpler and easier to extend.
The legacy runner can still be used to run shell system tests.
(cherry picked from commit 4af9bafb4e)
Since the legacy runner is no longer used in the automake test suite,
don't use the special GNU exit code indicating a skipped tests. Instead,
use 0 to avoid considering skipped tests as failed when using simpler
mechanism (such as xargs -P) to run the tests with the legacy runner.
The legacy runner no longer uses make check. Ensure the legacy runner
script doesn't interact with that automake target in any way. The legacy
runner script remains available to execute the legacy runner, but there
is no out-of-the box support for running tests in parallel. Other tools
such as xargs can be utilized for that.
(cherry picked from commit 0a9ee342e9)
Pytest provides JUnit output and uses different exit codes from
Automake. Use the conversion script to interpret the JUnit test results
from python rather than relying on the status code.
(cherry picked from commit 295890a16b)
It's important to parse the JUnit result file rather than relying on the
exit code from pytest, which has a different meaning. Include a .trs test
result for each test case and set an exit code which is most appropriate
as the aggregate result (e.g. it will be set to 77 (SKIP) if there's at
least one test case that was skipped).
(cherry picked from commit b96c8e8062)
Dependencies for these tests are already checked in prereq.sh - if the
dependencies are missing, these tests will be skipped. The extra
dependency check in Makefile.am is extraneous and only applied for the
legacy test runner.
(cherry picked from commit 807f67b1cd)
To allow concurrent invocations of pytest, it is necessary to assign
ports properly to avoid conflicts. In order to do that, pytest needs to
know a complete list of all test modules.
When pytest is invoked from run.sh, the current working directory is the
system test directory. To properly detect other tests, the conftest.py
has to look in the bin/tests/system directory, rather than the current
working directory.
(cherry picked from commit d4502206a1)
To conform with the expected naming convention, the pytest glue file for
the `allow-query` test should use underscore as the word separator in
the python file name: allow-query/tests_sh_allow_query.py
(cherry picked from commit c026a00696)
The _common directory is a special case directory which contains shared
files for other system test directories. Make sure it's tracked in git
and not deleted during temporary directory cleanup.
(cherry picked from commit 9e13c3dbcb)
The old name "common" clashes with the convention of system test
directory naming. It appears as a system test directory, but it only
contains helper files.
To reduce confusion and to allow automatic detection of issues with
possibly missing test files, rename the helper directory to "_common".
The leading underscore indicates the directory is different and the its
name can no longer be confused with regular system test directories.
(cherry picked from commit 168dba163c)
The covered type previously displayed as TYPE0 when it should
have reflected the records that was actually covered.
(cherry picked from commit 8ce359652a)
The dnssec-must-be-secure feature was added in the early days of BIND 9
and DNSSEC and it makes sense only as a debugging feature.
Remove the feature to simplify the code.
(cherry picked from commit 9e0b348a2b)
A negative or excessively large Content-Length could cause a crash
by making `INSIST(httpd->consume != 0)` fail.
(cherry picked from commit 26e10e8fb5)
The rrl system test has been unstable and producing false positive
results for years (GL #172). Allow the test to be re-run (once) to
reduce the noise it causes.
(cherry picked from commit 40289d5440)
The reclimit system test has been unstable and producing false positive
results for years (GL #1587). Allow the test to be re-run (once) to
reduce the noise it causes.
(cherry picked from commit 8c5833fe39)
The qmin test is inherently unstable. It fails quite often with failure
modes described in GL #904. Allow the pytest runner to re-run the test
up to 3 times to only detect a more persistent and reproducible failures
rather than random noise caused by the nature of the test.
(cherry picked from commit be2123a8e9)
It is better to disable the specific check that causes the test to fail
rather than mark the entire test as xfail, which can mask other issues
which the test is capable of detecting.
(cherry picked from commit 7522583b57)
The _last_test_run entry was accidentally added in !8194. It came from a
work-in-progress version of the MR and was left there during a rebase.
(cherry picked from commit 1b3db25adf)
nsupdate offers the switch -v to use TCP for update requests. But
before sending that update request nsupdate was using UDP connection
to gather the SOA for determining the zone if not given explicitly.
Only use TCP if not using the default servers, because the SOA
query lookup is a different server with different capabilities (and
usually not for the better of it).
Using check_PROGRAMS would postpone compiling the binaries needed by
system tests until `make check` would be called. Since it's preferable
to invoke pytest directly to run the system test suite, compile these
binaries without installing them during `make all` instead by using
noinst_PROGRAMS.
This removes the need to use TESTS= make -e check hack invoked from
pytest to work around this issue.
(cherry picked from commit 21980b43b8)
BIND 9.18 rndc does not know about the "-t" option, that results in an
silent "invalid argument" error.
The "-t" option originally came with
184eddb595 (a backport from the main
branch).
In some cases, BIND is not fast enough to fill the send buffer and
manages to answer all queries, contrary to what the test expects.
Repeat the check up to 3 times to limit this test instability.
(cherry picked from commit 681b23c398)
If the flaky plugin for pytest is available, use its decorator to
support re-running unstable tests. In case the package is missing,
execute the test as usual without attempts to re-run it in case of
failure.
This is mostly intended to increase the test stability in CI. Using a
custom decorator enables us to keep the flaky package as an optional
dependency.
(cherry picked from commit 5b703de733)
The command finds all directories in bin/tests/system which contain an
underscore. Underscore indicates either a temporary directory (_tmp_), a
symlink to test artifacts (TESTNAME_MODULENAME), or a python-related
cache. Using underscore for a system test name is invalid and a hyphen
must be used instead.
(cherry picked from commit d66ff81543)
While it'd be fairly easy to split the function up into smaller ones,
the readability wouldn't be improved in this case. Silence the
suggestions instead.
(cherry picked from commit 83ddca7690)
At the end of the test, display the symlink path to the artifact
directory in case it's preserved. Log the full tempdir name in debug
log.
(cherry picked from commit f91d0b13e8)
While temporary directories are useful for test execution to keep
everything clean, they are difficult to work with manually. Create a
symlink for each test artifact directory with a stable and predictable
path. The symlink always either points to the latest artifacts, or is
missing in case the last run succeeded.
Ensure these symlinked directories aren't detected as test suites by the
pytest runner.
(cherry picked from commit e1ca5c8071)
