Commit Graph
59 Commits
Author SHA1 Message Date
Michal Nowak c607237b77 Add rr-related common test artifacts 2024-12-10 17:31:35 +01:00
Nicki Křížek cf606a9ead Ignore test artifacts in .libs
Some tests may leave artifacts in the .libs directory. Ignore this
directory when detecting expected artifacts.
2024-11-26 09:05:15 +00:00
Nicki Křížek b66fb31dcb Ensure pytest runner get proper outcome from flaky reruns
When a test is re-run by the flaky plugin, the TestReport outcomes
collected in the pytest_runtest_makereport() hook should be overriden.
Each of the setup/call/teardown phases is reported again and since we
care about the overall outcome, their respective results should be
overriden so that only the outcome from the final test (re)run gets
reported.

Prior to this change, it lead to a situation where an extra_artifact
generated during the test might be ignored. This was caused because the
check was skipped, since the test was incorrectly considered as "failed"
in the case where the test would fail on the first run, but pass on a
subsequent flaky rerun.
2024-11-12 10:19:03 +01:00
Nicki Křížek f2cb2e5723 Remove invocations and mentions of clean.sh 2024-11-08 10:54:24 +01:00
Michał KępieńandNicki Křížek 3a9f4edddc Add pytest fixture for checking test artifacts
Prior to introducing the pytest runner, clean.sh files were used as a
list of files that the test is expected to leave around as artifacts and
check that no extra files were created.

With the pytest runner, those scripts are no longer used, but the
ability to detect extraneous files is still useful. Add a new
"extra_artifacts" mark which can be used for the same purpose.
2024-11-08 10:54:24 +01:00
Nicki Křížek 60e118c4fb Support jinja2 templates in pytest runner
Configuration files in system tests which require some variables (e.g.
port numbers) filled in during test setup, can now use jinja2 templates
when `jinja2` python package is available.

Any `*.j2` file found within the system test directory will be
automatically rendered with the environment variables into a file
without the `.j2` extension by the pytest runner. E.g.
`ns1/named.conf.j2` will become `ns1/named.conf` during test setup. To
avoid automatic rendering, use `.j2.manual` extension and render the
files manually at test time.

New `templates` pytest fixture has been added. Its `render()` function
can be used to render a template with custom test variables. This can be
useful to fill in different config options during the test. With
advanced jinja2 template syntax, it can also be used to include/omit
entire sections of the config file rather than using `named1.conf.in`,
`named2.conf.in` etc.
2024-10-30 18:00:20 +01:00
Nicki Křížek 7118cbed98 Make system tests compatible with pytest 8.0.0+
The pytest collection mechanism has been overhauled in pytest 8.0.0,
resulting in a different node tree when collecting the tests. Ensure the
paths / names we're using that are derived from the node tree are
consistent across different pytest versions.

Particularly, this has affected the convenience symlink name (which is
supposed to be in the form of e.g. dns64_sh_dns64 for the dns64 module
and tests_sh_dns64.py module) and the test name that's logged at the
start of the test, which is supposed to include the system test
directory relative to the root system test directory as well as the
module name (e.g. dns64/tests_sh_dns64.py).

Related https://github.com/pytest-dev/pytest/issues/7777
2024-10-24 13:13:13 +02:00
Nicki Křížek 7639c58c48 Disable too-many/too-few pylint checks
Enforcing pylint standards and default for our test code seems
counter-productive. Since most of the newly added code are tests or is
test-related, encountering these checks rarely make us refactor the code
in other ways and we just disable these checks individually. Code that
is too complex or convoluted will be pointed out in reviews anyways.
2024-10-17 11:01:44 +02:00
Tom KrizekandNicki Křížek 8302db407c Move test algorithm configuration to isctest
Instead of invoking get_algorithms.py script repeatedly (which may yield
different results), move the algorithm configuration to an isctest
module. This ensures the variables are consistent across the entire test
run.
2024-07-31 10:50:09 +02:00
Nicki Křížek ac7c657d19 Move isctest.var initialization to conftest.py
The environment variable initialization requires logging to be set up
first. Ensure the initialization is delayed until loggers have been set
up.
2024-05-09 17:08:11 +02:00
Tom KrizekandNicki Křížek 2d1f348483 Move env var initialization to isctest.vars
Make sure all initialization takes place in isctest.vars.__init__ and
export the initial env vars there. Remove the no longer needed env
fixture and use os.environ instead.
2024-05-09 17:08:10 +02:00
Tom KrizekandNicki Křížek 1f6f2234d8 Move dir env var handling to isctest.vars.dirs 2024-05-09 17:08:10 +02:00
Tom KrizekandNicki Křížek 41cb553bdd Move port env vars into isctest.vars.ports module
The fixture for port assignment isn't needed, replace it with the common
way of handling environment variables.
2024-05-09 17:08:10 +02:00
Tom KrizekandNicki Křížek 46433ae17b Coalesce system test variables
Provide a single point of access to all the variables used by tests. Use
a custom dict-like structure to access the underlying data without
making a copy. This allows the individual modules to update the contents
at runtime, which is used for some variables.
2024-05-09 17:08:09 +02:00
Tom KrizekandNicki Křížek ab27f504ca Move environment variables from conf.sh to pytest
Remove conf.sh.in and move the environment variables into isctest/vars
python package. This enabled the removal of an ugly pytest hack which
loaded and parsed these variables from the environment.
2024-05-09 17:08:08 +02:00
Tom KrizekandMichał Kępień 673387c4d5 Move conftest log initialization to conftest.py
Initializing the conftest logging upon importing the isctest package
isn't practical when there are standalone pieces which can be used
outside of the testing framework, such as the asyncdnsserver module.
2024-03-20 09:22:36 +01:00
Tom Krizek c60975f108 Add utility logging functions to isctest.log
Unify the different loggers (conftest, module, test) into a single
interface. Remove the need to select the proper logger by automatically
selecting the most-specific logger currently available.

This also removes the need to use the logger/mlogger fixtures manually
and pass these around. This was especially annoying and unwieldy when
splitting the test cases into functions, because logger had to always be
passed around. Instead, it is now possible to use the
isctest.log.(debug,info,warning,error) functions.
2024-02-16 14:56:00 +01:00
Michał Kępień 662a88055c Add wait_for_zones_loaded fixture
The wait_for_zones_loaded fixture waits for the "all zones loaded"
message in the named log file before the test function can proceed.
2023-12-22 15:02:58 +01:00
Michał KępieńandŠtěpán Balážik c38c29e84d Implement Python helpers for using RNDC in tests
Controlling named instances using RNDC is a common action in BIND 9
system tests.  However, there is currently no standardized way of doing
that from Python-based system tests, which leads to code duplication.
Add a set of Python classes and pytest fixtures which intend to simplify
and standardize use of RNDC in Python-based system tests.

For now, RNDC commands are sent to servers by invoking the rndc binary.
However, a switch to a native Python module able to send RNDC commands
without executing external binaries is expected to happen soon.  Even
when that happens, though, having the capability to invoke the rndc
binary (in order to test it) will remain useful.  Define a common Python
interface that such "RNDC executors" should implement (RNDCExecutor), in
order to make switching between them convenient.

Co-authored-by: Štěpán Balážik <stepan@isc.org>
2023-12-21 18:10:15 +00:00
Tom Krizek cba5a1d62c Add system test utility package isctest
Create a utility package for code shared by the python tests. The
utility functions should use reasonable defaults and be split up into
modules according to their functionality.

Ensure assert rewriting is enabled for the modules to get the most
useful output from pytest.
2023-12-05 13:26:49 +01:00
Tom Krizek fd13dfb097 Rename convenience symlink to pytest artifacts
The previous symlink name convention was prone to name collisions If a
system test contained both a shell test and a pytest module of the same
name (e.g. dnstap test has both tests.sh and tests_dnstap.py), then
these would have the same convenience symlink, which could cause test
setup issues as well as confusion when examining test artifacts.

Update the naming convention to include the full pytest module name.
This results in a slightly more verbose names for shell tests (e.g.
dnstap_sh_dnstap instead of the previous dnstap_dnstap), but it removes
the chance of a collision.
2023-09-19 17:20:32 +02:00
Tom Krizek 8deb026748 Use integers for ports fixtures in pytest
Reorganize individual port fixtures and re-use the ports fixture to
obtain their number. Store it as integer and only cast it to string when
setting it as environment variable.
2023-09-19 15:43:33 +02:00
Tom Krizek 197dac55a2 Reformat conftest.py with black 2023-09-19 15:43:33 +02:00
Tom Krizek d16e5fd5e5 Remove legacy runner support from conftest.py
Remove code fork for legacy runner, reorganize imports and move a
pylint-silencing snippet to the top of the file. The rest of the code
was just unindented.
2023-09-19 15:43:33 +02:00
Tom Krizek d4502206a1 Fix pytest module detection for run.sh
To allow concurrent invocations of pytest, it is necessary to assign
ports properly to avoid conflicts. In order to do that, pytest needs to
know a complete list of all test modules.

When pytest is invoked from run.sh, the current working directory is the
system test directory. To properly detect other tests, the conftest.py
has to look in the bin/tests/system directory, rather than the current
working directory.
2023-09-19 14:47:47 +02:00
Tom Krizek 21980b43b8 Compile system test binaries during make
Using check_PROGRAMS would postpone compiling the binaries needed by
system tests until `make check` would be called. Since it's preferable
to invoke pytest directly to run the system test suite, compile these
binaries without installing them during `make all` instead by using
noinst_PROGRAMS.

This removes the need to use TESTS= make -e check hack invoked from
pytest to work around this issue.
2023-08-21 20:04:48 +02:00
Tom Krizek 83ddca7690 Silence pylint's refactoring suggestions for system_test_dir()
While it'd be fairly easy to split the function up into smaller ones,
the readability wouldn't be improved in this case. Silence the
suggestions instead.
2023-08-21 15:54:40 +02:00
Tom Krizek f91d0b13e8 Improve tempdir logging for pytest runner
At the end of the test, display the symlink path to the artifact
directory in case it's preserved. Log the full tempdir name in debug
log.
2023-08-21 15:54:40 +02:00
Tom Krizek e1ca5c8071 Create symlinks to test artifacts for pytest runner
While temporary directories are useful for test execution to keep
everything clean, they are difficult to work with manually. Create a
symlink for each test artifact directory with a stable and predictable
path. The symlink always either points to the latest artifacts, or is
missing in case the last run succeeded.

Ensure these symlinked directories aren't detected as test suites by the
pytest runner.
2023-08-21 15:54:38 +02:00
Tom Krizek b1a9d1fe25 Improve pytest message when sanitizer report is found 2023-08-21 15:12:53 +02:00
Matthijs Mekking ee3ae4d7d5 Remove dupsigs system test
This test uses key timing metadata to do rollovers, this is no longer
applicable with 'dnssec-policy'. Note that with 'dnssec-policy' key
timing metadata is still written, but it is not used for determining
what and when to do key rollovers.
2023-07-20 11:04:23 +02:00
Michal Nowak 89c77daddb Detect core dumps after BIND failed to stop cleanly
With the pytest runner, when BIND crashed during test runtime, the
get_core_dumps.sh script hasn't been run, and core dumps were not
detected.
2023-07-13 15:09:55 +02:00
Tom Krizek e8ea6b610b Fix base_port calculation in pytest runner
The selected base port should be in the range <port_min, port_max), the
formula was incorrect.

Credit for discovering this fault goes to Ondrej Sury.
2023-05-30 14:11:14 +02:00
Tom Krizek 0f8a2b07a4 Tear down module logger handler in system tests
The module-level logger has a handler that writes into a temporary
directory. Ensure the logging output is flushed and the handler is
closed before attempting to remove this temporary directory.
2023-05-22 14:11:41 +02:00
Tom Krizek 68b1c6877b Remove "which" declaration from env vars in EL8+ tests
EL8+ systems declare "which" function using environment variables in the
/etc/profile.d/which2.sh file. Because of our suboptimal environment
variable detection, which is required in order to support the legacy
runner, these variables are picked up by the pytest runner.

If subprocesses are spawned with these environment variables set, it
will cause the following issue when they spawn yet another subprocess:

/bin/sh: which: line 1: syntax error: unexpected end of file
/bin/sh: error importing function definition for `which'
2023-05-22 14:11:41 +02:00
Tom Krizek 12c724ee07 Capture log output during pytest runner setup
Instantiate a new logger that is used during pytest initialization /
configuration. This logging isn't handled by pytest itself, since it
happens outside of any tests or fixtures.

Root logger can't be reused for this purpose, because that would
duplicate the logs. Instead, create a conftest-specific logger for this
purpose.

Unfortunately, this introduces another log file,
pytest.conftest.log.txt, which contains only the logging from pytest
initialization. However, unless one is debugging the runner /
environment, there should be no need to investigate this file.
2023-05-22 14:11:40 +02:00
Tom Krizek 99e2e50c0e Execute long running system tests first
In order to take the most advantage of parallel execution of tests,
ensure certain long running tests are scheduled first.

The list of tests considered long-running was created empirically. In
addition to the test run time, its position in the default
(alphabetical) ordering was also taken into account.
2023-05-22 14:11:40 +02:00
Tom Krizek 952776b61f Add test specific logger for pytests
The logger fixture is provided as a test-level logging facility which
can be easily passed to tests to enable capturing and/or displaying
messages from tests written in Python.

While this works optimally with the pytest runner, messages on INFO
level or above will also be visible when using the legacy runner.
2023-05-22 14:11:40 +02:00
Tom Krizek 3a20e8d990 Ensure assertions and exceptions end up in system test log
If a test fails with an assertion failure or exception, its content
along with traceback is displayed in pytest output. This information
should be preserved in the test-specific logger for a given system test
to make it easier to debug test failures.
2023-05-22 14:11:40 +02:00
Petr ŠpačekandTom Krizek 37ed9ad2f8 Use raw byte format of env variables in pytest
In order to avoid issues with decoding/encoding env variables due to
different encodings on different systems, deal with the environment
variables directly as bytes.
2023-05-22 14:11:40 +02:00
Tom Krizek 0a063f51d3 Handle missing test_results due to pytest runner interrupt
If pytest execution is interrupted, the hook that exposes test_results
to the pytest session is never called so the results can't be
interpreted.
2023-05-22 14:11:40 +02:00
Petr ŠpačekandTom Krizek d0619c7a18 Enable live logging for non-parallel pytest runs
This provides incremental output when test is running _without xdist_,
just like the old runner did.

With xdist the live output is not available, I believe because of
https://github.com/pytest-dev/pytest-xdist/issues/402
https://github.com/pytest-dev/pytest-xdist/pull/883 might help with
that, but I'm not going to hold my breath until it is available on
distros we use.
2023-05-22 14:11:39 +02:00
Tom Krizek 8f57bce7af Ensure --dist=loadscope is used when running pytest in parallel
The loadscope setting is required for parallel execution of our system
tests using pytest. The option ensure that all tests within a single
(module) scope will be assigned to the same worker.

This is neccessary because the worker sets up the nameservers for all
the tests within a module scope. If tests from the same module would be
assigned to different workers, then the setup could happen multiple
times, causing a race condition. This happens because each module uses
deterministic port numbers for the nameservers.
2023-05-22 14:11:39 +02:00
Tom Krizek 2f5bf6d971 Add pytest functions for shell system tests
In order to run the shell system tests, the pytest runner has to pick
them up somehow. Adding an extra python file with a single function
for the shell tests for each system test proved to be the most
compatible way of running the shell tests across older pytest/xdist
versions.

Modify the legacy run.sh script to ignore these pytest-runner specific
glue files when executing tests written in pytest.
2023-05-22 14:11:39 +02:00
Tom Krizek 8a406b73c9 Ignore tempdirs during pytest collection phase 2023-05-22 14:11:38 +02:00
Tom Krizek 527ac6ad26 Ensure compatiblity with older pytest
Special care needs to be taken to support older pytest / xdist versions.
The target versions are what is available in EL8, since that seems to
have the oldest versions that can be reasonably supported.
2023-05-22 14:11:38 +02:00
Tom Krizek 247e90c382 Keep the tempdir in case test setup/teardown fails
When an issue occurs inside a fixture (e.g. servers fail to start/stop),
the test result won't be detected as failed, but rather an error will be
thrown.

To ensure the tempdir is kept even if the test itself passes but the
system_test() fixture throws an error, a different mechanism is needed.
At the start of the critical test setup section, note that the fixture
hasn't finished yet. When this is detected in the system_test_dir()
fixture, it is recognized as error in test setup/teardown and the temp
directory is kept.

This may seem cumbersome, because it is. It's basically a workaround for
the way pytest handles fixtures and test errors in general.
2023-05-22 14:11:38 +02:00
Tom Krizek 2d93ed8546 Ensure tempdir is kept for failed system tests
The temporary directory contains artifacts for the pytest module. That
module may contain multiple individual tests which were executed
sequentially. The artifacts should be kept if even one of these tests
failed.

Since pytest doesn't have any facility to expose test results to
fixtures, customize the pytest_runtest_makereport() hook to enable that.
It stores the test results into a session scope variable which is
available in all fixtures.

When deciding whether to remove the temporary directory, find the
relevant test results for this module and don't remove the tmpdir if any
one the tests failed.
2023-05-22 14:11:38 +02:00
Tom Krizek 8b1a906b39 Add --noclean option to pytest runner
Support the --noclean option to allow the user to keep the artifacts
from any test run.
2023-05-22 14:11:38 +02:00
Tom Krizek c571af8c8f Run system tests inside temporary directories
For every pytest module, create a copy of its system test directory and
run the test from that directory. This makes it easier to clean up
afterwards and makes it less error-prone when re-running (failed) tests.

Configure the logger to capture the module's log output into a separate
file available from the temporary directory. This is quite convenient
for exploring failures.

Cases where temporary directory should be kept are handled in a
follow-up commits.
2023-05-22 14:11:38 +02:00