Ensure that the jinja2 templates have all the environment variables
which are available to the tests present.
This omission during the original 9.18 backport caused an issue where
port numbers (determined by the pytest framework) wouldn't be available
in jinja2 templates.
The DLZ modules are poorly maintained as we only ensure they can still
be compiled, the DLZ interface is blocking, so anything that blocks the
query to the database blocks the whole server and they should not be
used except in testing. The DLZ interface itself is going to be scheduled
for removal.
The DLZ modules now live in https://gitlab.isc.org/isc-projects/dlz-modules
repository.
Closes#4865
Backport of MR !9349
Merge branch 'backport-4865-remove-contributed-DLZ-modules-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9778
The DLZ modules are poorly maintained as we only ensure they can still
be compiled, the DLZ interface is blocking, so anything that blocks the
query to the database blocks the whole server and they should not be
used except in testing. The DLZ interface itself should be scheduled
for removal.
(cherry picked from commit a6cce753e2)
In two places, after linking the client to the manager's
"recursing-clients" list using the check_recursionquota()
function, the query.c module fails to unlink it on error
paths. Fix the bugs by unlinking the client from the list.
Backport of MR !9586
Merge branch 'backport-aram/unlink-recursing-clients-on-error-paths-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9605
In two places, after linking the client to the manager's
"recursing-clients" list using the check_recursionquota()
function, the query.c module fails to unlink it on error
paths. Fix the bugs by unlinking the client from the list.
Also make sure that unlinking happens before detaching the
client's handle, as it is the logically correct order, e.g.
in case if it's the last handle and ns__client_reset_cb()
can be called because of the detachment.
(cherry picked from commit 36c4808903)
Some tests may leave artifacts in the .libs directory. Ignore this
directory when detecting expected artifacts.
Closes#5055
Backport of MR !9766
Merge branch 'backport-5055-ignore-libs-artifacts-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9774
`dig` fails to parse a valid (as far as I can tell, and accepted by `kdig` and `Wireshark`) `SVCB` record with a `dohpath` URI template containing a `{&dns}`, like `dohpath=/some/path?key=value{&dns}"`. If the URI template contains a `{?dns}` instead `dig` is happy, but my understanding of rfc9461 and section 1.2. "Levels and Expression Types" of rfc6570 is that `{&dns}` is valid.
See for example section 1.2. "Levels and Expression Types" of rfc6570.
Note that Peter van Dijk suggested that `{dns}` and `{dns,someothervar}` might be valid forms as well, so my patch might be too restrictive, although it's anyone's guess how DoH clients would handle complex templates.
Closes https://gitlab.isc.org/isc-projects/bind9/-/issues/4922
Backport of MR !9455
Merge branch 'backport-svcb-dohpath-uri-template-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9770
The 'dns' variable in dohpath can be in various forms ({?dns},
{dns}, {&dns} etc.). To check for a valid dohpath it ends up
being simpler to just parse the URI template rather than looking
for all the various forms if substring.
(cherry picked from commit af54ef9f5d)
by logging SSL_CTX_use_certificate_chain_file and SSL_CTX_use_PrivateKey_file errors individually.
Closes#5008
Backport of MR !9683
Merge branch 'backport-5008-provide-more-visibility-into-ssl-errors-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9768
On some slow systems, the test might intermittently fail due to inherent
timing issues. In our CI, this most often happens in the
system:gcc:8fips:amd64 jobs.
Closes#3098
Backport of MR !9732
Merge branch 'backport-3098-allow-re-run-of-mkeys-test-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9765
On some slow systems, the test might intermittently fail due to inherent
timing issues. In our CI, this most often happens in the
system:gcc:8fips:amd64 jobs.
(cherry picked from commit a299650a6f)
The cross-version-config-tests job has never functioned in CI because
the testing framework changed after the testing was completed. To run
the new "named" binary using the old configurations, paths in the test
framework must be updated to point to the location of the new binaries.
Closes#4977
Backport of MR !9702
Merge branch 'backport-4977-fix-cross-version-config-tests-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9763
The cross-version-config-tests job has never functioned in CI because
the testing framework changed after the testing was completed. To run
the new "named" binary using the old configurations, paths in the test
framework must be updated to point to the location of the new binaries.
(cherry picked from commit 09d7e2430a)
Re-split format strings that had been poorly split by multiple
clang-format runs using different versions of clang-format.
Closes#5043
Backport of MR !9752
Merge branch 'backport-5043-re-split-format-strings-lib-ns-update-c-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9756
Re-split format strings that had been poorly split by multiple
clang-format runs using different versions of clang-format.
(cherry picked from commit a24d6e1654)
* Add new clang-format option to remove redundant semicolons
* Add new clang-format option to remove redundant parentheses
Backport of MR !9749
Merge branch 'backport-ondrej/code-style-2024-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9751
The new clang-format options (since clang-format 16) removes semicolons
after the closing braces of functions and constructors/destructors.
(cherry picked from commit c7420eccd9)
The new clang-format option (since 17) can remove redundant parentheses
in the return statement (with configuration value ReturnStatement).
(cherry picked from commit 3873b0c279)
Add an 'initial-ds' entry to bind.keys for the new root key, ID
38696, which is scheduled for publication in January 2025.
Closes#4896
Backport of MR !9422
Merge branch 'backport-4896-update-bind-keys-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9747
Add an 'initial-ds' entry to bind.keys for the new root key, ID
38696, scheduled for publication in January 2025.
(cherry picked from commit 609bf35075)
The fix for #4950 should have never been backported to 9.18. Revert the change.
This reverts MR !9632
History:
A performance improvement for NSEC3 closest encloser lookups (#4460) was introduced (in MR !9436) and backported to 9.20 (MR !9438) and to 9.18 in (MR !9439). It was released in 9.18.30 (and 9.20.2 and 9.21.1).
There was a bug in the code (#4950), so we reverted the change in !9611, !9613 and !9614 (not released).
Then a new attempt was merged in main (MR !9610) and backported to 9.20 (MR !9631) and 9.18 (MR !9632). The latter should not have been backported.
Furthermore, the initial MR used the wrong MR title so the change was never added to the release note. This is done in main with MR !9598 and backports to 9.20 (MR !9615) and 9.18 (MR !9616).
The new release notes for 9.21 and 9.20 should probably say that the bug is fixed. The new release notes for 9.18 should probably say that the change is reverted.
Merge branch 'revert-4950-bind-logs-expected-covering-nsec3-got-an-exact-match-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9689
Keeping the Known Issues as part of the rendered docs has the issue that
the list can't be updated on the official docs website until the next
release. This is unpractical is a high-priority issue is discovered
shortly after a release. Keep the Known Issues in wiki and simply link
to the list from the rendered docs. The wiki article can be updated at
any time as needed.
Merge branch 'nicki/move-known-issues-to-wiki-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9718
Keeping the Known Issues as part of the rendered docs has the issue that
the list can't be updated on the official docs website until the next
release. This is unpractical is a high-priority issue is discovered
shortly after a release. Keep the Known Issues in wiki and simply link
to the list from the rendered docs. The wiki article can be updated at
any time as needed.
Some omissions of !9426 discovered during the backports
Backport of MR !9739
Merge branch 'backport-nicki/extra-artifacts-fixups-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9742
When a test is re-run by the flaky plugin, the TestReport outcomes
collected in the pytest_runtest_makereport() hook should be overriden.
Each of the setup/call/teardown phases is reported again and since we
care about the overall outcome, their respective results should be
overriden so that only the outcome from the final test (re)run gets
reported.
Prior to this change, it lead to a situation where an extra_artifact
generated during the test might be ignored. This was caused because the
check was skipped, since the test was incorrectly considered as "failed"
in the case where the test would fail on the first run, but pass on a
subsequent flaky rerun.
(cherry picked from commit b66fb31dcb)
``clean.sh`` scripts have been replaced by lists of expected artifacts for each system test module. The list is defined using the custom ``pytest.mark.extra_artifacts`` mark, which can use both filenames and globs.
Closes#4261
Backport of MR !9426
Merge branch 'backport-4261-add-pytest-fixture-checking-test-artifacts-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9735
The artifact lists in clean.sh and extra_artifacts might be slightly
different. The list was updated for each test to reflect the current
state.
(cherry picked from commit 7c259fe254)
Prior to introducing the pytest runner, clean.sh files were used as a
list of files that the test is expected to leave around as artifacts and
check that no extra files were created.
With the pytest runner, those scripts are no longer used, but the
ability to detect extraneous files is still useful. Add a new
"extra_artifacts" mark which can be used for the same purpose.
(cherry picked from commit 3a9f4edddc)
Remove named_g_sessionkey and named_g_sessionkeyname as they are declared and cleaned up but otherwise are unused.
Closes#5023
Backport of MR !9720
Merge branch 'backport-5023-remove-named_g_sessionkey-as-it-is-unused-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9726
With the introduction of the generated changelog, the CHANGES file
became a symlink to doc/arm/changelog.rst. After the changes made in
!9549, the changelog file transitioned from being a wholly generated
file to one that includes versioned changelog files, which are
themselves generated. However, while implementing !9549, we overlooked
that the CHANGES file is copied to a release directory on an FTP server
and contains just "include" directives, not the changelog itself.
Therefore, in the same fashion as the "RELEASE-NOTES*.html" file, create
a "CHANGELOG*.html" file that redirects to the Changelog appendix of the
ARM.
Closes#5000
Backport of MR !9690
Merge branch 'backport-5000-provide-correct-changelog-on-ftp-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9704
With the introduction of the generated changelog, the CHANGES file
became a symlink to doc/arm/changelog.rst. After the changes made in
!9549, the changelog file transitioned from being a wholly generated
file to one that includes versioned changelog files, which are
themselves generated. However, while implementing !9549, we overlooked
that the CHANGES file is copied to a release directory on an FTP server
and contains just "include" directives, not the changelog itself.
Therefore, in the same fashion as the "RELEASE-NOTES*.html" file, create
a "CHANGELOG*.html" file that redirects to the Changelog appendix of the
ARM.
(cherry picked from commit e40bd273e4)
Configuration files in system tests which require some variables (e.g.
port numbers) filled in during test setup, can now use jinja2 templates
when `jinja2` python package is available.
Any `*.j2` file found within the system test directory will be
automatically rendered with the environment variables into a file
without the `.j2` extension by the pytest runner. E.g.
`ns1/named.conf.j2` will become `ns1/named.conf` during test setup. To
avoid automatic rendering, use `.j2.manual` extension and render the
files manually at test time.
New `templates` pytest fixture has been added. Its `render()` function
can be used to render a template with custom test variables. This can be
useful to fill in different config options during the test. With
advanced jinja2 template syntax, it can also be used to include/omit
entire sections of the config file rather than using `named1.conf.in`,
`named2.conf.in` etc.
Closes#4938
Backport of MR !9587
Merge branch 'backport-4938-use-jinja2-templates-in-system-tests-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9700
Configuration files in system tests which require some variables (e.g.
port numbers) filled in during test setup, can now use jinja2 templates
when `jinja2` python package is available.
Any `*.j2` file found within the system test directory will be
automatically rendered with the environment variables into a file
without the `.j2` extension by the pytest runner. E.g.
`ns1/named.conf.j2` will become `ns1/named.conf` during test setup. To
avoid automatic rendering, use `.j2.manual` extension and render the
files manually at test time.
New `templates` pytest fixture has been added. Its `render()` function
can be used to render a template with custom test variables. This can be
useful to fill in different config options during the test. With
advanced jinja2 template syntax, it can also be used to include/omit
entire sections of the config file rather than using `named1.conf.in`,
`named2.conf.in` etc.
(cherry picked from commit 60e118c4fb)
Use a stricter hazard check which ensures the audience tag is present in
the MR title and is one of the known values. This prevents siuations
where incorrect audience is accidentally used, resulting in a missing
changelog entry or a release note.
(cherry picked from commit cdb93bcbd4)
Make system tests symlinks and logged test names consistent across pytest versions.
Backport of MR !9071
Merge branch 'backport-nicki/pytest-v8-compat-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9686
The pytest collection mechanism has been overhauled in pytest 8.0.0,
resulting in a different node tree when collecting the tests. Ensure the
paths / names we're using that are derived from the node tree are
consistent across different pytest versions.
Particularly, this has affected the convenience symlink name (which is
supposed to be in the form of e.g. dns64_sh_dns64 for the dns64 module
and tests_sh_dns64.py module) and the test name that's logged at the
start of the test, which is supposed to include the system test
directory relative to the root system test directory as well as the
module name (e.g. dns64/tests_sh_dns64.py).
Related https://github.com/pytest-dev/pytest/issues/7777
(cherry picked from commit 7118cbed98)
