Fix DNS-over-HTTP(S) implementation issues that arise under heavy
query load. Optimize resource usage for :iscman:`named` instances
that accept queries over DNS-over-HTTP(S).
Previously, :iscman:`named` would process all incoming HTTP/2 data
at once, which could overwhelm the server, especially when dealing
with clients that send requests but don't wait for responses. That
has been fixed. Now, :iscman:`named` handles HTTP/2 data in smaller
chunks and throttles reading until the remote side reads the
response data. It also throttles clients that send too many requests
at once.
Additionally, :iscman:`named` now carefully processes data sent by
some clients, which can be considered "flooding." It logs these
clients and drops connections from them.
:gl:`#4795`
In some cases, :iscman:`named` could leave DNS-over-HTTP(S)
connections in the `CLOSE_WAIT` state indefinitely. That also has
been fixed. ISC would like to thank JF Billaud for thoroughly
investigating the issue and verifying the fix.
:gl:`#5083`
See https://gitlab.isc.org/isc-projects/bind9/-/issues/4795
Closes https://gitlab.isc.org/isc-projects/bind9/-/issues/5083
Backport of !732.
Merge branch 'artem-improve-doh-resource-usage-9.18' into 'v9.18.33-release'
See merge request isc-private/bind9!763
We started using isc_nm_bad_request() more actively throughout
codebase. In the case of HTTP/2 it can lead to a large count of
useless "Bad Request" messages in the BIND log, as often we attempt to
send such request over effectively finished HTTP/2 sessions.
This commit fixes that.
(cherry picked from commit 937b5f8349)
This commit introduces manual read timer control as used by StreamDNS
and its underlying transports. Before that, DoH code would rely on the
timer control provided by TCP, which would reset the timer any time
some data arrived. Now, the timer is restarted only when a full DNS
message is processed in line with other DNS transports.
That change is required because we should not stop the timer when
reading from the network is paused due to throttling. We need a way to
drop timed-out clients, particularly those who refuse to read the data
we send.
(cherry picked from commit 609a41517b)
This commit adds logic to make code better protected against clients
that send valid HTTP/2 data that is useless from a DNS server
perspective.
Firstly, it adds logic that protects against clients who send too
little useful (=DNS) data. We achieve that by adding a check that
eventually detects such clients with a nonfavorable useful to
processed data ratio after the initial grace period. The grace period
is limited to processing 128 KiB of data, which should be enough for
sending the largest possible DNS message in a GET request and then
some. This is the main safety belt that would detect even flooding
clients that initially behave well in order to fool the checks server.
Secondly, in addition to the above, we introduce additional checks to
detect outright misbehaving clients earlier:
The code will treat clients that open too many streams (50) without
sending any data for processing as flooding ones; The clients that
managed to send 1.5 KiB of data without opening a single stream or
submitting at least some DNS data will be treated as flooding ones.
Of course, the behaviour described above is nothing else but
heuristical checks, so they can never be perfect. At the same time,
they should be reasonable enough not to drop any valid clients,
realatively easy to implement, and have negligible computational
overhead.
(cherry picked from commit 3425e4b1d0)
Initially, our DNS-over-HTTP(S) implementation would try to process as
much incoming data from the network as possible. However, that might
be undesirable as we might create too many streams (each effectively
backed by a ns_client_t object). That is too forgiving as it might
overwhelm the server and trash its memory allocator, causing high CPU
and memory usage.
Instead of doing that, we resort to processing incoming data using a
chunk-by-chunk processing strategy. That is, we split data into small
chunks (currently 256 bytes) and process each of them
asynchronously. However, we can process more than one chunk at
once (up to 4 currently), given that the number of HTTP/2 streams has
not increased while processing a chunk.
That alone is not enough, though. In addition to the above, we should
limit the number of active streams: these streams for which we have
received a request and started processing it (the ones for which a
read callback was called), as it is perfectly fine to have more opened
streams than active ones. In the case we have reached or surpassed the
limit of active streams, we stop reading AND processing the data from
the remote peer. The number of active streams is effectively decreased
only when responses associated with the active streams are sent to the
remote peer.
Overall, this strategy is very similar to the one used for other
stream-based DNS transports like TCP and TLS.
(cherry picked from commit 9846f395ad)
This commit adds isc__nm_async_run() which is very similar to
isc_async_run() in newer versions of BIND: it allows calling a
callback asynchronously.
Potentially, it can be used to replace some other async operations in
other networking code, in particular the delayed I/O calls in TLS a
TCP DNS transports to name a few and remove quiet a lot of code, but
it we are unlikely to do that for the strictly maintenance only
branch, so it is protected with DoH-related #ifdefs.
It is implemented in a "universal" way mainly because doing it in the
specific code requires the same amount of code and is not simpler.
This commit adds a manual TLS read timer control mode which is
supposed to override automatic resetting of the timer when any data is
received.
It both depends and complements similar functionality in TCP.
This commit adds a manual TCP read timer control mode which is
supposed to override automatic resetting of the timer when any data is
received. That can be accomplished by
`isc__nmhandle_set_manual_timer()`.
This functionality is supposed to be used by multilevel networking
transports which require finer grained control over the read
timer (TLS Stream, DoH).
The commit is essentially an implementation of the functionality from
newer versions of BIND.
When answering queries, don't add data to the additional section if the answer has more than 13 names in the RDATA. This limits the number of lookups into the database(s) during a single client query, reducing query processing load.
Backport of MR !750
See isc-projects/bind9#5034
Merge branch '5034-security-limit-additional-9.18' into 'v9.18.33-release'
See merge request isc-private/bind9!759
When answering queries, don't add data to the additional section if
the answer has more than 13 names in the RDATA. This limits the
number of lookups into the database(s) during a single client query,
reducing query processing load.
Also, don't append any additional data to type=ANY queries. The
answer to ANY is already big enough.
(cherry picked from commit a1982cf1bb)
Instead of running the whole resolver/ns4 server with -T noaa flag,
use it only for the part where it is actually needed. The -T noaa
could interfere with other parts of the test because the answers don't
have the authoritative-answer bit set, and we could have false
positives (or false negatives) in the test because the authoritative
server doesn't follow the DNS protocol for all the tests in the resolver
system test.
(cherry picked from commit e51d4d3b88)
Add performance tests of DoH using the GET protocol to nightly pipelines.
Backport of MR !9926
Merge branch 'backport-nicki/ci-shotgun-doh-get-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9940
When isc_rwlock_trylock() fails to get a read lock because another
writer was faster, it should wake up other waiting writers in case
there are no other readers, but the current code forgets about
the currently active writer when evaluating 'cntflag'.
Unset the WRITER_ACTIVE bit in 'cntflag' before checking to see if
there are other readers, otherwise the waiting writers, if they exist,
might not wake up.
Closes#5121
Merge branch 'aram/isc_rwlock_trylock-bugfix-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9937
When isc_rwlock_trylock() fails to get a read lock because another
writer was faster, it should wake up other waiting writers in case
there are no other readers, but the current code forgets about
the currently active writer when evaluating 'cntflag'.
Unset the WRITER_ACTIVE bit in 'cntflag' before checking to see if
there are other readers, otherwise the waiting writers, if they exist,
might not wake up.
coccinelle v1.1 trips over a superfluous isc_mem_get() NULL check in
tests/libtest/ns.c and reports the following failure in CI:
EXN: Failure("rule starting on line 26: already tagged token:\nC code context\nFile \"./tests/libtest/ns.c\", line 350, column 1, charpos = 7939\n around = 'if',\n whole content = \tif (qctx != NULL) {") in ./tests/libtest/ns.c
(cherry picked from commit cf76851c75)
Fix the loop terminating condition to get consistent sample sizes and increase the minimum number of samples from 20 to 40.
Closes#5091
Backport of MR !9894
Merge branch 'backport-5091-investigate-checking-startup-notify-rate-limit-failure-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9910
The terminating conditions for the startup notify test would
occasionally get ~20 records or get +10 seconds of records due to
a bad terminating condition. Additionally 20 samples lead to test
failures. Fix the terminating condition to use the correct conditional
(-eq -> -ge) and increase the minimum number of log entries to
average over to 22.
(cherry picked from commit 46388d07a2)
Closes#5088
Backport of MR !9884
Merge branch 'backport-5088-tests-irs-resconf_test-c-is-missing-check-callbacks-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9908
Include the recent changes such as:
- changes to running system tests
- gitlab development workflow
- changelog and release note process
Closes#5045
Backport of MR !9784
Merge branch 'backport-5045-update-contributing-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9903
Include the recent changes such as:
- changes to running system tests
- gitlab development workflow
- changelog and release note process
(cherry picked from commit 39485c1f70)
After the rndc reload command finished, we might have queried the
database zone sooner than it was reloaded because rndc reloads zones
asynchronously if no specific zone was provided. We should wait for "all
zones loaded" in the ns1 log to be sure.
Closes#5075
Backport of MR !9829
Merge branch 'backport-5075-database-rndc-reload-ensure-all-zones-loaded-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9901
After the rndc reload command finished, we might have queried the
database zone sooner than it was reloaded because rndc reloads zones
asynchronously if no specific zone was provided. We should wait for "all
zones loaded" in the ns1 log to be sure.
(cherry picked from commit 0bdd03db66)
The style guide now mentions clang-format, doesn't parenthesize return values, and no longer calls for backward compatibility in public function names.
Backport of MR !9892
Merge branch 'backport-each-style-update-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9897
It now mentions clang-format, doesn't parenthesize return values,
and no longer calls for backward compatibility in public function names.
(cherry picked from commit 9f7314eaa4)
The December releases suffer from the ns2/managed1.conf file not being
in the mkeys extra_artifacts. This manifests only when pytest is run
with the --setup-only option, which is the case in the
cross-version-config-tests CI job. The original issue is fixed in !9815,
but the fix will be effective only when subsequent releases are out.
(cherry picked from commit 97a9d7287c)
This allows easier identification of which burst is which in
named.run.
Backport of MR !9881
Merge branch 'backport-marka-use-different-burst-name-for-forensics-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9883
This subtest exercises static stub behaviour when server-addresses has an address. This was misidentified in the description.
Closes!9799
Backport of MR !9799
Merge branch 'backport-marka-fix-stub-subtest-description-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9880
The line after an unknown directive in resolv.conf could accidentally be skipped, potentially affecting dig, host, nslookup, nsupdate, or delv. This has been fixed.
Closes#5084
Backport of MR !9865
Merge branch 'backport-5084-plain-unknown-keyword-in-resolv-conf-not-handled-propely-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9878
Update to the new unit test framework.
Add a test for an unknown directive without any arguments.
Add test for an unknown directive without arguments, followed
by a search directive.
(cherry picked from commit c44c4fcbfb)
Only call eatline() to skip to the next line if we're not
already at the end of a line when parsing an unknown directive.
We were accidentally skipping the next line when there was only
a single unknown directive on the current line.
(cherry picked from commit eb78ad2080)
Prereq: isc-projects/images!345
Backport of MR !9612
Merge branch 'backport-mnowak/fedora-41-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9876
Prereq: isc-projects/images!359
Backport of MR !9872
Merge branch 'backport-mnowak/alpine-3.21-9.18' into 'bind-9.18'
See merge request isc-projects/bind9!9874
