DNSRPS-enabled builds have recently been silently broken a few times due
to that feature not being tested in regular CI pipelines. Add the
--enable-dnsrps --enable-dnsrps-dl switches to the ./configure
invocation in one of the CI jobs run for all merge requests so that
DNSRPS-related build issues can be detected in advance.
It is important to note that this change by itself does NOT enable
actual testing of the DNSRPS feature as doing that requires a DNSRPS
provider library to be present on the test host.
Building the bin/tests/system/rpz/dnsrps helper binary is currently not
possible at all as the necessary compiler and linker flag definitions
are missing from bin/tests/system/Makefile.am. Add these as a basis for
addressing the problem.
Unfortunately, this is where the "mostly" bit mentioned in this commit's
subject line comes into play. The dlopen() parts of DNSRPS code have
not yet been reworked to use libuv's dlopen() API (uv_dlopen() etc.)
(See commit 37b9511ce1 for prior work in
this area.) While it is certainly possible to do that, implementing
such a change without testing it in practice against a usable librpz.so
(i.e. a DNSRPS provider library) is bound to cause more trouble and
confusion than keeping the code the way it is right now. However,
making that code buildable as-is requires linking against a C standard
library that exports the dlopen(), dlsym(), and dlclose() symbols used
by the DNSRPS dynamic loading code. glibc 2.34+ satisfies that
requirement, but older glibc versions do not (these come with a separate
libdl shared library that would need to be linked in as well). (Other
C standard library implementations have not been examined.) Since the
long-term plan is to rely on libuv's dlopen() API exclusively and
detecting the shared object containing dlopen() & friends would only
pull in build system complexity for no good reason, assume for now that
the target system provides the dlopen() API in its C standard library.
This change enables the system test suite to be run for a BIND 9 build
prepared using --enable-dnsrps --enable-dnsrps-dl (on systems satisfying
the requirement explained above). However, it is important to note that
this change by itself does NOT enable actual testing of the DNSRPS
feature as doing that requires a DNSRPS provider library to be present
on the test host.
This implements node reference tracing that passes all the internal
layers from dns_db API (and friends) to increment_reference() and
decrement_reference().
It can be enabled by #defining DNS_DB_NODETRACE in <dns/trace.h> header.
The output then looks like this:
incr:node:check_address_records:rootns.c:409:0x7f67f5a55a40->references = 1
decr:node:check_address_records:rootns.c:449:0x7f67f5a55a40->references = 0
incr:nodelock:check_address_records:rootns.c:409:0x7f67f5a55a40:0x7f68304d7040->references = 1
decr:nodelock:check_address_records:rootns.c:449:0x7f67f5a55a40:0x7f68304d7040->references = 0
There's associated python script to find the missing detach located at:
https://gitlab.isc.org/isc-projects/bind9/-/snippets/1038
Change the commandline option -G to take a string that determines what
sync records should be published. It is a comma-separated string with
each element being either "cdnskey", or "cds:<algorithm>", where
<algorithm> is a valid digest type. Duplicates are suppressed.
Now that we can configure a different digest type, update the code
to honor the configuration. Update 'dns_dnssec_syncupdate' so that
the correct CDS record is published, and also when deleting CDS records,
ensure that all possible CDS records are removed from the zone.
Change one of the test cases to use a different digest type (4). The
system tests and kasp script need to be updated to take into account
the new algorithm (instead of the hard coded 2).
Commits ffa4757c79 and
77e7eac54c inadvertently broke
DNSRPS-enabled builds:
- the new member of struct dns_db that holds a reference count for the
database is called 'references', not 'refcount',
- a syntax error was introduced in the designated initializer for
'rpsdb_rdataset_methods',
- rpsdb_destroy() no longer takes a 'dbp' argument.
Address all of the above issues to make DNSRPS-enabled builds work
again.
The test was setting a minimum count for recursive clients which
was not always being met (e.g. 91 instead of 100) producing a false
positive. Lower the lower bound on recursive clients for this
test to 1.
In general, it's better to do one thorough compaction when a batch of
work is complete, which is the way that `update` transactions work.
Conversely, `write` transactions are designed so that lots of little
transactions are not too inefficient, but they need explicit
compaction. This changes `dns_qp_compact()` so that it is easier to
compact any time that makes sense, if there isn't a better way to
schedule compaction. And `dns_qpmulti_commit()` only recycles garbage
when there is enough to make it worthwhile.
Add some qp-trie tracing macros which can be enabled by a
developer. These print a message when a leaf is attached or
detached, indicating which part of the qp-trie implementation
did so. The refcount methods must now return the refcount value
so it can be printed by the trace macros.
The first working multi-threaded qp-trie was stuck with an unpleasant
trade-off:
* Use `isc_rwlock`, which has acceptable write performance, but
terrible read scalability because the qp-trie made all accesses
through a single lock.
* Use `liburcu`, which has great read scalability, but terrible
write performance, because I was relying on `rcu_synchronize()`
which is rather slow. And `liburcu` is LGPL.
To get the best of both worlds, we need our own scalable read side,
which we now have with `isc_qsbr`. And we need to modify the write
side so that it is not blocked by readers.
Better write performance requires an async cleanup function like
`call_rcu()`, instead of the blocking `rcu_synchronize()`. (There
is no blocking cleanup in `isc_qsbr`, because I have concluded
that it would be an attractive nuisance.)
Until now, all my multithreading qp-trie designs have been based
around two versions, read-only and mutable. This is too few to
work with asynchronous cleanup. The bare minimum (as in epoch
based reclamation) is three, but it makes more sense to support an
arbitrary number. Doing multi-version support "properly" makes
fewer assumptions about how safe memory reclamation works, and it
makes snapshots and rollbacks simpler.
To avoid making the memory management even more complicated, I
have introduced a new kind of "packed reader node" to anchor the
root of a version of the trie. This is simpler because it re-uses
the existing chunk lifetime logic - see the discussion under
"packed reader nodes" in `qp_p.h`.
I have also made the chunk lifetime logic simpler. The idea of a
"generation" is gone; instead, chunks are either mutable or
immutable. And the QSBR phase number is used to indicate when a
chunk can be reclaimed.
Instead of the `shared_base` flag (which was basically a one-bit
reference count, with a two version limit) the base array now has a
refcount, which replaces the confusing ad-hoc lifetime logic with
something more familiar and systematic.
Adjust the dns_qp_memusage() and dns_qp_compact() functions
to be more informative and flexible about handling fragmentation.
Avoid wasting space in runt chunks.
Switch from twigs_mutable() to cells_immutable() because that is the
sense we usually want.
Drop the redundant evacuate() function and rename evacuate_twigs() to
evacuate(). Move some chunk test functions closer to their point of
use.
Clarify compact_recursive(). Some small cleanups to comments.
Use isc_time_monotonic() for qp-trie timing stats.
Use #define constants to control debug logging.
Set up DNS name label offsets in dns_qpkey_fromname() so it is easier
to use in cases where the name is not fully hydrated.
The main benchmark is `qpmulti`, which exercizes the qp-trie
transactional API with differing numbers of threads and differing data
sizes, to get some idea of how its performance scales.
The `load-names` benchmark compares the times to populate and query
and the memory used by various BIND data structures: qp-trie, hash
table (chained), hash map (closed), and red-black tree.
The `qp-dump` program is a test utility rather than a benchmark. It
populates a qp-trie and prints it out, either in an ad-hoc text
format, or as input to the graphviz `dot` program.
Ensure dns_qpkey_fromname() and dns_qpkey_toname() are inverses.
Excercise a single-threaded dns_qp_t with a fixed set of random keys
and a small chunk size. Use the table of names to ensure that the trie
is behaving as expected. This is (in effect) randomized testing like
the `qpmulti` unit test, but making use of coverage-guided fuzzing
and (in principle) test case minimization.
Randomized testing with intensive consistency and correctness checks
make it much easier to get good coverage and to shake out bugs than
hand-written unit tests for specific cases.
These tests only run in a single thread, but each test transaction
uses both a write/update and a query/snapshot, to ensure that
modifications are not visible to concurrent readers.
This change adds a number of support routines for the unit tests, and
for benchmarks and fuzz tests to be added later. It isn't necessary to
include the support routines in libdns, since they are not needed by
BIND's installed programs. So `libtest` seems like the best place for
them.
The tests themselves verify that dns_qpkey_fromname() behaves as
expected.
The error occurred when:
* The bump chunk was re-used across multiple write transactions.
In this situation the bump chunk is marked immutable, but the
immutable flag is disregarded for cells after the fender, which
were allocated in the current transaction.
* The bump chunk fills up during an insert operation, so that the
enlarged twigs vector is allocated from a new bump chunk.
* Before this happened, we should have (but didn't) made the twigs
vector mutable. This would have adjusted its refcounts as necessary.
* However, moving to a new bump chunk has a side effect: twigs that
were previously considered mutable because they are after the
fender become immutable.
* Because of this, the old twigs vector was not destroyed as expected.
* So leaves were duplicated without their refcounts being increased.
The effect is that the refcounts were lower than they should have
been, and underflowed. The tests failed to check for refcount
underflow, so this mistake was detected much later than it ideally
could have been.
After the fix, it is now correct not to ensure the twigs are mutable,
because they are about to be copied to a larger vector. Instead, we
need to find out whether `squash_twigs()` destroyed the old twigs, and
adjust the refcounts accordingly.
A qp-trie is a kind of radix tree that is particularly well-suited to
DNS servers. I invented the qp-trie in 2015, based on Dan Bernstein's
crit-bit trees and Phil Bagwell's HAMT. https://dotat.at/prog/qp/
This code incorporates some new ideas that I prototyped using
NLnet Labs NSD in 2020 (optimizations for DNS names as keys)
and 2021 (custom allocator and garbage collector).
https://dotat.at/cgi/git/nsd.git
The BIND version of my qp-trie code has a number of improvements
compared to the prototype developed for NSD.
* The main omission in the prototype was the very sketchy outline of
how locking might work. Now the locking has been implemented,
using a reader/writer lock and a mutex. However, it is designed to
benefit from liburcu if that is available.
* The prototype was designed for two-version concurrency, one
version for readers and one for the writer. The new code supports
multiversion concurrency, to provide a basis for BIND's dbversion
machinery, so that updates are not blocked by long-running zone
transfers.
* There are now two kinds of transaction that modify the trie: an
`update` aims to support many very small zones without wasting
memory; a `write` avoids unnecessary allocation to help the
performance of many small changes to the cache.
* There is also a single-threaded interface for situations where
concurrent access is not necessary.
* The API makes better use of types to make it more clear which
operations are permitted when.
* The lookup table used to convert a DNS name to a qp-trie key is
now initialized by a run-time constructor instead of a programmer
using copy-and-paste. Key conversion is more flexible, so the
qp-trie can be used with keys other than DNS names.
* There has been much refactoring and re-arranging things to improve
the terminology and order of presentation in the code, and the
internal documentation has been moved from a comment into a file
of its own.
Some of the required functionality has been stripped out, to be
brought back later after the basics are known to work.
* Garbage collector performance statistics are missing.
* Fancy searches are missing, such as longest match and
nearest match.
* Iteration is missing.
* Search for update is missing, for cases where the caller needs to
know if the value object is mutable or not.
Some qp-trie operations will need to know the maximum number of labels
in a name, so I wanted a standard macro definition with the right
value.
Replace DNS_MAX_LABELS from <dns/resolver.h with DNS_NAME_MAXLABELS in
<dns/name.h>, and add its counterpart DNS_NAME_LABELLEN.
Use these macros in `name.c` and `resolver.c`.
Fix an off-by-one error in an assertion in `dns_name_countlabels()`.
When detaching from the previous version of the database, make sure
that the update-notify callback is unregistered, otherwise there is
an INSIST check which can generate an assertion failure in free_rbtdb(),
which checks that there are no outstanding update listeners in the list.
There is a similar code already in place for RPZ.
The zone_postload() function can fail and unregister the callbacks.
Call dns_db_endload() only after calling zone_postload() to make
sure that the registered update-notify callbacks are not called
when the zone loading has failed during zone_postload().
Also, don't ignore the return value of zone_postload().
Add the 'ixfr-from-differences yes;' option to trigger a failed
zone postload operation when a zone is updated but the serial
number is not updated, then issue two successive 'rndc reload'
commands to trigger the bug, which causes an assertion failure.
The dbiterator read-locks the whole zone and it stayed locked during
whole processing time when catz is being read. Pause the iterator, so
the updates to catz zone are not being blocked while processing the catz
update.
Instead of holding the catzs->lock the whole time we process the catz
update, only hold it for hash table lookup and then release it. This
should unblock any other threads that might be processing updates to
catzs triggered by extra incoming transfer.
Offload catalog zone processing so that the network manager threads
are not interrupted by a large catalog zone update.
Introduce a new 'updaterunning' state alongside with 'updatepending',
like it is done in the RPZ module.
Note that the dns__catz_update_cb() function currently holds the
catzs->lock during the whole process, which is far from being optimal,
but the issue is going to be addressed separately.
This change should make sure that catalog zone update processing
doesn't happen when the catalog zone is being shut down. This
should help avoid races when offloading the catalog zone updates
in the follow-up commit.
The configure_catz() function creates the catalog zones structure
for the view even when it is not needed, in which case it then
discards it (by detaching) later.
Instead, call dns_catz_new_zones() only when it is needed, i.e. when
there is no existing "previous" view with an existing 'catzs', that
is going to be reused.
* Change 'dns_catz_new_zones()' function's prototype (the order of the
arguments) to synchronize it with the similar function in rpz.c.
* Rename 'refs' to 'references' in preparation of ISC_REFCOUNT_*
macros usage for reference tracking.
* Unify dns_catz_zone_t naming to catz, and dns_catz_zones_t naming to
catzs, following the logic of similar changes in rpz.c.
* Use C compound literals for structure initialization.
* Synchronize the "new zone version came too soon" log message with the
one in rpz.c.
* Use more of 'sizeof(*ptr)' style instead of the 'sizeof(type_t)' style
expressions when allocating or freeing memory for 'ptr'.
`libirs` used to be a reference implementation of `getaddrinfo` and
related modern resolver APIs. It was stripped down in BIND 9.18
leaving only the `irs_resconf` module, which parses
`/etc/resolv.conf`. I have kept its include path and namespace prefix,
so it remains a little fragment of libirs now embedded in libdns.
Add new SonarCloud GitHub Action and configuration; something (maybe
the way the builds were submitted) has apparently changed and the
project got deleted and the analysis wasn't working.
when a message arrives over a TCP connection matching an expected
QID, the dispatch is updated so it no longer expects that QID,
but continues reading. subsequent messages with the same QID are
ignored, unless the dispatch entry has called dns_dispatch_getnext()
or dns_dispatch_resume().
however, a coding error caused those functions to have no effect
when the dispatch was reading, so streams of messages with the same
QID could not be received over a single TCP connection, breaking *XFR.
this has been corrected by changing the order of operations in
tcp_dispatch_getnext() so that disp->reading isn't checked until
after the dispatch entry has been reactivated.
the dns_xfrin module was still using the network manager directly to
manage TCP connections and send and receive messages. this commit
changes it to use the dispatch manager instead.
use ISC_REFCOUNT_IMPL for dns_xfrin_ctx_t (which has been renamed
to dns_xfrin_t to keep the function names dns_xfrin_attach() and
dns_xfrin_detach() unchanged).
the 'dispatchmgr' member of the resolver object is used by both
the dns_resolver and dns_request modules, and may in the future
be used by others such as dns_xfrin. it doesn't make sense for it
to live in the resolver object; this commit moves it into dns_view.
This "quiescent state based reclamation" module provides support for
the qp-trie module in dns/qp. It is a replacement for liburcu, written
without reference to the urcu source code, and in fact it works in a
significantly different way.
A few specifics of BIND make this variant of QSBR somewhat simpler:
* We can require that wait-free access to a qp-trie only happens in
an isc_loop callback. The loop provides a natural quiescent state,
after the callbacks are done, when no qp-trie access occurs.
* We can dispense with any API like rcu_synchronize(). In practice,
it takes far too long to wait for a grace period to elapse for each
write to a data structure.
* We use the idea of "phases" (aka epochs or eras) from EBR to
reduce the amount of bookkeeping needed to track memory that is no
longer needed, knowing that the qp-trie does most of that work
already.
I considered hazard pointers for safe memory reclamation. They have
more read-side overhead (updating the hazard pointers) and it wasn't
clear to me how to nicely schedule the cleanup work. Another
alternative, epoch-based reclamation, is designed for fine-grained
lock-free updates, so it needs some rethinking to work well with the
heavily read-biased design of the qp-trie. QSBR has the fastest read
side of the basic SMR algorithms (with no barriers), and fits well
into a libuv loop. More recent hybrid SMR algorithms do not appear to
have enough benefits to justify the extra complexity.
the isc_glob module was originally needed to support posix-style glob
processing on Windows, but is now just an unnecessary wrapper around
glob(3). this commit removes it.
the parser could crash when "include" specified an empty string in place
of the filename. this has been fixed by returning ISC_R_FILENOTFOUND
when the string length is 0.
Previously, the async job queue would use a locked-list (ISC_LIST).
With introduction of atomic stack (that has to be drained at once), we
could use it to remove some contention between the threads and simplify
the async queue.
Fortunately, the reverse order still works for us - instead of append
and tail/prev operation on the list, we are now using prepend and
head/next operation on the atomic stack.
Add a singly-linked stack that supports lock-free prepend and drain (to
empty the list and clean up its elements). Intended for use with QSBR
to collect objects that need safe memory reclamation, or any other user
that works with adding objects to the stack and then draining them in
one go like various work queues.
In <isc/atomic.h>, add an `atomic_ptr()` macro to make type
declarations a little less abominable, and clean up a duplicate
definition of `atomic_compare_exchange_strong_acq_rel()`
[CVE-2022-3924] Add a reproducer for the serve-stale crash when recursive clients soft quota is reached
Closes#3619
See merge request isc-projects/bind9!7575
Reproduce the assertion by configuring a 'named' resolver with
'recursive-clients 10;' configuration option and running 20
queries is parallel.
Also tweak the 'ans2/ans.pl' to simulate a 50ms network latency
when qname starts with "latency". This makes sure that queries
running in parallel don't get served immediately, thus allowing
the configured recursive clients quota limitation to be activated.
removed references in code comments, doc/dev documentation, etc, to
isc_task, isc_timer_reset(), and isc_timertype_inactive. also removed a
coccinelle patch related to isc_timer_reset() that was no longer needed.
instead of using the SDB API as a wrapper to register and
unregister and provide a call framework for the builtin databases,
this commit flattens things so that the builtin databases implement
dns_db directly.
move all dns_sdb code into bin/named/builtin.c, which is the
only place from which it's called.
(note this is temporary: later we'll refactor builtin so that it's a
standalone dns_db implementation on its own instead of using SDB
as a wrapper.)
move database attach/detach functions to db.c, instead of
requiring them to be implemented for every database type.
instead, they must implement a 'destroy' function that is
called when references go to zero.
this enables us to use ISC_REFCOUNT_IMPL for databases,
with detailed tracing enabled by setting DNS_DB_TRACE to 1.
SDB is currently (and foreseeably) only used by the named
builtin databases, so it only needs as much of its API as
those databases use.
- removed three flags defined for the SDB API that were always
set the same by builtin databases.
- there were two different types of lookup functions defined for
SDB, using slightly different function signatures. since backward
compatibility is no longer a concern, we can eliminate the 'lookup'
entry point and rename 'lookup2' to 'lookup'.
- removed the 'allnodes' entry point and all database iterator
implementation code
- removed dns_sdb_putnamedrr() and dns_sdb_putnamedrdata() since
they were never used.
initialize dns_dbmethods, dns_sdbmethods and dns_rdatasetmethods
using explicit struct member names, so we don't have to keep track
of NULLs for unimplemented functions any longer.
some dns_db functions would have crashed if the DB implementation failed
to implement them, requiring the implementations to add functions that
did nothing but return ISC_R_NOTIMPLEMENTED or some obvious default
value. we can just have the dns_db wrapper functions themselves return
those values, and clean up the implementations accordingly.
make the private isc__rdatalist_* functions public dns_rdatalist
functions so that all the rdatalist primitives can be used by
callers to libdns. (this will be needed later for moving SDB and
SDLZ out of libdns.)
This was causing 'CID 436299: Null pointer dereferences (REVERSE_INULL)'
in Coverity. Also removed an 'INSIST(fctx != NULL);' that should
no longer be needed.
When switching to a new view during a reconfiguration (or reverting
to the old view), detach the 'rpzs' and 'catzs' from the previuos view.
The 'catzs' case was earlier solved slightly differently, by detaching
from the new view when reverting to the old view, but we can not solve
this the same way for 'rpzs', because now in BIND 9.19 and BIND 9.18
a dns_rpz_shutdown_rpzs() call was added in view's destroy() function
before detaching the 'rpzs', so we can not leave the 'rpzs' attached to
the previous view and let it be shut down when we intend to continue
using it with the new view.
Instead, "re-fix" the issue for the 'catzs' pointer the same way as
for 'rpzs' for consistency, and also because a similar shutdown call
is likely to be implemented for 'catzs' in the near future.
this function was just a front-end for gethostname(). it was
needed when we supported windows, which has a different function
for looking up the hostname; it's not needed any longer.
as every validator function is loop-synchronized, it should no longer be
necessary to use a validator lock.
calling dns_validator_send(), dns_validator_cancel() or
dns_validator_destroy() from a thread other than the one on which the
validator is running will now cause an assertion failure; this should be
fine since the validator and resolver are tightly coupled, and the fetch
contexts and validators run in the same loops.
refactor validator so that the validation status object (previously
called dns_valstatus_t, which was derived from dns_validatorevent_t), is
now part of the dns_validator object. when calling validator callbacks,
the validator itself is now sent as the argument.
(note: this necessitates caution in the callback functions that are
internal to validator.c validators spawn other validators, and it can be
confusing at times whether we need to be looking at val, val->subvalidator,
or val->parent.)
There was a code flow error that would remove the expired ADB entry from
the LRU list and then a check in the expire_entry() would cause
assertion error because it expect the ADB entry to be linked.
Additionally, the expire mechanism would loop for cases when we would
held only a read rwlock; in such case we need to upgrade the lock and
try again, not just try again.
as there is no further use of isc_task in BIND, this commit removes
it, along with isc_taskmgr, isc_event, and all other related types.
functions that accepted taskmgr as a parameter have been cleaned up.
as a result of this change, some functions can no longer fail, so
they've been changed to type void, and their callers have been
updated accordingly.
the tasks table has been removed from the statistics channel and
the stats version has been updated. dns_dyndbctx has been changed
to reference the loopmgr instead of taskmgr, and DNS_DYNDB_VERSION
has been udpated as well.
change functions using isc_taskmgr_beginexclusive() to use
isc_loopmgr_pause() instead.
also, removed an unnecessary use of exclusive mode in
named_server_tcptimeouts().
most functions that were implemented as task events because they needed
to be running in a task to use exclusive mode have now been changed
into loop callbacks instead. (the exception is catz, which is being
changed in a separate commit because it's a particularly complex change.)
callback events from dns_resolver_createfetch() are now posted
using isc_async_run.
other modules which called the resolver and maintained task/taskmgr
objects for this purpose have been cleaned up.
The callbacks from dns_abd_createfind() are now posted using
isc_async_run() instead of isc_task_send(). ADB event types
have been replaced with a new dns_adbstatus_t type which is
included as find->status.
(The ADB still uses a task for dns_resolver_createfetch().)
dns_request_create() and _createraw() now take a 'loop' parameter
and run the callback event on the specified loop.
as the task manager is no longer used, it has been removed from
the dns_requestmgr structure. the dns_resolver_taskmgr() function
is also no longer used and has been removed.
Some compilers have a built-in definition of the _FORTIFY_SOURCE macro
that differs from BIND's preferred setting. This causes errors like
the one quoted below. The solution is to undefine the macro before
defining it. A similar fix was recently committed to glibc.
<command line>: error: '_FORTIFY_SOURCE' macro redefined
#define _FORTIFY_SOURCE 2
^
<built-in>: note: previous definition is here
#define _FORTIFY_SOURCE 0
^
https://sourceware.org/git/glibc.git/commitdiff/35bcb08eaa953c9b
Free/detach tsigkey and sig0key when exiting and then call
dst_lib_destroy if we have previously called dst_lib_init. This will,
in theory, allow OPENSSL_cleanup to free all memory.
Replace the isc_mutex in the dns_adb unit with isc_rwlock for better
performance. Both ADB names and ADB entries hashtables and LRU are now
using isc_rwlock.
This changes the internal isc_rwlock implementation to:
Irina Calciu, Dave Dice, Yossi Lev, Victor Luchangco, Virendra
J. Marathe, and Nir Shavit. 2013. NUMA-aware reader-writer locks.
SIGPLAN Not. 48, 8 (August 2013), 157–166.
DOI:https://doi.org/10.1145/2517327.24425
(The full article available from:
http://mcg.cs.tau.ac.il/papers/ppopp2013-rwlocks.pdf)
The implementation is based on the The Writer-Preference Lock (C-RW-WP)
variant (see the 3.4 section of the paper for the rationale).
The implemented algorithm has been modified for simplicity and for usage
patterns in rbtdb.c.
The changes compared to the original algorithm:
* We haven't implemented the cohort locks because that would require a
knowledge of NUMA nodes, instead a simple atomic_bool is used as
synchronization point for writer lock.
* The per-thread reader counters are not being used - this would
require the internal thread id (isc_tid_v) to be always initialized,
even in the utilities; the change has a slight performance penalty,
so we might revisit this change in the future. However, this change
also saves a lot of memory, because cache-line aligned counters were
used, so on 32-core machine, the rwlock would be 4096+ bytes big.
* The readers use a writer_barrier that will raise after a while when
readers lock can't be acquired to prevent readers starvation.
* Separate ingress and egress readers counters queues to reduce both
inter and intra-thread contention.
The dns_badcache was pulling the <isc/atomic.h> header only indirectly
via <isc/rwlock.h>, add the direct include as the <isc/rwlock.h> no
longer pulls the header when pthread_rwlock is used.
Include MD5 feature detection in featuretest tool and use it in some
places. When RHEL distribution or Fedora ELN is in FIPS mode, then MD5
algorithm is unavailable completely and even hmac-md5 algorithm usage
will always fail. Work that around by checking MD5 works and if not,
skipping its usage.
Those changes were dragged as downstream patch bind-9.11-fips-tests.patch
in Fedora and RHEL.
I misunderstood the purpose of the `heap_index` rdataset header
member; I thought it identified which heap to use, and could therefore
be smaller, the same size as `locknum` indexes. But in fact it is a
position within a heap, so it needs to be able to count up to the
total number of rdatasets in the rbtdb.
So this changes `heap_index` from `uint16_t` back to `unsigned int`.
To avoid re-embiggening the rdatasetheader, shrink the `count` member
from `uint32` to `uint16`. The `count` is used to rotate RRsets in
`dns_rdataset_towiresorted()`, so 16 bits is more than large enough.
This change also means we no longer need to avoid colliding with
`DNS_RDATASET_COUNT_UNDEFINED` i.e. UINT32_MAX.
Closes#3862
Unfortunately, C still lacks a standard function for pause (x86,
sparc) or yeild (arm) instructions, for use in spin lock or CAS loops.
BIND has its own based on vendor intrinsics or inline asm.
Previously, it was buried in the `isc_rwlock` implementation. This
commit renames `isc_rwlock_pause()` to `isc_pause()` and moves
it into <isc/pause.h>.
This commit also fixes the configure script so that it detects ARM
yield support on systems that identify as `aarch*` instead of `arm*`.
On 64-bit ARM systems we now use the ISB (instruction synchronization
barrier) instruction in preference to yield. The ISB instruction
pauses the CPU for longer, several nanoseconds, which is more like the
x86 pause instruction. There are more details in a Rust pull request,
which also refers to MySQL making the same change:
https://github.com/rust-lang/rust/pull/84725
Tests using diff to compare outputs of dig +short shall ignore lines
starting with ";". In dig +short output, such lines should only be
present for errors such as network issues. Since we utilize dig's
default timeout/retry mechanisms, these transitory issues should be
ignored and only the final output should be considered during the diff
comparison.
A dns_rpz_unref_rpzs() call is missing when taking the 'goto unlock;'
path on shutdown, in order to compensate for the earlier
dns_rpz_ref_rpzs() call.
Move the dns_rpz_ref_rpzs() call after the shutdown check.
This adds an island of trust that is reachable from the root
where the trust anchors are added to island.conf.
This add an island of trust that is not reachable from the root
where the trust anchors are added to private.conf.
When there are multiple managed trust anchors we need to know the
name of the trust anchor that is failing. Extend the error message
to include the trust anchor name.
removed some functions that are no longer used and unlikely to
be resurrected, and also some that were only used to support Windows
and can now be replaced with generic versions.
Occasionally, the allotted 10 seconds for the "running" line to appear
in log after named is started proved insufficient in CI, especially
during increased load. Give named up to 60 seconds to start up to
mitigate this issue.
The script will start the named process configured as both an
authoritative and recursive server for each pairwise ./configure
configuration. The test is considered successful if the named process
runs until the 5-second timeout is triggered, and there is no named.lock
file present, indicating that named did not crash on shutdown.
Add magic value to the fctxcount, to check for completely invalid
counters, or counters that have been already destroyed.
Improve the locking around the counters, and because of that we can drop
the atomics and use simple integers - the counters were already locked
and the tiny bits that used the atomics were not worth the extra effort.
- removed documentation of -S option from named man page
- removed documentation of reserved-sockets from ARM
- simplified documentation of dnssec-secure-to-insecure - it
now just says it's obsolete rather than describing what it
doesn't do anymore
- marked three formerly obsolete options as ancient:
parent-registration-delay, reserved-sockets, and
suppress-initial-notify
Add an extra job for a build from tarball, but without sphinx-build and
enable RUN_MAKE_INSTALL to check that man pages were generated and
installed.
Disable the RUN_MAKE_INSTALL on the systems without sphinx-build (sid).
We don't need them in the repo, it's sufficient if we pregenerate them
while preparing the tarball. That way we don't have overhead while
modifying them but they are still available for installations without
Sphinx.
I assume that this will make rebases and cherry-picks across branches
easier, with less trial and error churn required in the CI.
It's implemented in the way that we build the manpages only when we
either have pregenerated pages available at the configure time or
sphinx-build is installed and working.
isc_bind9 was a global bool used to indicate whether the library
was being used internally by BIND or by an external caller. external
use is no longer supported, but the variable was retained for use
by dyndb, which needed it only when being built without libtool.
building without libtool is *also* no longer supported, so the variable
can go away.
libuv support for receiving multiple UDP messages in a single system
call (recvmmsg()) has been tweaked several times between libuv versions
1.35.0 and 1.40.0. Mixing and matching libuv versions within that span
may lead to assertion failures and is therefore considered harmful, so
try to limit potential damage be preventing users from mixing libuv
versions with distinct sets of recvmmsg()-related flags.
The implementation of UDP recvmmsg in libuv 1.35 and 1.36 is
incomplete and could cause assertion failure under certain
circumstances.
Modify the configure and runtime checks to report a fatal error when
trying to compile or run with the affected versions.
Send the test message from ns3 to ns2 instead of ns2 to ns3 as ns2
is started first and therefore the test doesn't have to wait on the
resend of the the NOTIFY message to be successful.
Move and give unique names to the dns_db_t, dns_dbnode_t and
dns_dbversion_t pointers, so they have global scope and therefore
are visible to cleanup. Unique names are not strictly necessary,
as none of the functions involved call each other.
Change free_db to handle NULL pointers and also an optional
(dns_dbversion_t **).
In match_keyset_dsset and free_keytable, ki to be handled
differently to prevent a false positive NULL pointer dereference
warning from scan.
In formatset moved dns_master_styledestroy earlier and freed
buf before calling check_result to prevent memory leak.
In append_new_ds_set freed ds on the default path before
calling check_result to prevent memory leak.
The RHEL (and clones) 7 will reach EOL in June 2024, shortly after BIND
9.20 will be released. Drop the support for building on those
platforms, so we can use features of modern operating systems - newer
compiler that supports at least subset of C23 and OpenSSL 1.1/3.0.
This will simplify some of the code that we are using in BIND 9.
Instead of using an extra rarely-used paramater to dns_clientinfo_init()
to set ECS information for a client, this commit adds a function
dns_clientinfo_setecs() which can be called only when ECS is needed.
the nsupdate system test was intermittently failing due to the update
quota not being exceeded when it should have been. this is most likely
a timing issue: the client is sending updates too slowly, or the server
is processing them too quickly, for the quota to fill. this commit
attempts to make that the failure less likely by increasing the number
of update transactions from 10 to 20.
Following deleting the root trust anchor and reconfiguring the
server it takes some time to for trust anchor to appear in 'rndc
managed-keys status' output. Retry several times.
A recent refactoring in 7e4e125e5e
had introduced a logical error which could result in calling the
dns_resolver_createfetch() function with 'nameservers' pointer set
to NULL, but with 'domain' not set to NULL, which is not allowed
by the function.
Make sure 'domain' is set only when 'nsrdataset' is valid.
It is trivial to fully cleanup memory on all the error paths in
named-rrchecker, many of which are triggered by bad user input.
This involves freeing lex and mctx if they exist when fatal is
called.
check in the log files of receiving servers that the originating
ports for notify and SOA query messages were set correctly from
configured notify-source and transfer-source options.
the optional 'port' option, when used with notify-source,
transfer-source, etc, is used to set up UDP dispatches with a
particular source port, but when the actual UDP connection was
established the port would be overridden with a random one. this
has been fixed.
(configuring source ports is deprecated in 9.20 and slated for
removal in 9.22, but should still work correctly until then.)
the built-in trust anchors in named and delv are sufficent for
validation. named still needs to be able to load trust anchors from
a bind.keys file for testing purposes, but it doesn't need to be
the default behavior.
we now only load trust anchors from a file if explicitly specified
via the "bindkeys-file" option in named or the "-a" command line
argument to delv. documentation has been cleaned up to remove references
to /etc/bind.keys.
Closes#3850.
it was possible for a managed trust anchor needing to send a key
refresh query to be unable to do so because an authoritative zone
was not yet loaded. this has been corrected by delaying the
synchronization of managed-keys zones until after all zones are
loaded.
* rbt node chains were sized to allow for bitstring labels, so they
had 256 levels; but in the absence of bistrings, 128 is enough.
* dns_byaddr_createptrname() had a redundant options argument,
and a very outdated doc comment.
* A number of comments referred to bitstring labels in a way that is
no longer helpful. (A few informative comments remain.)
ISC_MEM_ZERO requires great care to use when the space returned by
the allocator is larger than the requested space, and when memory is
reallocated. You must ensure that _every_ call to allocate or
reallocate a particular block of memory uses ISC_MEM_ZERO, to ensure
that the extra space is zeroed as expected. (When ISC_MEMFLAG_FILL
is set, the extra space will definitely be non-zero.)
When BIND is built without jemalloc, ISC_MEM_ZERO is implemented in
`jemalloc_shim.h`. This had a bug on systems that have malloc_size()
or malloc_usable_size(): memory was only zeroed up to the requested
size, not the allocated size. When an oversized allocation was
returned, and subsequently reallocated larger, memory between the
original requested size and the original allocated size could
contain unexpected nonzero junk. The realloc call does not know the
original requested size and only zeroes from the original allocated
size onwards.
After this change, `jemalloc_shim.h` always zeroes up to the
allocated size, not the requested size.
GL !7412 updated the set of counters exposed via the XML & JSON
statistics channels. Apply a corresponding version bump, which was
not included in that merge request.
It was agreed that the monthly CI container image rebuild should be done
manually rather than be automated. This allows us to have control over
when things could break and the end of the release cycle is the most
convenient time to have that happen.
Commit b69e783164 changed the scope of the
local 'view' variable in load_configuration(), but the code section
guarded by the #ifdef USE_DNSRPS directive was not adjusted accordingly,
causing build errors for DNSRPS-enabled builds. Fix the latter by
declaring the 'view' variable inside the loop in the DNSRPS-specific
block of code.
Commit 7695c36a5d added a new parameter,
'options', to the prototype of the 'allrdatasets' function pointer in
struct dns_dbmethods. Handle this new parameter accordingly in
rpsdb_allrdatasets().
The --enable-dnsrps-dl switch for ./configure enables preparing a
DNSRPS-enabled build of BIND 9 that is not directly linked against a
DNSRPS provider library (dlopen() at runtime is used instead). Employ
this switch to test DNSRPS-enabled builds in the pairwise testing job in
GitLab CI.
the rate limter now uses loop callbacks rather than task events.
the API for isc_ratelimiter_enqueue() has been changed; we now pass
in a loop, a callback function and a callback argument, and
receive back a rate limiter event object (isc_rlevent_t). it
is no longer necessary for the caller to allocate the event.
the callback argument needs to include a pointer to the rlevent
object so that it can be freed using isc_rlevent_free(), or by
dequeueing.
If the address lookup of the primary server fails just abort
the current update request rather than calling exit. This allows
nsupdate to cleanup gracefully.
The ADB hashmaps are stored in extra memory contexts, so the hash
tables are excluded from the overmem accounting. The new memory
context was unnamed, give it a proper name.
Same thing has happened with extra memory context used for named
global log context - give the extra memory context a proper name.
The ADB hashmaps are stored in extra memory contexts, so the hash
tables are excluded from the overmem accounting. The new memory
context was unnamed, give it a proper name.
Same thing has happened with extra memory context used for named
global log context - give the extra memory context a proper name.
Set the DS state after issuing 'rndc dnssec -checkds'. If the DS
was published, it should go in RUMOURED state, regardless whether it
is already safe to do so according to the state machine.
Leaving it in HIDDEN (or if it was magically already in OMNIPRESENT or
UNRETENTIVE) would allow for easy shoot in the foot situations.
Similar, if the DS was withdrawn, the state should be set to
UNRETENTIVE. Leaving it in OMNIPRESENT (or RUMOURED/HIDDEN)
would also allow for easy shoot in the foot situations.
The following are not also accepted as single-line commits without
generating warnings:
- CHANGES/release note may appear in the beginning of the commit message
- Release note may be capitalized
- Allow commits with "GL #" (e.g. Update documentation for [GL #XXXX])
Update the release checklist to incorporate some minor tweaks that we
have been applying manually for the past few months as a result of
release process evolution.
Rework the Security Incident Handling Checklist so that it does not only
contain the SWENG-side steps for handling a security incident, but also
all the other steps required by ISC procedures.
Artifacts of the "stress" jobs in GitLab CI are used for generating the
QA summary for a given set of releases. It happened in the past that
these artifacts were purged before the QA summary was prepared,
unnecessarily prolonging the release process. A complete set of
artifacts from all such jobs for a single pipeline should be less than 1
GB in size, so extend their lifetime from one day to one week as a more
reasonable compromise between availability and disk space usage. (Note:
these jobs are also run in scheduled pipelines, but that is expected to
be acceptable in the context of artifact lifetime.)
Artifacts of the "release" jobs in GitLab CI are what we eventually
publish on the FTP server. It happened in the past that these artifacts
were purged before we copied them to their destination, unnecessarily
prolonging the release process, even though we usually press the "Keep"
button for the "release" jobs manually to keep them around indefinitely
(as they are the source of what we publish in other places). Since
there is only one "release" job per tag pipeline and its artifacts only
take up tens of megabytes of space, keep the artifacts of the "release"
jobs around indefinitely by default.
commit --fixup=amend:<hash> produces a subject starting with amend!
by default. Have danger look for this to ensure that it is squashed
before merging.
The util/release-tarball-comparison.sh script compares a release-ready
BIND 9 tarball to a temporary BIND 9 tarball created from the same
signed Git tag to ensure that their content does not differ
(significantly).
Add check for extracting the public 'n' component on OpenSSL 3.0
path. This is mandatory component, and it's presence is checked
already on the other code path.
Also document the reason why private key component getting errors
are ignored.
This restores the Malloced memory counter and it's now always equal to
InUse counter. This is only for backwards compatibility reason and
there is no separate counter.
The commit also cleanups little things like structure with a single
item (summary.inuse), and shuts up a wrong cppcheck warning (the
notorious NULL check after assignment).
Instead of enforcing stronger synchronization between threads, make all
the atomic operations relaxed. We are not really interested in exact
numbers at all times - the single place where we need the exact number
is when the memory context is being destroyed. Even when there's a
overmem counter, we don't care about exact ordering or exact number.
The Lost memory counter would count the memory "lost" by external
libraries. There's really no such thing as `named` require the memory
contexts to be clean on destroy.
The stats buckets were again more useful for internal allocator, because
we would see the individual "block" caches where the allocations would
fall into. Remove the stats buckets, and if needed, we can pull more
detailed statistics out of the jemalloc.
The total memory counter had again little or no meaning when we removed
the internal memory allocator. It was just a monotonic counter that
would count add the allocation sizes but never subtracted anything, so
it would be just a "big number".
The maxinuse memory counter indicated the highest amount of
memory allocated in the past. Checking and updating this high-
water mark value every time memory was allocated had an impact
on server performance, so it has been removed. Memory size can
be monitored more efficiently via an external tool logging RSS.
The malloced and maxmalloced memory counters were mostly useless since
we removed the internal allocator blocks - it would only differ from
inuse by the memory context size itself.
Both receive_secure_serial() and setnsec3param() run on the same zone
loop, therefore they are serialized. Remove the mechanism to enqueue
the nsec3param and secure serial updates in case one of them is
running (as they can not) and replace it with sanity check.
Previously, the zone loading and dumping was effectively serialized by
the dns_io_t mechanism. In theory, more IO operations could be run in
parallel, but the zone manager .iolimit was set to 1 and never increased
as dns_zonemgr_setiolimit() was never ever called.
As the dns_master asynchronous load and dump was already offloaded to
non-worker threads with isc_work mechanism, drop the whole dns_io_t
and just rely on the isc_work to do the load and dump scheduling.
The signal handler in the isc_loop would wait for all the work to finish
before interrupting the signing. Add teardown handlers via
isc_loopmgr_teardown() to signal the assignwork() it should stop signing
and bail-out early.
NOTE: The dnssec-signzone binary still can't be interrupted during zone
loading, zone cleaning, nsec(3) chain generation or zone writing. This
might get addressed in the future if it becomes a problem.
Instead of dumping the signed zone contents node by node during the
signing, dump the entire zone at the end. This was already done for the
raw zone format, but it shows that the IO is better utilized when the
zone dump is done in one single write rather than in small chunks.
A side effect of dumping node by node was that all names were printed
relative to the zone origin rather than being grouped under different
$ORIGINs as would normally be the case when dumping a zone. Also, state
was not maintained from one node to the next regarding whether the CLASS
has already been printed, so it was always included with the first
record of each node.
Since dnssec-signzone uses the dns_master_style_explicittl text format
style, and is the only application that does so, we can revise that
style and add a new DNS_STYLEFLAG_CLASS_PERNAME flag to get the output
back to what it was before this change.
Use isc_job_run() instead of isc_task_send() for dnssec-signzone
worker threads.
Also fix the issue where the additional assignwork() would be run only
from the main thread effectively serializing all the signing.
The validity default days value of 1 was used for debugging and
left as such accidentally.
Use 10950 days, as used elsewhere (for example, in doth test CA).
This does not affect anything, the value will be effective when
generating new test certificates in the future.
Change the 'forward' system test to enable DoT on ns2 server,
and test that forwarding from ns4 to the DoT-enabled ns2 works.
In order to test different scenarios, create a test CA (based on
similar CAs for 'doth' and 'nsupdate' system tests), and test
both insecure (no certificate validation) and secure (also with
mutual TLS) TLS configurations, as well as a configuration with an
expired certificate.
Add support for loading and validating the 'tls' parameter from
the forwarders' configuration.
This prepares ground for adding support to forward queries to
DoT-enabled upstream servers.
A 'tls' statement can be specified both for individual addresses
and for the whole list (as a default value when an individual
address doesn't have its own 'tls' set), just as it was done
before for the 'port' value.
Create a new function 'print_rawqstring()' to print a string residing
in a 'isc_textregion_t' type parameter.
Create a new function 'copy_string()' to copy a string from a
'cfg_obj_t' object into a 'isc_textregion_t'.
The old code didn't handle race conditions and errors on systems
with non load balancing sockets gracefully. Look for an error on
any child socket and if found close all the child sockets and return
an error.
The old code didn't handle race conditions and errors on systems
with non load balancing sockets gracefully. Look for an error on
any child socket and if found close all the child sockets and return
an error.
This commit ensures that BIND and supplementary tools still can be
built on newer versions of DragonFly BSD. It used to be the case, but
somewhere between versions 6.2 and 6.4 the OS developers rearranged
headers and moved some function definitions around.
Before that the fact that it worked was more like a coincidence, this
time we, at least, looked at the related man pages included with the
OS.
No in depth testing has been done on this OS as we do not really
support this platform - so it is more like a goodwill act. We can,
however, use this platform for testing purposes, too. Also, we know
that the OS users do use BIND, as it is included in its ports
directory.
Building with './configure' and './configure --without-jemalloc' have
been fixed and are known to work at the time the commit is made.
The write node lock needs to be held when setting node->wild in
add_wildcard_magic except when being called from loading_addrdataset
which is used to load the zone without locking during its initial
load.
Return 'isc_result_t' type value instead of 'bool' to indicate
the actual failure. Rename the function to something not suggesting
a boolean type result. Make changes in the places where the API
function is being used to check for the result code instead of
a boolean value.
It is allowed to point parental-agents to a resolver. Therefore, the
RD bit should be set on requests.
Upon receiving a DS response, ensure that the message has either the
AA or the RA bit set.
Add a test case for a server that uses a resolver as an parental-agent.
We need two root servers, ns1 and ns10, one that delegates to the
'checkds' tld with the DS published (ns2), and one that delegates to
the 'checkds' tld with the DS removed (ns5). Both root zones are
being setup in the 'ns1/setup.sh' script.
We also need two resolvers, ns3 and ns8, that use different root hints
(one uses ns1 address as a hint, the other uses ns10).
Then add the checks to test_checkds.py is similar to the existing tests.
Update 'types' because for zones that have the DS withdrawn (or to be
withdrawn), the CDS and CDNSKEY records should not be published and
thus should not be in the NSEC bitmap.
Detaching the views in the zone_shutdown() could lead to
lock-order-inversion between adb->namelocks[bucket], adb->lock,
view->lock and zone->lock. Detach the views outside of the section that
zone-locked.
BIND 9.16 needs Python and PLY packages for configure to succeed.
Unless we want to tweak the build script to exclude python, we need to
add python3-ply package to the CodeQL configuration.
Cherry-pick small fixup commit from 9.18/9.16 branches needed for
thread-safety. This fixup commit is not needed for 9.19+ because of
reworked application setup, but it decouples isc_iterated_hash and
isc_md units and keeps all the branches in sync.
As this code is on hot path (NSEC3) this introduces an additional
optimization of the EVP_MD API - instead of calling EVP_MD_CTX_new() on
every call to isc_iterated_hash(), we create two thread_local objects
for each thread - a basectx and mdctx, initialize basectx once and then
use EVP_MD_CTX_copy_ex() to flip the initialized state into mdctx. This
saves us couple more valuable microseconds from the isc_iterated_hash()
call.
If the OpenSSL SHA1_{Init,Update,Final} API is still available, use it.
The API has been deprecated in OpenSSL 3.0, but it is significantly
faster than EVP_MD API, so make an exception here and keep using it
until we can't.
Instead of going through another layer, use OpenSSL EVP_MD API directly
in the isc_iterated_hash() implementation. This shaves off couple of
microseconds in the microbenchmark.
The implicit algorithm fetch causes a lock contention and significant
slowdown for small input buffers. For more details, see:
https://github.com/openssl/openssl/issues/19612
Instead of using EVP_DigestInit_ex() initialize empty MD_CTX objects for
each algorithm and use EVP_MD_CTX_copy_ex() to initialize MD_CTX from a
static copy. Additionally avoid implicit algorithm fetching by using
EVP_MD_fetch() for OpenSSL 3.0.
named formerly reserved a set of dispatch objects for use when
sending requests from user-specified source ports. this objects
are no longer used and have been removed.
Add 'port' token to deprecated.conf. Also add options
'use-v4-udp-ports', 'use-v6-udp-ports', 'avoid-v4-udp-ports',
and 'avoid-v6-udp-ports'.
All of these should trigger warnings (except when deprecation warnings
are being ignored).
Deprecate the use of "port" when configuring query-source(-v6),
transfer-source(-v6), notify-source(-v6), parental-source(-v6),
etc. Also deprecate use-{v4,v6}-udp-ports and avoid-{v4,v6}udp-ports.
When we change the view in the view->managed_keys, we never commit the
change, keeping the previous view possibly attached forever.
Call the dns_zone_setviewcommit() immediately after changing the view as
we are detaching the previous view anyway and there's no way to recover
from that.
The .view (and possibly .prev_view) would be kept attached to the
removed zone until the zone is fully removed from the memory in
zone_free(). If this process is delayed because server is busy
something else like doing constant `rndc reconfig`, it could take
seconds to detach the view, possibly keeping multiple dead views in the
memory. This could quickly lead to a massive memory bloat.
Release the views early in the zone_shutdown() call, and don't wait
until the zone is freed.
During XoT it is important to check for "dot" ALPN tag to be
negotiated (according to the RFC 9103). We were doing that, however, the
situation was not handled properly, leading to non-cancelled zone
transfers that would crash (abort()) BIND on shutdown.
In this particular case 'result' might equal 'ISC_R_SUCCESS'. When
this is the case, the part of the code supposed to handle failures
will not cancel the zone transfer.
This situation cannot happen when BIND is a secondary of other BIND
instance. Only primaries following the RFC not closely enough could
trigger such a behaviour.
The condition was accidentally reversed during refactoring in
9730ac4c56 . It would result in skipped
tests on builds with proper support and false negatives on builds
without proper feature support.
Credit for reporting the issue and the fix goes to Stanislav Levin.
The authoritative source for this value is in the project's CI/CD
Variables Setting. The reason to keep it in .gitlab-ci.yaml as well is
to have functional testing in forks without the need to manually specify
this variable in Settings.
The tests have been executed with 4 jobs for some time now. This
"change" only brings .gitlab-ci.yaml file up to date, it doesn't
actually change the number of jobs we currently use to test.
Instead of using the current working directory to find the ifconfig.sh
script, look for the ifconfig.sh.in template in the directory where the
testsock.pl script is located. This enables the testsock.pl script to be
called from any working directory.
Using the ifconfig.sh.in template is sufficient, since it contains
the necessary information to be extracted: the max= value (which is
hard-coded in the template).
Move the core dump detection functionality for system test runs into a
separate script. This enables reuse by the pytest runner. The
functionality remains the same.
Avoid creating any temporary files in the current workdir.
Additional/changing files in the bin/tests/system directory are
problematic for pytest/xdist collection phase, which assumes the list of
files doesn't change between the collection phase of the main pytest
thread and the subsequent collection phase of the xdist worker threads.
Since the testcrypto.sh is also called during pytest initialization
through conf.sh.common (to detect feature support), this could
occasionally cause a race condition when the list of files would be
different for the main pytest thread and the xdist worker.
Although 'dns_fetch_t' fetch can have two associated events, one for
each of 'DNS_EVENT_FETCHDONE' and 'DNS_EVENT_TRYSTALE' types, the
dns_resolver_cancelfetch() function is designed in a way that it
expects only one existing event, which it must cancel, and when it
happens so that 'stale-answer-client-timeout' is enabled and there
are two events, only one of them is canceled, and it results in an
assertion in dns_resolver_destroyfetch(), when it finds a dangling
event.
Change the logic of dns_resolver_cancelfetch() function so that it
cancels both the events (if they exist), and in the right order.
dns_db_findext() asserts if RRSIG is passed to it and
query_lookup_stale() failed to map RRSIG to ANY to prevent this. To
avoid cases like this in the future, move the mapping of SIG and RRSIG
to ANY for qctx->type to qctx_init().
verify that updates are refused when the client is disallowed by
allow-query, and update forwarding is refused when the client is
is disallowed by update-forwarding.
verify that "too many DNS UPDATEs" appears in the log file when too
many simultaneous updates are processing.
check allow-update, update-policy, and allow-update-forwarding before
consuming quota slots, so that unauthorized clients can't fill the
quota.
(this moves the access check before the prerequisite check, which
violates the precise wording of RFC 2136. however, RFC co-author Paul
Vixie has stated that the RFC is mistaken on this point; it should have
said that access checking must happen *no later than* the completion of
prerequisite checks, not that it must happen exactly then.)
limit the number of simultaneous DNS UPDATE events that can be
processed by adding a quota for update and update forwarding.
this quota currently, arbitrarily, defaults to 100.
also add a statistics counter to record when the update quota
has been exceeded.
As shown in the previous commit, using sizeof(type_t) is a little
bit more error-prone when copy-pasting code, so extracting the
size information from the pointer which is being dealt with seems
like a better alternative.
Free 'sizeof(dns_forwarder_t)' bytes of memory instead of
'sizeof(dns_sockaddr_t)' bytes, because `fwd` is a pointer
to a 'dns_forwarder_t' type structure.
Prefer the pthread_barrier implementation on platforms where it is
available over uv_barrier implementation. This also solves the problem
with thread sanitizer builds on macOS that doesn't have pthread barrier.
The target branch is added to the title of the MR, and a "Backport"
label is added. If the target branch starts with "v" a version label
is added too.
While I am here, clean up remnants of the old gitlab API (which has
been replaced by push options for our purposes) and improve the
command-line parsing.
We already have a synchronization mechanism when starting the UDP and
TCP listener children - barriers. Change how we start the first-born
child (tid == 0), so we don't have to race for sock->parent->result and
sock->parent->fd.
Change the per-socket inactive uvreq cache (implemented as isc_astack)
to per-worker memory pool.
Change the per-socket inactive nmhandle cache (implemented as
isc_astack) to unlocked per-socket ISC_LIST.
Always track the per-worker sockets in the .active_sockets field in the
isc__networker_t struct and always track the per-socket handles in the
.active_handles field ian the isc_nmsocket_t struct.
DSCP has not been fully working since the network manager was
introduced in 9.16, and has been completely broken since 9.18.
This seems to have caused very few difficulties for anyone,
so we have now marked it as obsolete and removed the
implementation.
To ensure that old config files don't fail, the code to parse
dscp key-value pairs is still present, but a warning is logged
that the feature is obsolete and should not be used. Nothing is
done with configured values, and there is no longer any
range checking.
- Use separate EVP_PKEY for public and private keys
- On private key load, generate public key allowing better consistency
- Support OpenSSL3 providers
- Clean up key construction abstraction
- Various other clean ups
The ns_client_aclchecksilent is used to check multiple ACLs before
the decision is made that a query is denied. It is also used to
determine if recursion is available. In those cases we should not
set the extended DNS error "Prohibited".
Instead of trying to optimize by using a stack local variable
with additional #ifdef logic, use identical implementations of
the upstream functions to reduce #ifdef clutter.
Move the definitions from dst_openssl.h to openssl_shim.h where
rest of the shim is.
Instead of trying to enforce one pkey to contain both a private
and a public key pair, refactor the code to have separate public
and private pkeys.
This is a prerequisite for proper openssl 3.0 providers support
and greatly simplifies the code.
The dns_zonemgr_releasezone() function makes a decision to destroy
'zmgr' (based on its references count, after decreasing it) inside
a lock, and then destroys the object outside of the lock.
This causes a race with dns_zonemgr_detach(), which could destroy
the object in the meantime.
Change dns_zonemgr_releasezone() to detach from 'zmgr' and destroy
the object (if needed) using dns_zonemgr_detach(), outside of the
lock.
- Make it a separate opensslrsa_check_exponent_bits() function to
clean up the code a bit
- Always use provider API first if using openssl 3.0, and fallback
to EVP API for older openssl or if built with engine support
- Use RSA_get0_key() (with shim for openssl 1.0) to avoid memory
allocations
In the previous refactoring, the findnodeintree() function could return
ISC_R_EXISTS (from dns_db_addnode() call) instead of ISC_R_SUCCESS
leading to node being attached, but never detached.
Change the ISC_R_EXISTS result code returned from dns_rbt_addnode() to
the ISC_R_SUCCESS in the findnodeintree() function (called internally by
dns_db_findnode() and dns_db_findnsec3node()).
With 'stale-answer-enable yes;' and 'stale-answer-client-timeout off;',
consider the following situation:
A CNAME record and its target record are in the cache, then the CNAME
record expires, but the target record is still valid.
When a new query for the CNAME record arrives, and the query fails,
the stale record is used, and then the query "restarts" to follow
the CNAME target. The problem is that the query's multiple stale
options (like DNS_DBFIND_STALEOK) are not reset, so 'query_lookup()'
treats the restarted query as a lookup following a failed lookup,
and returns a SERVFAIL answer when there is no stale data found in the
cache, even if there is valid non-stale data there available.
With this change, query_lookup() now considers non-stale data in the
cache in the first place, and returns it if it is available.
Prime the cache with the following records:
shortttl.cname.example. 1 IN CNAME longttl.target.example.
longttl.target.example. 600 IN A 10.53.0.2
Wait for the CNAME record to expire, disable the authoritative server,
and query 'shortttl.cname.example' again, expecting a stale answer.
On some platforms, when a synchronizing barrier is cleared, one
thread can progress while other threads are still in the process
of releasing the barrier. If a barrier is reused by the progressing
thread during this window, it can cause a deadlock. This can occur if,
for example, we stop listening immediately after we start, because the
stop and listen functions both use socket->barrier. This has been
addressed by using separate barrier objects for stop and listen.
There were couple of redundant macros on both sides of
DNS_RBTDB_STRONG_RWLOCK_CHECK #ifdef block. Use a single set of
macros, but disable the extra REQUIRES if the #define is not set.
Extend the expire_header() to accept the node lock type as one of the
arguments and check whether the the node lock is always write locked +
fix that bug.
While doing that, it was found that expire_header() invocation in
rdataset_expire() passes `false` as a type of tree lock instead of
`isc_rwlocktype_none`.
(Un)fortunately, both values mapped to 0, so no harm was done, but it
has been fixed nevertheless.
There was a repetetive pattern:
if (NODE_TRYUPGRADE(&nodelock->lock, nlocktypep) != ISC_R_SUCCESS)
{
NODE_UNLOCK(&nodelock->lock, nlocktypep);
NODE_WRLOCK(&nodelock->lock, nlocktypep);
}
Instead of doing that over again, introduce new NODE_FORCEUPGRADE()
and TREE_FORCEUPGRADE() that does exactly this code, and simplify
the aforementioned code with just:
NODE_FORCEUPGRADE(&nodelock->lock, nlocktypep);
Previously, dns_dispatch_gettcp() could pick a TCP connection created by
different thread - this breaks our contractual promise to DNS dispatch
by using the TCP connection on a different thread than it was created.
Add .tid member to the dns_dispatch_t struct and skip the dispatches
from other threads when looking up a TCP dispatch that we can reuse in
dns_request.
NOTE: This is going to be properly refactored, but this change could be
also backported to 9.18 for better stability and thread-affinity.
Since dns_master_dump() can no longer return DNS_R_CONTINUE,
the error recovery code in synchronous calls to zone_dump() had
branches that could no longer be reached. This has been cleaned
up by using a boolean variable to determine whether the write
is asynchronous rather than depending on the result code.
This bug was masked in the tests because the `catz` test script did an
`rndc addzone` before an `rndc delzone`. The `addzone` autovivified
the NZF config, so `delzone` worked OK.
This commit swaps the order of two sections of the `catz` test script
so that it uses `delzone` before `addzone`, which provokes a crash
when `delzone` requires a non-NULL NZF config.
To fix the crash, we now try to remove the zone from the NZF config
only if it was dynamically added but not by a catalog zone.
The following report suggests that the 'size' parameter in the two
calls to 'isc_mem_get()' should be set to the
'count * sizeof(*tlsnames)' and 'count * sizeof(*keynames)'
respectively.
/lib/dns/remote.c: 117 in dns_remote_init()
111 }
112 } else {
113 remote->keynames = NULL;
114 }
115
116 if (tlsnames != NULL) {
>>> CID 432259: (SIZEOF_MISMATCH)
>>> Passing argument "count * 8UL /* sizeof (tlsnames) */" to
>>> function "isc__mem_get" and then casting the return value to
>>> "dns_name_t **" is suspicious. In this particular case
>>> "sizeof (dns_name_t **)" happens to be equal to
>>> "sizeof (dns_name_t *)", but this is not a portable assumption.
117 remote->tlsnames = isc_mem_get(mctx, count * sizeof(tlsnames));
118 for (i = 0; i < count; i++) {
119 remote->tlsnames[i] = NULL;
120 }
121 for (i = 0; i < count; i++) {
122 if (tlsnames[i] != NULL) {
/lib/dns/remote.c: 99 in dns_remote_init()
93 memmove(remote->dscps, dscp, count * sizeof(isc_dscp_t));
94 } else {
95 remote->dscps = NULL;
96 }
97
98 if (keynames != NULL) {
>>> CID 432259: (SIZEOF_MISMATCH)
>>> Passing argument "count * 8UL /* sizeof (keynames) */" to
>>> function "isc__mem_get" and then casting the return value to
>>> "dns_name_t **" is suspicious. In this particular case
>>> "sizeof (dns_name_t **)" happens to be equal to "sizeof
>>> (dns_name_t *)", but this is not a portable assumption.
99 remote->keynames = isc_mem_get(mctx, count * sizeof(keynames));
100 for (i = 0; i < count; i++) {
101 remote->keynames[i] = NULL;
102 }
103 for (i = 0; i < count; i++) {
104 if (keynames[i] != NULL) {
When shutting down, the cleanup path should not try to destroy
'newnodes', because it is NULL at that point.
Introduce another label for the "shuttingdown" scenario.
The clean_namehooks() function does't hold the 'adb->entries_lock'
lock, so calling maybe_expire_entry() is not thread-safe.
Instead of adding a lock/unlock, leave the expiration to later,
e.g. by the get_attached_and_locked_entry() function.
Also fix a couple of comment typos.
This commit replaces ad-hoc code for send requests buffer management
within TLS with the one based on isc_buffer_t.
Previous version of the code was trying to use pre-allocated small
buffers to avoid extra allocations. The code would allocate a larger
dynamic buffer when needed. There is no need to have ad-hoc code for
this, as isc_buffer_t now provides this functionality internally.
Additionally to the above, the old version of the code lacked any
logic to reuse the dynamically allocated buffers. Now, as we do not
manage memory buffers, but isc_buffer_t objects, we can implement this
strategy. It can be in particular helpful for longer lasting
connections, as in this case the buffer will adjust itself to the size
of the messages being transferred. That is, it is in particular useful
for XoT, as Stream DNS happen to order send requests in such a way
that the send request will get reused.
Remove parsing the configuration options 'alt-transfer-source',
'alt-transfer-source-v6', and 'use-alt-transfer-source', and remove
the corresponding code that implements the feature.
Use the configured 'source' and 'source-v6' when initiating a zone
transfer, sending a notify, or when checking for the DS. Remove the
special code for using alternate transfer sources.
Update some system tests to use the new configuration and make sure
the tests still work.
Add a new way to configure the preferred source address when talking to
remote servers such as primaries and parental-agents. This will
eventually deprecate options such as 'parental-source',
'parental-source-v6', 'transfer-source', etc.
Example of the new configuration:
parental-agents "parents" port 5353 \
source 10.10.10.10 port 5354 dscp 54 \
source-v6 2001:db8::10 port 5355 dscp 55 {
10.10.10.11;
2001:db8::11;
};
The dns_remote_t structure is intended to replace the variables in
the structure that deals with remote server communication to primaries,
parental agents, forwarders, etc.
Ensure that no .gitignore file in the tree contains patterns matching
files tracked by Git by adding a simple script that detects such
situations and running that script as part of the "misc" job in the
"precheck" stage of every GitLab CI pipeline.
The Makefile used to be autogenerated prior to
67f76b1269 which is the reason why the
.gitignore existed in the first place. Since then, a static Makefile is
used and is supposed to be tracked in the git repo.
The pre-defined test cases use named.$TESTCASE.conf naming convention,
where TESTCASE is a human readable name contaning actual word(s). The
autogenerated test cases' names always start with a number from 1 to 6.
bin/tests/system/rrsetorder/dig.out* files match a gitignore expression
present in bin/tests/system/.gitignore. Since these are meant to be
reference files that are compared to the files generated when the
"rrsetorder" system test is run, rename them to avoid listing tracked
files in .gitignore files.
Normally, when a 'resquery_t' object is created in fctx_query(),
we call dns_adb_beginudpfetch() (which increases the ADB quota)
only if it's a UDP query. Then, in fctx_cancelquery(), we call
dns_adb_endudpfetch() to decreases back the ADB quota, again only
if it's a UDP query.
The problem is that a UDP query can become a TCP query, preventing
the quota from adjusting back in fctx_cancelquery() later.
Call dns_adb_beginudpfetch() also when switching the query type
from UDP to TCP.
Rename isc_tlsctx_cache_new() to isc_tlsctx_cache_create(), tlsctx_client_session_cache_new() to tlsctx_client_session_create()
Closes#3748
See merge request isc-projects/bind9!7269
Additionally to renaming, it changes the function definition so that
it accepts a pointer to pointer instead of returning a pointer to the
new object.
It is mostly done to make it in line with other functions in the
module.
Additionally to renaming, it changes the function definition so that
it accepts a pointer to pointer instead of returning a pointer to the
new object.
It is mostly done to make it in line with other functions in the
module.
The test expects a "connection timed out" message from DiG when it
experiences a timeout, while the current version of DiG prints just
a "timed out" message, like below:
;; communications error to 10.53.0.1#11314: timed out
;; communications error to 10.53.0.1#11314: timed out
;; communications error to 10.53.0.1#11314: timed out
; <<>> DiG 9.19.9-dev <<>> -p 11314 +tries +time +tcp +tries +time @10.53.0.1 dropedns. TXT
; (1 server found)
;; global options: +cmd
;; no servers could be reached
Change the expected string to match the current DiG output.
Use the '-F' switch for "grep" for matching a fixed string.
In order to have a common naming convention for system tests, rename the
only outlier "engine_pkcs11" to "enginepkcs11", which was the only
system test using an underscore in its name.
The only allowed word separators for system test names are either dash
or no separator.
It is better to use consistent file names to avoid issue with sorting
etc.
Using underscore in filenames as opposed to dash was chosen because it
seems more common in pytest/python to use underscore for filenames.
Also rename the bin/tests/system/timeouts/tests-tcp.py file to
bin/tests/system/timeouts/tests_tcp_timeouts.py to avoid pytest name
collision (there can't be two files named tests_tcp.py).
The dns_request code is very sensitive about calling the connected and
deadlocks when the timing is "right" in several places. Move the call
to the connected callback to the (udp|tcp)_connected() functions, so
they are called asynchronously instead of directly from
the (udp|tcp)_dispentry_cancel() functions.
The TCP dispatches are removed from the dispatchmgr->list in the
dispatch_destroy() and there's a brief period of time where
dns_dispatch_gettcp() can find a dispatch in connected state that's
being destroyed.
Set the dispatch state to DNS_DISPATCHSTATE_NONE in the TCP connection
callback if there are no responses waiting, and ignore TCP dispatches
with zero references in dns_dispatch_gettcp().
In tcp_connected() a typo has turned a DbC check into an assignment
breaking the state machine and making the dns_dispatch_gettcp() try to
attach to dispatch in process of destruction.
The TCP dispatches in DNS_DISPATCHSTATE_NONE could be either very
fresh or those could be dispatches that failed connecting to the
destination. Ignore them when trying to connect to an existing
TCP dispatch via dns_dispatch_gettcp().
This introduces a Python dependency for running system tests. It is
needed in order to:
- write new test control scripts in Python
- gradually rewrite old Perl scripts into Python if needed
- eventually introduce pytest as the new test runner framework
This commit is not intended to be backported to 9.16.
Nothing from conf.sh.common is required to set these values. On the
contrary, a Python interpreter needs to be set in order to randomize the
algorithm set (which happens in conf.sh.common).
When testcrypto.sh is used as a standalone script, always use quiet mode
to avoid using undefined commands (such as echo_i) which require
inclusion of the entire conf.sh machinery.
This commit modifies the Stream DNS message so that it uses the
optimised code path (isc__nm_senddns()) for sending DNS messages over
the underlying transport. This way we avoid allocating any
intermediate memory buffers needed to render a DNS message with its
length pre-pended ahead of the contents (TCP DNS message format).
The new internal function works in the same way as isc_nm_send()
except that it sends a DNS message size ahead of the DNS message
data (the format used in DNS over TCP).
The intention is to provide a fast path for sending DNS messages over
streams protocols - that is, without allocating any intermediate
memory buffers.
This commit optimises TLS send request object allocation to enable
send request object reuse, somewhat reducing pressure on the memory
manager. It is especially helpful in the case when Stream DNS uses the
TLS implementation as the transport.
This commit ensures that generic TLS unit tests are available in non
DoH-enabled builds, as isc_nm_tlssocket is not tied exclusively to the
DoH implementation anymore.
This commit unties generic TLS code (isc_nm_tlssocket) from DoH, so
that it will be available regardless of the fact if BIND was built
with DNS over HTTP support or not.
This commit ensures that Stream DNS code attempts to disable Nagle's
algorithm regardless of underlying stream transport (TCP or TLS), as
we are not interested in trading latency for throughout when dealing
with DNS messages.
This commit ensures that Nagle's algorithm is disabled by default for
TLS connections on best effort basis, just like other networking
software (e.g. NGINX) does, as, in the case of TLS, we are not
interested in trading latency for throughput, rather vice versa.
We attempt to disable it as early as we can, right after TCP
connections establishment, as an attempt to speed up handshake
handling.
This commit adds ability to turn the Nagle's algorithm on or off via
connections handle. It adds the isc_nmhandle_set_tcp_nodelay()
function as the public interface for this functionality.
This commit modifies the existing unit tests for TLS DNS and TCP DNS
in such a way that the new Stream DNS transport is used as it is
intended to be a drop-in replacement for these two transports.
This commit adds an initial implementation of isc_nm_streamdnssocket
transport: a unified transport for DNS over stream protocols messages,
which is capable of replacing both TCP DNS and TLS DNS
transports. Currently, the interface it provides is a unified set of
interfaces provided by both of the transports it attempts to replace.
The transport is built around "isc_dnsbuffer_t" and
"isc_dnsstream_assembler_t" objects and attempts to minimise both the
number of memory allocations during network transfers as well as
memory usage.
This commit adds the implementation for an "isc_dnsstream_assembler_t"
object. The object is built on top of "isc_dnsbuffer_t" and is
intended to encapsulate the state machine used for handling DNS
messages received in the format used for messages transmitted over
TCP.
The idea is that the object accepts the input data received from a
socket, tries to assemble DNS messages from the incoming data and
calls the callback which contains the status of the incoming data as
well as a pointer to the memory region referencing the data of the
assembled message. It is capable of assembling DNS messages no matter
how torn apart they are when sent over network.
The following statuses might be passed to the callback:
* ISC_R_SUCCESS - a message has been successfully assembled;
* ISC_R_NOMORE - not enough data has been processed to assemble a
message;
* ISC_R_RANGE - there was an attempt to process a zero-sized DNS
message (someone attempts to send us junk data).
One could say that the object replaces the implementation of
"isc__nm_*_processbuffer()" functions used by the old TCP DNS and TLS
DNS transports with a better defined state machine completely
decoupled from the networking code itself.
Such a design makes it trivial to write unit tests for it, leading to
better verification of its correctness.
Another important difference is directly related to the fact that it
is built on top of "isc_dnsbuffer_t", which tries to manage memory in
a smart way. In particular:
* It tries to use a static buffer for smaller messages, reducing
pressure on the memory manager (hot path);
* When allocating dynamic memory for larger messages, it tries to
allocate memory conservatively (generic path).
These characteristics is a significant upgrade over the older logic
where a 64KB(+2 bytes) buffer was allocated from dynamic memory
regardless of the fact if we need a buffer this large or not. That is,
lesser memory usage is expected in a generic case for DNS transports
built on top of "isc_dnsstream_assembler_t."
This commit adds "isc_dnsbuffer_t" object implementation, a thin
wrapper on top of "isc_buffer_t" which has the following
characteristics:
* provides interface specifically atuned for handling/generating DNS
messages, especially in the format used for DNS messages over TCP;
* avoids allocating dynamic memory when handling small DNS messages,
while transparently switching to using dynamic memory when handling
larger messages. This approach significantly reduces pressure on the
memory allocator, as most of the DNS messages are small.
The added function provides the interface for getting an ALPN tag
negotiated during TLS connection establishment.
The new function can be used by higher level transports.
This commit adds manual read timer control mode, similarly to TCP.
This way the read timer can be controlled manually using:
* isc__nmsocket_timer_start();
* isc__nmsocket_timer_stop();
* isc__nmsocket_timer_restart().
The change is required to make it possible to implement more
sophisticated read timer control policies in DNS transports, built on
top of TLS.
This commit adds a manual read timer control mode to the TCP
code (adding isc__nmhandle_set_manual_timer() as the interface to it).
Manual read timer control mode suppresses read timer restarting the
read timer when receiving any amount of data. This way the read timer
can be controlled manually using:
* isc__nmsocket_timer_start();
* isc__nmsocket_timer_stop();
* isc__nmsocket_timer_restart().
The change is required to make it possible to implement more
sophisticated read timer control policies in DNS transports, built on
top of TCP.
This commit adds implementation of isc__nmsocket_timer_restart() and
isc__nmsocket_timer_stop() for generic TLS code in order to make its
interface more compatible with that of TCP.
This commit adds implementations of isc_nm_bad_request() and
isc__nmsocket_reset() to the generic TLS stream code in order to make
it more compatible with TCP code.
The purpose of this commit is to aid compiler in generating better
code when working with `isc_buffer_t` objects by using restricted
pointers (and, to a lesser extent, 'const' modifier for read-only
arguments).
This way we, basically, instruct the compiler that the members of
structured passed by pointers into the functions can be treated as
local variables in the scope of a function. That should reduce the
number of load/store operations emitted by compilers when accessing
objects (e.g. 'isc_buffer_t') via pointers.
Add two extra functions needed by StreamDNS:
1. isc_buffer_setmctx() sets the buffer internal memory context, so we
can use isc_buffer_reserve() on the buffer. For this, we also need
to track whether the .base was dynamically allocated or not. This
needs to be called after isc_buffer_init() and before first
isc_buffer_reserve() call.
2. isc_buffer_clearmctx() clears the buffer internal memory context, and
frees any dynamically allocated buffer. This needs to be called
after the last isc_buffer_reserve() call and before calling the
isc_buffer_invalidate()
There are couple places where we use putstr(), putmem(), ... helpers
that tries to reserve space and only if successful puts the data onto
the buffer. Cleanup the double reference as it's not needed there.
When the buffer is allocated via isc_buffer_allocate() and the size is
smaller or equal ISC_BUFFER_STATIC_SIZE (currently 512 bytes), the
buffer will be allocated as a flexible array member in the buffer
structure itself instead of allocating it on the heap. This should help
when the buffer is used on the hot-path with small allocations.
When isc_buffer_t buffer is created with isc_buffer_allocate() assume
that we want it to always auto-reallocate instead of having an extra
call to enable auto-reallocation.
The isc_buffer_putdecint() could be easily replaced with
isc_buffer_printf() with just a small overhead of calling vsnprintf()
twice instead once. This is not on a hot-path (dns_catz unit), so we
can ignore the overhead and instead have less single-use code in favor
of using reusable more generic function.
The Stream DNS implementation needs a peek methods that read the value
from the buffer, but it doesn't advance the current position. Add
isc_buffer_peekuintX methods, refactor the isc_buffer_{get,put}uintN
methods to modern integer types, and move the isc_buffer_getuintN to the
header as static inline functions.
Move the U8TO{32,64}_LE and U{32,64}TO8_LE macros to endian.h and extend
the macros for 16-bit and Big-Endian variants.
Use the macros both in isc_siphash (LE) and isc_buffer (BE) units.
The isc_buffer_reserve() would be passed a reference to the buffer
pointer, which was unnecessary as the pointer would never be changed
in the current implementation. Remove the extra dereference.
The dispatches are not thread-bound, and used freely between various
threads (see the dns_resolver and dns_request units for details).
This refactoring make sure that all non-const dns_dispatch_t and
dns_dispentry_t members are accessed under a lock, and both object now
track their internal state (NONE, CONNECTING, CONNECTED, CANCELED)
instead of guessing the state from the state of various struct members.
During the refactoring, the artificial limit DNS_DISPATCH_SOCKSQUOTA on
UDP sockets per dispatch was removed as the limiting needs to happen and
happens on in dns_resolver and limiting the number of UDP sockets
artificially in dispatch could lead to unpredictable behaviour in case
one dispatch has the limit exhausted by others are idle.
The TCP artificial limit of DNS_DISPATCH_MAXREQUESTS makes even less
sense as the TCP connections are only reused in the dns_request API
that's not a heavy user of the outgoing connections.
As a side note, the fact that UDP and TCP dispatch pretends to be same
thing, but in fact the connected UDP is handled from dns_dispentry_t and
dns_dispatch_t acts as a broker, but connected TCP is handled from
dns_dispatch_t and dns_dispatchmgr_t acts as a broker doesn't really
help the clarity of this unit.
This refactoring kept to API almost same - only dns_dispatch_cancel()
and dns_dispatch_done() were merged into dns_dispatch_done() as we need
to cancel active netmgr handles in any case to not leave dangling
connections around. The functions handling UDP and TCP have been mostly
split to their matching counterparts and the dns_dispatch_<function>
functions are now thing wrappers that call <udp|tcp>_dispatch_<function>
based on the socket type.
More debugging-level logging was added to the unit to accomodate for
this fact.
When get_attached_entry() encounters entry that would be expired, it
needs to get reference to the entry before calling maybe_expire_entry(),
so the ADB entry doesn't get destroyed inside the its own lock.
This creeped into the code base again during review, so I am adding
an extra comment to prevent this.
A full backport must have all the commit from the original MR and the
original commit IDs must be referenced in the backport commit messages.
If the criteria above is not met, the MR should be marked as a partial
backport. In that case, any discrepencies are only logged as informative
messages rather than failures.
When checking a backport MR, ensure that the original MR has been merged
already. This is vital for followup checks that verify commit IDs from
original commits are present in backport commit messages.
When doing archeology, it is much easier to find stuff if it's properly
linked. This check ensures that backport MR are linked to their original
MR via a "Backport of !XXXX" message.
The regular expression is fairly broad and has been tested to accept the
following variants of the message:
Backport of MR !XXXX
Backport of: !XXXX
backport of mr !XXXX
Backport of !XXXX
Backport of https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/XXXX
Having the MR title clearly marked in its title can be very useful when
looking through older issues/MRs.
This check also ensures that the version from the version label matches
the proper version branch (i.e. v9.16 must be marked with [v9_16]).
The overmem cleaning in ADB could become overzealous and clean fresh ADB
names and entries. Add a safety check to not clean any ADB names and
entries that are below ADB_CACHE_MINIMUM threshold.
The ADB overmem accounting would include the memory used by hashtables
thus vastly reducing the space that can be used for ADB names and
entries when the hashtables would grow. Create own memory context for
the ADB names and entries hash tables.
There was a datarace that could expire a freshly created ADB names and
entries between the return from get_attached_{name,entry} and locking it
again. Lock the ADB name and ADB entry inside the hash table lock, so
they won't get expired until the full initialization has been complete.
The dns_catz_update_from_db() function prints serial number as a signed
number (with "%d" in the format string), but the `vers` variable's type
is 'uint32_t'. This breaks serials bigger than 2^31.
Use PRIu32 instead of "d" in the format string.
The dns_adb_getcookie() doesn't use the 'adb' parameter, remove it.
Refactor the dns_adb_getcookie() function to just return the size of
the cookie when the caller passes 'NULL' as the 'cookie' argument.
Add internal logging functions isc__netmgr_log, isc__nmsocket_log(), and
isc__nmhandle_log() that can be used to add logging messages to the
netmgr, and change all direct use of isc_log_write() to use those
logging functions to properly prefix them with netmgr, nmsocket and
nmsocket+nmhandle.
When the resolver was refactored, the statistics system test had to be
adjusted in c6b4d82557. Unfortunately,
this change had to be done because of an error in the resolver
refactoring where timeout would not retry next server, but keep trying
the same server. As we have now fixed this bug, revert the change to
the test back to the previous state.
In case, we are trying to hash the empty key into the hashmap, the key
is going to have zero length. This might happen in the unit test.
Allow this and add a unit test to ensure the empty zero-length key
doesn't hash to slot 0 as SipHash 2-4 (our hash function of choice) has
no problem with zero-length inputs.
This commit fixes TLS session resumption via session IDs when
client certificates are used. To do so it makes sure that session ID
contexts are set within server TLS contexts. See OpenSSL documentation
for 'SSL_CTX_set_session_id_context()', the "Warnings" section.
A rdataset could have been left unassociated on the error path in the
resume_dslookup() in the dns_resolver unit. Clone the rdataset after
the error check, so it's not cloned before we check whether we can make
further progress chasing DS records.
Remove the trailing '\0' so that the length field of the dns_name_t
structure is correct. The old data just happens to work with
dns_name_issubdomain but would fail with dns_name_equal.
Instead of relying on hash table search when using the keys, implement a
proper reference counting in dns_keyfileio_t objects, and attach/detach
the objects to the zone.
Due to off-by-one error in zonemgr_keymgmt_delete, unused key file IO
lock objects were never freed and they were kept until the server
shutdown. Adjust the returned value by -1 to accomodate the fact that
the atomic_fetch_*() functions return the value before the operation and
not current value after the operation.
Zero TTL handling does not need to be different for 'rdatasetiter_first'
and 'rdatasetiter_next' and it interacts badly with 'bind_rdatadataset'
which makes different determinations.
'DNS_DB_STALEOK' returns stale rdatasets as well as current rdatasets.
'DNS_DB_EXPIREDOK' returns expired rdatasets as well as current
rdatasets. This option is currently only set when DNS_DB_STALEOK is
also set.
The only function left in the isc_resource API was setting the file
limit. Replace the whole unit with a simple getrlimit to check the
maximum value of RLIMIT_NOFILE and set the maximum back to rlimit_cur.
This is more compatible than trying to set RLIMIT_UNLIMITED on the
RLIMIT_NOFILE as it doesn't work on Linux (see man 5 proc on
/proc/sys/fs/nr_open), neither it does on Darwin kernel (see man 2
getrlimit).
The only place where the maximum value could be raised under privileged
user would be BSDs, but the `named_os_adjustnofile()` were not called
there before. We would apply the increased limits only on Linux and Sun
platforms.
After deprecating the operating system limits settings (coresize,
datasize, files and stacksize), mark them as ancient and remove the code
that sets the values from config.
Send the ns_query_cancel() on the recursing clients when we initiate the
named shutdown for faster shutdown.
When we are shutting down the resolver, we cancel all the outstanding
fetches, and the ISC_R_CANCEL events doesn't propagate to the ns_client
callback.
In the future, the better solution how to fix this would be to look at
the shutdown paths and let them all propagate from bottom (loopmgr) to
top (f.e. ns_client).
When get_attached_entry() encounters entry that would be expired, it
needs to get reference to the entry before calling maybe_expire_entry(),
so the ADB entry doesn't get destroyed inside the its own lock.
Neither of the new CI jobs can reliably pass at the moment; hence they
are defined with "allow_failure: true" until issues in the code base are
resolved.
Using a restored catalog zone excercised a use-after-free bug.
The test checks that the use-after-free bug is gone and is just
a reasonable behaviour check in its own right.
dns_db_updatenotify_unregister needed to be called earlier to ensure
that listener->onupdate_arg always points to a valid object. The
existing lazy cleanup in rbtdb_free did not ensure that.
With 'stale-answer-enable yes;' and 'stale-answer-client-timeout off;',
consider the following situation:
A CNAME record and its target record are in the cache, then the CNAME
record expires, but the target record is still valid.
When a new query for the CNAME record arrives, and the query fails,
the stale record is used, and then the query "restarts" to follow
the CNAME target. The problem is that the query's multiple stale
options (like DNS_DBFIND_STALEOK) are not reset, so 'query_lookup()'
treats the restarted query as a lookup following a failed lookup,
and returns a SERVFAIL answer when there is no stale data found in the
cache, even if there is valid non-stale data there available.
With this change, query_lookup() now considers non-stale data in the
cache in the first place, and returns it if it is available.
Prime the cache with the following records:
shortttl.cname.example. 1 IN CNAME longttl.target.example.
longttl.target.example. 600 IN A 10.53.0.2
Wait for the CNAME record to expire, disable the authoritative server,
and query 'shortttl.cname.example' again, expecting a stale answer.
This commit adds a check if 'sock->recv_cb' might have been nullified
during the call to 'sock->recv_cb'. That could happen, e.g. by an
indirect call to 'isc_nmhandle_close()' from within the callback when
wrapping up.
In this case, let's close the TLS connection.
This commit ensures that the non-atomic flags inside a DoH listener
socket object (and associated worker) are accessed when doing accept
for a connection only from within the context of the dedicated thread,
but not other worker threads.
The purpose of this commit is to avoid TSAN errors during
isc__nmsocket_closing() calls. It is a continuation of
4b5559cd8f.
This commit ensures that the non-atomic flags inside a TLS listener
socket object (and associated worker) are accessed when doing
handshake for a connection only from within the context of the
dedicated thread, but not other worker threads.
The purpose of this commit is to avoid TSAN errors during
isc__nmsocket_closing() calls. It is a continuation of
4b5559cd8f.
While some of these tests are for DoT which doesn't require nghttp2,
the server configs won't allow the server to start without nghttp2
support during compile time.
It might be possible to split these tests into DoT and DoH and only
require nghttp2 for DoH tests, but since almost all of our CI jobs are
compiled with nghttp2, we wouldn't gain a lot of coverage, so it's
probably not worth the effort.
Avoid using the environment variables for feature detection and use the
feature-test utility instead.
Remove the obsolete environment variables from conf.sh, since they're no
longer used anywhere.
Previously, there were two different ways to detect feature support.
Either through an environment variable set by configure in conf.sh, or
using the feature-test utility.
It is more simple and consistent to have only one way of detecting the
feature support. Using the feature-test utility seems superior the the
environment variables set by configure.
This commit ensures that the flags inside a TLS listener socket
object (and associated worker) are accessed when accepting a
connection only from within the context of the dedicated thread, but
not other worker threads.
The TLSDNS transport was not honouring the single read callback for
TLSDNS client. It would call the read callbacks repeatedly in case the
single TLS read would result in multiple DNS messages in the decoded
buffer.
This is second in the series of fixing the usage of hashtables in the
dns_adb and the dns_resolver units.
Currently, the fetch buckets (used to hold the fetch context) and zone
buckets (used to hold per-domain counters) would never get cleaned from
the memory. Combined with the fact that the hashtable now grows as
needed (instead of using hashtable as buckets), the memory usage in the
resolver can just grow and it never drops down.
In this commit, the usage of hashtables (hashmaps) has been completely
rewritten, so there are no "buckets" and all the matching conditions are
directly mapped into the hashtable key:
1. For per-domain counter hashtable, this is simple as the lowercase
domain name is used directly as a counter.
2. For fetch context hashtable, this requires copying some extra flags
back and forth in the key.
As we don't hold the "buckets" forever, the cleaning mechanism has been
rewritten as well:
1. For per-domain counter hashtable, this is again much simpler, as we
only need to check whether the usage counter is still zero under the
lock and bail-out on cleaning if the counter is in use.
2. For fetch context hashtable, this is more complicated as the fetch
context cannot be reused after it has been finished. The algorithm
is different, the fetch context is always removed from the
hashtable, but if we find the fetch context that has been marked
as finished in the lookup function, we help with the cleaning from
the hashtable and try again.
Couple of additional changes have been implemented in this refactoring
as those were needed for correct functionality and could not be split
into individual commits (or would not make sense as seperate commits):
1. The dns_resolver_createfetch() has an option to create "unshared"
fetch. The "unshared" fetch will never get matched, so there's
little point in storing the "unshared" fetch in the hashtable.
Therefore the "unshared" fetches are now detached from the
hashtable and live just on their own.
2. Replace the custom reference counting with ISC_REFCOUNT_DECL/IMPL
macros for better tracing.
3. fctx_done_detach() is idempotent, it makes the "final" detach (the
one matching the create function) only once. But that also means
that it has to be called before the detach that kept the fetch
context alive in the callback. A new macro fctx_done_unref() has
been added to allow this code flow:
fctx_done_unref(fctx, result);
fctx_detach(&fctx);
Doing this the other way around could cause fctx to get destroyed in
the fctx_unref() first and fctx_done_detach() would cause UAF.
4. The resume_qmin() and resume_dslookup() callbacks have been
refactored for more readability and simpler code paths. The
validated() callback has also received some of the simplifications,
but it should be refactored in the future as it is bit of spaghetti
now.
The mechanism for associating a worker task to a database now
uses loops rather than tasks.
For this reason, the parameters to dns_cache_create() have been
updated to take a loop manager rather than a task manager.
This commit ensures that send callbacks are always called from within
the context of its worker thread even in the case of
shuttigdown/inactive socket, just like TCP transport does and with
which TLS attempts to be as compatible as possible.
This commit changes ISC_R_NOTCONNECTED error code to ISC_R_CANCELLED
when attempting to start reading data on the shutting down socket in
order to make its behaviour compatible with that of TCP and not break
the common code in the unit tests.
It turned out that after the latest Network Manager refactoring
'sock->reading' flag was not processed correctly. Due to this
isc_nm_read_stop() might not work as expected because reading from the
underlying TCP socket could have been resume in 'tls_do_bio()'
regardless of the 'sock->reading' value.
This bug did not seem to cause problems with DoH, so it was not
noticed, but Stream DNS has more strict expectations regarding the
underlying transport.
Additionally to the above, the 'sock->recv_read' flag was completely
ignored and corresponding logic was completely unimplemented. That did
not allow to implement one fine detail compared to TCP: once reading
is started, it could be satisfied by one datum reading.
This commit fixes the issues above.
After change 5995, zone transfers were using a small
compression context that only had space for the first
few dozen names in each message. They now use a large
compression context with enough space for every name.
The "final reference detached" message was meant to be DEBUG(1), but was
instead kept at INFO level. Move it to the DEBUG(1) logging level, so
it's not printed under normal operations.
Currently, the ADB uses TTL of 0 for ADB names that the server is
authoritative for and TTL of 10 seconds for HINT and GLUE ADB names.
This requires the unlinked ADB entries to be kept around, because they
would disappear too quickly. This especially affect the root zone as
the trust level is "ultimate" for the root zone nameservers.
This commit restores the ability to keep the unlinked ADB entries in the
database for later reuse, restores printing the unlinked entries and
adds some extra cleaning of the unlinked ADB entries on the tail of the
LRU list (similar to what we are doing for the ADB names).
The dns_adb unit has been refactored to be much simpler. Following
changes have been made:
1. Simplify the ADB to always allow GLUE and hints
There were only two places where dns_adb_createfind() was used - in
the dns_resolver unit where hints and GLUE addresses were ok, and in
the dns_zone where dns_adb_createfind() would be called without
DNS_ADBFIND_HINTOK and DNS_ADBFIND_GLUEOK set.
Simplify the logic by allowing hint and GLUE addresses when looking
up the nameserver addresses to notify. The difference is negligible
and would cause a difference in the notified addresses only when
there's mismatch between the parent and child addresses and we
haven't cached the child addresses yet.
2. Drop the namebuckets and entrybuckets
Formerly, the namebuckets and entrybuckets were used to reduced the
lock contention when accessing the double-linked lists stored in each
bucket. In the previous refactoring, the custom hashtable for the
buckets has been replaced with isc_ht/isc_hashmap, so only a single
item (mostly, see below) would end up in each bucket.
Removing the entrybuckets has been straightforward, the only matching
was done on the isc_sockaddr_t member of the dns_adbentry.
Removing the zonebuckets required GLUEOK and HINTOK bits to be
removed because the find could match entries with-or-without the bits
set, and creating a custom key that stores the
DNS_ADBFIND_STARTATZONE in the first byte of the key, so we can do a
straightforward lookup into the hashtable without traversing a list
that contains items with different flags.
3. Remove unassociated entries from ADB database
Previously, the adbentries could live in the ADB database even after
unlinking them from dns_adbnames. Such entries would show up as
"Unassociated entries" in the ADB dump. The benefit of keeping such
entries is little - the chance that we link such entry to a adbname
is small, and it's simpler to evict unlinked entries from the ADB
cache (and the hashtable) than create second LRU cleaning mechanism.
Unlinked ADB entries are now directly deleted from the hash
table (hashmap) upon destruction.
4. Cleanup expired entries from the hash table
When buckets were still in place, the code would keep the buckets
always allocated and never shrink the hash table (hashmap). With
proper reference counting in place, we can delete the adbnames from
the hash table and the LRU list.
5. Stop purging the names early when we hit the time limit
Because the LRU list is now time ordered, we can stop purging the
names when we find a first entry that doesn't fullfil our time-based
eviction criteria because no further entry on the LRU list will meet
the criteria.
Future work:
1. Lock contention
In this commit, the focus was on correctness of the data structure,
but in the future, the lock contention in the ADB database needs to
be addressed. Currently, we use simple mutex to lock the hash
tables, because we almost always need to use a write lock for
properly purging the hashtables. The ADB database needs to be
sharded (similar to the effect that buckets had in the past). Each
shard would contain own hashmap and own LRU list.
2. Time-based purging
The ADB names and entries stay intact when there are no lookups.
When we add separate shards, a timer needs to be added for time-based
cleaning in case there's no traffic hashing to the inactive shard.
3. Revisit the 30 minutes limit
The ADB cache is capped at 30 minutes. This needs to be revisited,
and at least the limit should be configurable (in both directions).
The dns_adb would serialize all fetches on a single task. Create a
per-thread task, so the fetches will stay local to the thread that
initiated the fetch.
Before the refactoring, there was only few buckets with many names in
them, so cleaning up stale ADB names per-bucket made sense. After the
refactoring, each bucket directly maps to ADB name, so purging has been
effectively disabled.
Create a global LRU list for ADB names (and ADB entries) and purge the
stale ADB names globally.
Previously, the name and entry buckets were much larger, so the dead
names and entries were moved to a secondary list to be cleaned
later (f.e. after the already running fetch has been canceled). After
the last refactoring, the bucket now contains only the name (entry)
itself and thus the extra list has a little use. Remove the .deadnames
and .deadentries from dns_adbnamebucket_t and dns_adbentrybucket_t
structures.
The dns_rpz_zones structure was using .refs and .irefs for strong and
weak reference counting. Rewrite the unit to use just a single
reference counting + shutdown sequence (dns_rpz_destroy_rpzs) that must
be called by the creator of the dns_rpz_zones_t object. Remove the
reference counting from the dns_rpz_zone structure as it is not needed
because the zone objects are fully embedded into the dns_rpz_zones
structure and dns_rpz_zones_t object must never be destroyed before all
dns_rpz_zone_t objects.
The dns_rps_zones_t reference counting uses the new ISC_REFCOUNT_TRACE
capability - enable by defining DNS_RPZ_TRACE in the dns/rpz.h header.
Additionally, add magic numbers to the dns_rpz_zone and dns_rpz_zones
structures.
The new ISC_REFCOUNT_TRACE_{IMPL,DECL} macros can be used to add a
reference tracing capability to any unit using the reference counting.
It requires a little bit of extra work in each header as you can't have
a define from inside a define (see rpz.h), but it's fairly easy to add
tracing to any struct using reference counting with these macros.
The dns_cache API contained a cache cleaning mechanism that would be
disabled for 'rbt' based cache. As named doesn't have any other cache
implementations, remove the cache cleaning mechanism from dns_cache API.
The RBTDB has own cache cleaning mechanism and therefor the iterator
.cleaning member would never be set to true. Remove the code that
checks for iterator->cleaning from the RBTDB.
This commit make TCP code use uv_try_write() on best effort basis,
just like TCP DNS and TLS DNS code does.
This optimisation was added in
'caa5b6548a11da6ca772d6f7e10db3a164a18f8d' but, similar change was
mistakenly omitted for generic TCP code. This commit fixes that.
The `gen` program was causing a lengthy single-threaded pause in
the BIND build. When generating RDATATYPE_FROMTEXT_SW(), `gen` hit
the inner loop of `find_typename()` over 1.2 billion times. This
change avoids long deeply-nested loops, so `gen` now runs in less
than 10ms, about 300x faster.
No changes to the output.
Don't restart reading in the send callback after the httpdmgr has been
shut down, and call httpd_request(..., ISC_R_SHUTDOWN, ...) when
shutting down the httpdmgr to reduce code duplication.
Previously, the send callback would be synchronous only on success. Add
an option (similar to what other callbacks have) to decide whether we
need the asynchronous send callback on a higher level.
On a general level, we need the asynchronous callbacks to happen only
when we are invoking the callback from the public API. If the path to
the callback went through the libuv callback or netmgr callback, we are
already on asynchronous path, and there's no need to make the call to
the callback asynchronous again.
For the send callback, this means we need the asynchronous path for
failure paths inside the isc_nm_send() (which calls isc__nm_udp_send(),
isc__nm_tcp_send(), etc...) - all other invocations of the send callback
could be synchronous, because those are called from the respective libuv
send callbacks.
Previously, the read callback would be synchronous only on success or
timeout. Add an option (similar to what other callbacks have) to decide
whether we need the asynchronous read callback on a higher level.
On a general level, we need the asynchronous callbacks to happen only
when we are invoking the callback from the public API. If the path to
the callback went through the libuv callback or netmgr callback, we are
already on asynchronous path, and there's no need to make the call to
the callback asynchronous again.
For the read callback, this means we need the asynchronous path for
failure paths inside the isc_nm_read() (which calls isc__nm_udp_read(),
isc__nm_tcp_read(), etc...) - all other invocations of the read callback
could be synchronous, because those are called from the respective libuv
or netmgr read callbacks.
The various factors like NS_PER_MS are now defined in a single place
and the names are no longer inconsistent. I chose the _PER_SEC names
rather than _PER_S because it is slightly more clear in isolation;
but the smaller units are always NS, US, and MS.
The system test should never attempt to start or stop any other server
than those that belong to that system test. Therefore, it is not
necessary to specify the system test name in function calls.
Additionally, this makes it possible to run the test inside a
differently named directory, as its name is automatically detected with
the $SYSTESTDIR variable. This enables running the system tests inside a
temporary directory.
Direct use of stop.pl was replaced with a more systematic approach to
use stop_servers helper function.
Extract the tlss values if present from the ipkeylist entry and add
the resulting tls setting to the constructed configuration for the
primary.
When comparing catalog zone entries for reuse also check the
masters.tlss values for equality.
the 'nupdates' field was originally used to track whether a client
was ready to shut down, along with other similar counters nreads,
nrecvs, naccepts and nsends. this is now tracked differently, but
nupdates was overlooked when the other counters were removed.
dlz_ldap_dynamic.c: In function ‘dlz_create’:
dlz_ldap_dynamic.c:971:20: warning: this statement may fall through [-Wimplicit-fallthrough=]
971 | if (result != ISC_R_SUCCESS) {
| ^
dlz_ldap_dynamic.c:974:9: note: here
974 | case 11:
| ^~~~
dlz_ldap_dynamic.c:976:20: warning: this statement may fall through [-Wimplicit-fallthrough=]
976 | if (result != ISC_R_SUCCESS) {
| ^
dlz_ldap_dynamic.c:979:9: note: here
979 | case 10:
| ^~~~
dlz_ldap_dynamic.c:980:20: warning: this statement may fall through [-Wimplicit-fallthrough=]
980 | if (strlen(argv[9]) > 0) {
| ^
dlz_ldap_dynamic.c:987:9: note: here
987 | case 9:
| ^~~~
dlz_wildcard_dynamic.c: In function ‘dlz_lookup’:
dlz_wildcard_dynamic.c:227:14: warning: variable ‘origin’ set but not used [-Wunused-but-set-variable]
227 | bool origin = true;
| ^~~~~~
dlz_wildcard_dynamic.c: In function ‘dlz_lookup’:
dlz_wildcard_dynamic.c:252:28: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
252 | cd->record = "@";
| ^
dlz_wildcard_dynamic.c: In function ‘dlz_authority’:
dlz_wildcard_dynamic.c:328:22: warning: unused variable ‘origin’ [-Wunused-variable]
328 | bool origin;
| ^~~~~~
dlz_wildcard_dynamic.c:312:25: warning: unused variable ‘name’ [-Wunused-variable]
312 | const char *p, *name = "@";
| ^~~~
dlz_wildcard_dynamic.c: In function ‘dlz_create’:
dlz_wildcard_dynamic.c:441:23: warning: comparison of integer expressions of different signedness: ‘int’ and ‘unsigned int’ [-Wsign-compare]
441 | for (i = 4; i < argc; i += 4) {
| ^
dlz_sqlite3_dynamic.c: In function ‘dlz_sqlite3_fetch_row’:
dlz_sqlite3_dynamic.c:447:31: warning: comparison of integer expressions of different signedness: ‘int’ and ‘unsigned int’ [-Wsign-compare]
447 | if (rs->pnRow > 0U && rs->curRow < rs->pnRow) {
| ^
dlz_sqlite3_dynamic.c:447:50: warning: comparison of integer expressions of different signedness: ‘unsigned int’ and ‘int’ [-Wsign-compare]
447 | if (rs->pnRow > 0U && rs->curRow < rs->pnRow) {
| ^
dlz_perl_driver.c: In function ‘dlz_version’:
dlz_perl_driver.c:116:27: warning: unused parameter ‘flags’ [-Wunused-parameter]
116 | dlz_version(unsigned int *flags) {
| ~~~~~~~~~~~~~~^~~~~
In file included from /usr/lib64/perl5/CORE/perl.h:5685,
from dlz_perl_driver.c:33:
dlz_perl_driver.c: In function ‘dlz_allnodes’:
/usr/lib64/perl5/CORE/pp.h:162:26: warning: value computed is not used [-Wunused-value]
162 | #define POPs (*sp--)
| ~^~~~~~
dlz_perl_driver.c:151:17: note: in expansion of macro ‘POPs’
151 | POPs;
| ^~~~
dlz_perl_driver.c: In function ‘dlz_allowzonexfr’:
/usr/lib64/perl5/CORE/pp.h:162:26: warning: value computed is not used [-Wunused-value]
162 | #define POPs (*sp--)
| ~^~~~~~
dlz_perl_driver.c:251:17: note: in expansion of macro ‘POPs’
251 | POPs;
| ^~~~
dlz_perl_driver.c: In function ‘dlz_findzonedb’:
/usr/lib64/perl5/CORE/pp.h:162:26: warning: value computed is not used [-Wunused-value]
162 | #define POPs (*sp--)
| ~^~~~~~
dlz_perl_driver.c:328:17: note: in expansion of macro ‘POPs’
328 | POPs;
| ^~~~
dlz_perl_driver.c: In function ‘dlz_lookup’:
/usr/lib64/perl5/CORE/pp.h:162:26: warning: value computed is not used [-Wunused-value]
162 | #define POPs (*sp--)
| ~^~~~~~
dlz_perl_driver.c:407:17: note: in expansion of macro ‘POPs’
407 | POPs;
| ^~~~
dlz_perl_driver.c:472:1: error: no previous prototype for ‘missing_perl_method’ [-Werror=missing-prototypes]
472 | missing_perl_method(const char *perl_class_name, PerlInterpreter *my_perl)
| ^~~~~~~~~~~~~~~~~~~
dlz_perl_driver.c: In function ‘missing_perl_method’:
dlz_perl_driver.c:485:9: error: ISO C90 forbids array ‘full_name’ whose size cannot be evaluated [-Werror=vla]
485 | char full_name[BUF_LEN];
| ^~~~
dlz_perl_driver.c: In function ‘dlz_create’:
dlz_perl_driver.c:613:13: warning: suggest parentheses around assignment used as truth value [-Wparentheses]
613 | if (missing_method_name = missing_perl_method(perl_class_name, my_perl))
| ^~~~~~~~~~~~~~~~~~~
/usr/lib64/perl5/CORE/pp.h:162:26: warning: value computed is not used [-Wunused-value]
162 | #define POPs (*sp--)
| ~^~~~~~
dlz_perl_driver.c:657:17: note: in expansion of macro ‘POPs’
657 | POPs;
| ^~~~
dlz_perl_driver.c:502:15: warning: unused variable ‘init_args’ [-Wunused-variable]
502 | char *init_args[] = { NULL, NULL };
| ^~~~~~~~~
gcovr fails to process fuzz/old.gcda and fuzz/old.gcno files after !7045
MR with:
(WARNING) GCOV produced the following errors processing /builds/isc-projects/bind9/fuzz/old.gcda:
Cannot open source file ../../fuzz/old.c
Cannot open source file ../../lib/dns/include/dns/compress.h
Cannot open source file ../../lib/isc/include/isc/buffer.h
...
(gcovr could not infer a working directory that resolved it.)
Given that code coverage inspection is meant only for BIND 9 code and
not its tests and auxiliary tools, the "fuzz" directory should be
excluded from being included in the code coverage report.
Remove the text from the ARM and DNSSEC guide that describes how to do
key and denial of existence operations using dynamic update. Add a new
section about DNSSEC multi-signer models, but no longer suggest using
dynamic update and auto-dnssec allow.
Remove code that triggers key and denial of existence management
operations. Dynamic update should no longer be used to do DNSSEC
maintenance (other than that of course signatures need to be
created for the new zone contents).
Remove test cases that rely upon key and denial of existence
management operations triggered by dynamic updates.
The autosign system test needed a bit more care than just removing
because the test cases are dependent on each other, so there are some
additional tweaks such as setting the NSEC3PARAM via rndc signing,
and renaming zone input files. In the process, some additional
debug output files have been added, and a 'ret' fail case overwrite
was fixed.
Since this is very sensitive code which has often had security
problems in many DNS implementations, it needs a decent amount of
validation. This fuzzer ensures that the new code has the same output
as the old code, and that it doesn't take longer than a second.
The benchmark uses the fuzzer's copy of the old dns_name_fromwire()
code to compare a number of scenarios: many compression pointers, many
labels, long labels, random data, with/without downcasing.
The aim is to do less work per byte:
* Check the bounds for each label, instead of checking the
bounds for each character.
* Instead of copying one character at a time from the wire to
the name, copy entire runs of sequential labels using memmove()
to make the most of its fast loop.
* To remember where the name ends, we only need to set the end
marker when we see a compression pointer or when we reach the
root label. There is no need to check if we jumped back and
conditionally update the counter for every character.
* To parse a compression pointer, we no longer take a diversion
around the outer loop in between reading the upper byte of the
pointer and the lower byte.
* The parser state machine is now implicit in the instruction
pointer, instead of being an explicit variable. Similarly,
when we reach the root label we break directly out of the loop
instead of setting a second state machine variable.
* DNS_NAME_DOWNCASE is never used with dns_name_fromwire() so
that option is no longer supported.
I have removed this comment which dated from January 1999 when
dns_name_fromwire() was first introduced:
/*
* Note: The following code is not optimized for speed, but
* rather for correctness. Speed will be addressed in the future.
*/
No functional change, apart from removing support for the unused
DNS_NAME_DOWNCASE option. The new code is about 2x faster than the
old code: best case 11x faster, worst case 1.4x faster.
C does not make any guarantees about the value of padding in a
structure, so bytewise comparison of two semantically equal structures
with padding can be spuriously non-equal due to non-equal padding
bytes.
Compare each member of name.attributes individually to avoid this
problem.
Create a zone that triggers DNAME owner name checks in a zone that
is only reachable using a dual stack server. The answer contains
a name that is higher in the tree than the query name.
e.g.
foo.v4only.net. CNAME v4only.net.
v4only.net. A 10.0.0.1
ns4 is serving the test zone (ipv4-only)
ns6 is the root server for this test (dual stacked)
ns7 is acting as the dual stack server (dual stacked)
ns9 is the server under test (ipv6-only)
When using dual-stack-servers the covering namespace to check whether
answers are in scope or not should be fctx->domain. To do this we need
to be able to distingish forwarding due to forwarders clauses and
dual-stack-servers. A new flag FCTX_ADDRINFO_DUALSTACK has been added
to signal this.
checkbashisms warns about possible reliance on HOSTNAME environmental
variable which Bash sets to the name of the current host, and some
commands may leverage it:
possible bashism in builtin/tests.sh line 199 ($HOST(TYPE|NAME)):
grep "^\"$HOSTNAME\"$" dig.out.ns1.$n > /dev/null || ret=1
possible bashism in builtin/tests.sh line 221 ($HOST(TYPE|NAME)):
grep "^\"$HOSTNAME\"$" dig.out.ns2.$n > /dev/null || ret=1
possible bashism in builtin/tests.sh line 228 ($HOST(TYPE|NAME)):
grep "^; NSID: .* (\"$HOSTNAME\")$" dig.out.ns2.$n > /dev/null || ret=1
We don't use the variable this way but rename it to HOST_NAME to silence
the tool.
"next_key_event_threshold" is assigned with
"next_key_event_threshold+i", but "i" is empty (never set, nor used
afterwards).
posh, the Policy-compliant Ordinary SHell, failed on this assignment
with:
tests.sh:253: : unexpected `end of expression'
checkbashisms gets confused by the rndc command being on two lines:
possible bashism in bin/tests/system/nzd2nzf/tests.sh line 37 (type):
rndccmd 10.53.0.1 addzone "added.example { type primary; file \"added.db\";
checkbashisms reports Bash-style ("==") string comparisons inside test/[
command:
possible bashism in bin/tests/system/checkconf/tests.sh line 105 (should be 'b = a'):
if [ $? == 0 ]; then echo_i "failed"; ret=1; fi
possible bashism in bin/tests/system/keyfromlabel/tests.sh line 62 (should be 'b = a'):
test $ret == 0 || continue
possible bashism in bin/tests/system/keyfromlabel/tests.sh line 79 (should be 'b = a'):
test $ret == 0 || continue
The checkbashisms script reports errors like this one:
script util/check-line-length.sh does not appear to have a #! interpreter line;
you may get strange results
It was possible to set operating system limits (RLIMIT_DATA,
RLIMIT_STACK, RLIMIT_CORE and RLIMIT_NOFILE) from named.conf. It's
better to leave these untouched as setting these is responsibility of
the operating system and/or supervisor.
Deprecate the configuration options and remove them in future BIND 9
release.
The small/large tuning has been completely removed from the code with
last remnant of the dead code in ns_interfacemgr. Remove the dead code
and the configure option.
There were a number of places where the zone table should have been
locked, but wasn't, when dns_zt_apply was called.
Added a isc_rwlocktype_t type parameter to dns_zt_apply and adjusted
all calls to using it. Removed locks in callers.
The retry 3 times when checking signatures did not make sense because
at this point the input file does not change.
Raise the number of retries when checking the apex DNSKEY response to
reduce the number of intermittent failures due to unexpected delays.
Despite the RFC says that the NSEC3PARAM is not something that is
intended for the resolver to be cached, and thus the TTL of 0 is most
logical, a zero TTL RRset can be abused by bad actors.
Change the default to SOA MINIMUM.
The call to dns_view_flushcache() is done under exclusive mode, but we
still need to check if view->adb is still attached before calling
dns_adb_flush() because the shutdown might have been already
initialized. This most likely only a theoretical problem on shutdown
because there's either no way how to initiate cache flush when shutting
down or very slim window where the `rndc flush` would have to hit the
slim time during named shutdown.
When starting priming from dns_view_find(), the dns_view shutdown could
be initiated by different thread, detaching from the resolver. Use
dns_view_getresolver() to attach to the resolver under view->lock, so we
don't try to call dns_resolver_prime() with NULL pointer.
There are more accesses to view->resolver, (and also view->adb and
view->requestmgr that suffer from the same problem) in the dns_view
module, but they are all done in exclusive mode or under a view->lock.
Firefox 90+ apparently sends more than 10 headers, so we need to bump
the number to some higher number. Bump it to 100 just to be on a save
side, this is for internal use only anyway.
Replace the use of isc_ht API with isc_hashmap API in the dns_resolver
implementation. This requires extending the fctxbucket_t structure to
include keysize and copy of the key because the isc_hashmap API needs
the raw key in case of resizing the hashmap table.
Replace the use of isc_ht API with isc_hashmap API in the dns_adb
database implementation. This requires extending the
dns_adbnamebucket_t and dns_adbentrybucket_t structures to include
keysize and copy of the key because the isc_hashmap API needs the raw
key in case of resizing the hashmap table.
Add new isc_hashmap API that differs from the current isc_ht API in
several aspects:
1. It implements Robin Hood Hashing which is open-addressing hash table
algorithm (e.g. no linked-lists)
2. No memory allocations - the array to store the nodes is made of
isc_hashmap_node_t structures instead of just pointers, so there's
only allocation on resize.
3. The key is not copied into the hashmap node and must be also stored
externally, either as part of the stored value or in any other
location that's valid as long the value is stored in the hashmap.
This makes the isc_hashmap_t a little less universal because of the key
storage requirements, but the inserts and deletes are faster because
they don't require memory allocation on isc_hashmap_add() and memory
deallocation on isc_hashmap_delete().
If 'set -x' is in effect file.prev gets populated with debugging output.
To prevent this open descriptor 3 and redirect stderr from the awk
command to descriptor 3. Debugging output will stay directed to stderr.
Using the -x option for cherry pick makes it easy to link commits across
branches and it is recommended to use for all backport commits (with
exceptions -- thus a warning level rather than failure).
To avoid accidentally merging unfinished work, detect prohibited
keywords at the start of the subject line. If the first word is any of
the following, fail the check:
WIP, wip, DROP, drop, TODO, todo
The only slightly controversial is the lowercase "drop" which might have
a legitimate use - seems like four commits in the history used it as a
start of a sentence. However, since people commonly use "drop" to
indicate a commit should be dropped before merging, let's prohibit it as
well. In case of false-positive, "Drop" with a capitalized first letter
can always be used.
Since the LGTM label was deprecated in favor of using the Approve button
in gitlab, adjust the detection in danger bot.
Unfortunately, danger-python seems no longer maintained since 2020 and
MR approvals aren't available in its Python API (even though they're
supported in its Ruby/JS APIs). Going forward, let's use the more
comprehensive python-gitlab API.
It still makes sense to utilize the danger-python, since it handles the
integration with gitlab which doesn't need to be reimplemented as long
as it works - same with the other checks.
The code change that entry 6012 describes (introduced in commit
be204bf4c7) was reverted shortly after (in
commit c429b52533). Remove that entry
from CHANGES as it is misleading.
From now on all per-version notes link to the global list
of Known Issues. If there is a new note it should be listed twice:
In the per-version list, and in the global list.
Previously, the tree read lock could be upgraded to a write lock in
decrement_reference() and then downgraded back to read lock in
dereference_iter_node(). When the use of isc_rwlock_downgrade() was
removed, the downgrade was changed to a simple unlock+lock. This allows
some delete operations to sneak in and delete nodes that the iterator
expects to be in place.
Expand decrement_reference() so the caller can indicate whether the
tree read lock should be upgraded, and disallow the upgrade when
calling from dereference_iter_node(), so there will be no need to
release the lock afterward.
The zone_refreshkeys() could run before the zone_shutdown(), but after
the last .erefs has been "detached" causing assertion failure when doing
dns_zone_attach(). Remove the use of .erefs (dns_zone_attach/detach)
and replace it with using the .irefs and additional checks whether the
zone is exiting in the callbacks.
If after a reconfig a zone is not reusable because inline-signing
was turned on/off, trigger a full resign. This is necessary because
otherwise the zone maintenance may decide to only apply the changes
in the journal, leaving the zone in an inconsistent DNSSEC state.
There was an exception for dnssec-policy that allowed DNSSEC in the
unsigned version of the zone. This however causes a crash if the
zone switches from dynamic to inline-signing in the case of NSEC3,
because we are now trying to add an NSEC3 record to a non-NSEC3 node.
This is because BIND expects none of the records in the unsigned
version of the zone to be NSEC3.
Remove the exception for dnssec-policy when copying non DNSSEC
records, but do allow for DNSKEY as this may be a published DNSKEY
from a different provider.
The changes in the code have the side effect that the CDNSKEY and CDS
records in the secure version of the zone are not reusable and thus
are thrashed from the zone. Remove the apex checks for this use case.
We only care about that the zone is not immediately goes bogus, but
a user really should use the built-in "insecure" policy when unsigning
a zone.
Add one more case that tests reconfiguring a zone to turn off
inline-signing. It should still be a valid DNSSEC zone and the NSEC3
parameters should not change.
Add another test to ensure that you cannot update the zone with a
NSEC3 record.
We no longer accept copying DNSSEC records from the raw zone to
the secure zone, so update the kasp system test that relies on this
accordingly.
Also add more debugging and store the dnssec-verify results in a file.
Add a kasp system test that reconfigures a dnssec-policy zone from
maintaining DNSSEC records directly to the zone to using inline-signing.
Add a similar test case to the nsec3 system test, testing the same
thing but now with NSEC3 in use.
The dead nodes might get reactivated during the db iterator walks the
version of the tree, so we can't cleanup the dead nodes while the db
version is open. Restore the previous behaviour that cleaned up the
dead nodes when we are closing the version.
While using mutrace, the phtread-rwlock based isc_rwlock implementation
would be all tracked in the rwlock.c unit losing all useful information
as all rwlocks would be traced in a single place. Rewrite the
pthread_rwlock based implementation to be header-only macros, so we can
use mutrace to properly track the rwlock contention without heavily
patching mutrace to understand the libisc synchronization primitives.
Instead of checking the PTHREAD_RUNTIME_CHECK from the header, move it
to the pthread_rwlock implementation functions. The internal isc_rwlock
actually cannot fail, so the checks in the header was useless anyway.
The dns_rbtdb unit already tracks the state of the node and tree rwlocks
during the top level function and passes the states of the locks to the
called functions.
Add the tree locking family of macros modeled after node locking macros,
and expand both to track the state of the lock in an external variable.
Additionally, in developer mode, add precondition to the macros, so the
lock is in required state - this should cause an assertion failure on
double locking instead of the thread getting stuck.
The only place where isc_rwlock_downgrade was being used was the
decrement_reference() where the code tries either relocks the node
rwlock to write and then tries to upgrade the tree lock. When returning
from the function it tries to restore the locks into a previous state
which is nice, but kind of moot, because at every use of
decrement_reference() the node locks is immediately or almost
immeditately unlocked, and same holds for the tree lock.
Instead of trying to restore the node and tree lock into the initial
state, the decrement_reference now returns the state of the locks, so
the caller can then use the right unlock operation (read or write).
Only when the tree lock was originally unlocked, the decrement_reference
unlocks the tree lock before returning to the caller.
On Linux, the libcap is now mandatory. It makes things simpler for us.
System without {set,get}res{uid,gid} now have compatibility shim using
setreuid/setregid or seteuid/setegid to setup effective UID/GID, so the
same code can be called all the time (including on Linux).
TLS DNS unit tests were sharing the port with TCP DNS tests by
mistake. That could have caused conflicts between the two, when
running the unit tests in parallel. This commit fixes that.
After the loop manager refactoring TCP DNS and TLS DNS unit tests
ended up broken.
The problem is that in these unit tests the code is written in such a
way that for establishing a new connection tcpdns_connect() and
tlsdns_connect() functions are used. However, in these tests as a
connection callback function connect_connect_cb() is used. The
function logic is responsible for determining the function for
establishing subsequent connection.
To do so, it called get_stream_connect_function() ... which can return
only tcp_connect() or tls_connect(), not tcpdns_connect() or
tlsdns_connect(). That is definitely *not* what was implied.
All this time the unit tests were testing something, but now what was
intended.
This commit fixes the problem by passing the tcpdns_connect() and
tlsdns_connect() function pointers to connect_connect_cb().
The ns_interfacemgr_scan() now requires the loopmgr to be running, so we
need to end exclusive mode for the rescan and then begin it again.
This is relatively safe operation (because the scan happens on the timer
anyway), but we need to ensure that we won't load the configuration from
different threads. This is already the case because the initial load
happens on the main thread and the control channel also listens just on
the main loop.
When named starts it creates an empty KEYDATA record in the managed-keys
zone as a placeholder, then schedules a key refresh. If key refresh
fails for some reason (e.g. connectivity problems), named will load the
placeholder key into secroots as a trusted key during the next startup,
which will break the chain of trust, and named will never recover from
that state until managed-keys.bind and managed-keys.bind.jnl files are
manually deleted before (re)starting named again.
Before calling load_secroots(), check that we are not dealing with a
placeholder.
Because dns_resolver_createfetch() locks the view, it was necessary
to unlock the zone in zone_refreshkeys() before calling it in order
to maintain the lock order, and relock afterward. this permitted a race
with dns_zone_synckeyzone().
This commit moves the call to dns_resolver_createfetch() into a separate
function which is called asynchronously after the zone has been
unlocked.
The keyfetch object now attaches to the zone to ensure that
it won't be shut down before the asynchronous call completes.
This necessitated refactoring dns_zone_detach() so it always runs
unlocked. For managed zones it now schedules zone_shutdown() to
run asynchronously, and for unmanaged zones, it requires the last
dns_zone_detach() to be run without loopmgr running.
The NetBSD system allocator is in fact based on the jemalloc, but it
doesn't export the extended interface, so we can't use that. Remove
the jemalloc enforcement for the NetBSD.
when more than one event was scheduled in the isc_aysnc queue,
they were executed in reverse order. we need to pull events
off the back of queue instead the front, so that uv_loop will
run them in the right order.
note that isc_job_run() has the same behavior, because it calls
uv_idle_start() directly. in that case we just document it so
it'll be less surprising in the future.
the dupsigs test is prone to failing on slow CI machines
because the first test can occur before the zone is fully
signed.
instead of just waiting ten seconds arbitrarily, we now
check every second, and allow up to 30 seconds before giving
up.
Instead of doing incremental zone loading with fixed quantum - 100
loaded lines per event, move the zone loading process to the offloaded
libuv threads using isc_work_enqueue() API.
This has the advantage that the thread scheduling is given back to the
operating system that understands blocking operations, and the zone
loading operation doesn't block the networking threads directly.
Incremental file loads now use loopmgr events instead of task events.
The dns_master_loadstreaminc(), _loadbufferinc(), _loadlexer() and
_loadlexerinc() functions were not used in BIND, and have been removed.
Checks that malformed _dns SVCB records are rejected unless
check-svcb is set to no, in which case they are accepted. Both
missing ALPN and missing DOHPATH are checked for.
_dns SVBC records have additional constrains which should be checked
when records are being added. This adds those constraint checks but
allows the user to override them using 'check-svcb no'.
dns_rdata_checksvcb performs data entry checks on SVCB records.
In particular that _dns SVBC record have an 'alpn' and if that 'alpn'
parameter indicates HTTP is in use that 'dophath' is present.
The wrong tls configuration was picked here. It should be of the
primary that is selected by forward->which, not zone->curprimary.
This bug may cause BIND to select the wrong primary when retrieving
the TLS settings, or cause a crash in case the wrongly selected primary
has no TLS settings.
Add a new upforwd system test that checks if update forwarding still
works if the first primary is badly configured.
We cannot reuse the 'example.' zone for this test because that
checks if update forwarding works for DoT. What transport is used
in the new test is of no relevance.
Update the system test to use different known good file names for
the different zones that are being tested.
Use the ALGORITHM_SET option to use randomly selected default algorithm
in this test. Make sure the test works by using variables instead of
hard-coding values.
Use the get_algorithms.py script to detect supported algorithms and
select random algorithms to use for the tests.
Make sure to load common.conf.sh after KEYGEN env var is exported.
Multiple algorithm sets can be defined in this script. These can be
selected via the ALGORITHM_SET environment variable. For compatibility
reasons, "stable" set contains the currently used algorithms, since our
system tests need some changes before being compatible with randomly
selected algorithms.
The script operation is similar to the get_ports.py - environment
variables are created and then printed out as `export NAME=VALUE`
commands, to be interpreted by shell. Once we support pytest runner for
system tests, this should be a fixture instead.
Certain variables have to be exported in order for the system tests to
work. It makes little sense to export the variables in one place/script
while they're defined in another place.
Since it makes no harm, export all the variables to make the behaviour
more predictable and consistent. Previously, some variables were
exported as environment variables, while others were just shell
variables which could be used once the configuration was sourced from
another script. However, they wouldn't be exposed to spawned processes.
For simplicity sake (and for the upcoming effort to run system tests
with pytest), export all variables that are used. TESTS, PARALLEL_UNIX
and SUBDIRS variables are automake-specific, aren't used anywhere else
and thus not exported.
The only variable really needed for the script to work is the path to
the $KEYGEN binary. Allow setting this via an environment variable to
avoid loading conf.sh (and causing a chicken-egg problem). Also make
testcrypto.sh executable to allow its use from conf.sh.
There are three levels there for the port value, with increasing
priority:
1. The default ports, defined by 'port' and 'tls-port' config options.
2. The primaries-level default port: primaries port <number> { ... };
3. The primaries element-level port: primaries { <address> port
<number>; ... };"
In 'named_config_getipandkeylist()', the 'def_port' and 'def_tlsport'
variables are extracted from level 1. The 'port' variable is extracted
from the level 2. Currently if that is unset, it defaults to the
default port ('def_port' or 'def_tlsport' depending on the transport
used), but overrides the level 2 port setting for the next primaries in
the list.
Update the code such that we inherit the port only if the level 3 port
is not set, and inherit from the default ports if the level 2 port is
also not set.
Add a test case that if the first primary fails, the fallback of a
second primary on plain DNS works. This is mainly to test that the port
configuration inheritance works correctly.
The netievent handler for isc_nmsocket_set_tlsctx() was inadvertently
ifdef'd out when BIND was built with --disable-doh, resulting in an
assertion failure on startup when DoT was configured.
Add a couple of tests that verify the serve-stale behavior when
stale-answer-client-timeout is set to 0 and a (stale) CNAME record is
queried.
Related #3517
Make the Sphinx version listed in doc/arm/requirements.txt match the
version currently used in GitLab CI, so that Read the Docs builds the
documentation using the same Python software versions as those used in
GitLab CI.
The "prefetch" setting is in "defaultconf" so it cannot fail, use
INSIST to confirm that.
The 'trigger' and 'eligible' variables are now prefixed with
'prefetch_' and their declaration moved to an upper level, because
there is no more additional code block after this change.
For the prefetch "trigger" parameter ARM states that when a cache
record with a lower TTL value is encountered during query processing,
it is refreshed. But in reality, the record is refreshed when the TTL
value is lower or equal to the configured "trigger" value.
Fix the documentation to make it match with with the code.
ARM states that the "eligibility" TTL is the smallest original TTL
value that is accepted for a record to be eligible for prefetching,
but the code, which implements the condition doesn't behave in that
manner for the edge case when the TTL is equal to the configured
eligibility value.
Fix the code to check that the TTL is greater than, or equal to the
configured eligibility value, instead of just greater than it.
The test triggers a prefetch, but fails to check if it acutally
happened, which prevented it from catching a bug when the record's
TTL value matches the configured prefetch eligibility value.
Check that prefetch happened by comparing the TTL values.
Instead of fixing a Coverity complaint (and other style nits),
delete it because it needs input data that can't be generated
with the tools that ship with BIND.
For UDP queries, after calling dns_adb_beginudpfetch() in fctx_query(),
make sure that dns_adb_endudpfetch() is also called on error path, in
order to adjust the quota back.
It is currently possible that dns_adb_endudpfetch() is not
called in fctx_cancelquery() for a UDP query, which results
in quotas not being adjusted back.
Always call dns_adb_endudpfetch() for UDP queries.
In the cleanup code of fctx_query() function there is a code path
where 'query' is linked to 'fctx' and it is being destroyed.
Make sure that 'query' is unlinked before destroying it.
This commit fixes a startup issue on Solaris systems with
many (reportedly > 510) CPUs by bumping RLIMIT_NOFILE. This appears to
be a regression from 9.11.
For tests where the TCP connection might get interrupted abruptly,
replace the nc with curl as the data sent from server to client might
get lost because of abrupt TCP connection. This happens when the TCP
connection gets closed during sending the large request to the server.
As we already require curl for other system tests, replace the nc usage
in the statschannel test with curl that actually understands the
HTTP/1.1 protocol, so the same connection is reused for sending the
consequtive requests, but without client-side "pipelining".
For the record, the server doesn't support parallel processing of the
pipelined request, so it's a bit misnomer here, because what we are
actually testing is that we process all requests received in a single
TCP read callback.
RPZ rewrites called dns_db_findext() without passing through the
client database options; as as result, if the client set CD=1,
DNS_DBFIND_PENDINGOK was not used as it should have been, and
cache lookups failed, resulting in failure of the rewrite.
The statschannel truncated test still terminates abruptly sometimes and
it doesn't return the answer for the first query. This might happen
when the second process_request() discovers there's not enough space
before the sending is complete and the connection is terminated before
the client gets the data.
Change the isc_http, so it pauses the reading when it receives the data
and resumes it only after the sending has completed or there's
incomplete request waiting for more data.
This makes the request processing slightly less efficient, but also less
taxing for the server, because previously all requests that has been
received via single TCP read would be processed in the loop and the
sends would be queued after the read callback has processed a full
buffer.
In non-developer build, a wrong condition prevented the
isc__tls_malloc_ex, isc__tls_realloc_ex and isc__tls_free_ex to be
defined. This was causing FTBFS on platforms with OpenSSL 1.0.2.
The 5 seconds requirement to finish the 'pipelined with truncated
stream' was causing spurious failures in the CI because the job runners
might be very busy and sending 128k of data might simply take some time.
Remove the time requirement altogether, there's actually no reason why
the test SHOULD or even MUST finish under 5 seconds.
Correctly source conf.sh in dupsigs test scripts (fix issue introduced
by 093af1c00a).
Update dupsigs test for dnssec-dnskey-kskonly default. Since v9.17.20,
the dnssec-dnskey-kskonly is set to yes. Update the test to not expect
the additional RRSIG with ZSK for DNSKEY.
Speed up the test from 20 minutes to 2.5 minutes and make it part of the
default test suite executed in CI.
- decrease number of records to sign from 2000 to 500
- decrease the signing interval by a factor of 6
- shorten the final part of the test after last signing (since nothing
new happens there)
Finally, clarify misleading comments about (in)sufficient time for zone
re-signing. The time used in the test is in fact sufficient for the
re-signing to happen. If it wasn't, the previous ZSK would end up being
deleted while its signatures would still be present, which is a
situation where duplicate signatures can still happen.
Ensure the port numbers are dynamically filled in with copy_setports.
Clarify test fail condition.
Make the stress test part of the default test suite since it doesn't
seem to run too long or interfere with other tests any more (the
original note claiming so is more than 20 years old).
Related !6883
Properly template the port number in config files with copy_setports.
The test takes two minutes on my machine which doesn't seem like a
proper justification to exclude it from the test suite, especially
considering we run these tests in parallel nowadays. The resource usage
doesn't seems significantly increased so it shouldn't interfere with
other system tests.
There also exists a precedent for longer running system tests that are
already part of the default system test suite (e.g. serve-stale takes
almost three minutes on the same machine).
When a target server is unreachable, the varying network conditions may
cause different ICMP message (or no message). The host unreachable
message was discovered when attempting to run the test locally while
connected to a VPN network which handles all traffic.
Extend the dig output check with "host unreachable" message to avoid a
false negative test result in certain network environments.
Add a test ensuring that the amount of work fctx_getaddresses() performs
for any encountered delegation is limited: delegate example.net to a set
of 1,000 name servers in the redirect.com zone, the names of which all
resolve to IP addresses that nothing listens on, and query for a name in
the example.net domain, checking the number of times the findname()
function gets executed in the process; fail if that count is excessively
large.
Since the size of the referral response sent by ans3 is about 20 kB, it
cannot be sent back over UDP (EMSGSIZE) on some operating systems in
their default configuration (e.g. FreeBSD - see the
net.inet.udp.maxdgram sysctl). To enable reliable reproduction of
CVE-2022-2795 (retry patterns vary across BIND 9 versions) and avoid
false positives at the same time (thread scheduling - and therefore the
number of fetch context restarts - vary across operating systems and
across test runs), extend bin/tests/system/resolver/ans3/ans.pl so that
it also listens on TCP and make "ns1" in the "resolver" system test
always use TCP when communicating with "ans3".
Also add a test (foo.bar.sub.tld1/TXT) that ensures the new limitations
imposed on the resolution process by the mitigation for CVE-2022-2795 do
not prevent valid, glueless delegation chains from working properly.
It was possible that accept callback can be called after listener
shutdown. In such a case the callback pointer equals NULL, leading to
segmentation fault. This commit fixes that.
add a test to compare the Content-Length of successive compressed
messages on a single HTTP connection that should contain the same
data; fail if the size grows by more than 100 bytes from one query
to the next.
This log happens when BIND checks the parental-agents if the DS has
been published. But if you don't have parental-agents set up, the list
of keys to check will be empty and the result will be ISC_R_NOTFOUND.
This is not an error, so change the log level to debug in this case.
Since we are using designated initializers, we were missing initializers
for ISC_LIST and ISC_LINK, add them, so you can do
*foo = (foo_t){ .list = ISC_LIST_INITIALIZER };
Instead of:
*foo = (foo_t){ 0 };
ISC_LIST_INIT(foo->list);
This commit introduces a primitive isc__nmsocket_stop() which performs
shutting down on a multilayered socket ensuring the proper order of
the operations.
The shared data within the socket object can be destroyed after the
call completed, as it is guaranteed to not be used from within the
context of other worker threads.
I.e. print the name of the function in BIND that called the system
function that returned an error. Since it was useful for pthreads
code, it seems worthwhile doing so everywhere.
Mostly generated automatically with the following semantic patch,
except where coccinelle was confused by #ifdef in lib/isc/net.c
@@ expression list args; @@
- UNEXPECTED_ERROR(__FILE__, __LINE__, args)
+ UNEXPECTED_ERROR(args)
@@ expression list args; @@
- FATAL_ERROR(__FILE__, __LINE__, args)
+ FATAL_ERROR(args)
The isccfg_duration_fromtext() function is truncating large numbers
to 32 bits instead of capping or rejecting them, i.e. 64424509445,
which is 0xf00000005, gets parsed as 32-bit value 5 (0x00000005).
Fail parsing a duration if any of its components is bigger than
32 bits. Using those kind of big numbers has no practical use case
for a duration.
The isccfg_duration_toseconds() function can overflow the 32 bit
seconds variable when calculating the duration from its component
parts.
To avoid that, use 64-bit calculation and return UINT32_MAX if the
calculated value is bigger than UINT32_MAX. Again, a number this big
has no practical use case anyway.
The buffer for the generated duration string is limited to 64 bytes,
which, in theory, is smaller than the longest possible generated
duration string.
Use 80 bytes instead, calculated by the '7 x (10 + 1) + 3' formula,
where '7' is the count of the duration's parts (year, month, etc.), '10'
is their maximum length when printed as a decimal number, '1' is their
indicator character (Y, M, etc.), and 3 is two more indicators (P and T)
and the terminating NUL character.
The cfg_print_duration() checks added previously in the 'duration_test'
unit test uncovered a bug in cfg_print_duration().
When calculating the current 'str' pointer of the generated text in the
buffer 'buf', it erroneously adds 1 byte to compensate for that part's
indicator character. For example, to add 12 minutes, it needs to add
2 + 1 = 3 characters, where 2 is the length of "12", and 1 is the length
of "M" (for minute). The mistake was that the length of the indicator
is already included in 'durationlen[i]', so there is no need to
calculate it again.
In the result of this mistake the current pointer can advance further
than needed and end up after the zero-byte instead of right on it, which
essentially cuts off any further generated text. For example, for a
5 minutes and 30 seconds duration, instead of having this:
'P', 'T', '5', 'M', '3', '0', 'S', '\0'
The function generates this:
'P', 'T', '5', 'M', '\0', '3', '0', 'S', '\0'
Fix the bug by adding to 'str' just 'durationlen[i]' instead of
'durationlen[i] + 1'.
Currently the 'duration_test' unit test checks only the
cfg_obj_asduration() function.
Extend the test so it checks also the reverse operation using the
cfg_print_duration() function, which is used in named-checkconf.
The cfg_print_duration() function prints a ISO 8601 duration value
converted from an array of integers, where the parts of the date and
time are stored.
durationlen[6], which holds the "seconds" part of the duration, has
a special case in cfg_print_duration() to ensure that when there are
no values in the duration, the result still can be printed as "PT0S",
instead of just "P", so it can be a valid ISO 8601 duration value.
There is a logical error in one of the two special case code paths,
when it checks that no value from the "date" part is defined, and no
"hour" or "minute" from the "time" part are defined.
Because of the error, durationlen[6] can be used uninitialized, in
which case the second parameter passed to snprintf() (which is the
maximum allowed length) can contain a garbage value.
This can not be exploited because the buffer is still big enough to
hold the maximum possible amount of characters generated by the "%u%c"
format string.
Fix the logical bug, and initialize the 'durationlen' array to zeros
to be a little safer from other similar errors.
[performance] A new algorithm for DNS name compression based on a
hash set of message offsets. Name compression is now
more complete as well as being generally faster, and
the implementation is less complicated and requires
much less memory.
The `render` benchmark loads some binary DNS message dumps and
repeatedly passes them to `dns_message_render`.
The `compress` benchmark loads a list of domain names and packs them
into 4KiB chunks using `dns_name_towire`.
Check that names are correctly added and deleted in the compression
context. Use many names with differing numerical prefixes to make it
relatively easy to identify and debug problems.
All we need for compression is a very small hash set of compression
offsets, because most of the information we need (the previously added
names) can be found in the message using the compression offsets.
This change combines dns_compress_find() and dns_compress_add() into
one function dns_compress_name() that both finds any existing suffix,
and adds any new prefix to the table. The old split led to performance
problems caused by duplicate names in the compression context.
Compression contexts are now either small or large, which the caller
chooses depending on the expected size of the message. There is no
dynamic resizing.
There is a behaviour change: compression now acts on all the labels in
each name, instead of just the last few.
A small benchmark suggests this is about 2x faster.
During loop manager refactoring isc_nmsocket_set_tlsctx() was not
properly adapted. The function is expected to broadcast the new TLS
context for every worker, but this behaviour was accidentally broken.
Replace all uses of RUNTIME_CHECK() in lib/isc/include/isc/once.h with
PTHEADS_RUNTIME_CHECK(), in order to improve error reporting for any
once-related run-time failures (by augmenting error messages with
file/line/caller information and the error string corresponding to
errno).
There are multiple reasons to remove this test as obsolete:
- The test may not possibly work for over 2.5 years, since
98b3b93791 removed the rndc.py python
tool on which this test relies.
- It isn't part of the test suite either in CI or locally unless it is
explicitly enabled. As a result, there are many issues which prevent
the test from being executed caused by various refactoring efforts
accumulated over time.
- Even if the test could be executed, it has no clear failure condition.
If the python script(s) fail, the test still passes.
Now that the artificial limit on the recv buffer has been removed, the
current system test always fails because it tests if the truncation has
happened.
Add test that sending more than 10 headers makes the connection to
closed; and add test that sending huge HTTP request makes the connection
to be closed.
Rewrite the isc_httpd to be more robust.
1. Replace the hand-crafted HTTP request parser with picohttpparser for
parsing the whole HTTP/1.0 and HTTP/1.1 requests. Limit the number
of allowed headers to 10 (arbitrary number).
2. Replace the hand-crafted URL parser with isc_url_parse for parsing
the URL from the HTTP request.
3. Increase the receive buffer to match the isc_netmgr buffers, so we
can at least receive two full isc_nm_read()s. This makes the
truncation processing much simpler.
4. Process the received buffer from single isc_nm_read() in a single
loop and schedule the sends to be independent of each other.
The first two changes makes the code simpler and rely on already
existing libraries that we already had (isc_url based on nodejs) or are
used elsewhere (picohttpparser).
The second two changes remove the artificial "truncation" limit on
parsing multiple request. Now only a request that has too many
headers (currently 10) or is too big (so, the receive buffer fills up
without reaching end of the request) will end the connection.
We can be benevolent here with the limites, because the statschannel
channel is by definition private and access must be allowed only to
administrators of the server. There are no timers, no rate-limiting, no
upper limit on the number of requests that can be served, etc.
PicoHTTPParser is a tiny, primitive, fast HTTP request/response parser.
Unlike most parsers, it is stateless and does not allocate memory by
itself. All it does is accept pointer to buffer and the output
structure, and setups the pointers in the latter to point at the
necessary portions of the buffer.
Works nicely together with:
git config --add blame.ignoreRevsFile .git-blame-ignore-revs
The list was generated by hand-picking from git log --oneline augmented
with:
--author=tbox
--grep=clang-format
--grep=copyright
--grep=reformat
--grep=whitespace
plus
git log --format='commit %H %s' --stat | grep -E 'commit|changed' | grep -B1 '[0-9][0-9][0-9] files changed'
plus some sanity checking.
Comments were added with:
for COMMIT in $(cat .git-blame-ignore-revs)
do git log -1 --format="# %s" "$COMMIT"
echo $COMMIT
done
sizeof(dns_name_t) did not change but the boolean attributes are now
separated as one-bit structure members. This allows debuggers to
pretty-print dns_name_t attributes without any special hacks, plus we
got rid of manual bit manipulation code.
Originally RBT node stored three lowest bits from dns_name_t attributes.
This had a curious side-effect noticed by Tony Finch:
If you create an rbt node from a DYNAMIC name then the flag will be
propagated through dns_rbt_namefromnode() ... if you subsequently call
dns_name_free() it will try to isc_mem_put() a piece of an rbt node ...
but dns_name_free() REQUIRE()s that the name is dynamic so in the usual
case where rbt nodes are created from non-dynamic names, this kind of
code will fail an assertion.
This is a bug it dates back to june 1999 when NAMEATTR_DYNAMIC was
invented.
Apparently it does not happen often :-)
I'm planning to get rid of DNS_NAMEATTR_ definitions and bit operations,
so removal of this "three-bit-subset" assignment is a first step.
We can keep only the ABSOLUTE flag in RBT node and nothing else because
names attached to rbt nodes are always readonly: The internal node_name()
function always sets the NAMEATTR_READONLY when making a dns_name that
refers to the node's name, so the READONLY flag will be set in the name
returned by dns_rbt_namefromnode().
Co-authored-by: Tony Finch <fanf@isc.org>
doth system test fixes - decrese the size of HTTP listener quota, increase transfer-in/out limits
Closes#3596
See merge request isc-projects/bind9!6898
Sometimes doth test could intermittently fail shortly after start due
to inability to complete a zone transfer in time. As it turned out, it
could happen due to transfers-in/out limits. Initially the defaults
were fine, but over time, especially when adding Strict/Mutual TLS, we
added more than 10 zones so it became possible to hit the limits.
This commit takes care of that by bumping the limits.
This commit reduces the size of HTTP listener quota from 300 (default)
to 100 so that it would make hitting any global limits in case of
running multiple tests in parallel in multiple containers unlikely.
This way the need in opening many file descriptors of different
kinds (e.g. client side connections and pipes) gets significantly
reduced while the required code paths are still verified.
In ac4cc8443d, the ISC_R_CONNREFUSED was
removed in connect_read_cb, but it can actually happen in the udp_test:
[ RUN ] udp_recv_send
connect_read_cb(0x7f2c2801a270, connection refused, (nil))
The ISC_R_SHUTTINGDOWN should be handled the same as ISC_R_CANCELED in
the udp__send_cb(), as we might be sending the data while the
loopmgr/netmgr shutdown has been initiated.
In rare circumstances, the UDP port for the listening socket and the UDP
port for the connecting socket might be the same. Because we use the
"reuse" port socket option, this isn't caught when binding the socket,
and thus the connected client socket could send a datagram to itself,
completely bypassing the server. This doesn't happen under normal
operation mode because `named` is listening on a privileged port (53),
and even if not, it doesn't usually talk to itself as the tests do.
Pick an arbitrary port for listening (9153-9156) that is outside the
ephemeral port range for the network manager related unit tests (except
the `doh_test).
The isc_nm_udpconnect() erroneously set the reuse port with
load-balancing on the outgoing connected UDP sockets. This socket
option makes only sense for the listening sockets. Don't set the
load-balancing reuse port option on the outgoing UDP sockets.
Since we are testing UDP on the localhost and the same interface, the
UDP datagrams can't get lost. Change the connect read callback, so it
starts reading again on the timeout instead of just getting stuck, and
fail when any other result codes than ISC_R_SUCCESS and ISC_R_TIMEDOUT
are received because we don't expect them to happen in these simple
tests.
This commit removes broken remnants of unit test slowdown logic, which
caused unit test hangs on platforms susceptible to "too many open
files" error, notably OpenBSD.
This commit fixes TLS DNS verification error message reporting which
we probably broke during one of the recent networking code
refactorings.
This prevent e.g. dig from producing useful error messages related to
TLS certificates verification.
Ensure that TLS error is empty before calling SSL_get_error() or doing
SSL I/O so that the result will not get affected by prior error
statuses.
In particular, the improper error handling led to intermittent unit
test failure and, thus, could be responsible for some of the system
test failures and other intermittent TLS-related issues.
See here for more details:
https://www.openssl.org/docs/man3.0/man3/SSL_get_error.html
In particular, it mentions the following:
> The current thread's error queue must be empty before the TLS/SSL
> I/O operation is attempted, or SSL_get_error() will not work
> reliably.
As we use the result of SSL_get_error() to decide on I/O operations,
we need to ensure that it works reliably by cleaning the error queue.
TLS DNS: empty error queue before attempting I/O
There was inconsistency in which error codes would get accepted and
ignored in the network manager unit test callbacks. Add following
results, so we just detach the handle instead of causing assertion
failure:
* ISC_R_SHUTTINGDOWN - when the network manager is shutting down
* ISC_R_CANCELED - the socket has been shut down
* ISC_R_EOF - the (TCP) communication has ended on the other side
* ISC_R_CONNECTIONRESET - the TCP connection was reset
This should fix some of the spurious unit test failures.
The check is left from when tcp_connect_direct() called isc__nm_socket()
and it was uncertain whether it had succeeded, but now isc__nm_socket()
is called before tcp_connect_direct(), so sock->fd cannot be -1.
*** CID 357292: (REVERSE_NEGATIVE)
/lib/isc/netmgr/tcp.c: 309 in isc_nm_tcpconnect()
303
304 atomic_store(&sock->active, true);
305
306 result = tcp_connect_direct(sock, req);
307 if (result != ISC_R_SUCCESS) {
308 atomic_store(&sock->active, false);
>>> CID 357292: (REVERSE_NEGATIVE)
>>> You might be using variable "sock->fd" before verifying that it is >= 0.
309 if (sock->fd != (uv_os_sock_t)(-1)) {
310 isc__nm_tcp_close(sock);
311 }
312 isc__nm_connectcb(sock, req, result, true);
313 }
314
If sending took too long the isc_nm_read() could timeout twice, leading
to extra 'cread' counter in the udp_cancel_read test. Increase the
cread counter only on ISC_R_EOF (canceled read) and deal with the
multiple ISC_R_TIMEOUTS gracefully.
The bin/tests/system/start.pl script waits until a "running" message is
logged by a given name server instance before attempting to send a
version.bind/CH/TXT query to it. The idea behind this was to make the
script wait until named loads all the zones it is configured to serve
before telling the system test framework that a given server is ready to
use; this prevents the need to add boilerplate code that waits for a
specific zone to be loaded to each test expecting that.
The problem is that when it looks for "running" messages, the
bin/tests/system/start.pl script assumes that the existence of any such
message in the named.run file indicates that a given named instance has
already finished loading all zones. Meanwhile, some system tests
restart all the named instances they use throughout their lifetime (some
even do that a few times), for example to run Python-based tests. The
bin/tests/system/start.pl script handles such a scenario incorrectly: as
soon as it finds any "running" message in the named.run file it inspects
and it gets a response to a version.bind/CH/TXT query, it tells the
system test framework that a given server is ready to use, which might
not be true - it is possible that only the "version.bind" zone is loaded
at that point and the "running" message found was logged by a
previously-shutdown named instance. This triggers intermittent failures
for Python-based tests.
Fix by improving the logic that the bin/tests/system/start.pl script
uses to detect server startup: check how many "running" lines are
present in a given named.run file before attempting to start a named
instance and only proceed with version.bind/CH/TXT queries when the
number of "running" lines found in that named.run file increases after
the server is started.
In the "rrsetorder" system test, the ns2 named instance is restarted
without passing the --restart option to bin/tests/system/start.pl. This
causes the log file for that named instance to be needlessly truncated.
Prevent this from happening by restarting the affected named instance
in the same way as all the other named instances used in system tests.
Getting the recorded value of 'edns-udp-size' from the resolver requires
strong attach to the dns_view because we are accessing `view->resolver`.
This is not the case in places (f.e. dns_zone unit) where `.udpsize` is
accessed. By moving the .udpsize field from `struct dns_resolver` to
`struct dns_view`, we can access the value directly even with weakly
attached dns_view without the need to lock the view because `.udpsize`
can be accessed after the dns_view object has been shut down.
The dns_view implements weak and strong reference counting. When strong
reference counting reaches zero, the adb, ntatable and resolver objects
are shut down and detached.
In dns_zone and dns_nta the dns_view was weakly attached, but the
view->resolver reference was accessed directly leading to dereferencing
the NULL pointer.
Add dns_view_getresolver() method which attaches to view->resolver
object under the lock (if it still exists) ensuring the dns_resolver
will be kept referenced until not needed.
The value of `sign_bit` is platform-dependent but constant at compile
time. Use a cast to convert the boolean `sign_bit` to 0 or 1 instead of
ternary `?:` because one branch of the conditional is dead code. (We
could leave out the cast to `size_t` but our style prefers to handle
booleans more explicitly, hence the `?:` that caused the issue.)
*** CID 358310: Possible Control flow issues (DEADCODE)
/lib/isc/resource.c: 118 in isc_resource_setlimit()
112 * rlim_t, and whether rlim_t has a sign bit.
113 */
114 isc_resourcevalue_t rlim_max = UINT64_MAX;
115 size_t wider = sizeof(rlim_max) - sizeof(rlim_t);
116 bool sign_bit = (double)(rlim_t)-1 < 0;
117
>>> CID 358310: Possible Control flow issues (DEADCODE)
>>> Execution cannot reach the expression "1" inside this statement: "rlim_max >>= 8UL * wider + ...".
118 rlim_max >>= CHAR_BIT * wider + (sign_bit ? 1 : 0);
119 rlim_value = ISC_MIN(value, rlim_max);
120 }
121
122 rl.rlim_cur = rl.rlim_max = rlim_value;
123 unixresult = setrlimit(unixresource, &rl);
While refactoring the isc_mem_getx(...) usage, couple places were
identified where the memory was resized manually. Use the
isc_mem_reget(...) that was introduced in [GL !5440] to resize the
arrays via function rather than a custom code.
In several places, the structures were cleaned with memset(...)) and
thus the semantic patch converted the isc_mem_get(...) to
isc_mem_getx(..., ISC_MEM_ZERO). Use the designated initializer to
initialized the structures instead of zeroing the memory with
ISC_MEM_ZERO flag as this better matches the intended purpose.
Add new semantic patch to replace the straightfoward uses of:
ptr = isc_mem_{get,allocate}(..., size);
memset(ptr, 0, size);
with the new API call:
ptr = isc_mem_{get,allocate}x(..., size, ISC_MEM_ZERO);
Previously, the isc_mem_get_aligned() and friends took alignment size as
one of the arguments. Replace the specific function with more generic
extended variant that now accepts ISC_MEM_ALIGN(alignment) for aligned
allocations and ISC_MEM_ZERO for allocations that zeroes
the (re-)allocated memory before returning the pointer to the caller.
Coverity is optimistic that we might do thousands of hashes in less
than a microsecond.
/tests/bench/siphash.c: 54 in main()
48 count++;
49 }
50
51 isc_time_now_hires(&finish);
52
53 us = isc_time_microdiff(&finish, &start);
>>> CID 358309: Integer handling issues (DIVIDE_BY_ZERO)
>>> In expression "count * 1000UL / us", division by expression "us" which may be zero has undefined behavior.
54 printf("%f us wide-lower len %3zu, %7llu kh/s (%llx)\n",
55 (double)us / 1000000.0, len,
56 (unsigned long long)(count * 1000 / us),
57 (unsigned long long)sum);
58 }
59
This is hopefully end of duplication. This batch did not cause clashes
in Sphinx but it was pointless nonetheless as we have auto-generated
anchors for all statements.
Some statement names like "allow-query" had manually defined link anchor
_allow-query and also implicit anchor created by
.. namedconf:statement:: syntax. This causes warnings if a ambiguous
reference is made using :any:`allow-query` syntax.
Remove (hopefully all) manually defined anchors which pointed to
identical place as the implicit anchor. This allows :any: to work.
In rare cases where manual anchor points to descriptive text separated
from statement definition the reference was disamguated by replacing
:any:`notify` with :ref:`notify` (for manual anchor)
vs. :namedconf:ref:`notify` (for statement definition).
Please note that `options` statement is a trap: It is ambiguous even
without manual anchor because rndc.conf has its own `options`. Use
:namedconf:ref:`options` vs. :rndcconf:ref:`options` to select
appropriate target.
Fixup for 2c3b2dabe9.
We forgot to update TSAN paths when moving all the unit tests to
/tests/. Let's remove paths from find to make it less dependent on
exact location, and store all untracked files as we do in the normal
unit test template.
Related: !6243
The previous commit failed some tests because we expect that if a
fetch fails and we have stale candidates in cache, the
stale-refresh-time window is started. This means that if we hit a stale
entry in cache and answering stale data is allowed, we don't bother
resolving it again for as long we are within the stale-refresh-time
window.
This is useful for two reasons:
- If we failed to fetch the RRset that we are looking for, we are not
hammering the authoritative servers.
- Successor clients don't need to wait for stale-answer-client-timeout
to get their DNS response, only the first one to query will take
the latency penalty.
The latter is not useful when stale-answer-client-timeout is 0 though.
So this exception code only to make sure we don't try to refresh the
RRset again if it failed to do so recently.
Refreshing a stale RRset is similar to prefetching an RRset, so
reuse the existing code. When refreshing an RRset we need to clear
all db options related to serve-stale so that stale RRsets in cache
are ignored during the refresh.
We no longer need to set the "nodetach" flag, because the refresh
fetch is now a "fetch and forget". So we can detach from the client
in the query_send().
This code will break some serve-stale test cases, this will be fixed
in the successor commit.
TODO: add explanation why the serve-stale test cases fail.
Formerly, the isc_hash32() would have to change the key in a local copy
to make it case insensitive. Change the isc_siphash24() and
isc_halfsiphash24() functions to lowercase the input directly when
reading it from the memory and converting the uint8_t * array to
64-bit (respectively 32-bit numbers).
dohpath is specfied in draft-ietf-add-svcb-dns and has a value
of 7. It must be a relative path (start with a /), be encoded
as UTF8 and contain the variable dns ({?dns}).
On systems with signed rlim_t the old code calculated its maximum
value by shifting 1 into the sign bit, which is undefined behaviour.
Avoid the bug by using an unsigned shift.
When the TCP test is run on the busy server, the server might take a
while to wind the server down because it might still be processing all
that 300k invalid XFR requests.
Increate the rncd wait time to 120 seconds, the SIGTERM time to 300
seconds, and reduce the time to wait for ans servers from 1200 second
to just 120 seconds.
The dns__nta_shutdown() could be run from different threads and it was
accessing nta->timer unlocked. Don't check and stop the timer from
dns__nta_shutdown() directly, but leave it for the async callback.
Because the dns_zonemgr_create() was run before the loopmgr was started,
the isc_ratelimiter API was more complicated that it had to be. Move
the dns_zonemgr_create() to run_server() task which is run on the main
loop, and simplify the isc_ratelimiter API implementation.
The isc_timer is now created in the isc_ratelimiter_create() and
starting the timer is now separate async task as is destroying the timer
in case it's not launched from the loop it was created on. The
ratelimiter tick now doesn't have to create and destroy timer logic and
just stops the timer when there's no more work to do.
This should also solve all the races that were causing the
isc_ratelimiter to be left dangling because the timer was stopped before
the last reference would be detached.
Access to the source tree is not available with oss_fuzz. Have
fuzz/dns_message_checksig build and populate a key directory for
the fuzzer to use. This contains a key pair and a zone file which
has the public key from the key pair. Clean it up on shutdown.
Unlike standard free(), isc_mem_free() is not a no-op when passed a
NULL pointer. For size accounting purposes it calls sallocx(), which
crashes when passed a NULL pointer. To get more helpful diagnostics,
REQUIRE() that the pointer is not NULL so that when the programmer
makes a mistake they get a backtrace that shows what went wrong.
Extra care must be taken when executing the callbacks to prevent the
deadlocks on the caller's side. Add a paragraph that addresses when we
can and when we cannot call the callbacks directly.
The isc__nm_udp_send() callback would be called synchronously when
shutting down or when the socket has been closed. This could lead to
double locking in the calling code and thus those callbacks needs to be
called asynchronously.
Add several test cases in the 'upforwd' system test to make sure
that different scenarios of Dynamic DNS update forwarding are
tested, in particular when both the original and forwarded requests
are over Do53, or DoT, or they use different transports.
Now that the 'dns_request' supports using TLS transport, implement
dynamic update forwarding using DoT when the primary server is
configured to use a TLS transport.
Previously, when using such configuration, the dynamic update forwarding
feature was broken.
There's a known memory leak in the engine_pkcs11 at the time of writing
this and it interferes with the named ability to check for memory leaks
in the OpenSSL memory context by default.
Add an autoconf option to explicitly enable the memory leak detection,
and use it in the CI except for pkcs11 enabled builds. When this gets
fixed in the engine_pkc11, the option can be enabled by default.
As we can't check the deallocations done in the library memory contexts
by default because it would always fail on non-clean exit (that happens
on error or by calling exit() early), we just want to enable the checks
to be done on normal exit.
The libxml2 library provides a way to replace the default allocator with
user supplied allocator (malloc, realloc, strdup and free).
Create a memory context specifically for libxml2 to allow tracking the
memory usage that has originated from within libxml2. This will provide
a separate memory context for libxml2 to track the allocations and when
shutting down the application it will check that all libxml2 allocations
were returned to the allocator.
Additionally, move the xmlInitParser() and xmlCleanupParser() calls from
bin/named/main.c to library constructor/destructor in libisc library.
The OpenSSL library provides a way to replace the default allocator with
user supplied allocator (malloc, realloc, and free).
Create a memory context specifically for OpenSSL to allow tracking the
memory usage that has originated from within OpenSSL. This will provide
a separate memory context for OpenSSL to track the allocations and when
shutting down the application it will check that all OpenSSL allocations
were returned to the allocator.
The libuv library provides a way to replace the default allocator with
user supplied allocator (malloc, realloc, calloc and free).
Create a memory context specifically for libuv to allow tracking the
memory usage that has originated from within libuv. This requires
libuv >= 1.38.0 which provides uv_library_shutdown() function that
assures no more allocations will be made.
Instead of using generic HAVE_BUILTIN_OVERFLOW, we need to check whether
the overflow functions actually work as there was a bug in GCC that it
would not detect mul overflow when compiled with `-m32` option without
optimizations and the bug was fixed only for GCC 6.5+ and 7.3+/8+.
For further details see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82274
Previously, the isc_mem_debugging would be single global variable that
would affect the behavior of the memory context whenever it would be
changed which could be after some allocation were already done.
Change the memory debugging options to be local to the memory context
and immutable, so all allocations within the same memory context are
treated the same.
By bumping the minimum libuv version to 1.34.0, it allows us to remove
all libuv shims we ever had and makes the code much cleaner. The
up-to-date libuv is available in all distributions supported by BIND
9.19+ either natively or as a backport.
The HMACs and GSSAPI are just using unallocated values.
Moving them around shouldn't cause issues.
Only the dnssec system test knew the internal number in use for hmacmd5.
Switch the primary to require 'next_key' for zone transfers then
update the catalog zone to say to use 'next_key'. Next update the
zones contents then check that those changes are seen on the
secondary.
When looking for changes in a catalog zone member zone we need to
also check if the TSIG key name associated with a primary server
has be added, removed or changed.
Previously stack with buffer for test dns message went out of scope
before the message was processed. For fuzz testing its better to avoid
allocation, so let's avoid allocations completely and use simplest
possible static buffer.
Fixes: #3565
previously, when ISC_BUFFER_USEINLINE was defined, macros were
used to implement isc_buffer primitives (isc_buffer_init(),
isc_buffer_region(), etc). these macros were missing the DbC
assertions for those primitives, which made it possible for
coding errors to go undetected.
adding the assertions to the macros caused compiler warnings on
some platforms. therefore, this commit converts the ISC__BUFFER
macros to static inline functions instead, with assertions included,
and eliminates the non-inline implementation from buffer.c.
the --enable-buffer-useinline configure option has been removed.
RHEL8 Sphinx does not support all features used in ARM building. But
with few emitted warnings it can build the documentation fine. Simplify
warnings acceptance by allowing make doc SPHINX_W=''.
Make documentation building successful even on RHEL9 sphinx 3.4.3. It
does not like case-insensitive matching of terms, so provide lowercase
text description with Uppercase word reference.
dns_message_checksig is called in a number of scenarios
* on requests and responses
* on multiple opcodes
* with and without signatures
* with TSIG signatures
* with SIG(0) signatures
* with and without configured TSIG keys
* with and without KEY records being present
* signing performed now, in the future and in the past
we use the first two octets of the seed to configure the calling
environment with the remainder of the seed being the rdata of the
TSIG/SIG(0) record.
When fuzzing it is useful for all signing operations to happen
at a specific time for reproducability. Add two variables to
the message structure (fuzzing and fuzztime) to specify if a
fixed time should be used and the value of that time.
The RAND_bytes() implementation differs between the OpenSSL versions and
uses the system entropy only for seeding its internal CSPRNG. The
uv_random() on the other hand uses the system provided CSPRNG.
Switch from RAND_bytes() to uv_random() to use system provided CSPRNG.
dns_rdata_tostruct doesn't need a mctx passed to it for SIG (the signer
is already expanded at this point). About the only time when mctx is
needed is when the structure is to be used after the rdata has been
destroyed.
Add a simple test PKI based on the existing one in the doth test.
Check ephemeral, forward-secrecy, and forward-secrecy-mutual-tls
TLS configurations with different scenarios.
The comments in CA.cfg file serve as a good tutorial for setting up
a simple PKI for a system test. There is a typo in one of the presented
commands, which results in openssl not exiting with an error message
instead of generating a certificate.
Fix the typo.
OpenSSL just cannot work with mixing ENGINE_* api mixed with OSSL_PARAM
builders. But it can be built in legacy mode, where deprecated but still
working API would be used.
It can work under OpenSSL 3.0, but only if using legacy code paths
matching OpenSSL 1.1 calls and functions.
Remove fromlabel processing by OpenSSL 3.0 only functions. They can
return later with a proper provider support for pkcs11.
OpenSSL has deprecated many things in version 3.0. If pkcs11 engine
should work then no builder from OpenSSL 3.0 API can be used.
Allow switching to OpenSSL 1.1 like calls even on OpenSSL 3.0 when
OPENSSL_API_COMPAT=10100 is defined. It would still compile and allow
working keys loading from the engine passed on command line.
After the loopmgr work has been merged, we can now cleanup the TCP and
TLS protocols a little bit, because there are stronger guarantees that
the sockets will be kept on the respective loops/threads. We only need
asynchronous call for listening sockets (start, stop) and reading from
the TCP (because the isc_nm_read() might be called from read callback
again.
This commit does the following changes (they are intertwined together):
1. Cleanup most of the asynchronous events in the TCP code, and add
comments for the events that needs to be kept asynchronous.
2. Remove isc_nm_resumeread() from the netmgr API, and replace
isc_nm_resumeread() calls with existing isc_nm_read() calls.
3. Remove isc_nm_pauseread() from the netmgr API, and replace
isc_nm_pauseread() calls with a new isc_nm_read_stop() call.
4. Disable the isc_nm_cancelread() for the streaming protocols, only the
datagram-like protocols can use isc_nm_cancelread().
5. Add isc_nmhandle_close() that can be used to shutdown the socket
earlier than after the last detach. Formerly, the socket would be
closed only after all reading and sending would be finished and the
last reference would be detached. The new isc_nmhandle_close() can
be used to close the underlying socket earlier, so all the other
asynchronous calls would call their respective callbacks immediately.
Co-authored-by: Ondřej Surý <ondrej@isc.org>
Co-authored-by: Artem Boldariev <artem@isc.org>
When looking for required glue, dns_message_rendersection() only
processes the first rdataset associated with the first name added to the
ADDITIONAL section. If the DNS_RDATASETATTR_REQUIRED attribute is set
for an rdataset which is located somewhere else (i.e. the name it is
associated with is preceded by another name in the ADDITIONAL section),
it will not be honored, i.e. the TC bit will not be set even if the
rdataset does not fit into the response.
Check the attributes of each processed rdataset while appending names to
a referral response based on a glue cache entry. If a given rdataset is
marked with DNS_RDATASETATTR_REQUIRED, make sure the name it is
associated with is added to the response at the beginning of the
ADDITIONAL section, not its end.
Note that using ISC_LIST_PREPEND() instead of ISC_LIST_APPEND() is not
necessary when associating the rdataset with its owner name because the
dns_name_t structures are initialized just before the glue rdatasets are
associated with them and therefore they are empty at that point, which
means no other (non-required) rdataset can precede the glue rdatasets
within the dns_name_t structure owning them.
If an NS RRset at the parent side of a delegation point only contains
in-bailiwick NS records, at least one glue record should be included in
every referral response sent for such a delegation point or else clients
will need to send follow-up queries in order to determine name server
addresses. In certain edge cases (when the total size of a referral
response without glue records was just below to the UDP packet size
limit), named failed to adhere to that rule by sending non-truncated,
glueless referral responses.
Fix the problem by marking all in-bailiwick NS records processed by
glue_nsdname_cb() (the dns_rdataset_additionaldata() callback used by
RBTDB code while iterating over an NS RRset when dns_rdataset_addglue()
is called) with the DNS_RDATASETATTR_REQUIRED flag. Note that for
simplicity, glue RRsets for all in-bailiwick NS records are marked this
way, even though dns_message_rendersection() only checks the attributes
for the first rdataset associated with the first name added to the
ADDITIONAL section.
If an NS RRset at the parent side of a delegation point only contains
in-bailiwick NS records, at least one glue record should be included in
every referral response sent for such a delegation point or else clients
will need to send follow-up queries in order to determine name server
addresses. In certain edge cases (when the total size of a referral
response without glue records was just below to the UDP packet size
limit), named failed to adhere to that rule by sending non-truncated,
glueless referral responses.
Add tests attempting to trigger that bug in several different scenarios,
covering all possible combinations of the following factors:
- type of zone (signed, unsigned),
- glue record type (A, AAAA, both).
Bring the "glue" system test up to speed with other system tests: add
check numbering, ensure test artifacts are preserved upon failure,
improve error reporting, make the test fail upon unexpected errors,
address ShellCheck warnings.
When jemalloc is the system allocator (on FreeBSD and NetBSD), trying
to build --without-jemalloc caused an obscure compiler error. Instead,
complain at configure time that --without-jemalloc cannot work. (It
needs to remain an error because it is vexing when configure quietly
ignores an explicit direction.)
Instead of expecting that telemetry check has already finished,
wait for it for maximum of three seconds, because named is run with
-tat=3, so the telemetry check must happen with 3 second window.
Co-authored-by: Evan Hunt <each@isc.org>
Each isc_timer needs to be created, started and destroyed on the current
loop. The isc_timer_stop() can be run on any loop, but when run from
different loop than the one associated with the timer, the request to
stop the timer will be recorded in atomic variable and the underlying
uv_timer_t will be stopped on next uv_timer_t callback call. This
allows any thread to stop the timer.
Instead of always creating the trust anchor timer (dns_nta_t) on the
main loop, create the timer on the current loop and associate each
dns_nta_t object to the loop it was created on. This simplifies the
timer handling as everything is run on the associated loop.
During the change, the dns_nta_t structure was renamed to dns__nta_t
and changed to be fully internal to the nta.c compilation unit, and the
dns_ntatable_t structure was made opaque. This required no change to
code using the API as dns_nta_t never had any external users and the
dns_ntatable_t was properly accessed only by using function calls.
Instead of creating the response policy zone deferred update timer when
creating the response policy zone object, create it on demand on the
current loop and destroy it as soon as the timer has finished its job.
There's a side-effect - the processing of the response policy zone
update is now done on the current loop - previously, it was always on
the main loop.
Instead of creating the catalog zone deferred update timer when creating
the catalog zone object, create it on demand on the current loop and
destroy it as soon as the timer has finished its job. There's a
side-effect - the processing of the catalog zone update is now done on
the current loop - previously, it was always on the main loop.
Instead of creating dns_resolver .spillattimer when the dns_resolver_t
object is created, create it on the current loop as needed and destroy
it as soon as the timer has finished its job. This avoids the need to
manipulate the timer from a different thread.
Instead of creating the zone timers at the zone creation time (which
could be any thread), create the zone timer from the isc_loop that has
beena assigned to the zone (zone->loop);
In preparation for the on-loop timers, the isc_ratelimiter API was
converted to use the timer on main loop and start and stop the timer
asynchronously on the main loop.
As it sometimes happens that the object using isc_timer_t is destroyed
via detaching all the references with no guarantee that the last thread
will be matching thread, add a helper isc_timer_async_destroy() function
that stops the timer and runs the destroy function via isc_async_run()
on the matching thread.
- use isc_buffer functions when appropriate, rather than converting
to and from isc_region unnecessarily
- use the zlib total_out value instead of calculating it
- use c99 struct initialization
In fuzzing mode, `isc_random` uses a fixed seed for reproducibility.
The particular seed chosen happened to produce zero as its first
number, however commit bd251de0 introduced an initialization check in
`random_test` that required it to be non-zero. This change adjusts the
seed to avoid spurious test failures.
Also, remove the temporary variable that was used for initialization
because it did not match the type of the thread-local seed array.
This change prepares ground for sending DNS requests using DoT,
which, in particular, will be used for forwarding dynamic updates
to TLS-enabled primaries.
In order to make xfrin.c:get_create_tlsctx() reusable, move the function
into transport.c, and make changes into its prototype to not use the
'dns_xfrin_ctx_t' type, thus making it more universal.
This change prepares ground for adding transport support into the
dispatch manager.
Also, move the typedefs for 'dns_transport_t' and 'dns_transport_list_t'
from transport.h into types.h.
Instead of checking if we need to re-seed for every isc_random call,
seed the random number generator in the libisc global initializer
and the per-thread initializer.
When Address Sanitizer is enabled in gcc-11+, number of false positives
might appear like this:
netmgr/udp.c: In function 'isc__nm_udp_send':
netmgr/udp.c:729:13: warning: 'uv_udp_send' reading 16 bytes from a region of size 8 [-Wstringop-overread]
729 | r = uv_udp_send(&uvreq->uv_req.udp_send, &sock->uv_handle.udp,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
730 | &uvreq->uvbuf, 1, sa, udp_send_cb);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
netmgr/udp.c:729:13: note: referencing argument 3 of type 'const uv_buf_t[0]'
In file included from ./include/isc/uv.h:17,
from ./include/isc/barrier.h:31,
from netmgr/udp.c:17:
/usr/include/uv.h:711:15: note: in a call to function 'uv_udp_send'
711 | UV_EXTERN int uv_udp_send(uv_udp_send_t* req,
| ^~~~~~~~~~~
Disable the warning globally in the autoconf, instead of just locally in
a single CI job, as it might affect people outside our GitLab CI.
The destructor for the isc__nmsocket_t was missing call to the
isc_refcount_destroy() on the reference counter, which might lead to
spurious ThreadSanitizer data race warnings if we ever change the
acquire-release memory order in the isc_refcount_decrement().
Simplify the closing code - during the loopmgr implementation, it was
discovered that the various lists used by the uv_loop_t aren't FIFO, but
LIFO. See doc/dev/libuv.md for more details.
With this knowledge, we can close the protocol handles (uv_udp_t and
uv_tcp_t) and uv_timer_t at the same time by reordering the uv_close()
calls, and thus making sure that after calling the
isc__nm_stoplistening(), the code will not issue any additional callback
calls (accept, read) on the socket that stopped listening.
This might help with the TLS and DoH shutting down sequence as described
in the [GL #3509] as we now stop the reading, stop the timer and call
the uv_close() as earliest as possible.
In the udp_shutdown_read unit test, delay the isc_loopmgr_shutdown() to
the send callback, and in the udp_cancel_read test wait for a single
timed out test, then read again, send an UDP packet and cancel the read
from the send callback.
The network manager UDP code was misinterpreting when the libuv called
the udp_recv_cb with nrecv == 0 and addr == NULL -> this doesn't really
mean that the "stream" has ended, but the libuv indicates that the
receive buffer can be freed. This could lead to assertion failure in
the code that calls isc_nm_read() from the network manager read callback
due to the extra spurious callbacks.
Properly handle the extra callback calls from the libuv in the client
read callback, and refactor the UDP isc_nm_read() implementation to be
synchronous, so no datagram is lost between the time that we stop the
reading from the UDP socket and we restart it again in the asychronous
udpread event.
Add a unit test that tests the isc_nm_read() call from the read
callback to receive two datagrams.
The tcp Pytest on OpenBSD fairly reliably fails when receive_tcp()
on a socket is attempted:
> (response, rtime) = dns.query.receive_tcp(sock, timeout())
tests-tcp.py:50:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/dns/query.py:659: in receive_tcp
ldata = _net_read(sock, 2, expiration)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sock = <socket.socket [closed] fd=-1, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6>
count = 2, expiration = 1662719959.8106785
def _net_read(sock, count, expiration):
"""Read the specified number of bytes from sock. Keep trying until we
either get the desired amount, or we hit EOF.
A Timeout exception will be raised if the operation is not completed
by the expiration time.
"""
s = b''
while count > 0:
try:
> n = sock.recv(count)
E socket.timeout: timed out
This is because the socket is already closed.
Bump the socket connection timeout to 10 seconds.
An assertion failure would be triggered when the TCP connection
is canceled during sending the data back to the client.
Don't require the state to be `RECV` on non successful read to
gracefully handle canceled TCP connection during the SEND state of the
HTTPD channel.
sd_notify() may be called by a service to notify the service manager
about state changes. It can be used to send arbitrary information,
encoded in an environment-block-like string. Most importantly, it can be
used for start-up completion notification.
Add libsystemd check to autoconf script and when the library is detected
add calls to sd_notify() around the server->reload_status changes.
Co-authored-by: Petr Špaček <pspacek@isc.org>
If there was a collision of key id across algorithms it was not
possible to determine where counter applies to which algorithm for
xml statistics while for json only one of the values was emitted.
The key names are now "<algorithm-number>+<id>" (e.g. "8+54274").
the 'resolve' binary was added for testing dns_client as part of
the export library. the export libraries are no longer supported,
and tests using 'delv' provide the same coverage, so 'resolve' can
be removed now.
dns_request_create() was a front-end to dns_request_createvia() that
was only used by test binaries. dns_request_createvia() has been
renamed to dns_request_create(), and the test programs that formerly
used dns_request_create() have been updated to use the new parameters.
The test expected `xn--ah-` to be treated as a syntax error (punycode
requires letters after the last hyphen) but libidn2 on buster
converted the label to `ah` instead. To avoid this bug, change the
invalid label to `xn--0000h` which translates to an out-of-range
unicode codepoint (beyond the maximum value) which is corectly
trated as invalid in older libidn2.
The change to `testsock.pl` in commit 258a896a broke the system
tests in out-of-tree builds because `ifconfig.sh.in` is not
copied to the worktree. Use `ifconfig.sh` instead.
Don't attempt to resolve DNS responses for intermediate results. This
may create multiple refreshes and can cause a crash.
One scenario is where for the query there is a CNAME and canonical
answer in cache that are both stale. This will trigger a refresh of
the RRsets because we encountered stale data and we prioritized it over
the lookup. It will trigger a refresh of both RRsets. When we start
recursing, it will detect a recursion loop because the recursion
parameters will eventually be the same. In 'dns_resolver_destroyfetch'
the sanity check fails, one of the callers did not get its event back
before trying to destroy the fetch.
Move the call to 'query_refresh_rrset' to 'ns_query_done', so that it
is only called once per client request.
Another scenario is where for the query there is a stale CNAME in the
cache that points to a record that is also in cache but not stale. This
will trigger a refresh of the RRset (because we encountered stale data
and we prioritized it over the lookup).
We mark RRsets that we add to the message with
DNS_RDATASETATTR_STALE_ADDED to prevent adding a duplicate RRset when
a stale lookup and a normal lookup conflict with each other. However,
the other non-stale RRset when following a CNAME chain will be added to
the message without setting that attribute, because it is not stale.
This is a variant of the bug in #2594. The fix covered the same crash
but for stale-answer-client-timeout > 0.
Fix this by clearing all RRsets from the message before refreshing.
This requires the refresh to happen after the query is send back to
the client.
When used with OpenSSL v3.0.0+, the `openssldh_compare()`,
`openssldh_paramcompare()`, and `openssldh_todns()` functions
fail to cleanup the used memory on some error paths.
Use `DST_RET` instead of `return`, when there is memory to be
released before returning from the functions.
when the compression buffer was reused for multiple statistics
requests, responses could grow beyond the correct size. this was
because the buffer was not cleared before reuse; compressed data
was still written to the beginning of the buffer, but then the size
of used region was increased by the amount written, rather than set
to the amount written. this caused responses to grow larger and
larger, potentially reading past the end of the allocated buffer.
Limit the amount of database lookups that can be triggered in
fctx_getaddresses() (i.e. when determining the name server addresses to
query next) by setting a hard limit on the number of NS RRs processed
for any delegation encountered. Without any limit in place, named can
be forced to perform large amounts of database lookups per each query
received, which severely impacts resolver performance.
The limit used (20) is an arbitrary value that is considered to be big
enough for any sane DNS delegation.
2022-09-08 11:11:30 +02:00
1235 changed files with 54477 additions and 51313 deletions
- test -z "${BUILD_CONTRIB}" || for DIR in contrib/dlz/modules/*; do test -f "${DIR}/Makefile" && CFLAGS="${CFLAGS} -Werror" make -C "${DIR}"; done
- test -z "${RUN_MAKE_INSTALL}" || make DESTDIR="${INSTALL_PATH}" install
- test -z "${RUN_MAKE_INSTALL}" -o -z "${BUILD_CONTRIB}" || for DIR in contrib/dlz/modules/*; do test -f "${DIR}/Makefile" && make -C "${DIR}" DESTDIR="${INSTALL_PATH}" install; done
- test -z "${RUN_MAKE_INSTALL}" || DESTDIR="${INSTALL_PATH}" sh util/check-make-install
- if [[ "${CFLAGS}" == *"-fsanitize=address"* ]]; then ( ! grep -F AddressSanitizer config.log ); fi
- test -z "${CROSS_COMPILATION}" || grep -F -A 1 "checking whether we are cross compiling" config.log | grep -q "result.*yes"
- [ ] Ensure the merge request from the previous step is reviewed by SWENG staff and has no outstanding discussions
- [ ] Ensure the documentation changes introduced by the merge request addressing the problem are reviewed by Support and Marketing staff
- [ ] Prepare backports of the merge request addressing the problem for all affected (and still maintained) BIND branches (backporting might affect the issue's scope and/or description)
- [ ] Prepare a standalone patch for the last stable release of each affected (and still maintained) BIND branch
:bulb: **Click [here][checklist_explanations] (internal resource) for general information about the security incident handling process.**
- [ ] Create/update the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle
- [ ] Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined
- [ ] Ensure the merge requests containing CVE fixes are merged into `security-*` branches in CVE identifier order
### Earlier Than T-5
### Post-disclosure actions
- [ ] [:link:][step_deputy] **(IM)** Pick a Deputy Incident Manager
- [ ] [:link:][step_respond] **(IM)** Respond to the bug reporter
- [ ] [:link:][step_etherpad] **(IM)** Create an Etherpad for post-mortem
- [ ] [:link:][step_public_mrs] **(SwEng)** Ensure there are no public merge requests which inadvertently disclose the issue
- [ ] [:link:][step_assign_cve_id] **(IM)** Assign a CVE identifier
- [ ] [:link:][step_note_cve_info] **(SwEng)** Update this issue with the assigned CVE identifier and the CVSS score
- [ ] [:link:][step_versions_affected] **(SwEng)** Determine the range of product versions affected (including the Subscription Edition)
- [ ] [:link:][step_workarounds] **(SwEng)** Determine whether workarounds for the problem exist
- [ ] [:link:][step_coordinate] **(SwEng)** If necessary, coordinate with other parties
- [ ] [:link:][step_earliest] **(Support)** Prepare and send out "earliest" notifications
- [ ] [:link:][step_advisory_mr] **(Support)** Create a merge request for the Security Advisory and include all readily available information in it
- [ ] [:link:][step_reproducer_mr] **(SwEng)** Prepare a private merge request containing a system test reproducing the problem
- [ ] [:link:][step_notify_support] **(SwEng)** Notify Support when a reproducer is ready
- [ ] [:link:][step_code_analysis] **(SwEng)** Prepare a detailed explanation of the code flow triggering the problem
- [ ] [:link:][step_fix_mr] **(SwEng)** Prepare a private merge request with the fix
- [ ] [:link:][step_review_fix] **(SwEng)** Ensure the merge request with the fix is reviewed and has no outstanding discussions
- [ ] [:link:][step_review_docs] **(Support)** Review the documentation changes introduced by the merge request with the fix
- [ ] [:link:][step_backports] **(SwEng)** Prepare backports of the merge request addressing the problem for all affected (and still maintained) branches of a given product
- [ ] [:link:][step_finish_advisory] **(Support)** Finish preparing the Security Advisory
- [ ] [:link:][step_meta_issue] **(QA)** Create (or update) the private issue containing links to fixes & reproducers for all CVEs fixed in a given release cycle
- [ ] [:link:][step_changes] **(QA)** (BIND 9 only) Reserve a block of `CHANGES` placeholders once the complete set of vulnerabilities fixed in a given release cycle is determined
- [ ] [:link:][step_merge_fixes] **(QA)** Merge the CVE fixes in CVE identifier order
- [ ] [:link:][step_patches] **(QA)** Prepare a standalone patch for the last stable release of each affected (and still maintained) product branch
- [ ] [:link:][step_asn_releases] **(QA)** Prepare ASN releases (as outlined in the Release Checklist)
- [ ] Merge a regression test reproducing the bug into all affected (and still maintained) BIND branches
### At T-5
- [ ] [:link:][step_send_asn] **(Support)** Send ASN to eligible customers
- [ ] [:link:][step_preannouncement] **(Support)** (BIND 9 only) Send a pre-announcement email to the *bind-announce* mailing list to alert users that the upcoming release will include security fixes
### At T-4
- [ ] [:link:][step_verify_asn] **(Support)** Verify that all ASN-eligible customers have received the notification email
### At T-1
- [ ] [:link:][step_check_customers] **(Support)** Verify that any new or reinstated customers have received the notification email
- [ ] [:link:][step_packager_emails] **(First IM)** Send notifications to OS packagers
### On the Day of Public Disclosure
- [ ] [:link:][step_clearance] **(IM)** Grant Support clearance to proceed with public release
- [ ] [:link:][step_publish] **(Support)** Publish the releases (as outlined in the release checklist)
- [ ] [:link:][step_matrix] **(Support)** (BIND 9 only) Update vulnerability matrix in the Knowledge Base
- [ ] [:link:][step_publish_advisory] **(Support)** Bump Document Version for the Security Advisory and publish it in the Knowledge Base
- [ ] [:link:][step_notifications] **(First IM)** Send notification emails to third parties
- [ ] [:link:][step_mitre] **(First IM)** Advise MITRE about the disclosed CVEs
- [ ]***(Support)*** Pre-publish ASN and/or Subscription Edition tarballs so that packages can be built.
- [ ]***(QA)*** Build and test ASN and/or Subscription Edition packages.
- [ ]***(QA)*** Prepare the `patches/` subdirectory for each security release (if applicable).
- [ ]***(QA)*** Notify Support that the releases have been prepared.
- [ ]***(Support)*** Send out ASNs (if applicable).
@@ -86,7 +88,8 @@
- [ ]***(QA)*** Merge published release tags (non-linearly) back into the their relevant development/maintenance branches.
- [ ]***(QA)*** Sanitize confidential issues which are assigned to the current release milestone and do not describe a security vulnerability, then make them public.
- [ ]***(QA)*** Sanitize confidential issues which are assigned to older release milestones and describe security vulnerabilities, then make them public if appropriate[^2].
- [ ]***(QA)*** Update QA tools used in GitLab CI (e.g. Flake8, PyLint) by modifying the relevant `Dockerfile`.
- [ ]***(QA)*** Update QA tools used in GitLab CI (e.g. Black, PyLint, Sphinx) by modifying the relevant `Dockerfile`.
- [ ]***(QA)*** Run a pipeline to rebuild all [images](https://gitlab.isc.org/isc-projects/images) used in GitLab CI.
[^1]: If not, use the time remaining until the tagging deadline to ensure all outstanding issues are either resolved or moved to a different milestone.
[^2]: As a rule of thumb, security vulnerabilities which have reproducers merged to the public repository are considered okay for full disclosure.
| `-DOPENSSL_API_COMPAT=10100` | Build using the deprecated OpenSSL APIs so that the `engine` API is available when building with OpenSSL 3.0.0 for PKCS#11 support |
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Tony Finch
Michał Kępień
Ondřej Surý
Michal Nowak
Matthijs Mekking
Mark AndrewsandMatthijs Mekking
Arаm Sаrgsyаn
Evan Hunt
Tom Krizek
Tony FinchandTony Finch
Ondřej SurýandOndřej Surý
Petr MenšíkandMark Andrews
Petr ŠpačekandOndřej Surý
Timo Teräs
Artem Boldariev
Artem Boldariev