isc_cfgmgr_foreachnodes/foreachproperties enable to list each direct
sub-nodes/properties form within a node.
This is needed in general, and will make it easier to have a dump API
which also will be needed to make materialization testing easier.
The implementation is not great because it's likely slow (lot of LMDB
lookups and string manipulation). But at least it enables to expose the
API so we can built up on top of it, and come back to a better
implementation later.
Builtin and inheritance code are removed. The way it will work now will
be specific to options, view and zone clauses. Materialization will be
used. In otherwords, 3 versions of the config co-exists:
- the builtin (the one hard-coded)
- the user (the one coming from named.conf)
- the running
The running configuration is a copy of the builtin one, and on top of
that, a copy of the user one overriding values already existing from the
builtin one. Then, each modifications applied override it.
In order to keep things simple initially, the next changes will be:
- removal of lists: the API feels brittle, it's non deterministic order
(might be problematic), it enforces a size of values to 511 bytes, and
it's very specific to LMDB implementation. The better approach will
need to be found later.
- removal of "repeatable clauses" this concepts was to be able to
iterate over views and zones, but it forces to have random IDs
generated for clause names, which make materialization over-complex.
(i.e. the ID for a clause "x" in the builtin wouldn't be the same in
user or running, and would force some cotortions).
Short term: removal of newclause/nextclause, this will also break
validation. This will need to be addressed. Probably by having an extra
LMDB DB which tells the list of top-level keys from a given prefix, so
we would be able to quickly jump from one clause to another.
cfgmgr can dump its configuration. It needs to have a configuration
format (passed to isc_cfgmgr_init, also used by the validator) and a
isc_cfgmgr_dumpcfg_t struct which contains various function pointers to
dump the various clauses and properties.
Add validator support. The validator takes a "tree"-ish structure
describing the configuration model, composed of clauses (repeatable or
not) and properties (mandatory or not, as well as their types).
The validator walk in this structure and check for each clause if it
exists, if it is repeated (and allowed or not), if the mandatory
properties are present and what's their type.
What is not handled by the validator (and probably okay) are
non-described properties and non-described clauses (so it is possible to
add more which is not in the configuration file).
Add unit test for builtin mode logic as well as few tidy-up in the code
(adding some assertions and API documentation), in particular, to make
it clear that builtin-mode is tight to an opened clause, so it must be
set to a currently opened clause.
In order to implement a generic inheritance mechanism of values,
isc_cfgmgr_getval attempts to lookup for the given name in the parent
clauses if the value is not found in the current clause.
This also introduce the notion of "builtin" mode, which enable to prefix
the builtin configuration (bin/named/config.c) with "default.", and
handle it as a specific case when the parent is a top-level clause,
effectively enabling to also support inheritance of options values to
the builtin-ones.
Environment variable NAMED_CFGMGR_PERSIST being set enable to keep LMDB
DB created by cfgmgr. For now usecase is only for debugging (i.e.
inspecting what has been actually written in the DB).
- pack the cfgmgr type enum, so it likely takes 1 byte rather than 4
bytes
- when putting a string in LMDB, let's allocate only the size of type +
strlen(string) + 1, instead of the size of the whole cfgmgr_val_t +
strlen(string) + 1. It needs a special threatment when reading, but
probably worth it considering cfgmgr_val_t is not tiny (and will
likely grows up)
- revert previous changes from configure.ac script in order to re-use
the existing logic, as well as the gitlab ci flag which check build
with two combinations.
- Conditionally build LMDB source code based on HAVE_LMDB. So if
--without-lmdb is used, HAVE_LMDB won't be set and LMDB source code
will be compiled and included in libisc. This enable code using LMDB
to link no matter LMDB is OS-provided or not.
- Regarding the LMDB headers, and in order to make future LMDB update
easier, all original OpenLDAP code is wrapped under
lib/isc/openldap-lmdb including the headers. This avoid extra
changes (putting aside liscence header) to mdb.c to find out its
headers. The counter-part is the relative inclusion of this header
from isc/lmdb.h header, but it's probably a good idea to not make
this header public anyway.
- update LMDB license to add BIND9 SPDX headers but mentioning LMDB
lisence
As several places uses custom LMDB macros, this centralize those
definitions inside a dedicated `lib/isc/include/isc/lmdb.h` header
file, so future uses cases not DNS-centrics could fit into there as
well.
MDB_CREATE is removed as well because:
- it was useless (it's needed only for named DB, which is not the case
for all use case so far)
- it was used in `mdb_env_open` which is not expecting this flag: it
should be passed (if needed) to `mdb_dbi_open`. It was likely
ignored so far.
Few points to note (and possibly discuss):
- cfgmgr is build on top of LMDB, as it brings transaction (and so
thread safe) support out of the box
- LMDB keys are build in a way that we can support repeatable (see
https://pad.isc.org/p/cfgmgr-proposal-v2 even if most of it is
outdated, the section 2.0.2 about the way it's build is still
relevant, otherwise I hope the code changes here are clear enough)
- Each thread own its own cfgmgr context (which basically means LMDB
transaction, and where the API points to inside the configuration)
- In order to avoid allocations everytime we get/set a value (as well
as few other operations) a single buffer is pre-allocated per-thread
and per-transaction. Now, few internal helper functions directly use
it (instead or, let's say, work on a parameter) and this might be
confusing and error prone, I'm happy to change that if it is a
worry.
- The data type which can be read/wrote from cfgmgr is not exhaustive,
more data type will be added (i.e. duration type, uint64 if needed,
etc.)
- This current implementation does not support inheritance (i.e. a
non-specified view option won't use the option one). It's something
we need to discuss, I see some options that could be put on top of
that's here.
- The "default" values, however, should be fine out-of-the-box: I
think the default configuration in bin/named/config.c can be
parse/"added" in cfgmgr first then user one on top of that (because
writting an existing value override it). Obviously there would be
no-way to "go back" to the default config only, but I don't see such
use case in existing code. Even if we'd need such thing, that would
be quite invasive and a full config reload would be advisable. (so
we could add an API to entirely drop the cfgmgr data and start from
scratch. But I don't see such use case right now anyway, so it's not
there)
- I hope the things a user of cfgmgr must know should be clearly
explained in the cfgmgr.h file (if it's not the case, then I need
to fix it -- That said I'll likely re-work the doc anyway).
- I initially implemented a mechanism which would dynamically re-size
key buffers in case keys are very long, but I remove this as LMDB
doesn't supports key more than 511 bytes anyway. Instead I made
assertions every time we build a key to make sure we don't exceed
this value.
Instead of relying on unreliable order of execution of the library
constructors and destructors, move them to individual binaries. The
advantage is that the execution time and order will remain constant and
will not depend on the dynamic load dependency solver.
Merge branch 'ondrej/move-the-constructors-destructors-to-binaries' into 'main'
See merge request isc-projects/bind9!10069
As the default_call_rcu_thread can't be forced to flush all the work
during the executable shutdown, create one call_rcu_thread explicitly
and assign it to the all created threads.
This allows this explicit call_rcu_thread to be unassociated from the
main thread and freed before the executable destructor exits.
Instead of relying on unreliable order of execution of the library
constructors and destructors, move them to individual binaries. The
advantage is that the execution time and order will remain constant and
will not depend on the dynamic load dependency solver.
This requires more work, but that was mitigated by a simple requirement,
any executable using libisc and libdns, must include <isc/lib.h> and
<dns/lib.h> respectively (in this particular order). In turn, these two
headers must not be included from within any library as they contain
inlined functions marked with constructor/destructor attributes.
Previously, active resolver fetches were only dumped when the `fetches-per-zone` configuration option was enabled. Now, active resolver fetches are dumped along with the number of `clients-per-server` counters per resolver fetch.
Merge branch 'ondrej/make-dns_resolver_dumpfetches-dump-fetches' into 'main'
See merge request isc-projects/bind9!10107
Previously, the dns_resolver_dumpfetches() would go over the fetch
counters. Alas, because of the earlier optimization, the fetch counters
would be increased only when fetches-per-zone was not 0, otherwise the
whole counting was skipped for performance reasons.
Instead of using the auxiliary fetch counters hash table, use the real
hash table that stores the fetch contexts to dump the ongoing fetches to
the recursing file.
Additionally print more information about the fetch context like start
and expiry times, number of fetch responses, number of queries and count
of allowed and dropped fetches.
Previously, a data race could cause a newly created fetch context for a new client to be used
before it had been fully initialized, which would cause the query to become stuck; queries for the same
data would be either paused indefinitely or dropped because of
the `clients-per-query` limit. This has been fixed.
Closes#5053
Merge branch '5053-fetch-context-create-data-race' into 'main'
See merge request isc-projects/bind9!10146
The order of the fetch context hash table rwlock and the individual
fetch context was reversed when calling the release_fctx() function.
This was causing a problem when iterating the hash table, and thus the
ordering has been corrected in a way that the hash table rwlock is now
always locked on the outside and the fctx lock is the interior lock.
In the next commit, we need to know whether the timer has been started
or stopped. Add isc_timer_running() function that returns true if the
timer has been started.
Update Sphinx-related Python packages to their current versions pulled
in by "pip install sphinx-rtd-theme" run in a fresh virtual environment.
Merge branch 'michal/update-sphinx-and-sphinx_rtd_theme' into 'main'
See merge request isc-projects/bind9!10138
With RPZ in use, `named` could terminate unexpectedly because of a race condition when a reconfiguration command was received using `rndc`. This has been fixed.
Closes#5146
Merge branch '5146-rpz-reconfig-bug-fix' into 'main'
See merge request isc-projects/bind9!10079
After a reconfiguration the old view can be left without a valid
'rpzs' member, because when the RPZ is not changed during the named
reconfiguration 'rpzs' "migrate" from the old view into the new
view, so when a query resumes it can find that 'qctx->view->rpzs'
is NULL which query_resume() currently doesn't expect to happen if
it's recursing and 'qctx->rpz_st' is not NULL.
Fix the issue by adding a NULL-check. In order to not split the log
message to two different log messages depending on whether
'qctx->view->rpzs' is NULL or not, change the message to not log
the RPZ policy's "version" which is just a runtime counter and is
most likely not very useful for the users.
The isc_counter_create() doesn't need the return value (it was always
ISC_R_SUCCESS), use the macros to implement the reference counting,
little style cleanup, and expand the unit test.
Merge branch 'ondrej/cleanup-isc_counter-unit' into 'main'
See merge request isc-projects/bind9!10126
The isc_counter_create() doesn't need the return value (it was always
ISC_R_SUCCESS), use the macros to implement the reference counting,
little style cleanup, and expand the unit test.
Previously, when parsing responses, named incorrectly rejected responses without matching RRSIG records for NSEC/DS/NSEC3 records in the authority section. This rejection, if appropriate, should have been left for the validator to determine and has been fixed.
Closes#5185
Merge branch '5185-remove-rrsig-check-from-dns_message_parse' into 'main'
See merge request isc-projects/bind9!10125
Checking whether the authority section is properly signed should
be left to the validator. Checking in getsection (dns_message_parse)
was way too early and resulted in resolution failures of lookups
that should have otherwise succeeded.
Previously a hard-coded limitation of maximum two key or message
verification checks were introduced when checking the message's
SIG(0) signature. It was done in order to protect against possible
DoS attacks. The logic behind choosing the number 2 was that more
than a single key should only be required during key rotations, and
in that case two keys are enough. But later it became apparent that
there are other use cases too where even more keys are required, see
issue number #5050 in GitLab.
This change introduces two new configuration options for the views,
`sig0key-checks-limit` and `sig0message-checks-limit`, which define how
many keys are allowed to be checked to find a matching key, and how
many message verifications are allowed to take place once a matching
key has been found. The latter protects against expensive cryptographic
operations when there are keys with colliding tags and algorithm
numbers, with default being 2, and the former protects against a bit
less expensive key parsing operations and defaults to 16.
Closes#5050
Merge branch '5050-sig0-let-considering-more-than-two-keys' into 'main'
See merge request isc-projects/bind9!9967
Colin Vidal
Evan HuntandColin Vidal
Ondřej Surý
Michał Kępień
Arаm Sаrgsyаn
Mark Andrews