When opening a code or closing it, index of the current node is
immediately added/cleared (if needed) from all DBs.
This makes materialization easier: adding a value in builtin mode (for
example) can be directly added into the running mode (if needed) w/o
having to worry about the indexes maintainance.
To support materialization, each value stored inside cfgmgr must have an
extra flag telling in which mode the value was inserted/updated (i.e.
was it in builtin mode, user mode or running mode).
Instead of playing with pointers arithmetic to squeeze this extra info
in the value being serialized/deserialized into LMDB, uses isc_buffer_t.
This commit enable to
- have all the properties of a node grouped together so it is possible
to list all properties of a node without lineary lookup through all
the keys until the find one is found. A node's property key has the
following syntax:
p<node prefix 32 bits hash><node prefix><property name>
So listing all propertes of a node consist to do a MDB_SET_RANGE on
"p<node prefix 32 bits hash><node prefix> then checking the first 5
bytes are equal to "v<node prefix 32bits" instead of doing a string
compare to the whole prefix, and linerary lookup the next LMDB value
(as key of properties are then groupped together).
- Similarly, the name of all direct sub-nodes of a node N1 are "indexed"
by having the following key,values as the following syntax:
i<N1 prefix hash><sub-node prefix>,<sub-node name>
So listing all sub-nodes of a node is as simple as listing all
properties of a node, only needs a 5 bytes comparaison (instead of
whole string compare of the prefix) for each entry and avoid lineray
LMDB lookups to find the next sub-node.
isc_cfgmgr_foreachnodes/foreachproperties enable to list each direct
sub-nodes/properties form within a node.
This is needed in general, and will make it easier to have a dump API
which also will be needed to make materialization testing easier.
The implementation is not great because it's likely slow (lot of LMDB
lookups and string manipulation). But at least it enables to expose the
API so we can built up on top of it, and come back to a better
implementation later.
Builtin and inheritance code are removed. The way it will work now will
be specific to options, view and zone clauses. Materialization will be
used. In otherwords, 3 versions of the config co-exists:
- the builtin (the one hard-coded)
- the user (the one coming from named.conf)
- the running
The running configuration is a copy of the builtin one, and on top of
that, a copy of the user one overriding values already existing from the
builtin one. Then, each modifications applied override it.
In order to keep things simple initially, the next changes will be:
- removal of lists: the API feels brittle, it's non deterministic order
(might be problematic), it enforces a size of values to 511 bytes, and
it's very specific to LMDB implementation. The better approach will
need to be found later.
- removal of "repeatable clauses" this concepts was to be able to
iterate over views and zones, but it forces to have random IDs
generated for clause names, which make materialization over-complex.
(i.e. the ID for a clause "x" in the builtin wouldn't be the same in
user or running, and would force some cotortions).
Short term: removal of newclause/nextclause, this will also break
validation. This will need to be addressed. Probably by having an extra
LMDB DB which tells the list of top-level keys from a given prefix, so
we would be able to quickly jump from one clause to another.
cfgmgr can dump its configuration. It needs to have a configuration
format (passed to isc_cfgmgr_init, also used by the validator) and a
isc_cfgmgr_dumpcfg_t struct which contains various function pointers to
dump the various clauses and properties.
Add validator support. The validator takes a "tree"-ish structure
describing the configuration model, composed of clauses (repeatable or
not) and properties (mandatory or not, as well as their types).
The validator walk in this structure and check for each clause if it
exists, if it is repeated (and allowed or not), if the mandatory
properties are present and what's their type.
What is not handled by the validator (and probably okay) are
non-described properties and non-described clauses (so it is possible to
add more which is not in the configuration file).
Add unit test for builtin mode logic as well as few tidy-up in the code
(adding some assertions and API documentation), in particular, to make
it clear that builtin-mode is tight to an opened clause, so it must be
set to a currently opened clause.
In order to implement a generic inheritance mechanism of values,
isc_cfgmgr_getval attempts to lookup for the given name in the parent
clauses if the value is not found in the current clause.
This also introduce the notion of "builtin" mode, which enable to prefix
the builtin configuration (bin/named/config.c) with "default.", and
handle it as a specific case when the parent is a top-level clause,
effectively enabling to also support inheritance of options values to
the builtin-ones.
Environment variable NAMED_CFGMGR_PERSIST being set enable to keep LMDB
DB created by cfgmgr. For now usecase is only for debugging (i.e.
inspecting what has been actually written in the DB).
- pack the cfgmgr type enum, so it likely takes 1 byte rather than 4
bytes
- when putting a string in LMDB, let's allocate only the size of type +
strlen(string) + 1, instead of the size of the whole cfgmgr_val_t +
strlen(string) + 1. It needs a special threatment when reading, but
probably worth it considering cfgmgr_val_t is not tiny (and will
likely grows up)
- revert previous changes from configure.ac script in order to re-use
the existing logic, as well as the gitlab ci flag which check build
with two combinations.
- Conditionally build LMDB source code based on HAVE_LMDB. So if
--without-lmdb is used, HAVE_LMDB won't be set and LMDB source code
will be compiled and included in libisc. This enable code using LMDB
to link no matter LMDB is OS-provided or not.
- Regarding the LMDB headers, and in order to make future LMDB update
easier, all original OpenLDAP code is wrapped under
lib/isc/openldap-lmdb including the headers. This avoid extra
changes (putting aside liscence header) to mdb.c to find out its
headers. The counter-part is the relative inclusion of this header
from isc/lmdb.h header, but it's probably a good idea to not make
this header public anyway.
- update LMDB license to add BIND9 SPDX headers but mentioning LMDB
lisence
As several places uses custom LMDB macros, this centralize those
definitions inside a dedicated `lib/isc/include/isc/lmdb.h` header
file, so future uses cases not DNS-centrics could fit into there as
well.
MDB_CREATE is removed as well because:
- it was useless (it's needed only for named DB, which is not the case
for all use case so far)
- it was used in `mdb_env_open` which is not expecting this flag: it
should be passed (if needed) to `mdb_dbi_open`. It was likely
ignored so far.
Few points to note (and possibly discuss):
- cfgmgr is build on top of LMDB, as it brings transaction (and so
thread safe) support out of the box
- LMDB keys are build in a way that we can support repeatable (see
https://pad.isc.org/p/cfgmgr-proposal-v2 even if most of it is
outdated, the section 2.0.2 about the way it's build is still
relevant, otherwise I hope the code changes here are clear enough)
- Each thread own its own cfgmgr context (which basically means LMDB
transaction, and where the API points to inside the configuration)
- In order to avoid allocations everytime we get/set a value (as well
as few other operations) a single buffer is pre-allocated per-thread
and per-transaction. Now, few internal helper functions directly use
it (instead or, let's say, work on a parameter) and this might be
confusing and error prone, I'm happy to change that if it is a
worry.
- The data type which can be read/wrote from cfgmgr is not exhaustive,
more data type will be added (i.e. duration type, uint64 if needed,
etc.)
- This current implementation does not support inheritance (i.e. a
non-specified view option won't use the option one). It's something
we need to discuss, I see some options that could be put on top of
that's here.
- The "default" values, however, should be fine out-of-the-box: I
think the default configuration in bin/named/config.c can be
parse/"added" in cfgmgr first then user one on top of that (because
writting an existing value override it). Obviously there would be
no-way to "go back" to the default config only, but I don't see such
use case in existing code. Even if we'd need such thing, that would
be quite invasive and a full config reload would be advisable. (so
we could add an API to entirely drop the cfgmgr data and start from
scratch. But I don't see such use case right now anyway, so it's not
there)
- I hope the things a user of cfgmgr must know should be clearly
explained in the cfgmgr.h file (if it's not the case, then I need
to fix it -- That said I'll likely re-work the doc anyway).
- I initially implemented a mechanism which would dynamically re-size
key buffers in case keys are very long, but I remove this as LMDB
doesn't supports key more than 511 bytes anyway. Instead I made
assertions every time we build a key to make sure we don't exceed
this value.
Instead of relying on unreliable order of execution of the library
constructors and destructors, move them to individual binaries. The
advantage is that the execution time and order will remain constant and
will not depend on the dynamic load dependency solver.
Merge branch 'ondrej/move-the-constructors-destructors-to-binaries' into 'main'
See merge request isc-projects/bind9!10069
As the default_call_rcu_thread can't be forced to flush all the work
during the executable shutdown, create one call_rcu_thread explicitly
and assign it to the all created threads.
This allows this explicit call_rcu_thread to be unassociated from the
main thread and freed before the executable destructor exits.
Instead of relying on unreliable order of execution of the library
constructors and destructors, move them to individual binaries. The
advantage is that the execution time and order will remain constant and
will not depend on the dynamic load dependency solver.
This requires more work, but that was mitigated by a simple requirement,
any executable using libisc and libdns, must include <isc/lib.h> and
<dns/lib.h> respectively (in this particular order). In turn, these two
headers must not be included from within any library as they contain
inlined functions marked with constructor/destructor attributes.
Previously, active resolver fetches were only dumped when the `fetches-per-zone` configuration option was enabled. Now, active resolver fetches are dumped along with the number of `clients-per-server` counters per resolver fetch.
Merge branch 'ondrej/make-dns_resolver_dumpfetches-dump-fetches' into 'main'
See merge request isc-projects/bind9!10107
Previously, the dns_resolver_dumpfetches() would go over the fetch
counters. Alas, because of the earlier optimization, the fetch counters
would be increased only when fetches-per-zone was not 0, otherwise the
whole counting was skipped for performance reasons.
Instead of using the auxiliary fetch counters hash table, use the real
hash table that stores the fetch contexts to dump the ongoing fetches to
the recursing file.
Additionally print more information about the fetch context like start
and expiry times, number of fetch responses, number of queries and count
of allowed and dropped fetches.
Previously, a data race could cause a newly created fetch context for a new client to be used
before it had been fully initialized, which would cause the query to become stuck; queries for the same
data would be either paused indefinitely or dropped because of
the `clients-per-query` limit. This has been fixed.
Closes#5053
Merge branch '5053-fetch-context-create-data-race' into 'main'
See merge request isc-projects/bind9!10146
The order of the fetch context hash table rwlock and the individual
fetch context was reversed when calling the release_fctx() function.
This was causing a problem when iterating the hash table, and thus the
ordering has been corrected in a way that the hash table rwlock is now
always locked on the outside and the fctx lock is the interior lock.
In the next commit, we need to know whether the timer has been started
or stopped. Add isc_timer_running() function that returns true if the
timer has been started.
Update Sphinx-related Python packages to their current versions pulled
in by "pip install sphinx-rtd-theme" run in a fresh virtual environment.
Merge branch 'michal/update-sphinx-and-sphinx_rtd_theme' into 'main'
See merge request isc-projects/bind9!10138
Colin Vidal
Evan HuntandColin Vidal
Ondřej Surý
Michał Kępień