Builtin and inheritance code are removed. The way it will work now will
be specific to options, view and zone clauses. Materialization will be
used. In otherwords, 3 versions of the config co-exists:
- the builtin (the one hard-coded)
- the user (the one coming from named.conf)
- the running
The running configuration is a copy of the builtin one, and on top of
that, a copy of the user one overriding values already existing from the
builtin one. Then, each modifications applied override it.
In order to keep things simple initially, the next changes will be:
- removal of lists: the API feels brittle, it's non deterministic order
(might be problematic), it enforces a size of values to 511 bytes, and
it's very specific to LMDB implementation. The better approach will
need to be found later.
- removal of "repeatable clauses" this concepts was to be able to
iterate over views and zones, but it forces to have random IDs
generated for clause names, which make materialization over-complex.
(i.e. the ID for a clause "x" in the builtin wouldn't be the same in
user or running, and would force some cotortions).
Short term: removal of newclause/nextclause, this will also break
validation. This will need to be addressed. Probably by having an extra
LMDB DB which tells the list of top-level keys from a given prefix, so
we would be able to quickly jump from one clause to another.
cfgmgr can dump its configuration. It needs to have a configuration
format (passed to isc_cfgmgr_init, also used by the validator) and a
isc_cfgmgr_dumpcfg_t struct which contains various function pointers to
dump the various clauses and properties.
Add validator support. The validator takes a "tree"-ish structure
describing the configuration model, composed of clauses (repeatable or
not) and properties (mandatory or not, as well as their types).
The validator walk in this structure and check for each clause if it
exists, if it is repeated (and allowed or not), if the mandatory
properties are present and what's their type.
What is not handled by the validator (and probably okay) are
non-described properties and non-described clauses (so it is possible to
add more which is not in the configuration file).
Add unit test for builtin mode logic as well as few tidy-up in the code
(adding some assertions and API documentation), in particular, to make
it clear that builtin-mode is tight to an opened clause, so it must be
set to a currently opened clause.
In order to implement a generic inheritance mechanism of values,
isc_cfgmgr_getval attempts to lookup for the given name in the parent
clauses if the value is not found in the current clause.
This also introduce the notion of "builtin" mode, which enable to prefix
the builtin configuration (bin/named/config.c) with "default.", and
handle it as a specific case when the parent is a top-level clause,
effectively enabling to also support inheritance of options values to
the builtin-ones.
Environment variable NAMED_CFGMGR_PERSIST being set enable to keep LMDB
DB created by cfgmgr. For now usecase is only for debugging (i.e.
inspecting what has been actually written in the DB).
- pack the cfgmgr type enum, so it likely takes 1 byte rather than 4
bytes
- when putting a string in LMDB, let's allocate only the size of type +
strlen(string) + 1, instead of the size of the whole cfgmgr_val_t +
strlen(string) + 1. It needs a special threatment when reading, but
probably worth it considering cfgmgr_val_t is not tiny (and will
likely grows up)
- revert previous changes from configure.ac script in order to re-use
the existing logic, as well as the gitlab ci flag which check build
with two combinations.
- Conditionally build LMDB source code based on HAVE_LMDB. So if
--without-lmdb is used, HAVE_LMDB won't be set and LMDB source code
will be compiled and included in libisc. This enable code using LMDB
to link no matter LMDB is OS-provided or not.
- Regarding the LMDB headers, and in order to make future LMDB update
easier, all original OpenLDAP code is wrapped under
lib/isc/openldap-lmdb including the headers. This avoid extra
changes (putting aside liscence header) to mdb.c to find out its
headers. The counter-part is the relative inclusion of this header
from isc/lmdb.h header, but it's probably a good idea to not make
this header public anyway.
- update LMDB license to add BIND9 SPDX headers but mentioning LMDB
lisence
As several places uses custom LMDB macros, this centralize those
definitions inside a dedicated `lib/isc/include/isc/lmdb.h` header
file, so future uses cases not DNS-centrics could fit into there as
well.
MDB_CREATE is removed as well because:
- it was useless (it's needed only for named DB, which is not the case
for all use case so far)
- it was used in `mdb_env_open` which is not expecting this flag: it
should be passed (if needed) to `mdb_dbi_open`. It was likely
ignored so far.
As the default_call_rcu_thread can't be forced to flush all the work
during the executable shutdown, create one call_rcu_thread explicitly
and assign it to the all created threads.
This allows this explicit call_rcu_thread to be unassociated from the
main thread and freed before the executable destructor exits.
Instead of relying on unreliable order of execution of the library
constructors and destructors, move them to individual binaries. The
advantage is that the execution time and order will remain constant and
will not depend on the dynamic load dependency solver.
This requires more work, but that was mitigated by a simple requirement,
any executable using libisc and libdns, must include <isc/lib.h> and
<dns/lib.h> respectively (in this particular order). In turn, these two
headers must not be included from within any library as they contain
inlined functions marked with constructor/destructor attributes.
In the next commit, we need to know whether the timer has been started
or stopped. Add isc_timer_running() function that returns true if the
timer has been started.
The isc_counter_create() doesn't need the return value (it was always
ISC_R_SUCCESS), use the macros to implement the reference counting,
little style cleanup, and expand the unit test.
Running jobs which were entered into the isc_quota queue is the
responsibility of the isc_quota_release() function, which, when
releasing a previously acquired quota, checks whether the queue
is empty, and if it's not, it runs a job from the queue without touching
the 'quota->used' counter. This mechanism is susceptible to a possible
hangup of a newly queued job in case when between the time a decision
has been made to queue it (because used >= max) and the time it was
actually queued, the last quota was released. Since there is no more
quotas to be released (unless arriving in the future), the newly
entered job will be stuck in the queue.
Fix the wrong memory ordering for 'quota->used', as the relaxed
ordering doesn't ensure that data modifications made by one thread
are visible in other threads.
Add checks in both isc_quota_release() and isc_quota_acquire_cb()
to make sure that the described hangup does not happen. Also see
code comments.
The value returned by http_send_outgoing() is not used anywhere, so we
make it not return anything (void). Probably it is an omission from
older times.
When handling outgoing data, there were a couple of rarely executed
code paths that would not take into account that the callback MUST be
called.
It could lead to potential memory leaks and consequent shutdown hangs.
This commit changes the way how the number of active HTTP streams is
calculated and allows it to scale with the values of the maximum
amount of streams per connection, instead of effectively capping at
STREAM_CLIENTS_PER_CONN.
The original limit, which is intended to define the pipelining limit
for TCP/DoT. However, it appeared to be too restrictive for DoH, as it
works quite differently and implements pipelining at protocol level by
the means of multiplexing multiple streams. That renders each stream
to be effectively a separate connection from the point of view of the
rest of the codebase.
Previously we would limit the amount of incoming data to process based
solely on the presence of not completed send requests. That worked,
however, it was found to severely degrade performance in certain
cases, as was revealed during extended testing.
Now we switch to keeping track of how much data is in flight (or ready
to be in flight) and limit the amount of processed incoming data when
the amount of in flight data surpasses the given threshold, similarly
to like we do in other transports.
This commit does several changes to isc_symtab:
1. Rewrite the isc_symtab to internally use isc_hashmap instead of
hand-stiched hashtable.
2. Create a new isc_symtab_define_and_return() api, which returns
the already defined symvalue on ISC_R_EXISTS; this allows users
of the API to skip the isc_symtab_lookup()+isc_symtab_define()
calls and directly call isc_symtab_define_and_return().
3. Merge isccc_symtab into isc_symtab - the only missing function
was isccc_symtab_foreach() that was merged into isc_symtab API.
4. Add full set of unit tests for the isc_symtab API.
Instead of having many node_lock_count * sizeof(<member>) arrays, pack
all the members into a qpcache_bucket_t struct that is cacheline aligned
and have a single array of those.
Additionaly, make both the head and the tail of isc_queue_t padded, not
just the head, to prevent false sharing of the lock-free structure with
the lock that follows it.
ISCCC_R_SYNTAX, ISCCC_R_EXPIRED, and ISCCC_R_CLOCKSKEW have the
same usage and text formats as DNS_R_SYNTAX, DNS_R_EXPIRED and
DNS_R_CLOCKSCREW respectively. this was originally done because
result codes were defined in separate libraries, and some tool
might be linked with libisccc but not libdns. as the result codes
are now defined in only one place, there's no need to retain the
duplicates.
the isc_mem allocation functions can no longer fail; as a result,
ISC_R_NOMEMORY is now rarely used: only when an external library
such as libjson-c or libfstrm could return NULL. (even in
these cases, arguably we should assert rather than returning
ISC_R_NOMEMORY.)
code and comments that mentioned ISC_R_NOMEMORY have been
cleaned up, and the following functions have been changed to
type void, since (in most cases) the only value they could
return was ISC_R_SUCCESS:
- dns_dns64_create()
- dns_dyndb_create()
- dns_ipkeylist_resize()
- dns_kasp_create()
- dns_kasp_key_create()
- dns_keystore_create()
- dns_order_create()
- dns_order_add()
- dns_peerlist_new()
- dns_tkeyctx_create()
- dns_view_create()
- dns_zone_setorigin()
- dns_zone_setfile()
- dns_zone_setstream()
- dns_zone_getdbtype()
- dns_zone_setjournal()
- dns_zone_setkeydirectory()
- isc_lex_openstream()
- isc_portset_create()
- isc_symtab_create()
(the exception is dns_view_create(), which could have returned
other error codes in the event of a crypto library failure when
calling isc_file_sanitize(), but that should be a RUNTIME_CHECK
anyway.)
We started using isc_nm_bad_request() more actively throughout
codebase. In the case of HTTP/2 it can lead to a large count of
useless "Bad Request" messages in the BIND log, as often we attempt to
send such request over effectively finished HTTP/2 sessions.
This commit fixes that.
A call to isc_nm_read_stop() would always stop reading timer even in
manual timer control mode which was added with StreamDNS in mind. That
looks like an omission that happened due to how timers are controlled
in StreamDNS where we always stop the timer before pausing reading
anyway (see streamdns_on_complete_dnsmessage()). That would not work
well for HTTP, though, where we might want pause reading without
stopping the timer in the case we want to split incoming data into
multiple chunks to be processed independently.
I suppose that it happened due to NM refactoring in the middle of
StreamDNS development (at the time isc_nm_cancelread() and
isc_nm_pauseread() were removed), as the StreamDNS code seems to be
written as if timers are not stoping during a call to
isc_nm_read_stop().
This commit introduces manual read timer control as used by StreamDNS
and its underlying transports. Before that, DoH code would rely on the
timer control provided by TCP, which would reset the timer any time
some data arrived. Now, the timer is restarted only when a full DNS
message is processed in line with other DNS transports.
That change is required because we should not stop the timer when
reading from the network is paused due to throttling. We need a way to
drop timed-out clients, particularly those who refuse to read the data
we send.
This commit adds logic to make code better protected against clients
that send valid HTTP/2 data that is useless from a DNS server
perspective.
Firstly, it adds logic that protects against clients who send too
little useful (=DNS) data. We achieve that by adding a check that
eventually detects such clients with a nonfavorable useful to
processed data ratio after the initial grace period. The grace period
is limited to processing 128 KiB of data, which should be enough for
sending the largest possible DNS message in a GET request and then
some. This is the main safety belt that would detect even flooding
clients that initially behave well in order to fool the checks server.
Secondly, in addition to the above, we introduce additional checks to
detect outright misbehaving clients earlier:
The code will treat clients that open too many streams (50) without
sending any data for processing as flooding ones; The clients that
managed to send 1.5 KiB of data without opening a single stream or
submitting at least some DNS data will be treated as flooding ones.
Of course, the behaviour described above is nothing else but
heuristical checks, so they can never be perfect. At the same time,
they should be reasonable enough not to drop any valid clients,
realatively easy to implement, and have negligible computational
overhead.
Initially, our DNS-over-HTTP(S) implementation would try to process as
much incoming data from the network as possible. However, that might
be undesirable as we might create too many streams (each effectively
backed by a ns_client_t object). That is too forgiving as it might
overwhelm the server and trash its memory allocator, causing high CPU
and memory usage.
Instead of doing that, we resort to processing incoming data using a
chunk-by-chunk processing strategy. That is, we split data into small
chunks (currently 256 bytes) and process each of them
asynchronously. However, we can process more than one chunk at
once (up to 4 currently), given that the number of HTTP/2 streams has
not increased while processing a chunk.
That alone is not enough, though. In addition to the above, we should
limit the number of active streams: these streams for which we have
received a request and started processing it (the ones for which a
read callback was called), as it is perfectly fine to have more opened
streams than active ones. In the case we have reached or surpassed the
limit of active streams, we stop reading AND processing the data from
the remote peer. The number of active streams is effectively decreased
only when responses associated with the active streams are sent to the
remote peer.
Overall, this strategy is very similar to the one used for other
stream-based DNS transports like TCP and TLS.
Extracting the exact address that each wildcard/TCP socket is bound to
locally requires issuing the getsockname() system call, which libuv
exposes via its uv_*_getsockname() functions. This is only required for
detailed logging and comes at a noticeable performance cost, so it
should not happen by default. However, it is useful for debugging
certain problems (e.g. cryptic system test failures), so a convenient
way of enabling that behavior should exist.
Update isc_nmhandle_localaddr() so that it calls uv_*_getsockname() when
the ISC_SOCKET_DETAILS preprocessor macro is set at compile time.
Ensure proper handling of sockets that wrap other sockets.
Set the new ISC_SOCKET_DETAILS macro by default when --enable-developer
is passed to ./configure. This enables detailed logging in the system
tests run in GitLab CI without affecting performance in non-development
BIND 9 builds.
Note that setting the ISC_SOCKET_DETAILS preprocessor macro at compile
time enables all callers of isc_nmhandle_localaddr() to extract the
exact address of a given local socket, which results e.g. in dnstap
captures containing more accurate information.
Mention the new preprocessor macro in the section of the ARM that
discusses why exact socket addresses may not be logged by default.
This commit adds support for setting SNI hostnames in outgoing
connections over TLS.
Most of the changes are related to either adapting the code to accept
and extra argument in *connect() functions and a couple of changes to
the TLS Stream to actually make use of the new SNI hostname
information.
Previously, we had an ISC_CONSTEXPR macro that was expanded to either
`constexpr` or `static const`, depending on compiler support. To make
the code cleaner, move `constexpr` support detection to Autoconf; if
`constexpr` support is missing from the compiler, define `constexpr` as
`static const` in config.h.
Since BIND 9 headers are not longer public, there's no reason to keep
the ISC_LANG_BEGINDECL and ISC_LANG_ENDDECL macros to support including
them from C++ projects.
- remove obsolete DNS_LOGMODULE_RBT and DNS_LOGMODULE_RBTDB
- correct the misuse of the wrong log modules in dns/rpz.c and
dns/catz.c, and add DNS_LOGMODULE_RPZ and DNS_LOGMODULE_CATZ
to support them.
`shutdown_trigger_close_cb` is not called in the main loop since
queued events in the `loop->async_trigger`, including loop teardown
(shutdown_server) are processed first, before the `uv_close` callback
is executed..
In order to pass the information to the queued events, it is necessary
to set the flag earlier in the process and not wait for the `uv_close`
callback to trigger.
The DNS_R_MUSTBESECURE lost its meaning with removal of
dnssec-must-be-secure option, so replace the few remaining (and a bit
confusing) use of this result code with DNS_R_NOVALIDSIG.
Upstream code doesn't do regular releases, so we need to regularly
sync the code from the upstream repository. This is synchronization up
to the commit f8d0513 from Jan 29, 2024.
The root cause is the fix for CVE-2024-0760 (part 3), which resets
the TCP connection on a failed send. Specifically commit
4b7c61381f stops reading on the socket
because the TCP connection is throttling.
When the tcpdns_send_cb callback thinks about restarting reading
on the socket, this fails because the socket is a client socket.
And nsupdate is a client and is using the same netmgr code.
This commit removes the requirement that the socket must be a server
socket, allowing reading on the socket again after being throttled.
Colin Vidal
Ondřej Surý
Aram Sargsyan
Artem Boldariev
alessio
Andoni Duarte Pintado
Evan Hunt
Artem BoldarievandAndoni Duarte
Michał Kępień
Pavel BřezinaandOndřej Surý
Matthijs Mekking