Copyright (C) 2000, 2001, 2004, 2010, 2011, 2013, 2015-2017 Internet Systems Consortium, Inc. ("ISC")
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
Introduction
===
This directory holds a simple test environment for running bind9 system tests
involving multiple name servers.
With the exception of "common" (which holds configuration information common to
multiple tests) and "win32" (which holds files needed to run the tests in a
Windows environment), each directory holds a set of scripts and configuration
files to test different parts of BIND. The directories are named for the
aspect of BIND they test, for example:
dnssec/ DNSSEC tests
forward/ Forwarding tests
glue/ Glue handling tests
etc.
Typically each set of tests sets up 2-5 name servers and then performs one or
more tests against them. Within the test subdirectory, each name server has a
separate subdirectory containing its configuration data. These subdirectories
are named "nsN" or "ansN" (where N is a number between 1 and 8, e.g. ns1, ans2
etc.)
The tests are completely self-contained and do not require access to the real
DNS. Generally, one of the test servers (usually ns1) is set up as a root name
server and is listed in the hints file of the others.
Preparing to Run the Tests
===
To enable all servers to run on the same machine, they bind to separate virtual
IP addresses on the loopback interface. ns1 runs on 10.53.0.1, ns2 on
10.53.0.2, etc. Before running any tests, you must set up these addresses by
running the command
sh ifconfig.sh up
as root. The interfaces can be removed by executing the command:
sh ifconfig.sh down
... also as root.
The servers use unprivileged ports (above 1024) instead of the usual port 53,
so they can be run without root privileges once the interfaces have been set
up.
Note for MacOS Users
---
If you wish to make the interfaces survive across reboots, copy
org.isc.bind.system and org.isc.bind.system.plist to /Library/LaunchDaemons
then run
launchctl load /Library/LaunchDaemons/org.isc.bind.system.plist
... as root.
Running the Tests
===
The tests can be run individually using the following command:
sh run.sh [flags] <test-name> [<test-arguments>]
e.g.
sh run.sh [flags] notify
Optional flags are:
-p <number> Sets the range of ports used by the test. A block of 100
ports is available for each test, the number given to the
"-p" switch being the number of the start of that block
(e.g. "-p 7900" will mean that the test is able to use
ports 7900 through 7999). If not specified, the test will
have ports 5000 to 5099 available to it.
-n Noclean - do not remove the output files if the test
completes successfully. By default, files created by the
test are deleted if it passes; they are not deleted if the
test fails.
-k Keep servers running after the test completes. Each test
usually starts a number of nameservers, either instances
of the "named" being tested, or custom servers (written in
Python or Perl) that feature test-specific behavior. The
servers are automatically started before the test is run
and stopped after it ends. This flag leaves them running
at the end of the test, so that additional queries can be
sent by hand. To stop the servers afterwards, use the
command "sh stop.sh <test-name>".
-d <arg> Arguments to the "date" command used to produce the
start and end time of the tests. For example, the
switch
-d "+%Y-%m-%d:%h:%M:%s"
would cause the "S" and "E" messages (see below) to have
the date looking like "2017-11-23:16:06:32" instead of the
default "Thu, 23 Nov 2017 16:06:32 +0000".
-r The "runall" flag. This is related to cleaning up after
all the tests. (See "Mainenance Notes" below.)
Arguments are:
test-name Mandatory. The name of the test, which is the name of the
subdirectory in bin/tests/system holding the test files.
test-arguments Optional arguments that are passed to each of the test's
scripts.
To run all the system tests, type either:
sh runall.sh [numproc]
or
make [-j numproc] test
When running all the tests, the output is sent to the file systests.output
(in the bin/tests/system) directory.
The "numproc" option specifies the maximum number of tests that can run in
parallel. The default is 1, which means that all of the tests run
sequentially. If greater than 1, up to "numproc" tests will run simultaneously,
new tests being started as tests finish. Each test will get a unique set of
ports, so there is no danger of tests interfering with one another.
It is also possible to run the test suites from two installations of BIND on
the same machine at the same time. To do this:
1. Each installation must have its own directory tree. The system tests create
files in the test directories, so separate directory trees are required to
avoid interference between the same test running in the different
installations.
2. For one of the test suites, the starting port number must be specified by
setting the environment variable STARTPORT before starting the test suite.
Each test suite comprises about 100 tests, each being allocated a set of 100
ports. The port ranges are allocated in sequence, so each test suite requires
about 10,000 ports to itself. By default, the port allocation starts at 5,000.
So the following set of commands:
cd <installation-1>/bin/tests/system
sh run.sh 4
cd <installation-2>/bin/tests/system
STARTPORT=20000 sh run.sh 4
... will start the test suite for installation-1 using the default base port
of 5,000, so the test suite will use ports 5,000 through 15,000 (or there
abouts). The use of "STARTPORT=20000" to prefix the run of the test suite for
installation-2 will mean the test suite uses ports 20,000 through 30,000 or so.
Parallel running will reduce the total time taken to run the BIND system tests,
but will mean that the output from all the tests pointed to the screen will be
mixed up with one another. However, the systests.output file produced at the
end of the run will contain the output from each test, one after the other.
Format of Test Output
---
All output from the system tests is in the form of lines with the following
structure:
<letter>:<test-name>:<message> [(<number>)]
e.g.
I:catz:checking that dom1.example is not served by master (1)
The meanings of the fields are as follows:
<letter>
This indicates the type of message. This is one of:
S Start of the test
A Start of test (retained for backwards compatibility)
T Start of test (retained for backwards compatibility)
E End of the test
I Information. A test will typically output many of these messages
during its run, indicating test progress. Note that such a message
may be of the form "I:testname:failed", indicating that a sub-test
has failed.
R Result. Each test will reult in one such message, which is of the
form:
R:<test-name>:<result>
where <result> is one of:
PASS The test passed
FAIL The test failed
SKIPPED The test was not run, usually because some
prerequisites required to run the test are missing.
UNTESTED The test was not run for some other reason, e.g.
a prerequiste is available but is not compatible with
the platform on which the test is run.
<test-name>
This is the name of the test from which the message emanated, which is also
the name of the subdirectory holding the test files.
<message>
This is text output by the test during its execution.
(<number>)
If present, this will correlate with a file created by the test. The tests
execute commands and route the output of each command to a file. The name
of this file depends on the command and the test, but will usually be of
the form:
<command>.out.<suffix><number>
e.g. nsupdate.out.test28, dig.out.q3. This aids diagnosis of problems by
allowing the output that caused the problem to be identified.
Re-running the Tests
===
If there is a requirement to re-run a test (or the entire test suite), the
files produced by the tests should be deleted first.
Deletion of files produced by an individual test can be done with the
command:
sh clean.sh <test-name>
Deletion of the files produced by the set of tests (e.g. after the execution
of "runall.sh") can be deleted by the command:
make testclean
(Note that the Makefile has two other targets for cleaning up files: "clean"
will delete all the files produced by the tests, as well as the object and
executable files used by the tests. "distclean" does all the work of "clean"
as well as deleting configuration files produced by "configure".)
Developer Notes
===
This section is intended for developers writing new tests.
Overview
---
As noted above, each test suite is in a separate directory. To interact with
the test framework, the directories contain the following standard files:
prereq.sh Run at the beginning to determine whether the test can be run at
all; if not, we see a result of R:SKIPPED or R:UNTESTED. This file
is optional: if not present, the test is assumed to have all its
prerequisties met.
setup.sh Run after prereq.sh, this sets up the preconditions for the tests.
Although optional, virtually all tests will require such a file to
set up the ports they should use for the test.
tests.sh Runs the actual tests.
clean.sh Run at the end to clean up temporary files, but only if the test
was completed successfully and its running was not inhibited by the
"-n" switch being passed to "run.sh". Otherwise the temporary files
are left in place for inspection.
ns<N> These subdirectories contain test name servers that can be queried
or can interact with each other. the value of N indicates the
address the server listens on: for example, ns2 listens on
10.53.0.2, and ns4 on 10.53.0.4. All test servers use an
unprivileged port, so they don't need to run as root. These servers
log at the highest debug level and the log is captured in the file
"named.run".
ans<N> Like ns[X], but these are simple mock name servers implemented in
Perl or Python. They are generally programmed to misbehave in ways
named would not so as to exercise named's ability to interoperate
with badly behaved name servers.
Port Usage
---
In order for the tests to run in parallel, each test requires a unique set of
ports. These are specified by the "-p" option passed to "run.sh", which sets
environment variables that the scripts listed above can reference.
The convention used in the system tests is that the number passed is the start
of a range of 10 ports. The test is free to use the ports as required,
although present usage is that the lowest port is used as the query port and
the highest is used as the control port. This is reinforced by the environment
variables set by run.sh. These are:
PORT Number to be used for the query port.
CONTROLPORT Number to be used as the RNDC control port.
EXTRAPORT1 - EXTRAPORT8 Eight port numbers that can be use as needed.
Two other environment variables are defined:
LOWPORT The lowest port number in the range.
HIGHPORT The highest port number in the range.
When running tests in paralel (i.e. giving a value of "numproc" greater than 1
in the "make" or "runall.sh" commands listed above), it is guaranteed that each
test will get a set of unique port numbers.
Writing a Test
---
The test framework requires up to four shell scripts (as well as a number of
nameserver instances) to run. Certain expectations are put on each script:
General
---
1. Each of the four scripts will be invoked with the command
(cd <test-directory> ; sh <script> [<arguments>] )
... so that working directory when the script starts executing is the test
directory.
2. Arguments can be only passed to the script if the test is being run as a
one-off with "run.sh". In this case, everything on the command line after the
name of the test is passed to each script. For example, the command:
sh run.sh -p 12300 mytest -D xyz
... will run "mytest" with a port range of 12300 to 12399. Each of the
framework scripts provided by the test will be invoked using the remaining
arguments, e.g.
(cd mytest ; sh prereq.sh -D xyz)
(cd mytest ; sh setup.sh -D xyz)
(cd mytest ; sh tests.sh -D xyz)
(cd mytest ; sh clean.sh -D xyz)
No arguments will be passed to the test scripts if the test is run as part of
a run of the full test suite (e.g. the tests are started with "runall.sh").
3. Each script should start with the following lines:
SYSTEMTESTTOP=..
. $SYSTEMTESTTOP/conf.sh
"conf.sh" defines a series of environment variables together with functions
useful for the test scripts.
prereq.sh
---
As noted above, this is optional. If present, it should check whether specific
software needed to run the test is available and/or whether BIND has been
configured with the appropriate options required.
* If the software required to run the test is present and the BIND
configure options are correct, prereq.sh should return with a status code
of 0.
* If the software required to run the test is not available and/or BIND
has not been configured with the appropriate options, prereq.sh should
return with a status code of 1.
* If there is some other problem (e.g. prerequistie software is available
but is not properly configured), a status code of 255 should be returned.
setup.sh
---
This is responsible for setting up the configuration files used in the test.
To cope with the varying port number, ports are not hard-coded into
configuration files (or, for that matter, scripts that emulate nameservers).
Instead, setup.sh is responsible for editing the configuration files to set the
port numbers.
To do this, configuration files should be supplied in the form of templates
containing tokens identifying ports. The tokens have the same name as the
shell variables listed above, but in upper-case and prefixed and suffixed by
the "@" symbol. For example, a fragment of a configuration file template might
look like:
controls {
inet 10.53.0.1 port @CONTROLPORT@ allow { any; } keys { rndc_key; };
};
options {
query-source address 10.53.0.1;
notify-source 10.53.0.1;
transfer-source 10.53.0.1;
port @PORT@;
allow-new-zones yes;
};
setup.sh should copy the template to the desired filename using the
"copy_setports" shell function defined in "conf.sh", i.e.
copy_setports ns1/named.conf.in ns1/named.conf
This replaces the tokens @PORT@, @CONTROLPORT@, @EXTRAPORT1@ through
@EXTRAPORT8@ with the contents of the shell variables listed above. setup.sh
should do this for all configuration files required when the test starts.
tests.sh
---
This is the main test file and the contents depend on the test. The contents
are completely up to the developer, although most test scripts have a form
similar to the following for each test:
1. n=`expr $n + 1`
2. echo_i "prime cache nodata.example ($n)"
3. ret=0
4. $DIG -p ${port} @10.53.0.1 nodata.example TXT > dig.out.test$n
5. grep "status: NOERROR" dig.out.test$n > /dev/null || ret=1
6. grep "ANSWER: 0," dig.out.test$n > /dev/null || ret=1
7. if [ $ret != 0 ]; then echo_i "failed"; fi
8. status=`expr $status + $ret`
1. Increment the test number "n" (initialized to zero at the start of the
script).
2. Indicate that the sub-test is about to begin. Note that "echo_i" instead
of "echo" is used. echo_i is a function defined in "conf.sh" which will
prefix the message with "I:<testname>:", so allowing the output from each
test to be identified within the output. The test number is included in the
message in order to tie the sub-test with its output.
3. Initialize return status.
4 - 6. Carry out the sub-test. In this case, a nameserver is queried (note
that the port used is given by the "port" shell variable, which was set by
the inclusion the file "getopts.sh" at the start of the script). The
output is routed to a file whose suffix includes the test number. The
response from the server is examined and in this case, if the required
string is not found, an error is indicated by setting "ret" to 1.
7. If the sub-test failed, a mesage is printed.
8. "status", used to track whether any of the sub-tests have failed, is
incremented accordingly. The value of "status" determines the status
returned by "tests.sh", which in turn determines whether the framework
prints the PASS or FAIL message.
Regardless of this, rules that should be followed are:
a. Use the variables produced by getopts.sh to determine the ports to use for
sending and receiving queries.
b. Store all output produced by queries/commands into files.
c. Use a counter to tag messages and to associate the messages with the output
files.
d. Use "echo_i" to output informational messages.
clean.sh
---
The inverse of "setup.sh", this is invoked by the framework to clean up the
test directory. It should delete all files that have been created by the test
during its run.A
Adding a Test to the System Test Suite
---
Once a set of tests has been created, the following files should be edited:
* conf.sh.in The name of the test should be added to the PARALLELDIRS
variable.
* Makefile.in The name of the test should be added to the PARALLEL variable.
(It is likely that a future iteration of the system test suite will remove the
need to edit two files to add a test.)
Valgrind
---
When running system tests, named can be run under Valgrind. The output from
Valgrind are sent to per-process files that can be reviewed after the test has
completed. To enable this, set the USE_VALGRIND environment variable to
"helgrind" to run the Helgrind tool, or any other value to run the Memcheck
tool. To use "helgrind" effectively, build BIND with --disable-atomic.
Maintenance Notes
===
This section is aimed at developers maintaining BIND's system test framework.
Notes on Parallel Execution
---
Although execution of an individual test is controlled by "run.sh", which
executes the above shell scripts (and starts the relevant servers) for each
test, the running of all tests in the test suite is controlled by the Makefile.
("runall.sh" does little more than invoke "make" on the Makefile.)
All system tests are capable of being run in parallel. For this to work,
each test needs to use a unique set of ports. To avoid the need to define
which tests use which ports (and so risk port clashes as further tests are
added), the ports are assigned when the tests are run. This is achieved by
having the "test" target in the Makefile depend on "parallel.mk". This file
is created when "make check" is run, and contains a target for each test of
the form:
<test-name>:
@$(SHELL) run.sh -r -p <baseport> <test-name>
The <baseport> is unique and the values of <baseport> for each test are
separated by at least 100 ports.
Cleaning up from Tests
---
When a test is run, files are produced in the test directory by both the test
itself and the test framework.
The former comprises the the logs from the running of nameservers, the output
from "dig" and "rndc" commands used in the test, and anything else that the
test developer feels is relevant. In the event of a test failure, they can be
useful in tracking down the problem.
At the moment, there is only one file in the latter category, the file
test.output. This is created when the test is run as part of a run of the
test suite (e.g. via "runall.sh") and comprises a copy of the output sent to
stdout. (If the test is running in parallel with other tests, the output sent
to stdout will be interleaved with their output. The output in test.output
will comprise solely the output from this test.)
If the test is run by itself with a direct invocation of "run.sh", the disposal
of the output depends on the test. If it fails, all output is retained. If it
succeeds, "run.sh" calls the framework's "clean.sh". This deletes the test's
test.output file then calls the test's own "clean.sh" to clean up the
test-specific files.
The clean-up path is different if the test is run as part of the test suite.
In this case, although the test is ultimately run by a call to "run.sh",
"run.sh" is invoked with the "-r" switch (the "runall" flag). Should the test
fail, all files are retained as before. If the test passes, the framework's
"clean.sh" with the "-r" switch. This causes it to only call the test's
"clean.sh", leaving the test.output file untouched. When all tests in the
testsuite have been run, the test suite code invokes "testsummary.sh". This
iterates through all test directory, concatenating the "test.output" file to a
single "systests.output", deleting "test.output" as it does so.
ec063cdc99