From 5d2dd94cf83026c64dcead83d629e51d6bda5f8b Mon Sep 17 00:00:00 2001 From: Michal Nowak Date: Tue, 18 Jan 2022 19:39:37 +0100 Subject: [PATCH] Drop runall.sh runall.sh was mainly used on Windows and as it's support was removed from the "main" branch the script is not needed anymore. Also, remove bin/tests/system/README text on running multiple system test suites simultaneously with runall.sh as that support was not present in the script anyway. --- bin/tests/system/README | 58 ++---------------- bin/tests/system/runall.sh | 99 ------------------------------- bin/tests/system/runsequential.sh | 6 +- doc/dev/dev.md | 5 -- 4 files changed, 9 insertions(+), 159 deletions(-) delete mode 100755 bin/tests/system/runall.sh diff --git a/bin/tests/system/README b/bin/tests/system/README index 5db29de277..f4248a38d5 100644 --- a/bin/tests/system/README +++ b/bin/tests/system/README @@ -117,13 +117,7 @@ Running All The System Tests --- To run all the system tests, enter the command: - sh runall.sh [-c] [-n] [numproc] - -The optional flag "-c" forces colored output (by default system test output is -not printed in color due to run.sh being piped through "tee"). - -The optional flag "-n" has the same effect as it does for "run.sh" - it causes -the retention of all output files from all tests. + make [-j numproc] test The optional "numproc" argument specifies the maximum number of tests that can run in parallel. The default is 1, which means that all of the tests run @@ -136,13 +130,6 @@ with one another. However, the systests.output file produced at the end of the run (in the bin/tests/system directory) will contain the output from each test in sequential order. -Note that it is not possible to pass arguments to tests though the "runall.sh" -script. - -A run of all the system tests can also be initiated via make: - - make [-j numproc] test - In this case, retention of the output files after a test completes successfully is specified by setting the environment variable SYSTEMTEST_NO_CLEAN to 1 prior to running make, e.g. @@ -153,38 +140,6 @@ while setting environment variable SYSTEMTEST_FORCE_COLOR to 1 forces system test output to be printed in color. -Running Multiple System Test Suites Simultaneously ---- -In some cases it may be desirable to have multiple instances of the system test -suite running simultaneously (e.g. from different terminal windows). To do -this: - -1. Each installation must have its own directory tree. The system tests create -files in the test directories, so separate directory trees are required to -avoid interference between the same test running in the different -installations. - -2. For one of the test suites, the starting port number must be specified by -setting the environment variable STARTPORT before starting the test suite. -Each test suite comprises about 100 tests, each being allocated a set of 100 -ports. The port ranges for each test are allocated sequentially, so each test -suite requires about 10,000 ports to itself. By default, the port allocation -starts at 5,000. So the following set of commands: - - Terminal Window 1: - cd /bin/tests/system - sh runall.sh 4 - - Terminal Window 2: - cd /bin/tests/system - STARTPORT=20000 sh runall.sh 4 - -... will start the test suite for installation-1 using the default base port -of 5,000, so the test suite will use ports 5,000 through 15,000 (or there -abouts). The use of "STARTPORT=20000" to prefix the run of the test suite for -installation-2 will mean the test suite uses ports 20,000 through 30,000 or so. - - Format of Test Output --- All output from the system tests is in the form of lines with the following @@ -247,8 +202,8 @@ deleted if the test succeeds but are retained on error. The run.sh script automatically calls a given test's clean.sh script before invoking its setup.sh script. -Deletion of the files produced by the set of tests (e.g. after the execution -of "runall.sh") can be carried out using the command: +Deletion of the files produced by the set of tests (e.g. after the execution of +make) can be carried out using the command: sh cleanall.sh @@ -337,7 +292,7 @@ port assignments would be: HIGHPORT = 5299 When running tests in parallel (i.e. giving a value of "numproc" greater than 1 -in the "make" or "runall.sh" commands listed above), it is guaranteed that each +in the "make" command listed above), it is guaranteed that each test will get a set of unique port numbers. @@ -373,7 +328,7 @@ arguments, e.g.: (cd mytest ; sh clean.sh -D xyz) No arguments will be passed to the test scripts if the test is run as part of -a run of the full test suite (e.g. the tests are started with "runall.sh"). +a run of the full test suite (e.g. the tests are started with make). 3. Each script should start with the following lines: @@ -673,7 +628,6 @@ Notes on Parallel Execution Although execution of an individual test is controlled by "run.sh", which executes the above shell scripts (and starts the relevant servers) for each test, the running of all tests in the test suite is controlled by the Makefile. -("runall.sh" does little more than invoke "make" on the Makefile.) All system tests are capable of being run in parallel. For this to work, each test needs to use a unique set of ports. To avoid the need to define which @@ -701,7 +655,7 @@ abnormally, e.g. core files, PID files etc., are stored in the test directory. 3. A file "test.output." containing the text written to stdout by the test is written to bin/tests/system/. This file is only produced when the test -is run as part of the entire test suite (e.g. via "runall.sh"). +is run as part of the entire test suite (e.g. via make). If the test fails, all these files are retained. But if the test succeeds, they are cleaned up at different times: diff --git a/bin/tests/system/runall.sh b/bin/tests/system/runall.sh deleted file mode 100755 index a3bd54b26c..0000000000 --- a/bin/tests/system/runall.sh +++ /dev/null @@ -1,99 +0,0 @@ -#!/bin/sh - -# Copyright (C) Internet Systems Consortium, Inc. ("ISC") -# -# SPDX-License-Identifier: MPL-2.0 -# -# This Source Code Form is subject to the terms of the Mozilla Public -# License, v. 2.0. If a copy of the MPL was not distributed with this -# file, you can obtain one at https://mozilla.org/MPL/2.0/. -# -# See the COPYRIGHT file distributed with this work for additional -# information regarding copyright ownership. - -# Run all the system tests. -# -# Usage: -# runall.sh [-c] [-n] [numprocesses] -# -# -c Force colored output. -# -# -n Noclean. Keep all output files produced by all tests. These -# can later be removed by running "cleanall.sh". -# -# numprocess Number of concurrent processes to use when running the tests. -# The default is one, which causes the tests to run sequentially. - -. ./conf.sh - -usage="Usage: ./runall.sh [-c] [-n] [numprocesses]" - -# Preserve values of environment variables which are already set. - -SYSTEMTEST_FORCE_COLOR=${SYSTEMTEST_FORCE_COLOR:-0} -SYSTEMTEST_NO_CLEAN=${SYSTEMTEST_NO_CLEAN:-0} - -# Handle command line switches if present. - -while getopts "cn-" flag; do - case "$flag" in - c) SYSTEMTEST_FORCE_COLOR=1 ;; - n) SYSTEMTEST_NO_CLEAN=1 ;; - -) break;; - *) exit 1;; - esac -done -export NOCLEAN -shift $((OPTIND-1)) - -# Obtain number of processes to use. - -if [ $# -eq 0 ]; then - numproc=1 -elif [ $# -eq 1 ]; then - if [ "$1" -ne "$1" ] 2>&1; then - # Value passed is not numeric - echo "$usage" >&2 - exit 1 - fi - numproc=$1 -else - echo "$usage" >&2 - exit 1 -fi - -# Run the tests. - -export SYSTEMTEST_FORCE_COLOR -export SYSTEMTEST_NO_CLEAN - -status=0 - -if [ "$NOPARALLEL" = "" ]; then - # use "make" to run tests in parallel. - make -j "$numproc" check - status=$? -else - # the NOPARALLEL environment variable indicates that tests must be - # run sequentially. - $PERL testsock.pl || { - cat <<-EOF - I:NOTE: System tests were skipped because they require the - I: test IP addresses 10.53.0.* to be configured as alias - I: addresses on the loopback interface. Please run - I: "bin/tests/system/ifconfig.sh up" as root to configure them. - EOF - exit 1 - } - ( - status=0 - for testdir in $SUBDIRS; do - $SHELL run.sh -r "$testdir" || status=1 - done - echo "$status" > systests.status - ) 2>&1 | tee "systests.output" - read -r status < systests.status - rm systests.status -fi - -exit "$status" diff --git a/bin/tests/system/runsequential.sh b/bin/tests/system/runsequential.sh index 4b99bd09b9..b3c4b00e7c 100755 --- a/bin/tests/system/runsequential.sh +++ b/bin/tests/system/runsequential.sh @@ -13,9 +13,9 @@ # Run system tests that must be run sequentially # -# Note: Use "make check" (or runall.sh) to run all the system tests. This -# script will just run those tests that require that each of their nameservers -# is the only one running on an IP address. +# Note: Use "make check" to run all the system tests. This script will just +# run those tests that require that each of their nameservers is the only one +# running on an IP address. # . ./conf.sh diff --git a/doc/dev/dev.md b/doc/dev/dev.md index f66284873b..1435423f4f 100644 --- a/doc/dev/dev.md +++ b/doc/dev/dev.md @@ -155,11 +155,6 @@ To run only the system tests, omitting unit tests: $ make test -Or: - - $ cd bin/tests/system - $ sh runall.sh - To run an individual system test: $ make -C bin/tests/system/ check TESTS= V=1