From 28bae02a96860c9fcbde780b26990f1a9a749f83 Mon Sep 17 00:00:00 2001
From: Michal Nowak <mnowak@isc.org>
Date: Wed, 16 Aug 2023 16:19:30 +0200
Subject: [PATCH] Report hung system tests

At times, a problem might occur where a test is not responding,
especially in the CI, determining the specific test responsible can be
difficult. Fortunately, when running tests with the pytest runner,
pytest sets the PYTEST_CURRENT_TEST environment variable to the current
test nodeid and stage. Afterward, the variable can be examined to
identify the test that has stopped responding.

The monitoring script needs to be started in the background. Still, the
shell executor used for BSD and FIPS testing can't handle the background
process cleanly, and the script step will wait for the background
process for the entire duration of the background process (currently
3000 seconds). Therefore, run the monitoring script only when the Docker
executor is used where this is not a problem.

(cherry picked from commit 35792b1700b404b2eac7f0c60f6dfbf9a6bb3e31)
---
 .gitlab-ci.yml                   |  2 ++
 util/get-running-system-tests.py | 25 +++++++++++++++++++++++++
 2 files changed, 27 insertions(+)
 create mode 100755 util/get-running-system-tests.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index adab5116a5..5fc7be6e3a 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -404,6 +404,8 @@ cross-version-config-tests:
     - *setup_interfaces
   script:
     - *find_pytest
+    - *find_python
+    - ( if [ "${CI_DISPOSABLE_ENVIRONMENT}" = "true" ]; then sleep 3000; "$PYTHON" "${CI_PROJECT_DIR}/util/get-running-system-tests.py"; fi ) &
     - cd bin/tests/system
     - >
       "$PYTEST" --junit-xml="$CI_PROJECT_DIR"/junit.xml -n "$TEST_PARALLEL_JOBS" | tee pytest.out.txt
diff --git a/util/get-running-system-tests.py b/util/get-running-system-tests.py
new file mode 100755
index 0000000000..1653e0a66d
--- /dev/null
+++ b/util/get-running-system-tests.py
@@ -0,0 +1,25 @@
+#!/usr/bin/env python3
+
+# Copyright (C) Internet Systems Consortium, Inc. ("ISC")
+#
+# SPDX-License-Identifier: MPL-2.0
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0.  If a copy of the MPL was not distributed with this
+# file, you can obtain one at https://mozilla.org/MPL/2.0/.
+#
+# See the COPYRIGHT file distributed with this work for additional
+# information regarding copyright ownership.
+
+import psutil
+
+for pid in psutil.pids():
+    try:
+        environ = psutil.Process(pid).environ()
+        if "PYTEST_CURRENT_TEST" in environ:
+            name = psutil.Process(pid).name()
+            print(
+                f'pytest process {name}/{pid} running: {environ["PYTEST_CURRENT_TEST"]}'
+            )
+    except (psutil.AccessDenied, psutil.NoSuchProcess):
+        pass