github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-07 00:22:43 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #11932] ollama unable to use GPU on popOS #7919

New Issue
Closed
opened 2026-04-12 20:05:07 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 20:05:07 -05:00
Owner
Copy Link

Originally created by @guusec on GitHub (Aug 16, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/11932

What is the issue?

I've tried loading and unloading the kernel module, I've installed the nvidia-cuda-toolkit package, my GPU is working fine and ollama even sees my libraries but is unable to use them. Ollama's able to run cuda 12.x so I'm at a loss here.

nvidia-smi output

Sat Aug 16 00:48:45 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4070 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8              1W /  115W |       2MiB /   8188MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Relevant log output

[giuseppe]~❯ OLLAMA_DEBUG=1 ollama serve
time=2025-08-16T00:52:05.659-04:00 level=INFO source=routes.go:1297 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/giuseppe/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:8 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy:https_proxy: no_proxy:]"
time=2025-08-16T00:52:05.661-04:00 level=INFO source=images.go:477 msg="total blobs: 0"
time=2025-08-16T00:52:05.661-04:00 level=INFO source=images.go:484 msg="total unused blobs removed: 0"
time=2025-08-16T00:52:05.661-04:00 level=INFO source=routes.go:1350 msg="Listening on 127.0.0.1:11434 (version 0.11.0)"
time=2025-08-16T00:52:05.661-04:00 level=DEBUG source=sched.go:106 msg="starting llm scheduler"
time=2025-08-16T00:52:05.661-04:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-08-16T00:52:05.663-04:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA"
time=2025-08-16T00:52:05.663-04:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so*
time=2025-08-16T00:52:05.663-04:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcuda.so* /home/giuseppe/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]"
time=2025-08-16T00:52:05.674-04:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/usr/lib/i386-linux-gnu/libcuda.so.570.172.08 /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08]"
initializing /usr/lib/i386-linux-gnu/libcuda.so.570.172.08
library /usr/lib/i386-linux-gnu/libcuda.so.570.172.08 load err: /usr/lib/i386-linux-gnu/libcuda.so.570.172.08: wrong ELF class: ELFCLASS32
time=2025-08-16T00:52:05.674-04:00 level=DEBUG source=gpu.go:609 msg="skipping 32bit library" library=/usr/lib/i386-linux-gnu/libcuda.so.570.172.08
initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08
dlsym: cuInit - 0x736c6bd0cab0
dlsym: cuDriverGetVersion - 0x736c6bd0cad0
dlsym: cuDeviceGetCount - 0x736c6bd0cb10
dlsym: cuDeviceGet - 0x736c6bd0caf0
dlsym: cuDeviceGetAttribute - 0x736c6bd0cbf0
dlsym: cuDeviceGetUuid - 0x736c6bd0cb50
dlsym: cuDeviceGetName - 0x736c6bd0cb30
dlsym: cuCtxCreate_v3 - 0x736c6bd0cdd0
dlsym: cuMemGetInfo_v2 - 0x736c6bd2d190
dlsym: cuCtxDestroy - 0x736c6bd6bae0
calling cuInit
cuInit err: 3
time=2025-08-16T00:52:05.707-04:00 level=INFO source=gpu.go:612 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08: cuda driver library init failure: 3"
time=2025-08-16T00:52:05.707-04:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so*
time=2025-08-16T00:52:05.707-04:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /home/giuseppe/libcudart.so* /usr/local/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]"
time=2025-08-16T00:52:05.709-04:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.8.90 /usr/lib/x86_64-linux-gnu/libcudart.so.12.0.146]"
cudaSetDevice err: 3
time=2025-08-16T00:52:05.734-04:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.8.90: cudart init failure: 3"
cudaSetDevice err: 3
time=2025-08-16T00:52:05.759-04:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcudart.so.12.0.146: cudart init failure: 3"
time=2025-08-16T00:52:05.759-04:00 level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu"
time=2025-08-16T00:52:05.759-04:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-08-16T00:52:05.759-04:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="31.2 GiB" available="24.4 GiB"

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

0.11.0

Originally created by @guusec on GitHub (Aug 16, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/11932 ### What is the issue? I've tried loading and unloading the kernel module, I've installed the nvidia-cuda-toolkit package, my GPU is working fine and ollama even sees my libraries but is unable to use them. Ollama's able to run cuda 12.x so I'm at a loss here. nvidia-smi output ``` Sat Aug 16 00:48:45 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 4070 ... Off | 00000000:01:00.0 Off | N/A | | N/A 47C P8 1W / 115W | 2MiB / 8188MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` ### Relevant log output ```shell [giuseppe]~❯ OLLAMA_DEBUG=1 ollama serve time=2025-08-16T00:52:05.659-04:00 level=INFO source=routes.go:1297 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:DEBUG OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/home/giuseppe/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:8 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy:https_proxy: no_proxy:]" time=2025-08-16T00:52:05.661-04:00 level=INFO source=images.go:477 msg="total blobs: 0" time=2025-08-16T00:52:05.661-04:00 level=INFO source=images.go:484 msg="total unused blobs removed: 0" time=2025-08-16T00:52:05.661-04:00 level=INFO source=routes.go:1350 msg="Listening on 127.0.0.1:11434 (version 0.11.0)" time=2025-08-16T00:52:05.661-04:00 level=DEBUG source=sched.go:106 msg="starting llm scheduler" time=2025-08-16T00:52:05.661-04:00 level=INFO source=gpu.go:217 msg="looking for compatible GPUs" time=2025-08-16T00:52:05.663-04:00 level=DEBUG source=gpu.go:98 msg="searching for GPU discovery libraries for NVIDIA" time=2025-08-16T00:52:05.663-04:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcuda.so* time=2025-08-16T00:52:05.663-04:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcuda.so* /home/giuseppe/libcuda.so* /usr/local/cuda*/targets/*/lib/libcuda.so* /usr/lib/*-linux-gnu/nvidia/current/libcuda.so* /usr/lib/*-linux-gnu/libcuda.so* /usr/lib/wsl/lib/libcuda.so* /usr/lib/wsl/drivers/*/libcuda.so* /opt/cuda/lib*/libcuda.so* /usr/local/cuda/lib*/libcuda.so* /usr/lib*/libcuda.so* /usr/local/lib*/libcuda.so*]" time=2025-08-16T00:52:05.674-04:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/usr/lib/i386-linux-gnu/libcuda.so.570.172.08 /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08]" initializing /usr/lib/i386-linux-gnu/libcuda.so.570.172.08 library /usr/lib/i386-linux-gnu/libcuda.so.570.172.08 load err: /usr/lib/i386-linux-gnu/libcuda.so.570.172.08: wrong ELF class: ELFCLASS32 time=2025-08-16T00:52:05.674-04:00 level=DEBUG source=gpu.go:609 msg="skipping 32bit library" library=/usr/lib/i386-linux-gnu/libcuda.so.570.172.08 initializing /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08 dlsym: cuInit - 0x736c6bd0cab0 dlsym: cuDriverGetVersion - 0x736c6bd0cad0 dlsym: cuDeviceGetCount - 0x736c6bd0cb10 dlsym: cuDeviceGet - 0x736c6bd0caf0 dlsym: cuDeviceGetAttribute - 0x736c6bd0cbf0 dlsym: cuDeviceGetUuid - 0x736c6bd0cb50 dlsym: cuDeviceGetName - 0x736c6bd0cb30 dlsym: cuCtxCreate_v3 - 0x736c6bd0cdd0 dlsym: cuMemGetInfo_v2 - 0x736c6bd2d190 dlsym: cuCtxDestroy - 0x736c6bd6bae0 calling cuInit cuInit err: 3 time=2025-08-16T00:52:05.707-04:00 level=INFO source=gpu.go:612 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcuda.so.570.172.08: cuda driver library init failure: 3" time=2025-08-16T00:52:05.707-04:00 level=DEBUG source=gpu.go:501 msg="Searching for GPU library" name=libcudart.so* time=2025-08-16T00:52:05.707-04:00 level=DEBUG source=gpu.go:525 msg="gpu library search" globs="[/usr/local/lib/ollama/libcudart.so* /home/giuseppe/libcudart.so* /usr/local/lib/ollama/cuda_v*/libcudart.so* /usr/local/cuda/lib64/libcudart.so* /usr/lib/x86_64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/x86_64-linux-gnu/libcudart.so* /usr/lib/wsl/lib/libcudart.so* /usr/lib/wsl/drivers/*/libcudart.so* /opt/cuda/lib64/libcudart.so* /usr/local/cuda*/targets/aarch64-linux/lib/libcudart.so* /usr/lib/aarch64-linux-gnu/nvidia/current/libcudart.so* /usr/lib/aarch64-linux-gnu/libcudart.so* /usr/local/cuda/lib*/libcudart.so* /usr/lib*/libcudart.so* /usr/local/lib*/libcudart.so*]" time=2025-08-16T00:52:05.709-04:00 level=DEBUG source=gpu.go:558 msg="discovered GPU libraries" paths="[/usr/local/lib/ollama/libcudart.so.12.8.90 /usr/lib/x86_64-linux-gnu/libcudart.so.12.0.146]" cudaSetDevice err: 3 time=2025-08-16T00:52:05.734-04:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/local/lib/ollama/libcudart.so.12.8.90: cudart init failure: 3" cudaSetDevice err: 3 time=2025-08-16T00:52:05.759-04:00 level=DEBUG source=gpu.go:574 msg="Unable to load cudart library /usr/lib/x86_64-linux-gnu/libcudart.so.12.0.146: cudart init failure: 3" time=2025-08-16T00:52:05.759-04:00 level=DEBUG source=amd_linux.go:419 msg="amdgpu driver not detected /sys/module/amdgpu" time=2025-08-16T00:52:05.759-04:00 level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered" time=2025-08-16T00:52:05.759-04:00 level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="31.2 GiB" available="24.4 GiB" ``` ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version 0.11.0
GiteaMirror added the bug label 2026-04-12 20:05:07 -05:00
GiteaMirror commented 2026-04-12 20:05:08 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 16, 2025):

cudaErrorInitializationError = 3
  The API call failed because the CUDA driver and runtime could not be initialized.

Does the driver write anything to the system log (dmesg, syslog, kern.log)?

<!-- gh-comment-id:3193618220 --> @rick-github commented on GitHub (Aug 16, 2025): ``` cudaErrorInitializationError = 3 The API call failed because the CUDA driver and runtime could not be initialized. ``` Does the driver write anything to the system log (dmesg, syslog, kern.log)?
GiteaMirror commented 2026-04-12 20:05:09 -05:00
Author
Owner
Copy Link

@guusec commented on GitHub (Aug 16, 2025):

Here's my dmesg output:

sudo dmesg|ug nvidia --no-line-number
[   18.972759] nvidia-nvlink: Nvlink Core is being initialized, major device number 509
[   18.973660] nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[   19.138392] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  570.172.08  Release Build  (dvs-builder@U22-I3-AF01-21-3)  Tue Jul  8 17:59:47 UTC 2025
[   19.146051] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[   21.551377] [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
[   21.553960] nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes

Lots of syslog output too:

2025-08-11T19:37:52.828120-04:00 localhost ollama[2322]: time=2025-08-11T19:37:52.828-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-11T19:39:30.467612-04:00 localhost pop-upgrade[5892]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-12T10:58:40.454910-04:00 localhost ollama[2294]: time=2025-08-12T10:58:40.454-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-12T11:00:27.246135-04:00 localhost pop-upgrade[5817]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-13T02:53:13.079725-04:00 localhost pop-upgrade[436619]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-13T03:02:36.129107-04:00 localhost pop-upgrade[454221]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-13T03:09:38.768730-04:00 localhost pop-upgrade[458987]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-13T03:13:00.614763-04:00 localhost ollama[2371]: time=2025-08-13T03:13:00.614-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-13T03:14:38.997322-04:00 localhost pop-upgrade[5058]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-13T03:29:21.649475-04:00 localhost pop-upgrade[32331]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-13T11:01:50.567127-04:00 localhost ollama[2400]: time=2025-08-13T11:01:50.567-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-13T11:03:24.682079-04:00 localhost pop-upgrade[5351]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-14T10:28:19.566459-04:00 localhost ollama[2392]: time=2025-08-14T10:28:19.566-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-14T10:31:34.813899-04:00 localhost pop-upgrade[3953]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-15T03:04:26.953855-04:00 localhost systemctl[91505]: Failed to enable unit: Unit file nvidia-hibernate.service does not exist.
2025-08-15T03:04:26.954282-04:00 localhost system76-power[1558]: [WARN] systemctl enable nvidia-hibernate.service: failed with exit status: 1 (not an error if service does not exist!)
2025-08-15T03:04:26.960527-04:00 localhost systemctl[91506]: Failed to enable unit: Unit file nvidia-resume.service does not exist.
2025-08-15T03:04:26.960789-04:00 localhost system76-power[1558]: [WARN] systemctl enable nvidia-resume.service: failed with exit status: 1 (not an error if service does not exist!)
2025-08-15T03:04:26.967331-04:00 localhost systemctl[91507]: Failed to enable unit: Unit file nvidia-suspend.service does not exist.
2025-08-15T03:04:26.967471-04:00 localhost system76-power[1558]: [WARN] systemctl enable nvidia-suspend.service: failed with exit status: 1 (not an error if service does not exist!)
2025-08-15T03:04:26.970311-04:00 localhost system76-power[1558]: [INFO] Enabling nvidia-fallback.service
2025-08-15T03:04:26.976515-04:00 localhost systemctl[91508]: Failed to enable unit: Unit file nvidia-fallback.service does not exist.
2025-08-15T03:05:27.794816-04:00 localhost ollama[2369]: time=2025-08-15T03:05:27.794-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-15T03:08:39.829249-04:00 localhost pop-upgrade[3865]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-15T11:50:29.181634-04:00 localhost ollama[2400]: time=2025-08-15T11:50:29.181-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77"
2025-08-15T11:52:09.699005-04:00 localhost pop-upgrade[5385]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-16T00:01:29.428097-04:00 localhost systemd[1]: Starting nvidia-hibernate.service - NVIDIA system hibernate actions...
2025-08-16T00:01:29.432457-04:00 localhost hibernate: nvidia-hibernate.service
2025-08-16T00:01:29.432600-04:00 localhost logger[82091]: <13>Aug 16 00:01:29 hibernate: nvidia-hibernate.service
2025-08-16T00:01:29.433166-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service.
2025-08-16T00:01:29.438517-04:00 localhost systemd[1]: Starting nvidia-resume.service - NVIDIA system resume actions...
2025-08-16T00:01:29.439641-04:00 localhost /usr/bin/nvidia-powerd[82092]: nvidia-powerd version:1.0(build 1)
2025-08-16T00:01:29.441267-04:00 localhost systemd[1]: Starting nvidia-suspend.service - NVIDIA system suspend actions...
2025-08-16T00:01:29.445047-04:00 localhost suspend: nvidia-resume.service
2025-08-16T00:01:29.445196-04:00 localhost logger[82095]: <13>Aug 16 00:01:29 suspend: nvidia-resume.service
2025-08-16T00:01:29.446920-04:00 localhost systemd[1]: nvidia-hibernate.service: Deactivated successfully.
2025-08-16T00:01:29.447024-04:00 localhost systemd[1]: Finished nvidia-hibernate.service - NVIDIA system hibernate actions.
2025-08-16T00:01:29.448834-04:00 localhost suspend: nvidia-suspend.service
2025-08-16T00:01:29.448914-04:00 localhost logger[82096]: <13>Aug 16 00:01:29 suspend: nvidia-suspend.service
2025-08-16T00:01:29.455438-04:00 localhost systemd[1]: nvidia-resume.service: Deactivated successfully.
2025-08-16T00:01:29.455779-04:00 localhost systemd[1]: Finished nvidia-resume.service - NVIDIA system resume actions.
2025-08-16T00:01:29.458789-04:00 localhost systemd[1]: nvidia-suspend.service: Deactivated successfully.
2025-08-16T00:01:29.458955-04:00 localhost systemd[1]: Finished nvidia-suspend.service - NVIDIA system suspend actions.
2025-08-16T00:01:29.467370-04:00 localhost /usr/bin/nvidia-powerd[82092]: Allocate client failed 38
2025-08-16T00:01:29.467414-04:00 localhost /usr/bin/nvidia-powerd[82092]: Failed to initialize RM Client
2025-08-16T00:01:29.467823-04:00 localhost systemd[1]: nvidia-powerd.service: Main process exited, code=exited, status=1/FAILURE
2025-08-16T00:01:29.467895-04:00 localhost systemd[1]: nvidia-powerd.service: Failed with result 'exit-code'.
2025-08-16T00:01:29.742795-04:00 localhost systemd[1]: Starting nvidia-hibernate.service - NVIDIA system hibernate actions...
2025-08-16T00:01:29.745377-04:00 localhost hibernate: nvidia-hibernate.service
2025-08-16T00:01:29.745455-04:00 localhost logger[82250]: <13>Aug 16 00:01:29 hibernate: nvidia-hibernate.service
2025-08-16T00:01:29.753471-04:00 localhost systemd[1]: nvidia-hibernate.service: Deactivated successfully.
2025-08-16T00:01:29.753642-04:00 localhost systemd[1]: Finished nvidia-hibernate.service - NVIDIA system hibernate actions.
2025-08-16T00:01:30.054064-04:00 localhost systemd[1]: Starting nvidia-resume.service - NVIDIA system resume actions...
2025-08-16T00:01:30.057628-04:00 localhost suspend: nvidia-resume.service
2025-08-16T00:01:30.057737-04:00 localhost logger[82393]: <13>Aug 16 00:01:30 suspend: nvidia-resume.service
2025-08-16T00:01:30.067726-04:00 localhost systemd[1]: nvidia-resume.service: Deactivated successfully.
2025-08-16T00:01:30.067829-04:00 localhost systemd[1]: Finished nvidia-resume.service - NVIDIA system resume actions.
2025-08-16T00:01:30.368646-04:00 localhost systemd[1]: Starting nvidia-suspend.service - NVIDIA system suspend actions...
2025-08-16T00:01:30.373128-04:00 localhost suspend: nvidia-suspend.service
2025-08-16T00:01:30.373194-04:00 localhost logger[82538]: <13>Aug 16 00:01:30 suspend: nvidia-suspend.service
2025-08-16T00:01:30.381251-04:00 localhost systemd[1]: nvidia-suspend.service: Deactivated successfully.
2025-08-16T00:01:30.381459-04:00 localhost systemd[1]: Finished nvidia-suspend.service - NVIDIA system suspend actions.
2025-08-16T00:02:04.274862-04:00 localhost adduser[98515]: Adding system user `nvidia-persistenced' (UID 125) ...
2025-08-16T00:02:04.276594-04:00 localhost adduser[98515]: Adding new group `nvidia-persistenced' (GID 133) ...
2025-08-16T00:02:04.292157-04:00 localhost adduser[98515]: Adding new user `nvidia-persistenced' (UID 125) with group `nvidia-persistenced' ...
2025-08-16T00:02:32.488275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:32.489328-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:32.605246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:32.605272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:32.661788-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:32.713240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:32.713251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:32.779587-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:32.830243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:32.831245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:32.894848-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:32.915088-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:32.917531-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:32.917841-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:32.917905-04:00 localhost nvidia-persistenced: Started (103877)
2025-08-16T00:02:32.964242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:32.964254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.034353-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:33.038790-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:33.038885-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:33.039136-04:00 localhost nvidia-persistenced[103873]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:33.039187-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:33.039295-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:33.039331-04:00 localhost nvidia-persistenced: Shutdown (103877)
2025-08-16T00:02:33.039850-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:33.058662-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:33.058711-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:33.096244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.096262-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.146014-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:33.195243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.196263-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.270046-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:33.292749-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:33.295334-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:33.295807-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:33.295859-04:00 localhost nvidia-persistenced: Started (103897)
2025-08-16T00:02:33.354239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.355303-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.435743-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:33.440219-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:33.440374-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:33.440774-04:00 localhost nvidia-persistenced[103893]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:33.440852-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:33.440969-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:33.441045-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:33.441098-04:00 localhost nvidia-persistenced: Shutdown (103897)
2025-08-16T00:02:33.462390-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:33.462454-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:33.493271-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.494304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.567760-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:33.628271-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.629331-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.698976-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:33.725774-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:33.728794-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:33.729043-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:33.729097-04:00 localhost nvidia-persistenced: Started (103917)
2025-08-16T00:02:33.777275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.778295-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.863294-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:33.867321-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:33.867424-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:33.867784-04:00 localhost nvidia-persistenced[103913]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:33.867850-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:33.867952-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:33.868021-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:33.869466-04:00 localhost nvidia-persistenced: Shutdown (103917)
2025-08-16T00:02:33.890653-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:33.890698-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:33.918242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:33.918254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:33.991423-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:34.042274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.043255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.115419-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:34.133033-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:34.135378-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:34.135617-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:34.135677-04:00 localhost nvidia-persistenced: Started (103937)
2025-08-16T00:02:34.180270-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.180342-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.238830-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:34.242734-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:34.242859-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:34.243258-04:00 localhost nvidia-persistenced[103933]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:34.243333-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:34.243444-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:34.243517-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:34.243567-04:00 localhost nvidia-persistenced: Shutdown (103937)
2025-08-16T00:02:34.265682-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:34.265731-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:34.296283-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.297321-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.349727-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:34.407251-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.407267-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.473833-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:34.503737-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:34.505144-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:34.506714-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:34.506764-04:00 localhost nvidia-persistenced: Started (103958)
2025-08-16T00:02:34.545283-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.546272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.620308-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:34.623442-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:34.623535-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:34.623862-04:00 localhost nvidia-persistenced[103954]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:34.623925-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:34.624010-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:34.624069-04:00 localhost nvidia-persistenced: Shutdown (103958)
2025-08-16T00:02:34.624123-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:34.641721-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:34.641873-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:34.690240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.690253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.786057-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:34.834290-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.834302-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:34.909999-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:34.916152-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:34.916321-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:34.916382-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:34.995282-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:34.995347-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.049757-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:35.105249-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.106272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.171516-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:35.226280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.227269-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.292293-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:35.298014-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:35.298094-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:35.298145-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:35.370245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.371331-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.435664-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:35.502267-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.503254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.569843-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:35.624240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.625300-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.689744-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:35.695362-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:35.695473-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:35.695531-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:35.763238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.764235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.840332-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:35.901244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:35.902318-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:35.963840-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:36.013245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.014250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.083775-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:36.088250-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:36.088327-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:36.088395-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:36.156239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.157251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.232077-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:36.292242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.293279-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.347630-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:36.404274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.405245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.474258-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:36.479591-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:36.479677-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:36.479747-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:36.551275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.552237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.608942-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:36.663241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.663252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.712690-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:36.766241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.766253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.822953-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:36.827160-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:36.827267-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:36.827330-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:36.891241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:36.891251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:36.961085-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:37.020241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.021305-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.125741-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:37.175275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.176239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.275741-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:37.280305-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:37.280381-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:37.280417-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:37.358243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.359275-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.424207-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:37.490281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.491235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.563922-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:37.623259-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.624243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.677880-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:37.682195-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:37.682288-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:37.683642-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:37.758300-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.759285-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.816595-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:37.875242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.876256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:37.926712-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:37.990261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:37.990290-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.066164-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:38.069363-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:38.069449-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:38.069492-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:38.136275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.137237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.215793-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:38.261279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.262271-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.345776-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:38.403242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.404261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.455585-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:38.460897-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:38.460992-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:38.461024-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:38.527270-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.527304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.576483-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:38.642245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.643245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.701353-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:38.758242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.759300-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.824682-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:38.827063-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:38.827115-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:38.827140-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:38.891242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:38.891261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:38.976103-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:39.025241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.026298-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.097675-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:39.157242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.158336-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.220602-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:39.225044-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:39.225150-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:39.225198-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:39.321240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.321250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.385694-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:39.441239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.442335-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.515176-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:39.570241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.571235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.629548-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:39.634366-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:39.634490-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:39.634537-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:39.709277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.710295-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.780371-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:39.839262-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.840299-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:39.910139-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:39.977240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:39.977250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.035592-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:40.041097-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:40.041216-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:40.041294-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:40.103277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.104277-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.167090-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:40.218264-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.219243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.284272-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:40.346238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.346250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.418573-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:40.423650-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:40.423723-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:40.423770-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:40.493241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.493259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.562477-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:40.624237-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.625259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.689611-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:40.754239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.754251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.815643-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:40.820172-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:40.820242-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:40.820275-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:40.895278-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:40.896236-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:40.961429-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:41.025285-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.026278-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.107626-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:41.162242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.162255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.243541-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:41.248904-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:41.249049-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:41.249119-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:41.316245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.317322-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.396956-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:41.457242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.457253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.534590-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:41.599244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.599254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.678885-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:41.683544-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:41.683646-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:41.683727-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:41.753269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.754235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.837273-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:41.890324-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:41.890343-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:41.939245-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:42.017294-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.018263-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.115585-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:42.149830-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:42.149884-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:42.149925-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:42.189245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.190261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.272291-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:42.331277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.332308-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.417063-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:42.460281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.461334-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.531778-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:42.535316-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:42.535385-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:42.535441-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:42.604242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.605242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.670944-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:42.729242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.730273-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.807112-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:42.862240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:42.863281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:42.933051-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:42.952910-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:42.955108-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:42.955395-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:42.955445-04:00 localhost nvidia-persistenced: Started (104263)
2025-08-16T00:02:43.006239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.007245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.072285-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:43.075859-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:43.075966-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:43.076317-04:00 localhost nvidia-persistenced[104259]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:43.076383-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:43.076451-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:43.076542-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:43.076591-04:00 localhost nvidia-persistenced: Shutdown (104263)
2025-08-16T00:02:43.093895-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:43.093934-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:43.125240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.126285-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.217868-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:43.275265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.276137-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.339722-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:43.362882-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:43.365594-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:43.365815-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:43.365842-04:00 localhost nvidia-persistenced: Started (104284)
2025-08-16T00:02:43.412262-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.413290-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.481278-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:43.484616-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:43.484673-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:43.484696-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:43.484735-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:43.484954-04:00 localhost nvidia-persistenced[104280]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:43.484986-04:00 localhost nvidia-persistenced: Shutdown (104284)
2025-08-16T00:02:43.485900-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:43.507908-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:43.508071-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:43.544239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.545301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.641086-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:43.699238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.700293-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.756960-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:43.775101-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:43.778160-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:43.778435-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:43.778484-04:00 localhost nvidia-persistenced: Started (104304)
2025-08-16T00:02:43.834265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.835312-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:43.898589-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:43.901974-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:43.902070-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:43.902434-04:00 localhost nvidia-persistenced[104300]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:43.902526-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:43.902609-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:43.902663-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:43.902722-04:00 localhost nvidia-persistenced: Shutdown (104304)
2025-08-16T00:02:43.924665-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:43.924728-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:43.952276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:43.952288-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.020039-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:44.076261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.077281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.142428-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:44.170011-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:44.172879-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:44.173145-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:44.173198-04:00 localhost nvidia-persistenced: Started (104324)
2025-08-16T00:02:44.220274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.221283-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.298148-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:44.299931-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:44.299965-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:44.300112-04:00 localhost nvidia-persistenced[104320]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:44.300137-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:44.300173-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:44.300191-04:00 localhost nvidia-persistenced: Shutdown (104324)
2025-08-16T00:02:44.300302-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:44.315365-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:44.315416-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:44.349261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.350233-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.442007-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:44.499244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.500303-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.588680-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:44.610745-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:44.613787-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:44.614061-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:44.614118-04:00 localhost nvidia-persistenced: Started (104344)
2025-08-16T00:02:44.659240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.660351-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.724357-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:44.728344-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:44.728414-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:44.728468-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:44.728744-04:00 localhost nvidia-persistenced[104340]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:44.728795-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:44.728864-04:00 localhost nvidia-persistenced: Shutdown (104344)
2025-08-16T00:02:44.729014-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:44.751800-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:44.751924-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:44.780289-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.780344-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.864995-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:44.923243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:44.924315-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:44.989368-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:44.994997-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:44.995070-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:44.995119-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:45.061240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.062308-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.130714-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:45.181241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.181251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.267446-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:45.326241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.327298-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.410081-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:45.415308-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:45.415414-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:45.415505-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:45.487274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.488242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.563071-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:45.628238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.628255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.689974-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:45.743279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.744250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.811267-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:45.816388-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:45.816472-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:45.816507-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:45.881254-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:45.882311-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:45.949170-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:46.005244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.006268-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.058924-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:46.111275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.112271-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.182192-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:46.184765-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:46.184813-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:46.184834-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:46.253273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.254289-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.348911-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:46.397243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.397253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.457400-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:46.511243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.511259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.569322-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:46.572149-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:46.572204-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:46.572233-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:46.629240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.630251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.685156-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:46.740313-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.741266-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.822023-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:46.871249-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:46.871264-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:46.952959-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:46.955358-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:46.955387-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:46.955426-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:47.011272-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.012281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.118516-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:47.172251-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.173235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.250087-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:47.308286-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.309289-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.369303-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:47.374544-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:47.374626-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:47.374658-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:47.449243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.449253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.543111-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:47.597242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.598315-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.656295-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:47.712305-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.712324-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.780988-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:47.785401-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:47.785477-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:47.785525-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:47.854288-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.855294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:47.936073-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:47.991273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:47.992239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.039104-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:48.096309-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.097230-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.159155-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:48.164038-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:48.164156-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:48.164216-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:48.230243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.231321-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.301485-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:48.351242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.352301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.422977-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:48.475247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.476243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.546162-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:48.551138-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:48.551263-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:48.551324-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:48.626277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.627288-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.726312-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:48.769242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.769258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:48.853149-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:48.915280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:48.915320-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.001577-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:49.006486-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:49.006568-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:49.006601-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:49.070279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.071319-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.147176-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:49.199290-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.199328-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.274715-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:49.331277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.332297-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.404825-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:49.409831-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:49.409938-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:49.410008-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:49.485287-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.485327-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.562018-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:49.622288-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.622333-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.662387-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:49.720288-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.721256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.787702-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:49.792199-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:49.792286-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:49.792332-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:49.859244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.859258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:49.938647-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:49.996243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:49.997287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.050732-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:50.105831-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.107238-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.179913-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:50.186799-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:50.186934-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:50.186978-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:50.250275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.251287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.331740-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:50.387293-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.387330-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.466687-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:50.521257-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.522242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.613010-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:50.617282-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:50.617387-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:50.617419-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:50.689273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.690338-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.791520-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:50.839177-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.839799-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:50.890781-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:50.944242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:50.945278-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.015946-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:51.019679-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:51.019730-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:51.019754-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:51.088238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.088249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.164445-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:51.227239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.227249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.303018-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:51.352243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.352255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.429141-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:51.433940-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:51.434044-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:51.434097-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:51.501276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.502282-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.563821-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:51.621277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.622244-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.713270-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:51.768247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.769276-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.822688-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:51.827081-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:51.827176-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:51.827244-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:51.891281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:51.891304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:51.948052-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:52.002238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.003330-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.075950-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:52.131245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.131260-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.187535-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:52.192260-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:52.192339-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:52.192403-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:52.250243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.250253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.300775-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:52.356281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.357269-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.416577-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:52.472261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.473242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.537264-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:52.541630-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:52.541715-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:52.541801-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:52.612261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.613285-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.672378-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:52.730265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.731281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.793171-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:52.854280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:52.855325-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:52.945663-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:52.971936-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:52.975295-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:52.975578-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:52.975634-04:00 localhost nvidia-persistenced: Started (104653)
2025-08-16T00:02:53.022240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.023263-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.080220-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:53.083674-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:53.083777-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:53.084113-04:00 localhost nvidia-persistenced[104649]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:53.084179-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:53.084255-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:53.084380-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:53.084429-04:00 localhost nvidia-persistenced: Shutdown (104653)
2025-08-16T00:02:53.100864-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:53.100951-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:53.131267-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.132287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.205738-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:53.255277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.256277-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.310893-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:53.329884-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:53.332666-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:53.332877-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:53.332909-04:00 localhost nvidia-persistenced: Started (104673)
2025-08-16T00:02:53.385274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.386297-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.472673-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:53.476834-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:53.476966-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:53.477472-04:00 localhost nvidia-persistenced[104669]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:53.477600-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:53.477701-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:53.477737-04:00 localhost nvidia-persistenced: Shutdown (104673)
2025-08-16T00:02:53.478452-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:53.498007-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:53.498047-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:53.528239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.529305-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.613730-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:53.671242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.672238-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.738597-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:53.760642-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:53.763743-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:53.763968-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:53.764000-04:00 localhost nvidia-persistenced: Started (104693)
2025-08-16T00:02:53.819239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.820276-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:53.893667-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:53.897857-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:53.897923-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:53.898177-04:00 localhost nvidia-persistenced[104689]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:53.898223-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:53.898313-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:53.898343-04:00 localhost nvidia-persistenced: Shutdown (104693)
2025-08-16T00:02:53.898570-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:53.917321-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:53.917379-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:53.949277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:53.950256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.046346-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:54.097265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.098325-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.165695-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:54.179700-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:54.182096-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:54.182373-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:54.182421-04:00 localhost nvidia-persistenced: Started (104713)
2025-08-16T00:02:54.243264-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.243309-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.309048-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:54.311737-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:54.311833-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:54.312137-04:00 localhost nvidia-persistenced[104710]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:54.312195-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:54.312299-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:54.312341-04:00 localhost nvidia-persistenced: Shutdown (104713)
2025-08-16T00:02:54.312404-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:54.326736-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:54.326918-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:54.371240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.372392-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.453021-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:54.507248-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.508274-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.576669-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:54.602713-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:02:54.604685-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:02:54.604774-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:02:54.604791-04:00 localhost nvidia-persistenced: Started (104734)
2025-08-16T00:02:54.653263-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.654246-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.743466-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:54.747441-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:02:54.747571-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:02:54.747980-04:00 localhost nvidia-persistenced[104730]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:02:54.748084-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:02:54.748137-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:02:54.748205-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:02:54.748256-04:00 localhost nvidia-persistenced: Shutdown (104734)
2025-08-16T00:02:54.770520-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:54.770721-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:54.794268-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.794329-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:54.883199-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:54.931244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:54.932304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.007932-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:55.013018-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:55.013109-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:55.013148-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:55.092242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.093287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.174933-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:55.224238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.225304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.307275-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:55.362265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.362324-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.446471-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:55.449926-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:55.449993-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:55.450029-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:55.505245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.506261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.584728-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:55.645240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.646274-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.700115-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:55.759241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.760266-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.840969-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:55.846298-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:55.846357-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:55.846409-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:55.916238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:55.916249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:55.994099-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:56.051240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.051253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.118988-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:56.177244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.177256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.241832-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:56.246630-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:56.246702-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:56.246735-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:56.320243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.320253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.392460-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:56.450268-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.451248-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.523098-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:56.584272-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.585239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.684454-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:56.688041-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:56.688110-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:56.688149-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:56.745246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.745258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.835843-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:56.890284-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:56.891286-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:56.958646-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:57.017276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.018284-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.083579-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:57.088475-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:57.088575-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:57.088643-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:57.153241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.153260-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.220434-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:57.279239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.280237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.351535-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:57.409239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.409249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.477339-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:57.479790-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:57.479826-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:57.487850-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:57.534275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.534328-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.610492-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:57.667247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.668261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.746221-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:57.797240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.797252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:57.893896-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:57.899604-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:57.899725-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:57.899774-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:57.972854-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:57.974270-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.040685-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:58.103242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.104343-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.180111-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:58.234268-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.235244-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.294607-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:58.300122-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:58.300177-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:58.300252-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:58.374238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.374253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.440323-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:58.491287-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.491326-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.566264-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:58.623253-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.624384-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.704843-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:58.709519-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:58.709634-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:58.709686-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:58.777815-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.779294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.848728-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:58.908269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:58.909267-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:58.975162-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:59.025241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.026313-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.092933-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:59.097183-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:59.097272-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:59.097329-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:59.155245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.156294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.251993-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:59.315280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.316255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.372680-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:59.432247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.432258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.489266-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:59.493510-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:59.493619-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:59.493668-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:59.557269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.557314-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.642461-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:02:59.695243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.696308-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.767790-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:02:59.825242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.826264-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:02:59.903432-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:02:59.909098-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:02:59.909168-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:02:59.909224-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:02:59.978287-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:02:59.979301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.051919-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:00.110281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.111279-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.198395-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:00.251284-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.252304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.347883-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:00.353074-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:00.353157-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:00.353204-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:00.423285-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.423365-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.510790-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:00.564245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.565299-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.641102-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:00.699242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.699253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.766587-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:00.772373-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:00.772471-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:00.772519-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:00.843244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.843261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:00.906918-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:00.978244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:00.978256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.035008-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:01.090246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.091296-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.169585-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:01.174965-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:01.175045-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:01.175089-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:01.233279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.233312-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.284201-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:01.337267-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.338243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.401931-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:01.452273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.453272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.546993-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:01.601244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.602254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.706109-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:01.751247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.752237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.855867-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:01.901247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:01.902242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:01.907943-04:00 localhost system76-power[1533]: [INFO] Enabling nvidia-fallback.service
2025-08-16T00:03:01.908224-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:01.908267-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:01.908307-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:01.911184-04:00 localhost systemctl[105398]: Failed to enable unit: Unit file nvidia-fallback.service does not exist.
2025-08-16T00:03:02.017512-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:02.019804-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:02.019836-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:02.019851-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:02.070240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.071237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:02.159788-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:02.218243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.218254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:02.314789-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:02.372302-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.372330-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:02.443333-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:02.446707-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:02.446770-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:02.446798-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:02.513274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.514277-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:02.604671-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:02.660242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.660253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:02.758659-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:02.819244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.819259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:02.908162-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:02.912358-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:02.912476-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:02.912527-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:02.978241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:02.979251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.064829-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:03.118292-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.119248-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.210705-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:03.258237-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.259249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.331262-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:03.354626-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:03:03.357635-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:03:03.357907-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:03:03.357966-04:00 localhost nvidia-persistenced: Started (106225)
2025-08-16T00:03:03.400243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.400254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.481698-04:00 localhost (udev-worker)[106090]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:03.483719-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:03:03.483820-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:03:03.484076-04:00 localhost nvidia-persistenced[106216]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:03:03.484131-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:03:03.484205-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:03:03.484290-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:03:03.484328-04:00 localhost nvidia-persistenced: Shutdown (106225)
2025-08-16T00:03:03.497286-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:03.497429-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:03.536250-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.536262-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.630624-04:00 localhost (udev-worker)[106090]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:03.686242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.687291-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.758131-04:00 localhost (udev-worker)[106090]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:03.776860-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:03:03.779564-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:03:03.779820-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:03:03.779877-04:00 localhost nvidia-persistenced: Started (106460)
2025-08-16T00:03:03.820241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.820252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:03.934056-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:03.936659-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:03:03.936754-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:03:03.936797-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:03:03.936844-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:03:03.937121-04:00 localhost nvidia-persistenced[106447]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:03:03.937180-04:00 localhost nvidia-persistenced: Shutdown (106460)
2025-08-16T00:03:03.937586-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:03:03.959524-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:03.959639-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:03.988243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:03.989250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.070671-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:04.126245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:04.126256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.222699-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:04.244899-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:03:04.247328-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:03:04.247588-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:03:04.247645-04:00 localhost nvidia-persistenced: Started (106816)
2025-08-16T00:03:04.295241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:04.295253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.359739-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:04.362524-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:03:04.362856-04:00 localhost nvidia-persistenced[106796]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:03:04.362920-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:03:04.363004-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:03:04.363060-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:03:04.363117-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:03:04.363162-04:00 localhost nvidia-persistenced: Shutdown (106816)
2025-08-16T00:03:04.379759-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:04.379857-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:04.417248-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:04.418249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.507663-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:04.571244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:04.572239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.670715-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:04.691743-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:03:04.694257-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:03:04.694486-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:03:04.694534-04:00 localhost nvidia-persistenced: Started (107431)
2025-08-16T00:03:04.741241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:04.741252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.844551-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:04.846997-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:03:04.847056-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:03:04.847280-04:00 localhost nvidia-persistenced[107411]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:03:04.847316-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:03:04.847378-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:03:04.847410-04:00 localhost nvidia-persistenced: Shutdown (107431)
2025-08-16T00:03:04.848110-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:03:04.864039-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:04.864190-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:04.899269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:04.900284-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:04.965228-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:05.023242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.023253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:05.109103-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:05.133104-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:03:05.135773-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:03:05.136036-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:03:05.136091-04:00 localhost nvidia-persistenced: Started (107958)
2025-08-16T00:03:05.181242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.181254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:05.258100-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:05.260941-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files.
2025-08-16T00:03:05.261030-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:03:05.261390-04:00 localhost nvidia-persistenced[107938]: nvidia-persistenced failed to initialize. Check syslog for more details.
2025-08-16T00:03:05.261475-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:03:05.261549-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:03:05.261577-04:00 localhost nvidia-persistenced: Shutdown (107958)
2025-08-16T00:03:05.261820-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE
2025-08-16T00:03:05.287018-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:05.287128-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:05.318296-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.318337-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:05.431005-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:05.490241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.490257-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:05.584824-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:05.589185-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:05.589282-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:05.589315-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:05.654240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.654251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:05.742688-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:05.796245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.797338-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:05.892717-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:05.952289-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:05.952347-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.020698-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:06.023528-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:06.023592-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:06.023612-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:06.086240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.087242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.185689-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:06.243255-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.244307-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.347916-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:06.402242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.402254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.481798-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:06.486676-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:06.486756-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:06.486831-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:06.549241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.550329-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.636971-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:06.690289-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.690301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.772502-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:06.826240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.827242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:06.906639-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:06.910584-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:06.910666-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:06.910698-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:06.990280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:06.991271-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.098455-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:07.145246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:07.145264-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.257987-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:07.307245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:07.308283-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.387000-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:07.389433-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:07.389451-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:07.389467-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:07.441286-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:07.442239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.512422-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:07.557282-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:07.557319-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.661860-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:07.711282-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:07.711312-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.810856-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:07.816202-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:07.816348-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:07.816397-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:07.873239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:07.874235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:07.947849-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:08.003239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.003250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.067665-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:08.123239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.123254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.194635-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:08.199097-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:08.199174-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:08.199217-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:08.264242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.264252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.350133-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:08.405258-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.405272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.482899-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:08.536241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.536251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.615894-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:08.619806-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:08.619879-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:08.619914-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:08.688238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.688248-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.772896-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:08.820276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.821280-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:08.914958-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:08.980244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:08.980254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.046628-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:09.051205-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:09.051309-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:09.051355-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:09.110269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.111294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.212009-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:09.270274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.271236-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.347957-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:09.411276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.412313-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.479650-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:09.486471-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:09.486612-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:09.486690-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:09.558239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.559244-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.626611-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:09.680274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.681279-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.731773-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:09.784244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.785283-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.848546-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1.
2025-08-16T00:03:09.853762-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly.
2025-08-16T00:03:09.853858-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'.
2025-08-16T00:03:09.853915-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:09.919281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:09.919318-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:09.977202-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1.
2025-08-16T00:03:10.032275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:10.033282-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:10.095633-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1.
2025-08-16T00:03:10.154251-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:10.155243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509
2025-08-16T00:03:46.312350-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service.
2025-08-16T00:03:46.312366-04:00 localhost /usr/bin/nvidia-powerd[1605]: nvidia-powerd version:1.0(build 1)
2025-08-16T00:03:46.312589-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:03:46.312590-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
2025-08-16T00:03:46.312611-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  570.172.08  Release Build  (dvs-builder@U22-I3-AF01-21-3)  Tue Jul  8 17:59:47 UTC 2025
2025-08-16T00:03:46.312613-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
2025-08-16T00:03:47.900988-04:00 localhost /usr/bin/nvidia-powerd[1605]: DBus Connection is established
2025-08-16T00:03:47.947440-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:03:47.949121-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:03:47.949235-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:03:47.949267-04:00 localhost nvidia-persistenced: Started (1975)
2025-08-16T00:03:47.974947-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered
2025-08-16T00:03:47.974982-04:00 localhost nvidia-persistenced: Local RPC services initialized
2025-08-16T00:03:47.975327-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:03:47.984387-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
2025-08-16T00:03:47.986293-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes
2025-08-16T00:04:01.015128-04:00 localhost systemd[3865]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings...
2025-08-16T00:04:01.027782-04:00 localhost systemd[3865]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings.
2025-08-16T00:05:03.557174-04:00 localhost pop-upgrade[6904]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-16T00:28:46.655514-04:00 localhost nvidia-persistenced: Received signal 15
2025-08-16T00:28:46.655578-04:00 localhost systemd[1]: Stopping nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:28:46.655613-04:00 localhost systemd[1]: Stopping nvidia-powerd.service - nvidia-powerd service...
2025-08-16T00:28:46.656106-04:00 localhost nvidia-persistenced: Socket closed.
2025-08-16T00:28:46.656145-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T00:28:46.656161-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T00:28:46.656174-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T00:28:46.656189-04:00 localhost nvidia-persistenced: Shutdown (1975)
2025-08-16T00:29:37.686108-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service.
2025-08-16T00:29:37.686131-04:00 localhost /usr/bin/nvidia-powerd[1624]: nvidia-powerd version:1.0(build 1)
2025-08-16T00:29:37.686604-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T00:29:37.686605-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
2025-08-16T00:29:37.686614-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  570.172.08  Release Build  (dvs-builder@U22-I3-AF01-21-3)  Tue Jul  8 17:59:47 UTC 2025
2025-08-16T00:29:37.686616-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
2025-08-16T00:29:37.993971-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:29:37.995796-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:29:37.995862-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:29:37.995894-04:00 localhost nvidia-persistenced: Started (1976)
2025-08-16T00:29:39.178898-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered
2025-08-16T00:29:39.178967-04:00 localhost nvidia-persistenced: Local RPC services initialized
2025-08-16T00:29:39.179123-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:29:39.185315-04:00 localhost /usr/bin/nvidia-powerd[1624]: DBus Connection is established
2025-08-16T00:29:39.265710-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
2025-08-16T00:29:39.268729-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes
2025-08-16T00:29:51.469690-04:00 localhost systemd[3913]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings...
2025-08-16T00:29:51.489307-04:00 localhost systemd[3913]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings.
2025-08-16T00:30:52.732125-04:00 localhost pop-upgrade[5783]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-16T00:38:39.113831-04:00 localhost nvidia-persistenced: Received signal 15
2025-08-16T00:38:39.113890-04:00 localhost systemd[1]: Stopping nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:38:39.113919-04:00 localhost systemd[1]: Stopping nvidia-powerd.service - nvidia-powerd service...
2025-08-16T00:38:39.114390-04:00 localhost nvidia-persistenced: Socket closed.
2025-08-16T00:39:16.152624-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service.
2025-08-16T00:39:16.152638-04:00 localhost /usr/bin/nvidia-powerd[1595]: nvidia-powerd version:1.0(build 1)
2025-08-16T00:39:16.153165-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 510
2025-08-16T00:39:16.153165-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
2025-08-16T00:39:16.153172-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  570.172.08  Release Build  (dvs-builder@U22-I3-AF01-21-3)  Tue Jul  8 17:59:47 UTC 2025
2025-08-16T00:39:16.153172-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
2025-08-16T00:39:16.455685-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T00:39:16.457353-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T00:39:16.457400-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T00:39:16.457414-04:00 localhost nvidia-persistenced: Started (1945)
2025-08-16T00:39:17.592827-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered
2025-08-16T00:39:17.592903-04:00 localhost nvidia-persistenced: Local RPC services initialized
2025-08-16T00:39:17.593385-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T00:39:17.599692-04:00 localhost /usr/bin/nvidia-powerd[1595]: DBus Connection is established
2025-08-16T00:39:17.677502-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
2025-08-16T00:39:17.680480-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes
2025-08-16T00:39:29.487188-04:00 localhost systemd[3817]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings...
2025-08-16T00:39:29.504186-04:00 localhost systemd[3817]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings.
2025-08-16T00:40:31.795530-04:00 localhost pop-upgrade[6163]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-16T01:05:45.513712-04:00 localhost nvidia-persistenced: Received signal 15
2025-08-16T01:05:45.513775-04:00 localhost systemd[1]: Stopping nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T01:05:45.513807-04:00 localhost systemd[1]: Stopping nvidia-powerd.service - nvidia-powerd service...
2025-08-16T01:05:45.515130-04:00 localhost nvidia-persistenced: Socket closed.
2025-08-16T01:05:45.515187-04:00 localhost nvidia-persistenced: PID file unlocked.
2025-08-16T01:05:45.515207-04:00 localhost nvidia-persistenced: PID file closed.
2025-08-16T01:05:45.515230-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
2025-08-16T01:06:24.758006-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service.
2025-08-16T01:06:24.758025-04:00 localhost /usr/bin/nvidia-powerd[1584]: nvidia-powerd version:1.0(build 1)
2025-08-16T01:06:24.758611-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
2025-08-16T01:06:24.758611-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
2025-08-16T01:06:24.758626-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64  570.172.08  Release Build  (dvs-builder@U22-I3-AF01-21-3)  Tue Jul  8 17:59:47 UTC 2025
2025-08-16T01:06:24.758627-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
2025-08-16T01:06:25.034464-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon...
2025-08-16T01:06:25.036456-04:00 localhost nvidia-persistenced: Verbose syslog connection opened
2025-08-16T01:06:25.036511-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133
2025-08-16T01:06:25.036527-04:00 localhost nvidia-persistenced: Started (1928)
2025-08-16T01:06:26.204158-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered
2025-08-16T01:06:26.204200-04:00 localhost nvidia-persistenced: Local RPC services initialized
2025-08-16T01:06:26.204456-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon.
2025-08-16T01:06:26.221079-04:00 localhost /usr/bin/nvidia-powerd[1584]: DBus Connection is established
2025-08-16T01:06:26.295496-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0
2025-08-16T01:06:26.297417-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes
2025-08-16T01:06:38.034677-04:00 localhost systemd[3832]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings...
2025-08-16T01:06:38.053892-04:00 localhost systemd[3832]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings.
2025-08-16T01:07:42.281343-04:00 localhost pop-upgrade[5748]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia
2025-08-16T11:50:02.424665-04:00 localhost system76-power[1574]: [INFO] Enabling nvidia-fallback.service
2025-08-16T11:50:02.470143-04:00 localhost systemctl[63779]: Failed to enable unit: Unit file nvidia-fallback.service does not exist.
<!-- gh-comment-id:3193756815 --> @guusec commented on GitHub (Aug 16, 2025): Here's my dmesg output: ``` sudo dmesg|ug nvidia --no-line-number [ 18.972759] nvidia-nvlink: Nvlink Core is being initialized, major device number 509 [ 18.973660] nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none [ 19.138392] nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 570.172.08 Release Build (dvs-builder@U22-I3-AF01-21-3) Tue Jul 8 17:59:47 UTC 2025 [ 19.146051] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver [ 21.551377] [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0 [ 21.553960] nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes ``` Lots of syslog output too: ``` 2025-08-11T19:37:52.828120-04:00 localhost ollama[2322]: time=2025-08-11T19:37:52.828-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-11T19:39:30.467612-04:00 localhost pop-upgrade[5892]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-12T10:58:40.454910-04:00 localhost ollama[2294]: time=2025-08-12T10:58:40.454-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-12T11:00:27.246135-04:00 localhost pop-upgrade[5817]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-13T02:53:13.079725-04:00 localhost pop-upgrade[436619]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-13T03:02:36.129107-04:00 localhost pop-upgrade[454221]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-13T03:09:38.768730-04:00 localhost pop-upgrade[458987]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-13T03:13:00.614763-04:00 localhost ollama[2371]: time=2025-08-13T03:13:00.614-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-13T03:14:38.997322-04:00 localhost pop-upgrade[5058]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-13T03:29:21.649475-04:00 localhost pop-upgrade[32331]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-13T11:01:50.567127-04:00 localhost ollama[2400]: time=2025-08-13T11:01:50.567-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-13T11:03:24.682079-04:00 localhost pop-upgrade[5351]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-14T10:28:19.566459-04:00 localhost ollama[2392]: time=2025-08-14T10:28:19.566-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-14T10:31:34.813899-04:00 localhost pop-upgrade[3953]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-15T03:04:26.953855-04:00 localhost systemctl[91505]: Failed to enable unit: Unit file nvidia-hibernate.service does not exist. 2025-08-15T03:04:26.954282-04:00 localhost system76-power[1558]: [WARN] systemctl enable nvidia-hibernate.service: failed with exit status: 1 (not an error if service does not exist!) 2025-08-15T03:04:26.960527-04:00 localhost systemctl[91506]: Failed to enable unit: Unit file nvidia-resume.service does not exist. 2025-08-15T03:04:26.960789-04:00 localhost system76-power[1558]: [WARN] systemctl enable nvidia-resume.service: failed with exit status: 1 (not an error if service does not exist!) 2025-08-15T03:04:26.967331-04:00 localhost systemctl[91507]: Failed to enable unit: Unit file nvidia-suspend.service does not exist. 2025-08-15T03:04:26.967471-04:00 localhost system76-power[1558]: [WARN] systemctl enable nvidia-suspend.service: failed with exit status: 1 (not an error if service does not exist!) 2025-08-15T03:04:26.970311-04:00 localhost system76-power[1558]: [INFO] Enabling nvidia-fallback.service 2025-08-15T03:04:26.976515-04:00 localhost systemctl[91508]: Failed to enable unit: Unit file nvidia-fallback.service does not exist. 2025-08-15T03:05:27.794816-04:00 localhost ollama[2369]: time=2025-08-15T03:05:27.794-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-15T03:08:39.829249-04:00 localhost pop-upgrade[3865]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-15T11:50:29.181634-04:00 localhost ollama[2400]: time=2025-08-15T11:50:29.181-04:00 level=INFO source=gpu.go:602 msg="no nvidia devices detected by library /usr/lib/x86_64-linux-gnu/libcuda.so.565.77" 2025-08-15T11:52:09.699005-04:00 localhost pop-upgrade[5385]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-16T00:01:29.428097-04:00 localhost systemd[1]: Starting nvidia-hibernate.service - NVIDIA system hibernate actions... 2025-08-16T00:01:29.432457-04:00 localhost hibernate: nvidia-hibernate.service 2025-08-16T00:01:29.432600-04:00 localhost logger[82091]: <13>Aug 16 00:01:29 hibernate: nvidia-hibernate.service 2025-08-16T00:01:29.433166-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service. 2025-08-16T00:01:29.438517-04:00 localhost systemd[1]: Starting nvidia-resume.service - NVIDIA system resume actions... 2025-08-16T00:01:29.439641-04:00 localhost /usr/bin/nvidia-powerd[82092]: nvidia-powerd version:1.0(build 1) 2025-08-16T00:01:29.441267-04:00 localhost systemd[1]: Starting nvidia-suspend.service - NVIDIA system suspend actions... 2025-08-16T00:01:29.445047-04:00 localhost suspend: nvidia-resume.service 2025-08-16T00:01:29.445196-04:00 localhost logger[82095]: <13>Aug 16 00:01:29 suspend: nvidia-resume.service 2025-08-16T00:01:29.446920-04:00 localhost systemd[1]: nvidia-hibernate.service: Deactivated successfully. 2025-08-16T00:01:29.447024-04:00 localhost systemd[1]: Finished nvidia-hibernate.service - NVIDIA system hibernate actions. 2025-08-16T00:01:29.448834-04:00 localhost suspend: nvidia-suspend.service 2025-08-16T00:01:29.448914-04:00 localhost logger[82096]: <13>Aug 16 00:01:29 suspend: nvidia-suspend.service 2025-08-16T00:01:29.455438-04:00 localhost systemd[1]: nvidia-resume.service: Deactivated successfully. 2025-08-16T00:01:29.455779-04:00 localhost systemd[1]: Finished nvidia-resume.service - NVIDIA system resume actions. 2025-08-16T00:01:29.458789-04:00 localhost systemd[1]: nvidia-suspend.service: Deactivated successfully. 2025-08-16T00:01:29.458955-04:00 localhost systemd[1]: Finished nvidia-suspend.service - NVIDIA system suspend actions. 2025-08-16T00:01:29.467370-04:00 localhost /usr/bin/nvidia-powerd[82092]: Allocate client failed 38 2025-08-16T00:01:29.467414-04:00 localhost /usr/bin/nvidia-powerd[82092]: Failed to initialize RM Client 2025-08-16T00:01:29.467823-04:00 localhost systemd[1]: nvidia-powerd.service: Main process exited, code=exited, status=1/FAILURE 2025-08-16T00:01:29.467895-04:00 localhost systemd[1]: nvidia-powerd.service: Failed with result 'exit-code'. 2025-08-16T00:01:29.742795-04:00 localhost systemd[1]: Starting nvidia-hibernate.service - NVIDIA system hibernate actions... 2025-08-16T00:01:29.745377-04:00 localhost hibernate: nvidia-hibernate.service 2025-08-16T00:01:29.745455-04:00 localhost logger[82250]: <13>Aug 16 00:01:29 hibernate: nvidia-hibernate.service 2025-08-16T00:01:29.753471-04:00 localhost systemd[1]: nvidia-hibernate.service: Deactivated successfully. 2025-08-16T00:01:29.753642-04:00 localhost systemd[1]: Finished nvidia-hibernate.service - NVIDIA system hibernate actions. 2025-08-16T00:01:30.054064-04:00 localhost systemd[1]: Starting nvidia-resume.service - NVIDIA system resume actions... 2025-08-16T00:01:30.057628-04:00 localhost suspend: nvidia-resume.service 2025-08-16T00:01:30.057737-04:00 localhost logger[82393]: <13>Aug 16 00:01:30 suspend: nvidia-resume.service 2025-08-16T00:01:30.067726-04:00 localhost systemd[1]: nvidia-resume.service: Deactivated successfully. 2025-08-16T00:01:30.067829-04:00 localhost systemd[1]: Finished nvidia-resume.service - NVIDIA system resume actions. 2025-08-16T00:01:30.368646-04:00 localhost systemd[1]: Starting nvidia-suspend.service - NVIDIA system suspend actions... 2025-08-16T00:01:30.373128-04:00 localhost suspend: nvidia-suspend.service 2025-08-16T00:01:30.373194-04:00 localhost logger[82538]: <13>Aug 16 00:01:30 suspend: nvidia-suspend.service 2025-08-16T00:01:30.381251-04:00 localhost systemd[1]: nvidia-suspend.service: Deactivated successfully. 2025-08-16T00:01:30.381459-04:00 localhost systemd[1]: Finished nvidia-suspend.service - NVIDIA system suspend actions. 2025-08-16T00:02:04.274862-04:00 localhost adduser[98515]: Adding system user `nvidia-persistenced' (UID 125) ... 2025-08-16T00:02:04.276594-04:00 localhost adduser[98515]: Adding new group `nvidia-persistenced' (GID 133) ... 2025-08-16T00:02:04.292157-04:00 localhost adduser[98515]: Adding new user `nvidia-persistenced' (UID 125) with group `nvidia-persistenced' ... 2025-08-16T00:02:32.488275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:32.489328-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:32.605246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:32.605272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:32.661788-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:32.713240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:32.713251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:32.779587-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:32.830243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:32.831245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:32.894848-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:32.915088-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:32.917531-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:32.917841-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:32.917905-04:00 localhost nvidia-persistenced: Started (103877) 2025-08-16T00:02:32.964242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:32.964254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.034353-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:33.038790-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:33.038885-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:33.039136-04:00 localhost nvidia-persistenced[103873]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:33.039187-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:33.039295-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:33.039331-04:00 localhost nvidia-persistenced: Shutdown (103877) 2025-08-16T00:02:33.039850-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:33.058662-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:33.058711-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:33.096244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.096262-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.146014-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:33.195243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.196263-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.270046-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:33.292749-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:33.295334-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:33.295807-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:33.295859-04:00 localhost nvidia-persistenced: Started (103897) 2025-08-16T00:02:33.354239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.355303-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.435743-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:33.440219-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:33.440374-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:33.440774-04:00 localhost nvidia-persistenced[103893]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:33.440852-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:33.440969-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:33.441045-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:33.441098-04:00 localhost nvidia-persistenced: Shutdown (103897) 2025-08-16T00:02:33.462390-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:33.462454-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:33.493271-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.494304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.567760-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:33.628271-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.629331-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.698976-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:33.725774-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:33.728794-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:33.729043-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:33.729097-04:00 localhost nvidia-persistenced: Started (103917) 2025-08-16T00:02:33.777275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.778295-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.863294-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:33.867321-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:33.867424-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:33.867784-04:00 localhost nvidia-persistenced[103913]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:33.867850-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:33.867952-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:33.868021-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:33.869466-04:00 localhost nvidia-persistenced: Shutdown (103917) 2025-08-16T00:02:33.890653-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:33.890698-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:33.918242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:33.918254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:33.991423-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:34.042274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.043255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.115419-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:34.133033-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:34.135378-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:34.135617-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:34.135677-04:00 localhost nvidia-persistenced: Started (103937) 2025-08-16T00:02:34.180270-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.180342-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.238830-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:34.242734-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:34.242859-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:34.243258-04:00 localhost nvidia-persistenced[103933]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:34.243333-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:34.243444-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:34.243517-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:34.243567-04:00 localhost nvidia-persistenced: Shutdown (103937) 2025-08-16T00:02:34.265682-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:34.265731-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:34.296283-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.297321-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.349727-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:34.407251-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.407267-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.473833-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:34.503737-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:34.505144-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:34.506714-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:34.506764-04:00 localhost nvidia-persistenced: Started (103958) 2025-08-16T00:02:34.545283-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.546272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.620308-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:34.623442-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:34.623535-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:34.623862-04:00 localhost nvidia-persistenced[103954]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:34.623925-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:34.624010-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:34.624069-04:00 localhost nvidia-persistenced: Shutdown (103958) 2025-08-16T00:02:34.624123-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:34.641721-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:34.641873-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:34.690240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.690253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.786057-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:34.834290-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.834302-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:34.909999-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:34.916152-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:34.916321-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:34.916382-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:34.995282-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:34.995347-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.049757-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:35.105249-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.106272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.171516-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:35.226280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.227269-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.292293-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:35.298014-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:35.298094-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:35.298145-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:35.370245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.371331-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.435664-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:35.502267-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.503254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.569843-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:35.624240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.625300-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.689744-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:35.695362-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:35.695473-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:35.695531-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:35.763238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.764235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.840332-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:35.901244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:35.902318-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:35.963840-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:36.013245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.014250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.083775-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:36.088250-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:36.088327-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:36.088395-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:36.156239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.157251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.232077-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:36.292242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.293279-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.347630-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:36.404274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.405245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.474258-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:36.479591-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:36.479677-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:36.479747-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:36.551275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.552237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.608942-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:36.663241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.663252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.712690-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:36.766241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.766253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.822953-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:36.827160-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:36.827267-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:36.827330-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:36.891241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:36.891251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:36.961085-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:37.020241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.021305-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.125741-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:37.175275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.176239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.275741-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:37.280305-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:37.280381-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:37.280417-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:37.358243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.359275-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.424207-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:37.490281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.491235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.563922-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:37.623259-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.624243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.677880-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:37.682195-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:37.682288-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:37.683642-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:37.758300-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.759285-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.816595-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:37.875242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.876256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:37.926712-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:37.990261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:37.990290-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.066164-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:38.069363-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:38.069449-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:38.069492-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:38.136275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.137237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.215793-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:38.261279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.262271-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.345776-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:38.403242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.404261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.455585-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:38.460897-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:38.460992-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:38.461024-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:38.527270-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.527304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.576483-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:38.642245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.643245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.701353-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:38.758242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.759300-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.824682-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:38.827063-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:38.827115-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:38.827140-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:38.891242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:38.891261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:38.976103-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:39.025241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.026298-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.097675-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:39.157242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.158336-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.220602-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:39.225044-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:39.225150-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:39.225198-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:39.321240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.321250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.385694-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:39.441239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.442335-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.515176-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:39.570241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.571235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.629548-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:39.634366-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:39.634490-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:39.634537-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:39.709277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.710295-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.780371-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:39.839262-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.840299-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:39.910139-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:39.977240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:39.977250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.035592-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:40.041097-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:40.041216-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:40.041294-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:40.103277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.104277-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.167090-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:40.218264-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.219243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.284272-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:40.346238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.346250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.418573-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:40.423650-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:40.423723-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:40.423770-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:40.493241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.493259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.562477-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:40.624237-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.625259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.689611-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:40.754239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.754251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.815643-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:40.820172-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:40.820242-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:40.820275-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:40.895278-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:40.896236-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:40.961429-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:41.025285-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.026278-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.107626-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:41.162242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.162255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.243541-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:41.248904-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:41.249049-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:41.249119-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:41.316245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.317322-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.396956-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:41.457242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.457253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.534590-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:41.599244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.599254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.678885-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:41.683544-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:41.683646-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:41.683727-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:41.753269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.754235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.837273-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:41.890324-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:41.890343-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:41.939245-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:42.017294-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.018263-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.115585-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:42.149830-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:42.149884-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:42.149925-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:42.189245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.190261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.272291-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:42.331277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.332308-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.417063-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:42.460281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.461334-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.531778-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:42.535316-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:42.535385-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:42.535441-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:42.604242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.605242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.670944-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:42.729242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.730273-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.807112-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:42.862240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:42.863281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:42.933051-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:42.952910-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:42.955108-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:42.955395-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:42.955445-04:00 localhost nvidia-persistenced: Started (104263) 2025-08-16T00:02:43.006239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.007245-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.072285-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:43.075859-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:43.075966-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:43.076317-04:00 localhost nvidia-persistenced[104259]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:43.076383-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:43.076451-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:43.076542-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:43.076591-04:00 localhost nvidia-persistenced: Shutdown (104263) 2025-08-16T00:02:43.093895-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:43.093934-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:43.125240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.126285-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.217868-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:43.275265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.276137-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.339722-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:43.362882-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:43.365594-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:43.365815-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:43.365842-04:00 localhost nvidia-persistenced: Started (104284) 2025-08-16T00:02:43.412262-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.413290-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.481278-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:43.484616-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:43.484673-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:43.484696-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:43.484735-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:43.484954-04:00 localhost nvidia-persistenced[104280]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:43.484986-04:00 localhost nvidia-persistenced: Shutdown (104284) 2025-08-16T00:02:43.485900-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:43.507908-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:43.508071-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:43.544239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.545301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.641086-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:43.699238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.700293-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.756960-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:43.775101-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:43.778160-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:43.778435-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:43.778484-04:00 localhost nvidia-persistenced: Started (104304) 2025-08-16T00:02:43.834265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.835312-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:43.898589-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:43.901974-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:43.902070-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:43.902434-04:00 localhost nvidia-persistenced[104300]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:43.902526-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:43.902609-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:43.902663-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:43.902722-04:00 localhost nvidia-persistenced: Shutdown (104304) 2025-08-16T00:02:43.924665-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:43.924728-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:43.952276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:43.952288-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.020039-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:44.076261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.077281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.142428-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:44.170011-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:44.172879-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:44.173145-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:44.173198-04:00 localhost nvidia-persistenced: Started (104324) 2025-08-16T00:02:44.220274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.221283-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.298148-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:44.299931-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:44.299965-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:44.300112-04:00 localhost nvidia-persistenced[104320]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:44.300137-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:44.300173-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:44.300191-04:00 localhost nvidia-persistenced: Shutdown (104324) 2025-08-16T00:02:44.300302-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:44.315365-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:44.315416-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:44.349261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.350233-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.442007-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:44.499244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.500303-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.588680-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:44.610745-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:44.613787-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:44.614061-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:44.614118-04:00 localhost nvidia-persistenced: Started (104344) 2025-08-16T00:02:44.659240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.660351-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.724357-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:44.728344-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:44.728414-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:44.728468-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:44.728744-04:00 localhost nvidia-persistenced[104340]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:44.728795-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:44.728864-04:00 localhost nvidia-persistenced: Shutdown (104344) 2025-08-16T00:02:44.729014-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:44.751800-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:44.751924-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:44.780289-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.780344-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.864995-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:44.923243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:44.924315-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:44.989368-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:44.994997-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:44.995070-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:44.995119-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:45.061240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.062308-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.130714-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:45.181241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.181251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.267446-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:45.326241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.327298-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.410081-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:45.415308-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:45.415414-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:45.415505-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:45.487274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.488242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.563071-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:45.628238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.628255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.689974-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:45.743279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.744250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.811267-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:45.816388-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:45.816472-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:45.816507-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:45.881254-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:45.882311-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:45.949170-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:46.005244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.006268-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.058924-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:46.111275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.112271-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.182192-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:46.184765-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:46.184813-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:46.184834-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:46.253273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.254289-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.348911-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:46.397243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.397253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.457400-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:46.511243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.511259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.569322-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:46.572149-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:46.572204-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:46.572233-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:46.629240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.630251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.685156-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:46.740313-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.741266-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.822023-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:46.871249-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:46.871264-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:46.952959-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:46.955358-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:46.955387-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:46.955426-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:47.011272-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.012281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.118516-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:47.172251-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.173235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.250087-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:47.308286-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.309289-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.369303-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:47.374544-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:47.374626-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:47.374658-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:47.449243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.449253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.543111-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:47.597242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.598315-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.656295-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:47.712305-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.712324-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.780988-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:47.785401-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:47.785477-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:47.785525-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:47.854288-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.855294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:47.936073-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:47.991273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:47.992239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.039104-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:48.096309-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.097230-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.159155-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:48.164038-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:48.164156-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:48.164216-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:48.230243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.231321-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.301485-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:48.351242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.352301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.422977-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:48.475247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.476243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.546162-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:48.551138-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:48.551263-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:48.551324-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:48.626277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.627288-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.726312-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:48.769242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.769258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:48.853149-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:48.915280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:48.915320-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.001577-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:49.006486-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:49.006568-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:49.006601-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:49.070279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.071319-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.147176-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:49.199290-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.199328-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.274715-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:49.331277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.332297-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.404825-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:49.409831-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:49.409938-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:49.410008-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:49.485287-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.485327-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.562018-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:49.622288-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.622333-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.662387-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:49.720288-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.721256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.787702-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:49.792199-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:49.792286-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:49.792332-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:49.859244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.859258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:49.938647-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:49.996243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:49.997287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.050732-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:50.105831-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.107238-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.179913-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:50.186799-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:50.186934-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:50.186978-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:50.250275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.251287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.331740-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:50.387293-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.387330-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.466687-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:50.521257-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.522242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.613010-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:50.617282-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:50.617387-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:50.617419-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:50.689273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.690338-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.791520-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:50.839177-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.839799-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:50.890781-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:50.944242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:50.945278-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.015946-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:51.019679-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:51.019730-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:51.019754-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:51.088238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.088249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.164445-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:51.227239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.227249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.303018-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:51.352243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.352255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.429141-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:51.433940-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:51.434044-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:51.434097-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:51.501276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.502282-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.563821-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:51.621277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.622244-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.713270-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:51.768247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.769276-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.822688-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:51.827081-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:51.827176-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:51.827244-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:51.891281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:51.891304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:51.948052-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:52.002238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.003330-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.075950-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:52.131245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.131260-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.187535-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:52.192260-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:52.192339-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:52.192403-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:52.250243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.250253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.300775-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:52.356281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.357269-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.416577-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:52.472261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.473242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.537264-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:52.541630-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:52.541715-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:52.541801-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:52.612261-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.613285-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.672378-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:52.730265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.731281-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.793171-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:52.854280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:52.855325-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:52.945663-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:52.971936-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:52.975295-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:52.975578-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:52.975634-04:00 localhost nvidia-persistenced: Started (104653) 2025-08-16T00:02:53.022240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.023263-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.080220-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:53.083674-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:53.083777-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:53.084113-04:00 localhost nvidia-persistenced[104649]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:53.084179-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:53.084255-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:53.084380-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:53.084429-04:00 localhost nvidia-persistenced: Shutdown (104653) 2025-08-16T00:02:53.100864-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:53.100951-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:53.131267-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.132287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.205738-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:53.255277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.256277-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.310893-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:53.329884-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:53.332666-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:53.332877-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:53.332909-04:00 localhost nvidia-persistenced: Started (104673) 2025-08-16T00:02:53.385274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.386297-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.472673-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:53.476834-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:53.476966-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:53.477472-04:00 localhost nvidia-persistenced[104669]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:53.477600-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:53.477701-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:53.477737-04:00 localhost nvidia-persistenced: Shutdown (104673) 2025-08-16T00:02:53.478452-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:53.498007-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:53.498047-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:53.528239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.529305-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.613730-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:53.671242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.672238-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.738597-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:53.760642-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:53.763743-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:53.763968-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:53.764000-04:00 localhost nvidia-persistenced: Started (104693) 2025-08-16T00:02:53.819239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.820276-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:53.893667-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:53.897857-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:53.897923-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:53.898177-04:00 localhost nvidia-persistenced[104689]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:53.898223-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:53.898313-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:53.898343-04:00 localhost nvidia-persistenced: Shutdown (104693) 2025-08-16T00:02:53.898570-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:53.917321-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:53.917379-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:53.949277-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:53.950256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.046346-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:54.097265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.098325-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.165695-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:54.179700-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:54.182096-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:54.182373-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:54.182421-04:00 localhost nvidia-persistenced: Started (104713) 2025-08-16T00:02:54.243264-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.243309-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.309048-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:54.311737-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:54.311833-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:54.312137-04:00 localhost nvidia-persistenced[104710]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:54.312195-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:54.312299-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:54.312341-04:00 localhost nvidia-persistenced: Shutdown (104713) 2025-08-16T00:02:54.312404-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:54.326736-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:54.326918-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:54.371240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.372392-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.453021-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:54.507248-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.508274-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.576669-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:54.602713-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:02:54.604685-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:02:54.604774-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:02:54.604791-04:00 localhost nvidia-persistenced: Started (104734) 2025-08-16T00:02:54.653263-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.654246-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.743466-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:54.747441-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:02:54.747571-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:02:54.747980-04:00 localhost nvidia-persistenced[104730]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:02:54.748084-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:02:54.748137-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:02:54.748205-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:02:54.748256-04:00 localhost nvidia-persistenced: Shutdown (104734) 2025-08-16T00:02:54.770520-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:54.770721-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:54.794268-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.794329-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:54.883199-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:54.931244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:54.932304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.007932-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:55.013018-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:55.013109-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:55.013148-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:55.092242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.093287-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.174933-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:55.224238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.225304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.307275-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:55.362265-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.362324-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.446471-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:55.449926-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:55.449993-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:55.450029-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:55.505245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.506261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.584728-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:55.645240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.646274-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.700115-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:55.759241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.760266-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.840969-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:55.846298-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:55.846357-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:55.846409-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:55.916238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:55.916249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:55.994099-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:56.051240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.051253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.118988-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:56.177244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.177256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.241832-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:56.246630-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:56.246702-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:56.246735-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:56.320243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.320253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.392460-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:56.450268-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.451248-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.523098-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:56.584272-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.585239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.684454-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:56.688041-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:56.688110-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:56.688149-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:56.745246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.745258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.835843-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:56.890284-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:56.891286-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:56.958646-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:57.017276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.018284-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.083579-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:57.088475-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:57.088575-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:57.088643-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:57.153241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.153260-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.220434-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:57.279239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.280237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.351535-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:57.409239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.409249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.477339-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:57.479790-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:57.479826-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:57.487850-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:57.534275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.534328-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.610492-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:57.667247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.668261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.746221-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:57.797240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.797252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:57.893896-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:57.899604-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:57.899725-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:57.899774-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:57.972854-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:57.974270-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.040685-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:58.103242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.104343-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.180111-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:58.234268-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.235244-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.294607-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:58.300122-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:58.300177-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:58.300252-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:58.374238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.374253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.440323-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:58.491287-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.491326-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.566264-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:58.623253-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.624384-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.704843-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:58.709519-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:58.709634-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:58.709686-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:58.777815-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.779294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.848728-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:58.908269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:58.909267-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:58.975162-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:59.025241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.026313-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.092933-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:59.097183-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:59.097272-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:59.097329-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:59.155245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.156294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.251993-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:59.315280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.316255-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.372680-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:59.432247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.432258-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.489266-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:59.493510-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:59.493619-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:59.493668-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:59.557269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.557314-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.642461-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:02:59.695243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.696308-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.767790-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:02:59.825242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.826264-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:02:59.903432-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:02:59.909098-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:02:59.909168-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:02:59.909224-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:02:59.978287-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:02:59.979301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.051919-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:00.110281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.111279-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.198395-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:00.251284-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.252304-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.347883-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:00.353074-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:00.353157-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:00.353204-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:00.423285-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.423365-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.510790-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:00.564245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.565299-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.641102-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:00.699242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.699253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.766587-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:00.772373-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:00.772471-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:00.772519-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:00.843244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.843261-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:00.906918-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:00.978244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:00.978256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.035008-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:01.090246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.091296-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.169585-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:01.174965-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:01.175045-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:01.175089-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:01.233279-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.233312-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.284201-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:01.337267-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.338243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.401931-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:01.452273-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.453272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.546993-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:01.601244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.602254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.706109-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:01.751247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.752237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.855867-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:01.901247-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:01.902242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:01.907943-04:00 localhost system76-power[1533]: [INFO] Enabling nvidia-fallback.service 2025-08-16T00:03:01.908224-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:01.908267-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:01.908307-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:01.911184-04:00 localhost systemctl[105398]: Failed to enable unit: Unit file nvidia-fallback.service does not exist. 2025-08-16T00:03:02.017512-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:02.019804-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:02.019836-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:02.019851-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:02.070240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.071237-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:02.159788-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:02.218243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.218254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:02.314789-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:02.372302-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.372330-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:02.443333-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:02.446707-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:02.446770-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:02.446798-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:02.513274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.514277-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:02.604671-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:02.660242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.660253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:02.758659-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:02.819244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.819259-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:02.908162-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:02.912358-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:02.912476-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:02.912527-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:02.978241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:02.979251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.064829-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:03.118292-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.119248-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.210705-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:03.258237-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.259249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.331262-04:00 localhost (udev-worker)[103859]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:03.354626-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:03:03.357635-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:03:03.357907-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:03:03.357966-04:00 localhost nvidia-persistenced: Started (106225) 2025-08-16T00:03:03.400243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.400254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.481698-04:00 localhost (udev-worker)[106090]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:03.483719-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:03:03.483820-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:03:03.484076-04:00 localhost nvidia-persistenced[106216]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:03:03.484131-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:03:03.484205-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:03:03.484290-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:03:03.484328-04:00 localhost nvidia-persistenced: Shutdown (106225) 2025-08-16T00:03:03.497286-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:03.497429-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:03.536250-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.536262-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.630624-04:00 localhost (udev-worker)[106090]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:03.686242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.687291-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.758131-04:00 localhost (udev-worker)[106090]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:03.776860-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:03:03.779564-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:03:03.779820-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:03:03.779877-04:00 localhost nvidia-persistenced: Started (106460) 2025-08-16T00:03:03.820241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.820252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:03.934056-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:03.936659-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:03:03.936754-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:03:03.936797-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:03:03.936844-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:03:03.937121-04:00 localhost nvidia-persistenced[106447]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:03:03.937180-04:00 localhost nvidia-persistenced: Shutdown (106460) 2025-08-16T00:03:03.937586-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:03:03.959524-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:03.959639-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:03.988243-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:03.989250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.070671-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:04.126245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:04.126256-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.222699-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:04.244899-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:03:04.247328-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:03:04.247588-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:03:04.247645-04:00 localhost nvidia-persistenced: Started (106816) 2025-08-16T00:03:04.295241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:04.295253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.359739-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:04.362524-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:03:04.362856-04:00 localhost nvidia-persistenced[106796]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:03:04.362920-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:03:04.363004-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:03:04.363060-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:03:04.363117-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:03:04.363162-04:00 localhost nvidia-persistenced: Shutdown (106816) 2025-08-16T00:03:04.379759-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:04.379857-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:04.417248-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:04.418249-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.507663-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:04.571244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:04.572239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.670715-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:04.691743-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:03:04.694257-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:03:04.694486-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:03:04.694534-04:00 localhost nvidia-persistenced: Started (107431) 2025-08-16T00:03:04.741241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:04.741252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.844551-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:04.846997-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:03:04.847056-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:03:04.847280-04:00 localhost nvidia-persistenced[107411]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:03:04.847316-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:03:04.847378-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:03:04.847410-04:00 localhost nvidia-persistenced: Shutdown (107431) 2025-08-16T00:03:04.848110-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:03:04.864039-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:04.864190-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:04.899269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:04.900284-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:04.965228-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:05.023242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.023253-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:05.109103-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:05.133104-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:03:05.135773-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:03:05.136036-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:03:05.136091-04:00 localhost nvidia-persistenced: Started (107958) 2025-08-16T00:03:05.181242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.181254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:05.258100-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:05.260941-04:00 localhost nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 125 has read and write permissions for those files. 2025-08-16T00:03:05.261030-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:03:05.261390-04:00 localhost nvidia-persistenced[107938]: nvidia-persistenced failed to initialize. Check syslog for more details. 2025-08-16T00:03:05.261475-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:03:05.261549-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:03:05.261577-04:00 localhost nvidia-persistenced: Shutdown (107958) 2025-08-16T00:03:05.261820-04:00 localhost systemd[1]: nvidia-persistenced.service: Control process exited, code=exited, status=1/FAILURE 2025-08-16T00:03:05.287018-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:05.287128-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:05.318296-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.318337-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:05.431005-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:05.490241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.490257-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:05.584824-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:05.589185-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:05.589282-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:05.589315-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:05.654240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.654251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:05.742688-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:05.796245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.797338-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:05.892717-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:05.952289-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:05.952347-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.020698-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:06.023528-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:06.023592-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:06.023612-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:06.086240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.087242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.185689-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:06.243255-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.244307-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.347916-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:06.402242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.402254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.481798-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:06.486676-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:06.486756-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:06.486831-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:06.549241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.550329-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.636971-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:06.690289-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.690301-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.772502-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:06.826240-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.827242-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:06.906639-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:06.910584-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:06.910666-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:06.910698-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:06.990280-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:06.991271-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.098455-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:07.145246-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:07.145264-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.257987-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:07.307245-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:07.308283-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.387000-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:07.389433-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:07.389451-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:07.389467-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:07.441286-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:07.442239-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.512422-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:07.557282-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:07.557319-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.661860-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:07.711282-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:07.711312-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.810856-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:07.816202-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:07.816348-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:07.816397-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:07.873239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:07.874235-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:07.947849-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:08.003239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.003250-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.067665-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:08.123239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.123254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.194635-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:08.199097-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:08.199174-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:08.199217-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:08.264242-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.264252-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.350133-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:08.405258-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.405272-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.482899-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:08.536241-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.536251-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.615894-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:08.619806-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:08.619879-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:08.619914-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:08.688238-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.688248-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.772896-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:08.820276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.821280-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:08.914958-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:08.980244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:08.980254-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.046628-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:09.051205-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:09.051309-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:09.051355-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:09.110269-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.111294-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.212009-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:09.270274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.271236-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.347957-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:09.411276-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.412313-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.479650-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:09.486471-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:09.486612-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:09.486690-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:09.558239-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.559244-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.626611-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:09.680274-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.681279-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.731773-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:09.784244-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.785283-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.848546-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-uvm' failed with exit code 1. 2025-08-16T00:03:09.853762-04:00 localhost systemd[1]: nvidia-persistenced.service: Start request repeated too quickly. 2025-08-16T00:03:09.853858-04:00 localhost systemd[1]: nvidia-persistenced.service: Failed with result 'exit-code'. 2025-08-16T00:03:09.853915-04:00 localhost systemd[1]: Failed to start nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:09.919281-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:09.919318-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:09.977202-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-modeset' failed with exit code 1. 2025-08-16T00:03:10.032275-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:10.033282-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:10.095633-04:00 localhost (udev-worker)[106246]: nvidia: Process '/sbin/modprobe nvidia-drm' failed with exit code 1. 2025-08-16T00:03:10.154251-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:10.155243-04:00 localhost kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 509 2025-08-16T00:03:46.312350-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service. 2025-08-16T00:03:46.312366-04:00 localhost /usr/bin/nvidia-powerd[1605]: nvidia-powerd version:1.0(build 1) 2025-08-16T00:03:46.312589-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:03:46.312590-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none 2025-08-16T00:03:46.312611-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 570.172.08 Release Build (dvs-builder@U22-I3-AF01-21-3) Tue Jul 8 17:59:47 UTC 2025 2025-08-16T00:03:46.312613-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver 2025-08-16T00:03:47.900988-04:00 localhost /usr/bin/nvidia-powerd[1605]: DBus Connection is established 2025-08-16T00:03:47.947440-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:03:47.949121-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:03:47.949235-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:03:47.949267-04:00 localhost nvidia-persistenced: Started (1975) 2025-08-16T00:03:47.974947-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered 2025-08-16T00:03:47.974982-04:00 localhost nvidia-persistenced: Local RPC services initialized 2025-08-16T00:03:47.975327-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:03:47.984387-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0 2025-08-16T00:03:47.986293-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes 2025-08-16T00:04:01.015128-04:00 localhost systemd[3865]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings... 2025-08-16T00:04:01.027782-04:00 localhost systemd[3865]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings. 2025-08-16T00:05:03.557174-04:00 localhost pop-upgrade[6904]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-16T00:28:46.655514-04:00 localhost nvidia-persistenced: Received signal 15 2025-08-16T00:28:46.655578-04:00 localhost systemd[1]: Stopping nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:28:46.655613-04:00 localhost systemd[1]: Stopping nvidia-powerd.service - nvidia-powerd service... 2025-08-16T00:28:46.656106-04:00 localhost nvidia-persistenced: Socket closed. 2025-08-16T00:28:46.656145-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T00:28:46.656161-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T00:28:46.656174-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T00:28:46.656189-04:00 localhost nvidia-persistenced: Shutdown (1975) 2025-08-16T00:29:37.686108-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service. 2025-08-16T00:29:37.686131-04:00 localhost /usr/bin/nvidia-powerd[1624]: nvidia-powerd version:1.0(build 1) 2025-08-16T00:29:37.686604-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T00:29:37.686605-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none 2025-08-16T00:29:37.686614-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 570.172.08 Release Build (dvs-builder@U22-I3-AF01-21-3) Tue Jul 8 17:59:47 UTC 2025 2025-08-16T00:29:37.686616-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver 2025-08-16T00:29:37.993971-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:29:37.995796-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:29:37.995862-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:29:37.995894-04:00 localhost nvidia-persistenced: Started (1976) 2025-08-16T00:29:39.178898-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered 2025-08-16T00:29:39.178967-04:00 localhost nvidia-persistenced: Local RPC services initialized 2025-08-16T00:29:39.179123-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:29:39.185315-04:00 localhost /usr/bin/nvidia-powerd[1624]: DBus Connection is established 2025-08-16T00:29:39.265710-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0 2025-08-16T00:29:39.268729-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes 2025-08-16T00:29:51.469690-04:00 localhost systemd[3913]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings... 2025-08-16T00:29:51.489307-04:00 localhost systemd[3913]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings. 2025-08-16T00:30:52.732125-04:00 localhost pop-upgrade[5783]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-16T00:38:39.113831-04:00 localhost nvidia-persistenced: Received signal 15 2025-08-16T00:38:39.113890-04:00 localhost systemd[1]: Stopping nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:38:39.113919-04:00 localhost systemd[1]: Stopping nvidia-powerd.service - nvidia-powerd service... 2025-08-16T00:38:39.114390-04:00 localhost nvidia-persistenced: Socket closed. 2025-08-16T00:39:16.152624-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service. 2025-08-16T00:39:16.152638-04:00 localhost /usr/bin/nvidia-powerd[1595]: nvidia-powerd version:1.0(build 1) 2025-08-16T00:39:16.153165-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 510 2025-08-16T00:39:16.153165-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none 2025-08-16T00:39:16.153172-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 570.172.08 Release Build (dvs-builder@U22-I3-AF01-21-3) Tue Jul 8 17:59:47 UTC 2025 2025-08-16T00:39:16.153172-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver 2025-08-16T00:39:16.455685-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T00:39:16.457353-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T00:39:16.457400-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T00:39:16.457414-04:00 localhost nvidia-persistenced: Started (1945) 2025-08-16T00:39:17.592827-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered 2025-08-16T00:39:17.592903-04:00 localhost nvidia-persistenced: Local RPC services initialized 2025-08-16T00:39:17.593385-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T00:39:17.599692-04:00 localhost /usr/bin/nvidia-powerd[1595]: DBus Connection is established 2025-08-16T00:39:17.677502-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0 2025-08-16T00:39:17.680480-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes 2025-08-16T00:39:29.487188-04:00 localhost systemd[3817]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings... 2025-08-16T00:39:29.504186-04:00 localhost systemd[3817]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings. 2025-08-16T00:40:31.795530-04:00 localhost pop-upgrade[6163]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-16T01:05:45.513712-04:00 localhost nvidia-persistenced: Received signal 15 2025-08-16T01:05:45.513775-04:00 localhost systemd[1]: Stopping nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T01:05:45.513807-04:00 localhost systemd[1]: Stopping nvidia-powerd.service - nvidia-powerd service... 2025-08-16T01:05:45.515130-04:00 localhost nvidia-persistenced: Socket closed. 2025-08-16T01:05:45.515187-04:00 localhost nvidia-persistenced: PID file unlocked. 2025-08-16T01:05:45.515207-04:00 localhost nvidia-persistenced: PID file closed. 2025-08-16T01:05:45.515230-04:00 localhost nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced 2025-08-16T01:06:24.758006-04:00 localhost systemd[1]: Started nvidia-powerd.service - nvidia-powerd service. 2025-08-16T01:06:24.758025-04:00 localhost /usr/bin/nvidia-powerd[1584]: nvidia-powerd version:1.0(build 1) 2025-08-16T01:06:24.758611-04:00 localhost kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 2025-08-16T01:06:24.758611-04:00 localhost kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none 2025-08-16T01:06:24.758626-04:00 localhost kernel: nvidia-modeset: Loading NVIDIA UNIX Open Kernel Mode Setting Driver for x86_64 570.172.08 Release Build (dvs-builder@U22-I3-AF01-21-3) Tue Jul 8 17:59:47 UTC 2025 2025-08-16T01:06:24.758627-04:00 localhost kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver 2025-08-16T01:06:25.034464-04:00 localhost systemd[1]: Starting nvidia-persistenced.service - NVIDIA Persistence Daemon... 2025-08-16T01:06:25.036456-04:00 localhost nvidia-persistenced: Verbose syslog connection opened 2025-08-16T01:06:25.036511-04:00 localhost nvidia-persistenced: Now running with user ID 125 and group ID 133 2025-08-16T01:06:25.036527-04:00 localhost nvidia-persistenced: Started (1928) 2025-08-16T01:06:26.204158-04:00 localhost nvidia-persistenced: device 0000:01:00.0 - registered 2025-08-16T01:06:26.204200-04:00 localhost nvidia-persistenced: Local RPC services initialized 2025-08-16T01:06:26.204456-04:00 localhost systemd[1]: Started nvidia-persistenced.service - NVIDIA Persistence Daemon. 2025-08-16T01:06:26.221079-04:00 localhost /usr/bin/nvidia-powerd[1584]: DBus Connection is established 2025-08-16T01:06:26.295496-04:00 localhost kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:01:00.0 on minor 0 2025-08-16T01:06:26.297417-04:00 localhost kernel: nvidia 0000:01:00.0: [drm] Cannot find any crtc or sizes 2025-08-16T01:06:38.034677-04:00 localhost systemd[3832]: Starting app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings... 2025-08-16T01:06:38.053892-04:00 localhost systemd[3832]: Started app-nvidia\x2dsettings\x2dautostart@autostart.service - NVIDIA X Server Settings. 2025-08-16T01:07:42.281343-04:00 localhost pop-upgrade[5748]: [#033[1mINFO #033[0m] release_api.rs#033[0m#033[1m:#033[0m#033[1;35m58#033[0m: checking for build 24.04 in channel nvidia 2025-08-16T11:50:02.424665-04:00 localhost system76-power[1574]: [INFO] Enabling nvidia-fallback.service 2025-08-16T11:50:02.470143-04:00 localhost systemctl[63779]: Failed to enable unit: Unit file nvidia-fallback.service does not exist. ```
GiteaMirror commented 2026-04-12 20:05:10 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 16, 2025):

This is the dmesg off my 4070 machine:

[    6.141058] kernel: nvidia: loading out-of-tree module taints kernel.
[    6.141066] kernel: nvidia: module license 'NVIDIA' taints kernel.
[    6.141069] kernel: nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    6.141070] kernel: nvidia: module license taints kernel.
[    6.261883] kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509
[    6.263106] kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[    6.305586] kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module  550.107.02  Wed Jul 24 23:53:00 UTC 2024
[    6.327451] kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  550.107.02  Wed Jul 24 23:24:27 UTC 2024
[    6.330884] kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[    6.449760] kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input5
[    6.449803] kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input6
[    6.449836] kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input7
[    6.449873] kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input8
[    6.964069] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1
[    6.983090] kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
[    7.007689] kernel: nvidia-uvm: Loaded the UVM driver, major device number 507.

Differences seem to be the loading of the kernel module at the start, and then the loading of nvidia-uvm. What's the output of lsmod | grep -i nvidia?

$ lsmod | grep -i nvidia
nvidia_uvm           4956160  6
nvidia_drm            122880  0
nvidia_modeset       1355776  1 nvidia_drm
nvidia              54296576  54 nvidia_uvm,nvidia_modeset
video                  73728  3 xe,i915,nvidia_modeset
<!-- gh-comment-id:3193943009 --> @rick-github commented on GitHub (Aug 16, 2025): This is the dmesg off my 4070 machine: ``` [ 6.141058] kernel: nvidia: loading out-of-tree module taints kernel. [ 6.141066] kernel: nvidia: module license 'NVIDIA' taints kernel. [ 6.141069] kernel: nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 6.141070] kernel: nvidia: module license taints kernel. [ 6.261883] kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 509 [ 6.263106] kernel: nvidia 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none [ 6.305586] kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 550.107.02 Wed Jul 24 23:53:00 UTC 2024 [ 6.327451] kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 550.107.02 Wed Jul 24 23:24:27 UTC 2024 [ 6.330884] kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver [ 6.449760] kernel: input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input5 [ 6.449803] kernel: input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input6 [ 6.449836] kernel: input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input7 [ 6.449873] kernel: input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:01.0/0000:01:00.1/sound/card2/input8 [ 6.964069] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1 [ 6.983090] kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint. [ 7.007689] kernel: nvidia-uvm: Loaded the UVM driver, major device number 507. ``` Differences seem to be the loading of the kernel module at the start, and then the loading of nvidia-uvm. What's the output of `lsmod | grep -i nvidia`? ``` $ lsmod | grep -i nvidia nvidia_uvm 4956160 6 nvidia_drm 122880 0 nvidia_modeset 1355776 1 nvidia_drm nvidia 54296576 54 nvidia_uvm,nvidia_modeset video 73728 3 xe,i915,nvidia_modeset ```
GiteaMirror commented 2026-04-12 20:05:11 -05:00
Author
Owner
Copy Link

@guusec commented on GitHub (Aug 17, 2025):

I'm seeing this

lsmod | grep -i nvidia
nvidia_uvm           2125824  0
nvidia_drm            131072  6
nvidia_modeset       1724416  2 nvidia_drm
nvidia              11649024  21 nvidia_uvm,nvidia_modeset
drm_ttm_helper         16384  2 nvidia_drm,xe
video                  77824  3 xe,i915,nvidia_modeset
<!-- gh-comment-id:3194032324 --> @guusec commented on GitHub (Aug 17, 2025): I'm seeing this ``` lsmod | grep -i nvidia nvidia_uvm 2125824 0 nvidia_drm 131072 6 nvidia_modeset 1724416 2 nvidia_drm nvidia 11649024 21 nvidia_uvm,nvidia_modeset drm_ttm_helper 16384 2 nvidia_drm,xe video 77824 3 xe,i915,nvidia_modeset ```
GiteaMirror commented 2026-04-12 20:05:12 -05:00
Author
Owner
Copy Link

@guusec commented on GitHub (Aug 17, 2025):

I seem to have gotten it working. The default installed nvidia package on pop-os cosmic is the open-source one; nvidia-driver-570-open. This seems to be missing nvidia-uvm support? To fix install nvidia-driver-570. Now running at 58%/42% CPU/GPU

<!-- gh-comment-id:3194037633 --> @guusec commented on GitHub (Aug 17, 2025): I seem to have gotten it working. The default installed nvidia package on pop-os cosmic is the open-source one; `nvidia-driver-570-open`. This seems to be missing nvidia-uvm support? To fix install `nvidia-driver-570`. Now running at 58%/42% CPU/GPU
GiteaMirror commented 2026-04-12 20:05:13 -05:00
Author
Owner
Copy Link

@eliot-akira commented on GitHub (Dec 15, 2025):

Thank you for posting the solution, @guusec. I was experiencing the same issue after an upgrade to PopOS 24. Running sudo apt install nvidia-driver-570 solved it for me also.

<!-- gh-comment-id:3656307238 --> @eliot-akira commented on GitHub (Dec 15, 2025): Thank you for posting the solution, @guusec. I was experiencing the same issue after an upgrade to PopOS 24. Running `sudo apt install nvidia-driver-570` solved it for me also.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#7919
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?