github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 08:02:14 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #6515] "ollama run qwen2" return "the resource allocation failed" #4102

New Issue
Closed
opened 2026-04-12 15:00:13 -05:00 by GiteaMirror · 6 comments
GiteaMirror commented 2026-04-12 15:00:13 -05:00
Owner
Copy Link

Originally created by @fenggaobj on GitHub (Aug 26, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/6515

Originally assigned to: @dhiltgen on GitHub.

What is the issue?

When executing "ollama run qwen2" in the Nvidia Jetson AGX Orin environment, a "terminated" error was returned. Could you please help me identify the cause?

(ollama) nvidia@ubuntu:~$ ollama run qwen2
Error: llama runner process has terminated: CUDA error: the resource allocation failed
  current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644
  cublasCreate_v2(&cublas_handles[device])
/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error

(1) Environment and version information:

Device: Nvidia Jetson AGX Orin
CPU: 12 cores with a frequency of 2.2GHz
Memory: 64GB
GPU: 1.3GHz
Operating System: Ubuntu 22.04.4 LTS
Ollama Version: 0.3.7-rc6

cuda information(cuda-12)：

(ollama) nvidia@ubuntu:~$ jetson_release
Software part of jetson-stats 4.2.9 - (c) 2024, Raffaello Bonghi
Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0]
NV Power Mode[0]: MAXN
Serial Number: [XXX Show with: jetson_release -s XXX]
Hardware:
 - P-Number: p3701-0005
 - Module: NVIDIA Jetson AGX Orin (64GB ram)
Platform:
 - Distribution: Ubuntu 22.04 Jammy Jellyfish
 - Release: 5.15.136-tegra
jtop:
 - Version: 4.2.9
 - Service: Active
Libraries:
 - CUDA: 12.2.140
 - cuDNN: 8.9.4.25
 - TensorRT: 8.6.2.3
 - VPI: 3.1.5
 - Vulkan: 1.3.204
 - OpenCV: 4.8.0 - with CUDA: NO

(2) Log from ollama.service in DEBUG mode:

Aug 26 20:32:03 ubuntu ollama[31981]: CUDA error: the resource allocation failed
Aug 26 20:32:03 ubuntu ollama[31981]:   current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644
Aug 26 20:32:03 ubuntu ollama[31981]:   cublasCreate_v2(&cublas_handles[device])
Aug 26 20:32:03 ubuntu ollama[31981]: /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error
Aug 26 20:32:03 ubuntu ollama[31981]: Could not attach to process.  If your uid matches the uid of the target
Aug 26 20:32:03 ubuntu ollama[31981]: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
Aug 26 20:32:03 ubuntu ollama[31981]: again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
Aug 26 20:32:03 ubuntu ollama[31981]: ptrace: Inappropriate ioctl for device.
Aug 26 20:32:03 ubuntu ollama[31981]: No stack.
Aug 26 20:32:03 ubuntu ollama[31981]: The program is not being run.
Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.025+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.025+08:00 level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server error"
Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.053+08:00 level=DEBUG source=server.go:431 msg="llama runner terminated" error="signal: aborted (core dumped)"
Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.275+08:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: CUDA error: the resource allocation failed\n  current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644\n  cublasCreate_v2(&cublas_handles[device])\n/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error"
Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.275+08:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5

OS

Linux

GPU

Nvidia

CPU

Other

Ollama version

0.3.7-rc6

Originally created by @fenggaobj on GitHub (Aug 26, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/6515 Originally assigned to: @dhiltgen on GitHub. ### What is the issue? When executing "ollama run qwen2" in the Nvidia Jetson AGX Orin environment, a "terminated" error was returned. Could you please help me identify the cause? ``` (ollama) nvidia@ubuntu:~$ ollama run qwen2 Error: llama runner process has terminated: CUDA error: the resource allocation failed current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644 cublasCreate_v2(&cublas_handles[device]) /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error ``` **(1) Environment and version information:** Device: Nvidia Jetson AGX Orin CPU: 12 cores with a frequency of 2.2GHz Memory: 64GB GPU: 1.3GHz Operating System: Ubuntu 22.04.4 LTS Ollama Version: 0.3.7-rc6 cuda information(cuda-12)： ``` (ollama) nvidia@ubuntu:~$ jetson_release Software part of jetson-stats 4.2.9 - (c) 2024, Raffaello Bonghi Model: NVIDIA Jetson AGX Orin Developer Kit - Jetpack 6.0 [L4T 36.3.0] NV Power Mode[0]: MAXN Serial Number: [XXX Show with: jetson_release -s XXX] Hardware: - P-Number: p3701-0005 - Module: NVIDIA Jetson AGX Orin (64GB ram) Platform: - Distribution: Ubuntu 22.04 Jammy Jellyfish - Release: 5.15.136-tegra jtop: - Version: 4.2.9 - Service: Active Libraries: - CUDA: 12.2.140 - cuDNN: 8.9.4.25 - TensorRT: 8.6.2.3 - VPI: 3.1.5 - Vulkan: 1.3.204 - OpenCV: 4.8.0 - with CUDA: NO ``` **(2) Log from ollama.service in DEBUG mode:** ``` Aug 26 20:32:03 ubuntu ollama[31981]: CUDA error: the resource allocation failed Aug 26 20:32:03 ubuntu ollama[31981]: current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644 Aug 26 20:32:03 ubuntu ollama[31981]: cublasCreate_v2(&cublas_handles[device]) Aug 26 20:32:03 ubuntu ollama[31981]: /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error Aug 26 20:32:03 ubuntu ollama[31981]: Could not attach to process. If your uid matches the uid of the target Aug 26 20:32:03 ubuntu ollama[31981]: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try Aug 26 20:32:03 ubuntu ollama[31981]: again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf Aug 26 20:32:03 ubuntu ollama[31981]: ptrace: Inappropriate ioctl for device. Aug 26 20:32:03 ubuntu ollama[31981]: No stack. Aug 26 20:32:03 ubuntu ollama[31981]: The program is not being run. Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.025+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.025+08:00 level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server error" Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.053+08:00 level=DEBUG source=server.go:431 msg="llama runner terminated" error="signal: aborted (core dumped)" Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.275+08:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: CUDA error: the resource allocation failed\n current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644\n cublasCreate_v2(&cublas_handles[device])\n/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error" Aug 26 20:32:04 ubuntu ollama[31981]: time=2024-08-26T20:32:04.275+08:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 ``` ### OS Linux ### GPU Nvidia ### CPU Other ### Ollama version 0.3.7-rc6
GiteaMirror added the nvidiabug labels 2026-04-12 15:00:13 -05:00
GiteaMirror commented 2026-04-12 15:00:14 -05:00
Author
Owner
Copy Link

@igorschlum commented on GitHub (Aug 26, 2024):

@fenggaobj it works with MacOS version of Ollama. Could you try

ollama run llama3.1

And share if it works?

<!-- gh-comment-id:2310212325 --> @igorschlum commented on GitHub (Aug 26, 2024): @fenggaobj it works with MacOS version of Ollama. Could you try ```bash ollama run llama3.1 ``` And share if it works?
GiteaMirror commented 2026-04-12 15:00:15 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 26, 2024):

resource allocation failed might indicate a lack of resources. If you show the full log it will be easier to diagnose.

<!-- gh-comment-id:2311106429 --> @rick-github commented on GitHub (Aug 26, 2024): `resource allocation failed` might indicate a lack of resources. If you show the full log it will be easier to diagnose.
GiteaMirror commented 2026-04-12 15:00:15 -05:00
Author
Owner
Copy Link

@fenggaobj commented on GitHub (Aug 27, 2024):

resource allocation failed might indicate a lack of resources. If you show the full log it will be easier to diagnose.
@rick-github
thanks for your help, the full log is following:

Aug 27 09:42:58 ubuntu ollama[31981]: [GIN] 2024/08/27 - 09:42:58 | 200 |      50.655µs |       127.0.0.1 | HEAD     "/"
Aug 27 09:42:58 ubuntu ollama[31981]: [GIN] 2024/08/27 - 09:42:58 | 200 |   26.874329ms |       127.0.0.1 | POST     "/api/show"
Aug 27 09:42:58 ubuntu ollama[31981]: time=2024-08-27T09:42:58.959+08:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="61.4 GiB" before.free="51.7 GiB" before.free_swap="30.7 GiB" now.total="61.4 GiB" now.free="51.7 GiB" now.free_swap="30.7 GiB"
Aug 27 09:42:58 ubuntu ollama[31981]: CUDA driver version: 12.2
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.151+08:00 level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 name=Orin overhead="0 B" before.total="61.4 GiB" before.free="51.5 GiB" now.total="61.4 GiB" now.free="51.4 GiB" now.used="9.9 GiB"
Aug 27 09:42:59 ubuntu ollama[31981]: releasing cuda driver library
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.190+08:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.190+08:00 level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[51.4 GiB]"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.191+08:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 parallel=4 available=55215149056 required="5.3 GiB"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.191+08:00 level=DEBUG source=server.go:101 msg="system memory" total="61.4 GiB" free="51.7 GiB" free_swap="30.7 GiB"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.191+08:00 level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[51.4 GiB]"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[51.4 GiB]" memory.required.full="5.3 GiB" memory.required.partial="5.3 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[5.3 GiB]" memory.weights.total="3.9 GiB" memory.weights.repeating="3.4 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="478.0 MiB" memory.graph.partial="730.4 MiB"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cpu/ollama_llama_server
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v11/ollama_llama_server
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v12/ollama_llama_server
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cpu/ollama_llama_server
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v11/ollama_llama_server
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v12/ollama_llama_server
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=INFO source=server.go:174 msg="user override" OLLAMA_LLM_LIBRARY=cuda_v12 path=/tmp/ollama2044817588/runners/cuda_v12
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.193+08:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama2044817588/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 29 --verbose --numa numactl --parallel 4 --port 37685"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.193+08:00 level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin LD_LIBRARY_PATH=/usr/local/lib/ollama:/tmp/ollama2044817588/runners/cuda_v12 CUDA_VISIBLE_DEVICES=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122]"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.194+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=1
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.194+08:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding"
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.195+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
Aug 27 09:42:59 ubuntu ollama[44625]: INFO [main] build info | build=1 commit="1e6f655" tid="281472211150912" timestamp=1724722979
Aug 27 09:42:59 ubuntu ollama[44625]: INFO [main] system info | n_threads=12 n_threads_batch=-1 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="281472211150912" timestamp=1724722979 total_threads=12
Aug 27 09:42:59 ubuntu ollama[44625]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="11" port="37685" tid="281472211150912" timestamp=1724722979
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: loaded meta data with 21 key-value pairs and 339 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 (version GGUF V3 (latest))
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   0:                       general.architecture str              = qwen2
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   1:                               general.name str              = Qwen2-7B-Instruct
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   2:                          qwen2.block_count u32              = 28
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   3:                       qwen2.context_length u32              = 32768
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   4:                     qwen2.embedding_length u32              = 3584
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   5:                  qwen2.feed_forward_length u32              = 18944
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   6:                 qwen2.attention.head_count u32              = 28
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   7:              qwen2.attention.head_count_kv u32              = 4
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   8:                       qwen2.rope.freq_base f32              = 1000000.000000
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv   9:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  10:                          general.file_type u32              = 2
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = gpt2
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  12:                         tokenizer.ggml.pre str              = qwen2
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  14:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  15:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 151645
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  17:            tokenizer.ggml.padding_token_id u32              = 151643
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 151643
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  19:                    tokenizer.chat_template str              = {% for message in messages %}{% if lo...
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv  20:               general.quantization_version u32              = 2
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - type  f32:  141 tensors
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - type q4_0:  197 tensors
Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - type q6_K:    1 tensors
Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.446+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model"
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_vocab: special tokens cache size = 421
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_vocab: token to piece cache size = 0.9352 MB
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: format           = GGUF V3 (latest)
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: arch             = qwen2
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: vocab type       = BPE
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_vocab          = 152064
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_merges         = 151387
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: vocab_only       = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_ctx_train      = 32768
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd           = 3584
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_layer          = 28
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_head           = 28
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_head_kv        = 4
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_rot            = 128
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_swa            = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_head_k    = 128
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_head_v    = 128
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_gqa            = 7
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_k_gqa     = 512
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_v_gqa     = 512
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_norm_eps       = 0.0e+00
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_clamp_kqv      = 0.0e+00
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_logit_scale    = 0.0e+00
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_ff             = 18944
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_expert         = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_expert_used    = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: causal attn      = 1
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: pooling type     = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: rope type        = 2
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: rope scaling     = linear
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: freq_base_train  = 1000000.0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: freq_scale_train = 1
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_ctx_orig_yarn  = 32768
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: rope_finetuned   = unknown
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_d_conv       = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_d_inner      = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_d_state      = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_dt_rank      = 0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model type       = ?B
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model ftype      = Q4_0
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model params     = 7.62 B
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model size       = 4.12 GiB (4.65 BPW)
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: general.name     = Qwen2-7B-Instruct
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: LF token         = 148848 'ÄĬ'
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: max token length = 256
Aug 27 09:42:59 ubuntu ollama[31981]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
Aug 27 09:42:59 ubuntu ollama[31981]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
Aug 27 09:42:59 ubuntu ollama[31981]: ggml_cuda_init: found 1 CUDA devices:
Aug 27 09:42:59 ubuntu ollama[31981]:   Device 0: Orin, compute capability 8.7, VMM: yes
Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_tensors: ggml ctx size =    0.30 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: offloading 28 repeating layers to GPU
Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: offloading non-repeating layers to GPU
Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: offloaded 29/29 layers to GPU
Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors:        CPU buffer size =   292.36 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors:      CUDA0 buffer size =  3928.07 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.202+08:00 level=DEBUG source=server.go:636 msg="model load progress 0.24"
Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.453+08:00 level=DEBUG source=server.go:636 msg="model load progress 0.60"
Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.705+08:00 level=DEBUG source=server.go:636 msg="model load progress 0.94"
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: n_ctx      = 8192
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: n_batch    = 512
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: n_ubatch   = 512
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: flash_attn = 0
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: freq_base  = 1000000.0
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: freq_scale = 1
Aug 27 09:43:00 ubuntu ollama[31981]: llama_kv_cache_init:      CUDA0 KV buffer size =   448.00 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: KV self size  =  448.00 MiB, K (f16):  224.00 MiB, V (f16):  224.00 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model:  CUDA_Host  output buffer size =     2.38 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.957+08:00 level=DEBUG source=server.go:636 msg="model load progress 1.00"
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model:      CUDA0 compute buffer size =   492.00 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model:  CUDA_Host compute buffer size =    23.01 MiB
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: graph nodes  = 986
Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: graph splits = 2
Aug 27 09:43:01 ubuntu ollama[31981]: CUDA error: the resource allocation failed
Aug 27 09:43:01 ubuntu ollama[31981]:   current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644
Aug 27 09:43:01 ubuntu ollama[31981]:   cublasCreate_v2(&cublas_handles[device])
Aug 27 09:43:01 ubuntu ollama[31981]: /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error
Aug 27 09:43:01 ubuntu ollama[31981]: Could not attach to process.  If your uid matches the uid of the target
Aug 27 09:43:01 ubuntu ollama[31981]: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
Aug 27 09:43:01 ubuntu ollama[31981]: again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
Aug 27 09:43:01 ubuntu ollama[31981]: ptrace: Inappropriate ioctl for device.
Aug 27 09:43:01 ubuntu ollama[31981]: No stack.
Aug 27 09:43:01 ubuntu ollama[31981]: The program is not being run.
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.393+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error"
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.393+08:00 level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server error"
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.426+08:00 level=DEBUG source=server.go:431 msg="llama runner terminated" error="signal: aborted (core dumped)"
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.644+08:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: CUDA error: the resource allocation failed\n  current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644\n  cublasCreate_v2(&cublas_handles[device])\n/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error"
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=sched.go:376 msg="got lock to unload" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:43:01 ubuntu ollama[31981]: [GIN] 2024/08/27 - 09:43:01 | 500 |  2.710768583s |       127.0.0.1 | POST     "/api/chat"
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="61.4 GiB" before.free="51.7 GiB" before.free_swap="30.7 GiB" now.total="61.4 GiB" now.free="51.7 GiB" now.free_swap="30.7 GiB"
Aug 27 09:43:01 ubuntu ollama[31981]: CUDA driver version: 12.2
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.730+08:00 level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 name=Orin overhead="0 B" before.total="61.4 GiB" before.free="51.4 GiB" now.total="61.4 GiB" now.free="51.4 GiB" now.used="9.9 GiB"
Aug 27 09:43:01 ubuntu ollama[31981]: releasing cuda driver library
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.730+08:00 level=DEBUG source=server.go:1048 msg="stopping llama server"
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.730+08:00 level=DEBUG source=sched.go:381 msg="runner released" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.981+08:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="61.4 GiB" before.free="51.7 GiB" before.free_swap="30.7 GiB" now.total="61.4 GiB" now.free="51.7 GiB" now.free_swap="30.7 GiB"
Aug 27 09:43:01 ubuntu ollama[31981]: CUDA driver version: 12.2
Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 name=Orin overhead="0 B" before.total="61.4 GiB" before.free="51.4 GiB" now.total="61.4 GiB" now.free="51.4 GiB" now.used="9.9 GiB"
Aug 27 09:43:02 ubuntu ollama[31981]: releasing cuda driver library
Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=sched.go:660 msg="gpu VRAM free memory converged after 0.42 seconds" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=sched.go:385 msg="sending an unloaded event" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5
Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests"
<!-- gh-comment-id:2311408927 --> @fenggaobj commented on GitHub (Aug 27, 2024): > `resource allocation failed` might indicate a lack of resources. If you show the full log it will be easier to diagnose. @rick-github **thanks for your help, the full log is following:** ``` Aug 27 09:42:58 ubuntu ollama[31981]: [GIN] 2024/08/27 - 09:42:58 | 200 | 50.655µs | 127.0.0.1 | HEAD "/" Aug 27 09:42:58 ubuntu ollama[31981]: [GIN] 2024/08/27 - 09:42:58 | 200 | 26.874329ms | 127.0.0.1 | POST "/api/show" Aug 27 09:42:58 ubuntu ollama[31981]: time=2024-08-27T09:42:58.959+08:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="61.4 GiB" before.free="51.7 GiB" before.free_swap="30.7 GiB" now.total="61.4 GiB" now.free="51.7 GiB" now.free_swap="30.7 GiB" Aug 27 09:42:58 ubuntu ollama[31981]: CUDA driver version: 12.2 Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.151+08:00 level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 name=Orin overhead="0 B" before.total="61.4 GiB" before.free="51.5 GiB" now.total="61.4 GiB" now.free="51.4 GiB" now.used="9.9 GiB" Aug 27 09:42:59 ubuntu ollama[31981]: releasing cuda driver library Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.190+08:00 level=DEBUG source=sched.go:224 msg="loading first model" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.190+08:00 level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[51.4 GiB]" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.191+08:00 level=INFO source=sched.go:715 msg="new model will fit in available VRAM in single GPU, loading" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 parallel=4 available=55215149056 required="5.3 GiB" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.191+08:00 level=DEBUG source=server.go:101 msg="system memory" total="61.4 GiB" free="51.7 GiB" free_swap="30.7 GiB" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.191+08:00 level=DEBUG source=memory.go:101 msg=evaluating library=cuda gpu_count=1 available="[51.4 GiB]" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=INFO source=memory.go:309 msg="offload to cuda" layers.requested=-1 layers.model=29 layers.offload=29 layers.split="" memory.available="[51.4 GiB]" memory.required.full="5.3 GiB" memory.required.partial="5.3 GiB" memory.required.kv="448.0 MiB" memory.required.allocations="[5.3 GiB]" memory.weights.total="3.9 GiB" memory.weights.repeating="3.4 GiB" memory.weights.nonrepeating="426.4 MiB" memory.graph.full="478.0 MiB" memory.graph.partial="730.4 MiB" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cpu/ollama_llama_server Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v11/ollama_llama_server Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v12/ollama_llama_server Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cpu/ollama_llama_server Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v11/ollama_llama_server Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=DEBUG source=payload.go:71 msg="availableServers : found" file=/tmp/ollama2044817588/runners/cuda_v12/ollama_llama_server Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.192+08:00 level=INFO source=server.go:174 msg="user override" OLLAMA_LLM_LIBRARY=cuda_v12 path=/tmp/ollama2044817588/runners/cuda_v12 Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.193+08:00 level=INFO source=server.go:391 msg="starting llama server" cmd="/tmp/ollama2044817588/runners/cuda_v12/ollama_llama_server --model /usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 --ctx-size 8192 --batch-size 512 --embedding --log-disable --n-gpu-layers 29 --verbose --numa numactl --parallel 4 --port 37685" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.193+08:00 level=DEBUG source=server.go:408 msg=subprocess environment="[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin LD_LIBRARY_PATH=/usr/local/lib/ollama:/tmp/ollama2044817588/runners/cuda_v12 CUDA_VISIBLE_DEVICES=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122]" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.194+08:00 level=INFO source=sched.go:450 msg="loaded runners" count=1 Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.194+08:00 level=INFO source=server.go:591 msg="waiting for llama runner to start responding" Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.195+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" Aug 27 09:42:59 ubuntu ollama[44625]: INFO [main] build info | build=1 commit="1e6f655" tid="281472211150912" timestamp=1724722979 Aug 27 09:42:59 ubuntu ollama[44625]: INFO [main] system info | n_threads=12 n_threads_batch=-1 system_info="AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="281472211150912" timestamp=1724722979 total_threads=12 Aug 27 09:42:59 ubuntu ollama[44625]: INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="11" port="37685" tid="281472211150912" timestamp=1724722979 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: loaded meta data with 21 key-value pairs and 339 tensors from /usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 (version GGUF V3 (latest)) Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 0: general.architecture str = qwen2 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 1: general.name str = Qwen2-7B-Instruct Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 2: qwen2.block_count u32 = 28 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 3: qwen2.context_length u32 = 32768 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 4: qwen2.embedding_length u32 = 3584 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 5: qwen2.feed_forward_length u32 = 18944 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 6: qwen2.attention.head_count u32 = 28 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 7: qwen2.attention.head_count_kv u32 = 4 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 8: qwen2.rope.freq_base f32 = 1000000.000000 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 9: qwen2.attention.layer_norm_rms_epsilon f32 = 0.000001 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 10: general.file_type u32 = 2 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 11: tokenizer.ggml.model str = gpt2 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 12: tokenizer.ggml.pre str = qwen2 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 13: tokenizer.ggml.tokens arr[str,152064] = ["!", "\"", "#", "$", "%", "&", "'", ... Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,152064] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 15: tokenizer.ggml.merges arr[str,151387] = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",... Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 151645 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 17: tokenizer.ggml.padding_token_id u32 = 151643 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 151643 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 19: tokenizer.chat_template str = {% for message in messages %}{% if lo... Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - kv 20: general.quantization_version u32 = 2 Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - type f32: 141 tensors Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - type q4_0: 197 tensors Aug 27 09:42:59 ubuntu ollama[31981]: llama_model_loader: - type q6_K: 1 tensors Aug 27 09:42:59 ubuntu ollama[31981]: time=2024-08-27T09:42:59.446+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server loading model" Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_vocab: special tokens cache size = 421 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_vocab: token to piece cache size = 0.9352 MB Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: format = GGUF V3 (latest) Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: arch = qwen2 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: vocab type = BPE Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_vocab = 152064 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_merges = 151387 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: vocab_only = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_ctx_train = 32768 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd = 3584 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_layer = 28 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_head = 28 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_head_kv = 4 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_rot = 128 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_swa = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_head_k = 128 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_head_v = 128 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_gqa = 7 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_k_gqa = 512 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_embd_v_gqa = 512 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_norm_eps = 0.0e+00 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_norm_rms_eps = 1.0e-06 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_clamp_kqv = 0.0e+00 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_max_alibi_bias = 0.0e+00 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: f_logit_scale = 0.0e+00 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_ff = 18944 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_expert = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_expert_used = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: causal attn = 1 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: pooling type = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: rope type = 2 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: rope scaling = linear Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: freq_base_train = 1000000.0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: freq_scale_train = 1 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: n_ctx_orig_yarn = 32768 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: rope_finetuned = unknown Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_d_conv = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_d_inner = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_d_state = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: ssm_dt_rank = 0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model type = ?B Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model ftype = Q4_0 Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model params = 7.62 B Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: model size = 4.12 GiB (4.65 BPW) Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: general.name = Qwen2-7B-Instruct Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: BOS token = 151643 '<|endoftext|>' Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: EOS token = 151645 '<|im_end|>' Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: PAD token = 151643 '<|endoftext|>' Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: LF token = 148848 'ÄĬ' Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: EOT token = 151645 '<|im_end|>' Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_print_meta: max token length = 256 Aug 27 09:42:59 ubuntu ollama[31981]: ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no Aug 27 09:42:59 ubuntu ollama[31981]: ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no Aug 27 09:42:59 ubuntu ollama[31981]: ggml_cuda_init: found 1 CUDA devices: Aug 27 09:42:59 ubuntu ollama[31981]: Device 0: Orin, compute capability 8.7, VMM: yes Aug 27 09:42:59 ubuntu ollama[31981]: llm_load_tensors: ggml ctx size = 0.30 MiB Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: offloading 28 repeating layers to GPU Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: offloading non-repeating layers to GPU Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: offloaded 29/29 layers to GPU Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: CPU buffer size = 292.36 MiB Aug 27 09:43:00 ubuntu ollama[31981]: llm_load_tensors: CUDA0 buffer size = 3928.07 MiB Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.202+08:00 level=DEBUG source=server.go:636 msg="model load progress 0.24" Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.453+08:00 level=DEBUG source=server.go:636 msg="model load progress 0.60" Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.705+08:00 level=DEBUG source=server.go:636 msg="model load progress 0.94" Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: n_ctx = 8192 Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: n_batch = 512 Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: n_ubatch = 512 Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: flash_attn = 0 Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: freq_base = 1000000.0 Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: freq_scale = 1 Aug 27 09:43:00 ubuntu ollama[31981]: llama_kv_cache_init: CUDA0 KV buffer size = 448.00 MiB Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: KV self size = 448.00 MiB, K (f16): 224.00 MiB, V (f16): 224.00 MiB Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: CUDA_Host output buffer size = 2.38 MiB Aug 27 09:43:00 ubuntu ollama[31981]: time=2024-08-27T09:43:00.957+08:00 level=DEBUG source=server.go:636 msg="model load progress 1.00" Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: CUDA0 compute buffer size = 492.00 MiB Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: CUDA_Host compute buffer size = 23.01 MiB Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: graph nodes = 986 Aug 27 09:43:00 ubuntu ollama[31981]: llama_new_context_with_model: graph splits = 2 Aug 27 09:43:01 ubuntu ollama[31981]: CUDA error: the resource allocation failed Aug 27 09:43:01 ubuntu ollama[31981]: current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644 Aug 27 09:43:01 ubuntu ollama[31981]: cublasCreate_v2(&cublas_handles[device]) Aug 27 09:43:01 ubuntu ollama[31981]: /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error Aug 27 09:43:01 ubuntu ollama[31981]: Could not attach to process. If your uid matches the uid of the target Aug 27 09:43:01 ubuntu ollama[31981]: process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try Aug 27 09:43:01 ubuntu ollama[31981]: again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf Aug 27 09:43:01 ubuntu ollama[31981]: ptrace: Inappropriate ioctl for device. Aug 27 09:43:01 ubuntu ollama[31981]: No stack. Aug 27 09:43:01 ubuntu ollama[31981]: The program is not being run. Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.393+08:00 level=INFO source=server.go:625 msg="waiting for server to become available" status="llm server error" Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.393+08:00 level=DEBUG source=server.go:639 msg="model load completed, waiting for server to become available" status="llm server error" Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.426+08:00 level=DEBUG source=server.go:431 msg="llama runner terminated" error="signal: aborted (core dumped)" Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.644+08:00 level=ERROR source=sched.go:456 msg="error loading llama server" error="llama runner process has terminated: CUDA error: the resource allocation failed\n current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644\n cublasCreate_v2(&cublas_handles[device])\n/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error" Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=sched.go:459 msg="triggering expiration for failed load" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=sched.go:360 msg="runner expired event received" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=sched.go:376 msg="got lock to unload" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:43:01 ubuntu ollama[31981]: [GIN] 2024/08/27 - 09:43:01 | 500 | 2.710768583s | 127.0.0.1 | POST "/api/chat" Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.645+08:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="61.4 GiB" before.free="51.7 GiB" before.free_swap="30.7 GiB" now.total="61.4 GiB" now.free="51.7 GiB" now.free_swap="30.7 GiB" Aug 27 09:43:01 ubuntu ollama[31981]: CUDA driver version: 12.2 Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.730+08:00 level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 name=Orin overhead="0 B" before.total="61.4 GiB" before.free="51.4 GiB" now.total="61.4 GiB" now.free="51.4 GiB" now.used="9.9 GiB" Aug 27 09:43:01 ubuntu ollama[31981]: releasing cuda driver library Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.730+08:00 level=DEBUG source=server.go:1048 msg="stopping llama server" Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.730+08:00 level=DEBUG source=sched.go:381 msg="runner released" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:43:01 ubuntu ollama[31981]: time=2024-08-27T09:43:01.981+08:00 level=DEBUG source=gpu.go:359 msg="updating system memory data" before.total="61.4 GiB" before.free="51.7 GiB" before.free_swap="30.7 GiB" now.total="61.4 GiB" now.free="51.7 GiB" now.free_swap="30.7 GiB" Aug 27 09:43:01 ubuntu ollama[31981]: CUDA driver version: 12.2 Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=gpu.go:407 msg="updating cuda memory data" gpu=GPU-7f13a82f-7464-5494-b70d-704b2a3e9122 name=Orin overhead="0 B" before.total="61.4 GiB" before.free="51.4 GiB" now.total="61.4 GiB" now.free="51.4 GiB" now.used="9.9 GiB" Aug 27 09:43:02 ubuntu ollama[31981]: releasing cuda driver library Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=sched.go:660 msg="gpu VRAM free memory converged after 0.42 seconds" model=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=sched.go:385 msg="sending an unloaded event" modelPath=/usr/share/ollama/.ollama/models/blobs/sha256-43f7a214e5329f672bb05404cfba1913cbb70fdaa1a17497224e1925046b0ed5 Aug 27 09:43:02 ubuntu ollama[31981]: time=2024-08-27T09:43:02.067+08:00 level=DEBUG source=sched.go:308 msg="ignoring unload event with no pending requests" ```
GiteaMirror commented 2026-04-12 15:00:15 -05:00
Author
Owner
Copy Link

@fenggaobj commented on GitHub (Aug 27, 2024):

it works with MacOS version of Ollama. Could you try
@igorschlum
Thank you for your suggestion. I have tried it in other X86 environments and there is no problem. This issue only occurs on the Orin device. I want to use the Orin GPU for inference. According to your method, I downloaded the llama3.1 model and ran it, but the result was still the same problem. However, thank you very much for your advice

(ollama) nvidia@ubuntu:~$ ollama run llama3.1
Error: llama runner process has terminated: CUDA error: the resource allocation failed
  current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644
  cublasCreate_v2(&cublas_handles[device])
/go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error
<!-- gh-comment-id:2311412585 --> @fenggaobj commented on GitHub (Aug 27, 2024): > it works with MacOS version of Ollama. Could you try @igorschlum Thank you for your suggestion. I have tried it in other X86 environments and there is no problem. This issue only occurs on the Orin device. I want to use the Orin GPU for inference. According to your method, I downloaded the llama3.1 model and ran it, but the result was still the same problem. However, thank you very much for your advice ``` (ollama) nvidia@ubuntu:~$ ollama run llama3.1 Error: llama runner process has terminated: CUDA error: the resource allocation failed current device: 0, in function cublas_handle at /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda/common.cuh:644 cublasCreate_v2(&cublas_handles[device]) /go/src/github.com/ollama/ollama/llm/llama.cpp/ggml/src/ggml-cuda.cu:101: CUDA error ```
GiteaMirror commented 2026-04-12 15:00:16 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Aug 27, 2024):

Jetson Orin support is a work in progress: https://github.com/ollama/ollama/issues/2408, https://github.com/ollama/ollama/issues/4693

<!-- gh-comment-id:2311967679 --> @rick-github commented on GitHub (Aug 27, 2024): Jetson Orin support is a work in progress: https://github.com/ollama/ollama/issues/2408, https://github.com/ollama/ollama/issues/4693
GiteaMirror commented 2026-04-12 15:00:16 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Aug 27, 2024):

Dup of #2408 which will be resolved by #6400

<!-- gh-comment-id:2313541707 --> @dhiltgen commented on GitHub (Aug 27, 2024): Dup of #2408 which will be resolved by #6400
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#4102
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?