github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-06 16:11:34 -05:00
Code Issues 14.5k Packages Projects Releases 120 Wiki Activity

[GH-ISSUE #5361] Ollama running very slow on Windows #49869

New Issue
Closed
opened 2026-04-28 13:15:00 -05:00 by GiteaMirror · 20 comments
GiteaMirror commented 2026-04-28 13:15:00 -05:00
Owner
Copy Link

Originally created by @AbhisheakSaraswat on GitHub (Jun 28, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/5361

What is the issue?

I have pulled a couple of LLMs via Ollama. When I run any LLM, the response is very slow – so much so that I can type faster than the responses I am getting.

My system specifications are: 13th Gen Intel(R) Core(TM) i5-1345U, 1600 MHz, 10 cores, and 12 logical processors.

OS

Windows

GPU

Intel

CPU

Intel

Ollama version

0.1.47

Originally created by @AbhisheakSaraswat on GitHub (Jun 28, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/5361 ### What is the issue? I have pulled a couple of LLMs via Ollama. When I run any LLM, the response is very slow – so much so that I can type faster than the responses I am getting. My system specifications are: 13th Gen Intel(R) Core(TM) i5-1345U, 1600 MHz, 10 cores, and 12 logical processors. ### OS Windows ### GPU Intel ### CPU Intel ### Ollama version 0.1.47
GiteaMirror added the bug label 2026-04-28 13:15:00 -05:00
GiteaMirror commented 2026-04-28 13:15:03 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Jun 28, 2024):

If you add logs, somebody might be able to tell you what's happening.

<!-- gh-comment-id:2197472333 --> @rick-github commented on GitHub (Jun 28, 2024): If you add logs, somebody might be able to tell you what's happening.
GiteaMirror commented 2026-04-28 13:15:03 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jul 2, 2024):

Intel GPU is not yet integrated into our official builds, which means you'll be running on CPU, not your Intel GPU. CPU token rate will generally be slow on everything but the tiny models. Intel GPU support tracked via #1590

<!-- gh-comment-id:2204420057 --> @dhiltgen commented on GitHub (Jul 2, 2024): Intel GPU is not yet integrated into our official builds, which means you'll be running on CPU, not your Intel GPU. CPU token rate will generally be slow on everything but the tiny models. Intel GPU support tracked via #1590
GiteaMirror commented 2026-04-28 13:15:05 -05:00
Author
Owner
Copy Link

@aprisma2008 commented on GitHub (Nov 12, 2024):

I have the same problem.
If I use OOLAMA its about 33% of the speed that I get from KobaldCCC. I tested for example with mradermacher/Nautilus-RP-18B-v2-GGUF.
NVIDIA RTX 4070 TI, 12 GB

<!-- gh-comment-id:2470989016 --> @aprisma2008 commented on GitHub (Nov 12, 2024): I have the same problem. If I use OOLAMA its about 33% of the speed that I get from KobaldCCC. I tested for example with mradermacher/Nautilus-RP-18B-v2-GGUF. NVIDIA RTX 4070 TI, 12 GB
GiteaMirror commented 2026-04-28 13:15:05 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 12, 2024):

You have a different problem, because you are not using an Intel GPU. Create a new issue in the tracker. Include server logs.

<!-- gh-comment-id:2471029224 --> @rick-github commented on GitHub (Nov 12, 2024): You have a different problem, because you are not using an Intel GPU. Create a new issue in the tracker. Include [server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues).
GiteaMirror commented 2026-04-28 13:15:06 -05:00
Author
Owner
Copy Link

@aprisma2008 commented on GitHub (Nov 14, 2024):

Hello. I found it out. It seems to depend on the content size. So if I make
in ollama a new LLM with the create command and set city to lower its the
same fast. How ever it would be great if you can set it just in ollama
instead of creating always a new model. Thanx for your great work. Have a
good time

frob @.***> schrieb am Di., 12. Nov. 2024, 17:40:

You have a different problem, because you are not using an Intel GPU.
Create a new issue in the tracker. Include server logs
https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues
.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/5361#issuecomment-2471029224,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BGX4PLHKCTCBRLFKOHVQLFL2AIVOXAVCNFSM6AAAAABKCJTCCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRGAZDSMRSGQ
.
You are receiving this because you commented.Message ID:
@.***>

<!-- gh-comment-id:2475185596 --> @aprisma2008 commented on GitHub (Nov 14, 2024): Hello. I found it out. It seems to depend on the content size. So if I make in ollama a new LLM with the create command and set city to lower its the same fast. How ever it would be great if you can set it just in ollama instead of creating always a new model. Thanx for your great work. Have a good time frob ***@***.***> schrieb am Di., 12. Nov. 2024, 17:40: > You have a different problem, because you are not using an Intel GPU. > Create a new issue in the tracker. Include server logs > <https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues> > . > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama/issues/5361#issuecomment-2471029224>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BGX4PLHKCTCBRLFKOHVQLFL2AIVOXAVCNFSM6AAAAABKCJTCCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRGAZDSMRSGQ> > . > You are receiving this because you commented.Message ID: > ***@***.***> >
GiteaMirror commented 2026-04-28 13:15:07 -05:00
Author
Owner
Copy Link

@aprisma2008 commented on GitHub (Nov 14, 2024):

CTX

aprisma aprisma @.***> schrieb am Do., 14. Nov. 2024,
02:47:

Hello. I found it out. It seems to depend on the content size. So if I
make in ollama a new LLM with the create command and set city to lower its
the same fast. How ever it would be great if you can set it just in ollama
instead of creating always a new model. Thanx for your great work. Have a
good time

frob @.***> schrieb am Di., 12. Nov. 2024, 17:40:

You have a different problem, because you are not using an Intel GPU.
Create a new issue in the tracker. Include server logs
https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues
.

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/5361#issuecomment-2471029224,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BGX4PLHKCTCBRLFKOHVQLFL2AIVOXAVCNFSM6AAAAABKCJTCCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRGAZDSMRSGQ
.
You are receiving this because you commented.Message ID:
@.***>

<!-- gh-comment-id:2475186806 --> @aprisma2008 commented on GitHub (Nov 14, 2024): CTX aprisma aprisma ***@***.***> schrieb am Do., 14. Nov. 2024, 02:47: > Hello. I found it out. It seems to depend on the content size. So if I > make in ollama a new LLM with the create command and set city to lower its > the same fast. How ever it would be great if you can set it just in ollama > instead of creating always a new model. Thanx for your great work. Have a > good time > > frob ***@***.***> schrieb am Di., 12. Nov. 2024, 17:40: > >> You have a different problem, because you are not using an Intel GPU. >> Create a new issue in the tracker. Include server logs >> <https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues> >> . >> >> — >> Reply to this email directly, view it on GitHub >> <https://github.com/ollama/ollama/issues/5361#issuecomment-2471029224>, >> or unsubscribe >> <https://github.com/notifications/unsubscribe-auth/BGX4PLHKCTCBRLFKOHVQLFL2AIVOXAVCNFSM6AAAAABKCJTCCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRGAZDSMRSGQ> >> . >> You are receiving this because you commented.Message ID: >> ***@***.***> >> >
GiteaMirror commented 2026-04-28 13:15:08 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 14, 2024):

https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size

<!-- gh-comment-id:2476419285 --> @rick-github commented on GitHub (Nov 14, 2024): https://github.com/ollama/ollama/blob/main/docs/faq.md#how-can-i-specify-the-context-window-size
GiteaMirror commented 2026-04-28 13:15:09 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 19, 2024):

i'm trying ollama on Intel CPU Xeon E5, 12 core CPU with 264 GB in a CPU only configuration, the response from Ollama is very slow compared to LM studio on the same machine. i tried to create the model using the same GGUF model file from LM studio but still the performance is poor. any help is appreciated.

<!-- gh-comment-id:2486609287 --> @wixyskywriter commented on GitHub (Nov 19, 2024): i'm trying ollama on Intel CPU Xeon E5, 12 core CPU with 264 GB in a CPU only configuration, the response from Ollama is very slow compared to LM studio on the same machine. i tried to create the model using the same GGUF model file from LM studio but still the performance is poor. any help is appreciated.
GiteaMirror commented 2026-04-28 13:15:11 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 19, 2024):

Server logs will aid in debugging. Comparable logs from LM Studio would also be helpful.

<!-- gh-comment-id:2486639878 --> @rick-github commented on GitHub (Nov 19, 2024): [Server logs](https://github.com/ollama/ollama/blob/main/docs/troubleshooting.md#how-to-troubleshoot-issues) will aid in debugging. Comparable logs from LM Studio would also be helpful.
GiteaMirror commented 2026-04-28 13:15:15 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 20, 2024):

Below the log from ollama debug. It took 19m and no response was generated. Prompt is Hello.
ollama serve
2024/11/20 10:18:30 routes.go:1189: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\Users\Administrator\.ollama\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]"
time=2024-11-20T10:18:30.206+04:00 level=INFO source=images.go:755 msg="total blobs: 19"
time=2024-11-20T10:18:30.212+04:00 level=INFO source=images.go:762 msg="total unused blobs removed: 0"
time=2024-11-20T10:18:30.217+04:00 level=INFO source=routes.go:1240 msg="Listening on 127.0.0.1:11434 (version 0.4.2)"
time=2024-11-20T10:18:30.220+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe
time=2024-11-20T10:18:30.220+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe
time=2024-11-20T10:18:30.221+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe
time=2024-11-20T10:18:30.223+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11\ollama_llama_server.exe
time=2024-11-20T10:18:30.223+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12\ollama_llama_server.exe
time=2024-11-20T10:18:30.224+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm\ollama_llama_server.exe
time=2024-11-20T10:18:30.224+04:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 rocm cpu cpu_avx cpu_avx2 cuda_v11]"
time=2024-11-20T10:18:30.226+04:00 level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY"
time=2024-11-20T10:18:30.226+04:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler"
time=2024-11-20T10:18:30.226+04:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs"
time=2024-11-20T10:18:30.226+04:00 level=INFO source=gpu_windows.go:167 msg=packages count=2
time=2024-11-20T10:18:30.229+04:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=32 efficiency=0 threads=64
time=2024-11-20T10:18:30.229+04:00 level=INFO source=gpu_windows.go:214 msg="" package=1 cores=32 efficiency=0 threads=64
time=2024-11-20T10:18:30.229+04:00 level=DEBUG source=gpu.go:94 msg="searching for GPU discovery libraries for NVIDIA"
time=2024-11-20T10:18:30.229+04:00 level=DEBUG source=gpu.go:509 msg="Searching for GPU library" name=nvml.dll
time=2024-11-20T10:18:30.229+04:00 level=DEBUG source=gpu.go:532 msg="gpu library search" globs="[C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\nvml.dll C:\Windows\system32\nvml.dll C:\Windows\nvml.dll C:\Windows\System32\Wbem\nvml.dll C:\Windows\System32\WindowsPowerShell\v1.0\nvml.dll C:\Windows\System32\OpenSSH\nvml.dll C:\Program Files\SUT\bin\nvml.dll C:\Program Files\McAfee\Solidcore\Tools\GatherInfo\nvml.dll C:\Program Files\McAfee\Solidcore\Tools\Scanalyzer\nvml.dll C:\Program Files\McAfee\Solidcore\nvml.dll C:\Program Files\McAfee\Solidcore\Tools\ScGetCerts\nvml.dll C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps\nvml.dll C:\Users\Administrator\nvml.dll C:\Users\Administrator\.cache\lm-studio\bin\nvml.dll C:\Users\Administrator\AppData\Local\Programs\Ollama\nvml.dll c:\Windows\System32\nvml.dll]"
time=2024-11-20T10:18:30.235+04:00 level=DEBUG source=gpu.go:566 msg="discovered GPU libraries" paths=[]
time=2024-11-20T10:18:30.235+04:00 level=DEBUG source=gpu.go:509 msg="Searching for GPU library" name=nvcuda.dll
time=2024-11-20T10:18:30.235+04:00 level=DEBUG source=gpu.go:532 msg="gpu library search" globs="[C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\nvcuda.dll C:\Windows\system32\nvcuda.dll C:\Windows\nvcuda.dll C:\Windows\System32\Wbem\nvcuda.dll C:\Windows\System32\WindowsPowerShell\v1.0\nvcuda.dll C:\Windows\System32\OpenSSH\nvcuda.dll C:\Program Files\SUT\bin\nvcuda.dll C:\Program Files\McAfee\Solidcore\Tools\GatherInfo\nvcuda.dll C:\Program Files\McAfee\Solidcore\Tools\Scanalyzer\nvcuda.dll C:\Program Files\McAfee\Solidcore\nvcuda.dll C:\Program Files\McAfee\Solidcore\Tools\ScGetCerts\nvcuda.dll C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps\nvcuda.dll C:\Users\Administrator\nvcuda.dll C:\Users\Administrator\.cache\lm-studio\bin\nvcuda.dll C:\Users\Administrator\AppData\Local\Programs\Ollama\nvcuda.dll c:\windows\system\nvcuda.dll]"
time=2024-11-20T10:18:30.247+04:00 level=DEBUG source=gpu.go:566 msg="discovered GPU libraries" paths=[]
time=2024-11-20T10:18:30.247+04:00 level=DEBUG source=gpu.go:509 msg="Searching for GPU library" name=cudart64_.dll
time=2024-11-20T10:18:30.248+04:00 level=DEBUG source=gpu.go:532 msg="gpu library search" globs="[C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\cudart64_.dll C:\Windows\system32\cudart64_.dll C:\Windows\cudart64_.dll C:\Windows\System32\Wbem\cudart64_.dll C:\Windows\System32\WindowsPowerShell\v1.0\cudart64_.dll C:\Windows\System32\OpenSSH\cudart64_.dll C:\Program Files\SUT\bin\cudart64_.dll C:\Program Files\McAfee\Solidcore\Tools\GatherInfo\cudart64_.dll C:\Program Files\McAfee\Solidcore\Tools\Scanalyzer\cudart64_.dll C:\Program Files\McAfee\Solidcore\cudart64_.dll C:\Program Files\McAfee\Solidcore\Tools\ScGetCerts\cudart64_.dll C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps\cudart64_.dll C:\Users\Administrator\cudart64_.dll C:\Users\Administrator\.cache\lm-studio\bin\cudart64_.dll C:\Users\Administrator\AppData\Local\Programs\Ollama\cudart64_.dll C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\cudart64_.dll c:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v\bin\cudart64_*.dll]"
time=2024-11-20T10:18:30.263+04:00 level=DEBUG source=gpu.go:566 msg="discovered GPU libraries" paths="[C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\cudart64_110.dll C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\cudart64_12.dll]"
cudaSetDevice err: 35
time=2024-11-20T10:18:30.269+04:00 level=DEBUG source=gpu.go:582 msg="Unable to load cudart library C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\cudart64_110.dll: your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama"
cudaSetDevice err: 35
time=2024-11-20T10:18:30.273+04:00 level=DEBUG source=gpu.go:582 msg="Unable to load cudart library C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\cudart64_12.dll: your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama"
time=2024-11-20T10:18:30.275+04:00 level=DEBUG source=amd_windows.go:35 msg="unable to load amdhip64_6.dll, please make sure to upgrade to the latest amd driver: The specified module could not be found."
time=2024-11-20T10:18:30.275+04:00 level=INFO source=gpu.go:386 msg="no compatible GPUs were discovered"
time=2024-11-20T10:18:30.275+04:00 level=INFO source=types.go:123 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="255.9 GiB" available="239.9 GiB"
[GIN] 2024/11/20 - 10:19:09 | 200 | 563.2µs | 127.0.0.1 | HEAD "/"
[GIN] 2024/11/20 - 10:19:09 | 200 | 10.9182ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2024/11/20 - 10:19:35 | 200 | 537.3µs | 127.0.0.1 | HEAD "/"
[GIN] 2024/11/20 - 10:19:35 | 200 | 21.5998ms | 127.0.0.1 | POST "/api/show"
time=2024-11-20T10:19:35.868+04:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="255.9 GiB" before.free="239.9 GiB" before.free_swap="277.4 GiB" now.total="255.9 GiB" now.free="239.8 GiB" now.free_swap="277.3 GiB"
time=2024-11-20T10:19:35.868+04:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x7ff7a5d7cf20 gpu_count=1
time=2024-11-20T10:19:35.883+04:00 level=DEBUG source=sched.go:211 msg="cpu mode with first model, loading"
time=2024-11-20T10:19:35.883+04:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="255.9 GiB" before.free="239.8 GiB" before.free_swap="277.3 GiB" now.total="255.9 GiB" now.free="239.8 GiB" now.free_swap="277.3 GiB"
time=2024-11-20T10:19:35.883+04:00 level=INFO source=server.go:105 msg="system memory" total="255.9 GiB" free="239.8 GiB" free_swap="277.3 GiB"
time=2024-11-20T10:19:35.888+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe
time=2024-11-20T10:19:35.889+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe
time=2024-11-20T10:19:35.889+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11\ollama_llama_server.exe
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12\ollama_llama_server.exe
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm\ollama_llama_server.exe
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=memory.go:107 msg=evaluating library=cpu gpu_count=1 available="[239.8 GiB]"
time=2024-11-20T10:19:35.891+04:00 level=INFO source=memory.go:343 msg="offload to cpu" layers.requested=-1 layers.model=33 layers.offload=0 layers.split="" memory.available="[239.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.5 GiB" memory.required.partial="0 B" memory.required.kv="3.0 GiB" memory.required.allocations="[5.5 GiB]" memory.weights.total="4.8 GiB" memory.weights.repeating="4.7 GiB" memory.weights.nonrepeating="77.1 MiB" memory.graph.full="512.0 MiB" memory.graph.partial="512.0 MiB"
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe
time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe
time=2024-11-20T10:19:35.899+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11\ollama_llama_server.exe
time=2024-11-20T10:19:35.904+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12\ollama_llama_server.exe
time=2024-11-20T10:19:35.904+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm\ollama_llama_server.exe
time=2024-11-20T10:19:35.928+04:00 level=DEBUG source=gpu.go:703 msg="no filter required for library cpu"
time=2024-11-20T10:19:35.928+04:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe --model C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 --ctx-size 8192 --batch-size 512 --verbose --threads 64 --no-mmap --parallel 4 --port 53194"
time=2024-11-20T10:19:35.930+04:00 level=DEBUG source=server.go:400 msg=subprocess environment="[PATH=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama;C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\\SUT\bin;C:\Program Files\McAfee\Solidcore\Tools\GatherInfo;C:\Program Files\McAfee\Solidcore\Tools\Scanalyzer;C:\Program Files\McAfee\Solidcore\;C:\Program Files\McAfee\Solidcore\Tools\ScGetCerts;C:\Users\Administrator\AppData\Local\Microsoft\WindowsApps;;C:\Users\Administrator\.cache\lm-studio\bin;C:\Users\Administrator\AppData\Local\Programs\Ollama]"
time=2024-11-20T10:19:35.944+04:00 level=INFO source=sched.go:449 msg="loaded runners" count=1
time=2024-11-20T10:19:35.949+04:00 level=INFO source=server.go:562 msg="waiting for llama runner to start responding"
time=2024-11-20T10:19:35.953+04:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server error"
time=2024-11-20T10:19:35.980+04:00 level=INFO source=runner.go:883 msg="starting go runner"
time=2024-11-20T10:19:35.982+04:00 level=INFO source=runner.go:884 msg=system info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | cgo(clang)" threads=64
time=2024-11-20T10:19:35.984+04:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:53194"
llama_model_loader: loaded meta data with 40 key-value pairs and 197 tensors from C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = phi3
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Phi 3 Mini 128k Instruct
llama_model_loader: - kv 3: general.finetune str = 128k-instruct
llama_model_loader: - kv 4: general.basename str = Phi-3
llama_model_loader: - kv 5: general.size_label str = mini
llama_model_loader: - kv 6: general.license str = mit
llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/microsoft/Phi-...
llama_model_loader: - kv 8: general.tags arr[str,3] = ["nlp", "code", "text-generation"]
llama_model_loader: - kv 9: general.languages arr[str,1] = ["en"]
llama_model_loader: - kv 10: phi3.context_length u32 = 131072
llama_model_loader: - kv 11: phi3.rope.scaling.original_context_length u32 = 4096
llama_model_loader: - kv 12: phi3.embedding_length u32 = 3072
llama_model_loader: - kv 13: phi3.feed_forward_length u32 = 8192
llama_model_loader: - kv 14: phi3.block_count u32 = 32
llama_model_loader: - kv 15: phi3.attention.head_count u32 = 32
llama_model_loader: - kv 16: phi3.attention.head_count_kv u32 = 32
llama_model_loader: - kv 17: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 18: phi3.rope.dimension_count u32 = 96
llama_model_loader: - kv 19: phi3.rope.freq_base f32 = 10000.000000
llama_model_loader: - kv 20: general.file_type u32 = 30
llama_model_loader: - kv 21: phi3.attention.sliding_window u32 = 262144
llama_model_loader: - kv 22: phi3.rope.scaling.attn_factor f32 = 1.190238
llama_model_loader: - kv 23: tokenizer.ggml.model str = llama
llama_model_loader: - kv 24: tokenizer.ggml.pre str = default
llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,32064] = ["", "", "", "<0x00>", "<...
llama_model_loader: - kv 26: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 32000
llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 32000
llama_model_loader: - kv 32: tokenizer.ggml.add_bos_token bool = false
llama_model_loader: - kv 33: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 34: tokenizer.chat_template str = {% for message in messages %}{% if me...
llama_model_loader: - kv 35: general.quantization_version u32 = 2
llama_model_loader: - kv 36: quantize.imatrix.file str = /models_out/Phi-3.1-mini-128k-instruc...
llama_model_loader: - kv 37: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt
llama_model_loader: - kv 38: quantize.imatrix.entries_count i32 = 128
llama_model_loader: - kv 39: quantize.imatrix.chunks_count i32 = 151
llama_model_loader: - type f32: 67 tensors
llama_model_loader: - type q6_K: 1 tensors
llama_model_loader: - type iq4_xs: 129 tensors
llm_load_vocab: special tokens cache size = 14
llm_load_vocab: token to piece cache size = 0.1685 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = phi3
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32064
llm_load_print_meta: n_merges = 0
llm_load_print_meta: vocab_only = 0
llm_load_print_meta: n_ctx_train = 131072
llm_load_print_meta: n_embd = 3072
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 32
llm_load_print_meta: n_rot = 96
llm_load_print_meta: n_swa = 262144
llm_load_print_meta: n_embd_head_k = 96
llm_load_print_meta: n_embd_head_v = 96
llm_load_print_meta: n_gqa = 1
llm_load_print_meta: n_embd_k_gqa = 3072
llm_load_print_meta: n_embd_v_gqa = 3072
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 8192
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 2
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn = 4096
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: ssm_dt_b_c_rms = 0
llm_load_print_meta: model type = 3B
llm_load_print_meta: model ftype = IQ4_XS - 4.25 bpw
llm_load_print_meta: model params = 3.82 B
llm_load_print_meta: model size = 1.92 GiB (4.31 BPW)
llm_load_print_meta: general.name = Phi 3 Mini 128k Instruct
llm_load_print_meta: BOS token = 1 ''
llm_load_print_meta: EOS token = 32000 '<|endoftext|>'
llm_load_print_meta: UNK token = 0 ''
llm_load_print_meta: PAD token = 32000 '<|endoftext|>'
llm_load_print_meta: LF token = 13 '<0x0A>'
llm_load_print_meta: EOT token = 32007 '<|end|>'
llm_load_print_meta: EOG token = 32000 '<|endoftext|>'
llm_load_print_meta: EOG token = 32007 '<|end|>'
llm_load_print_meta: max token length = 48
llm_load_tensors: ggml ctx size = 0.10 MiB
llm_load_tensors: CPU buffer size = 1963.74 MiB
time=2024-11-20T10:19:36.208+04:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server loading model"
time=2024-11-20T10:19:36.208+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.10"
time=2024-11-20T10:19:36.462+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.29"
time=2024-11-20T10:19:36.717+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.46"
time=2024-11-20T10:19:36.968+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.67"
time=2024-11-20T10:19:37.226+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.87"
llama_new_context_with_model: n_ctx = 8192
llama_new_context_with_model: n_batch = 2048
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
time=2024-11-20T10:19:37.484+04:00 level=DEBUG source=server.go:607 msg="model load progress 1.00"
time=2024-11-20T10:19:37.740+04:00 level=DEBUG source=server.go:610 msg="model load completed, waiting for server to become available" status="llm server loading model"
llama_kv_cache_init: CPU KV buffer size = 3072.00 MiB
llama_new_context_with_model: KV self size = 3072.00 MiB, K (f16): 1536.00 MiB, V (f16): 1536.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.54 MiB
llama_new_context_with_model: CPU compute buffer size = 564.01 MiB
llama_new_context_with_model: graph nodes = 1286
llama_new_context_with_model: graph splits = 1
time=2024-11-20T10:19:39.275+04:00 level=INFO source=server.go:601 msg="llama runner started in 3.33 seconds"
time=2024-11-20T10:19:39.275+04:00 level=DEBUG source=sched.go:462 msg="finished setting up runner" model=C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054
[GIN] 2024/11/20 - 10:19:39 | 200 | 3.4274906s | 127.0.0.1 | POST "/api/generate"
time=2024-11-20T10:19:39.278+04:00 level=DEBUG source=sched.go:466 msg="context for request finished"
time=2024-11-20T10:19:39.279+04:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 duration=5m0s
time=2024-11-20T10:19:39.279+04:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 refCount=0
time=2024-11-20T10:19:53.512+04:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054
time=2024-11-20T10:19:53.514+04:00 level=DEBUG source=routes.go:1457 msg="chat request" images=0 prompt="<|user|>\nHello<|end|>\n<|assistant|>\n"
time=2024-11-20T10:19:53.518+04:00 level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=0 prompt=10 used=0 remaining=10
time=2024-11-20T10:39:40.416+04:00 level=DEBUG source=sched.go:407 msg="context for request finished"
[GIN] 2024/11/20 - 10:39:40 | 200 | 19m46s | 127.0.0.1 | POST "/api/chat"
time=2024-11-20T10:39:40.416+04:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 duration=5m0s
time=2024-11-20T10:39:40.425+04:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 refCount=0

<!-- gh-comment-id:2487627107 --> @wixyskywriter commented on GitHub (Nov 20, 2024): Below the log from ollama debug. It took 19m and no response was generated. Prompt is Hello. ollama serve 2024/11/20 10:18:30 routes.go:1189: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_DEBUG:true OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:C:\\Users\\Administrator\\.ollama\\models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://*] OLLAMA_SCHED_SPREAD:false OLLAMA_TMPDIR: ROCR_VISIBLE_DEVICES:]" time=2024-11-20T10:18:30.206+04:00 level=INFO source=images.go:755 msg="total blobs: 19" time=2024-11-20T10:18:30.212+04:00 level=INFO source=images.go:762 msg="total unused blobs removed: 0" time=2024-11-20T10:18:30.217+04:00 level=INFO source=routes.go:1240 msg="Listening on 127.0.0.1:11434 (version 0.4.2)" time=2024-11-20T10:18:30.220+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe time=2024-11-20T10:18:30.220+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe time=2024-11-20T10:18:30.221+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe time=2024-11-20T10:18:30.223+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11\ollama_llama_server.exe time=2024-11-20T10:18:30.223+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12\ollama_llama_server.exe time=2024-11-20T10:18:30.224+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm\ollama_llama_server.exe time=2024-11-20T10:18:30.224+04:00 level=INFO source=common.go:49 msg="Dynamic LLM libraries" runners="[cuda_v12 rocm cpu cpu_avx cpu_avx2 cuda_v11]" time=2024-11-20T10:18:30.226+04:00 level=DEBUG source=common.go:50 msg="Override detection logic by setting OLLAMA_LLM_LIBRARY" time=2024-11-20T10:18:30.226+04:00 level=DEBUG source=sched.go:105 msg="starting llm scheduler" time=2024-11-20T10:18:30.226+04:00 level=INFO source=gpu.go:221 msg="looking for compatible GPUs" time=2024-11-20T10:18:30.226+04:00 level=INFO source=gpu_windows.go:167 msg=packages count=2 time=2024-11-20T10:18:30.229+04:00 level=INFO source=gpu_windows.go:214 msg="" package=0 cores=32 efficiency=0 threads=64 time=2024-11-20T10:18:30.229+04:00 level=INFO source=gpu_windows.go:214 msg="" package=1 cores=32 efficiency=0 threads=64 time=2024-11-20T10:18:30.229+04:00 level=DEBUG source=gpu.go:94 msg="searching for GPU discovery libraries for NVIDIA" time=2024-11-20T10:18:30.229+04:00 level=DEBUG source=gpu.go:509 msg="Searching for GPU library" name=nvml.dll time=2024-11-20T10:18:30.229+04:00 level=DEBUG source=gpu.go:532 msg="gpu library search" globs="[C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvml.dll C:\\Windows\\system32\\nvml.dll C:\\Windows\\nvml.dll C:\\Windows\\System32\\Wbem\\nvml.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\nvml.dll C:\\Windows\\System32\\OpenSSH\\nvml.dll C:\\Program Files\\SUT\\bin\\nvml.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\GatherInfo\\nvml.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\Scanalyzer\\nvml.dll C:\\Program Files\\McAfee\\Solidcore\\nvml.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\ScGetCerts\\nvml.dll C:\\Users\\Administrator\\AppData\\Local\\Microsoft\\WindowsApps\\nvml.dll C:\\Users\\Administrator\\nvml.dll C:\\Users\\Administrator\\.cache\\lm-studio\\bin\\nvml.dll C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\nvml.dll c:\\Windows\\System32\\nvml.dll]" time=2024-11-20T10:18:30.235+04:00 level=DEBUG source=gpu.go:566 msg="discovered GPU libraries" paths=[] time=2024-11-20T10:18:30.235+04:00 level=DEBUG source=gpu.go:509 msg="Searching for GPU library" name=nvcuda.dll time=2024-11-20T10:18:30.235+04:00 level=DEBUG source=gpu.go:532 msg="gpu library search" globs="[C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\nvcuda.dll C:\\Windows\\system32\\nvcuda.dll C:\\Windows\\nvcuda.dll C:\\Windows\\System32\\Wbem\\nvcuda.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\nvcuda.dll C:\\Windows\\System32\\OpenSSH\\nvcuda.dll C:\\Program Files\\SUT\\bin\\nvcuda.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\GatherInfo\\nvcuda.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\Scanalyzer\\nvcuda.dll C:\\Program Files\\McAfee\\Solidcore\\nvcuda.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\ScGetCerts\\nvcuda.dll C:\\Users\\Administrator\\AppData\\Local\\Microsoft\\WindowsApps\\nvcuda.dll C:\\Users\\Administrator\\nvcuda.dll C:\\Users\\Administrator\\.cache\\lm-studio\\bin\\nvcuda.dll C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\nvcuda.dll c:\\windows\\system*\\nvcuda.dll]" time=2024-11-20T10:18:30.247+04:00 level=DEBUG source=gpu.go:566 msg="discovered GPU libraries" paths=[] time=2024-11-20T10:18:30.247+04:00 level=DEBUG source=gpu.go:509 msg="Searching for GPU library" name=cudart64_*.dll time=2024-11-20T10:18:30.248+04:00 level=DEBUG source=gpu.go:532 msg="gpu library search" globs="[C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll C:\\Windows\\system32\\cudart64_*.dll C:\\Windows\\cudart64_*.dll C:\\Windows\\System32\\Wbem\\cudart64_*.dll C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\cudart64_*.dll C:\\Windows\\System32\\OpenSSH\\cudart64_*.dll C:\\Program Files\\SUT\\bin\\cudart64_*.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\GatherInfo\\cudart64_*.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\Scanalyzer\\cudart64_*.dll C:\\Program Files\\McAfee\\Solidcore\\cudart64_*.dll C:\\Program Files\\McAfee\\Solidcore\\Tools\\ScGetCerts\\cudart64_*.dll C:\\Users\\Administrator\\AppData\\Local\\Microsoft\\WindowsApps\\cudart64_*.dll C:\\Users\\Administrator\\cudart64_*.dll C:\\Users\\Administrator\\.cache\\lm-studio\\bin\\cudart64_*.dll C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\cudart64_*.dll C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_*.dll c:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v*\\bin\\cudart64_*.dll]" time=2024-11-20T10:18:30.263+04:00 level=DEBUG source=gpu.go:566 msg="discovered GPU libraries" paths="[C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_110.dll C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_12.dll]" cudaSetDevice err: 35 time=2024-11-20T10:18:30.269+04:00 level=DEBUG source=gpu.go:582 msg="Unable to load cudart library C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_110.dll: your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama" cudaSetDevice err: 35 time=2024-11-20T10:18:30.273+04:00 level=DEBUG source=gpu.go:582 msg="Unable to load cudart library C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\cudart64_12.dll: your nvidia driver is too old or missing. If you have a CUDA GPU please upgrade to run ollama" time=2024-11-20T10:18:30.275+04:00 level=DEBUG source=amd_windows.go:35 msg="unable to load amdhip64_6.dll, please make sure to upgrade to the latest amd driver: The specified module could not be found." time=2024-11-20T10:18:30.275+04:00 level=INFO source=gpu.go:386 msg="no compatible GPUs were discovered" time=2024-11-20T10:18:30.275+04:00 level=INFO source=types.go:123 msg="inference compute" id=0 library=cpu variant=avx2 compute="" driver=0.0 name="" total="255.9 GiB" available="239.9 GiB" [GIN] 2024/11/20 - 10:19:09 | 200 | 563.2µs | 127.0.0.1 | HEAD "/" [GIN] 2024/11/20 - 10:19:09 | 200 | 10.9182ms | 127.0.0.1 | GET "/api/tags" [GIN] 2024/11/20 - 10:19:35 | 200 | 537.3µs | 127.0.0.1 | HEAD "/" [GIN] 2024/11/20 - 10:19:35 | 200 | 21.5998ms | 127.0.0.1 | POST "/api/show" time=2024-11-20T10:19:35.868+04:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="255.9 GiB" before.free="239.9 GiB" before.free_swap="277.4 GiB" now.total="255.9 GiB" now.free="239.8 GiB" now.free_swap="277.3 GiB" time=2024-11-20T10:19:35.868+04:00 level=DEBUG source=sched.go:181 msg="updating default concurrency" OLLAMA_MAX_LOADED_MODELS=0x7ff7a5d7cf20 gpu_count=1 time=2024-11-20T10:19:35.883+04:00 level=DEBUG source=sched.go:211 msg="cpu mode with first model, loading" time=2024-11-20T10:19:35.883+04:00 level=DEBUG source=gpu.go:398 msg="updating system memory data" before.total="255.9 GiB" before.free="239.8 GiB" before.free_swap="277.3 GiB" now.total="255.9 GiB" now.free="239.8 GiB" now.free_swap="277.3 GiB" time=2024-11-20T10:19:35.883+04:00 level=INFO source=server.go:105 msg="system memory" total="255.9 GiB" free="239.8 GiB" free_swap="277.3 GiB" time=2024-11-20T10:19:35.888+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe time=2024-11-20T10:19:35.889+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe time=2024-11-20T10:19:35.889+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11\ollama_llama_server.exe time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12\ollama_llama_server.exe time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm\ollama_llama_server.exe time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=memory.go:107 msg=evaluating library=cpu gpu_count=1 available="[239.8 GiB]" time=2024-11-20T10:19:35.891+04:00 level=INFO source=memory.go:343 msg="offload to cpu" layers.requested=-1 layers.model=33 layers.offload=0 layers.split="" memory.available="[239.8 GiB]" memory.gpu_overhead="0 B" memory.required.full="5.5 GiB" memory.required.partial="0 B" memory.required.kv="3.0 GiB" memory.required.allocations="[5.5 GiB]" memory.weights.total="4.8 GiB" memory.weights.repeating="4.7 GiB" memory.weights.nonrepeating="77.1 MiB" memory.graph.full="512.0 MiB" memory.graph.partial="512.0 MiB" time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu\ollama_llama_server.exe time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx\ollama_llama_server.exe time=2024-11-20T10:19:35.891+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe time=2024-11-20T10:19:35.899+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v11\ollama_llama_server.exe time=2024-11-20T10:19:35.904+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cuda_v12\ollama_llama_server.exe time=2024-11-20T10:19:35.904+04:00 level=DEBUG source=common.go:294 msg="availableServers : found" file=C:\Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\rocm\ollama_llama_server.exe time=2024-11-20T10:19:35.928+04:00 level=DEBUG source=gpu.go:703 msg="no filter required for library cpu" time=2024-11-20T10:19:35.928+04:00 level=INFO source=server.go:383 msg="starting llama server" cmd="C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\runners\\cpu_avx2\\ollama_llama_server.exe --model C:\\Users\\Administrator\\.ollama\\models\\blobs\\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 --ctx-size 8192 --batch-size 512 --verbose --threads 64 --no-mmap --parallel 4 --port 53194" time=2024-11-20T10:19:35.930+04:00 level=DEBUG source=server.go:400 msg=subprocess environment="[PATH=C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama;C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama\\lib\\ollama\\runners\\cpu_avx2;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\\\SUT\\bin;C:\\Program Files\\McAfee\\Solidcore\\Tools\\GatherInfo;C:\\Program Files\\McAfee\\Solidcore\\Tools\\Scanalyzer;C:\\Program Files\\McAfee\\Solidcore\\;C:\\Program Files\\McAfee\\Solidcore\\Tools\\ScGetCerts;C:\\Users\\Administrator\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\Users\\Administrator\\.cache\\lm-studio\\bin;C:\\Users\\Administrator\\AppData\\Local\\Programs\\Ollama]" time=2024-11-20T10:19:35.944+04:00 level=INFO source=sched.go:449 msg="loaded runners" count=1 time=2024-11-20T10:19:35.949+04:00 level=INFO source=server.go:562 msg="waiting for llama runner to start responding" time=2024-11-20T10:19:35.953+04:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server error" time=2024-11-20T10:19:35.980+04:00 level=INFO source=runner.go:883 msg="starting go runner" time=2024-11-20T10:19:35.982+04:00 level=INFO source=runner.go:884 msg=system info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | cgo(clang)" threads=64 time=2024-11-20T10:19:35.984+04:00 level=INFO source=.:0 msg="Server listening on 127.0.0.1:53194" llama_model_loader: loaded meta data with 40 key-value pairs and 197 tensors from C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = phi3 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Phi 3 Mini 128k Instruct llama_model_loader: - kv 3: general.finetune str = 128k-instruct llama_model_loader: - kv 4: general.basename str = Phi-3 llama_model_loader: - kv 5: general.size_label str = mini llama_model_loader: - kv 6: general.license str = mit llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/microsoft/Phi-... llama_model_loader: - kv 8: general.tags arr[str,3] = ["nlp", "code", "text-generation"] llama_model_loader: - kv 9: general.languages arr[str,1] = ["en"] llama_model_loader: - kv 10: phi3.context_length u32 = 131072 llama_model_loader: - kv 11: phi3.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 12: phi3.embedding_length u32 = 3072 llama_model_loader: - kv 13: phi3.feed_forward_length u32 = 8192 llama_model_loader: - kv 14: phi3.block_count u32 = 32 llama_model_loader: - kv 15: phi3.attention.head_count u32 = 32 llama_model_loader: - kv 16: phi3.attention.head_count_kv u32 = 32 llama_model_loader: - kv 17: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 18: phi3.rope.dimension_count u32 = 96 llama_model_loader: - kv 19: phi3.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 20: general.file_type u32 = 30 llama_model_loader: - kv 21: phi3.attention.sliding_window u32 = 262144 llama_model_loader: - kv 22: phi3.rope.scaling.attn_factor f32 = 1.190238 llama_model_loader: - kv 23: tokenizer.ggml.model str = llama llama_model_loader: - kv 24: tokenizer.ggml.pre str = default llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 26: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 32000 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 32000 llama_model_loader: - kv 32: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 33: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 34: tokenizer.chat_template str = {% for message in messages %}{% if me... llama_model_loader: - kv 35: general.quantization_version u32 = 2 llama_model_loader: - kv 36: quantize.imatrix.file str = /models_out/Phi-3.1-mini-128k-instruc... llama_model_loader: - kv 37: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt llama_model_loader: - kv 38: quantize.imatrix.entries_count i32 = 128 llama_model_loader: - kv 39: quantize.imatrix.chunks_count i32 = 151 llama_model_loader: - type f32: 67 tensors llama_model_loader: - type q6_K: 1 tensors llama_model_loader: - type iq4_xs: 129 tensors llm_load_vocab: special tokens cache size = 14 llm_load_vocab: token to piece cache size = 0.1685 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = phi3 llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32064 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 3072 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 32 llm_load_print_meta: n_rot = 96 llm_load_print_meta: n_swa = 262144 llm_load_print_meta: n_embd_head_k = 96 llm_load_print_meta: n_embd_head_v = 96 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 3072 llm_load_print_meta: n_embd_v_gqa = 3072 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 8192 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: ssm_dt_b_c_rms = 0 llm_load_print_meta: model type = 3B llm_load_print_meta: model ftype = IQ4_XS - 4.25 bpw llm_load_print_meta: model params = 3.82 B llm_load_print_meta: model size = 1.92 GiB (4.31 BPW) llm_load_print_meta: general.name = Phi 3 Mini 128k Instruct llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 32000 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: PAD token = 32000 '<|endoftext|>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_print_meta: EOT token = 32007 '<|end|>' llm_load_print_meta: EOG token = 32000 '<|endoftext|>' llm_load_print_meta: EOG token = 32007 '<|end|>' llm_load_print_meta: max token length = 48 llm_load_tensors: ggml ctx size = 0.10 MiB llm_load_tensors: CPU buffer size = 1963.74 MiB time=2024-11-20T10:19:36.208+04:00 level=INFO source=server.go:596 msg="waiting for server to become available" status="llm server loading model" time=2024-11-20T10:19:36.208+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.10" time=2024-11-20T10:19:36.462+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.29" time=2024-11-20T10:19:36.717+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.46" time=2024-11-20T10:19:36.968+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.67" time=2024-11-20T10:19:37.226+04:00 level=DEBUG source=server.go:607 msg="model load progress 0.87" llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: n_batch = 2048 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 time=2024-11-20T10:19:37.484+04:00 level=DEBUG source=server.go:607 msg="model load progress 1.00" time=2024-11-20T10:19:37.740+04:00 level=DEBUG source=server.go:610 msg="model load completed, waiting for server to become available" status="llm server loading model" llama_kv_cache_init: CPU KV buffer size = 3072.00 MiB llama_new_context_with_model: KV self size = 3072.00 MiB, K (f16): 1536.00 MiB, V (f16): 1536.00 MiB llama_new_context_with_model: CPU output buffer size = 0.54 MiB llama_new_context_with_model: CPU compute buffer size = 564.01 MiB llama_new_context_with_model: graph nodes = 1286 llama_new_context_with_model: graph splits = 1 time=2024-11-20T10:19:39.275+04:00 level=INFO source=server.go:601 msg="llama runner started in 3.33 seconds" time=2024-11-20T10:19:39.275+04:00 level=DEBUG source=sched.go:462 msg="finished setting up runner" model=C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 [GIN] 2024/11/20 - 10:19:39 | 200 | 3.4274906s | 127.0.0.1 | POST "/api/generate" time=2024-11-20T10:19:39.278+04:00 level=DEBUG source=sched.go:466 msg="context for request finished" time=2024-11-20T10:19:39.279+04:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 duration=5m0s time=2024-11-20T10:19:39.279+04:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 refCount=0 time=2024-11-20T10:19:53.512+04:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 time=2024-11-20T10:19:53.514+04:00 level=DEBUG source=routes.go:1457 msg="chat request" images=0 prompt="<|user|>\nHello<|end|>\n<|assistant|>\n" time=2024-11-20T10:19:53.518+04:00 level=DEBUG source=cache.go:104 msg="loading cache slot" id=0 cache=0 prompt=10 used=0 remaining=10 time=2024-11-20T10:39:40.416+04:00 level=DEBUG source=sched.go:407 msg="context for request finished" [GIN] 2024/11/20 - 10:39:40 | 200 | 19m46s | 127.0.0.1 | POST "/api/chat" time=2024-11-20T10:39:40.416+04:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 duration=5m0s time=2024-11-20T10:39:40.425+04:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 refCount=0
GiteaMirror commented 2026-04-28 13:15:16 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 20, 2024):

Which model in LM studio? Can you supply the HF path?

<!-- gh-comment-id:2488128029 --> @rick-github commented on GitHub (Nov 20, 2024): Which model in LM studio? Can you supply the HF path?
GiteaMirror commented 2026-04-28 13:15:17 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 20, 2024):

Hello, I use the same model in LM studio. I used the GGUF file to create the the custom model on ollama. what is the HF path?

<!-- gh-comment-id:2488836539 --> @wixyskywriter commented on GitHub (Nov 20, 2024): Hello, I use the same model in LM studio. I used the GGUF file to create the the custom model on ollama. what is the HF path?
GiteaMirror commented 2026-04-28 13:15:18 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 20, 2024):

LM Studio downloads the model from HuggingFace (HF). If you reveal the full model name you used, it can be downloaded and tested independently of LM Studio.

<!-- gh-comment-id:2488843140 --> @rick-github commented on GitHub (Nov 20, 2024): LM Studio downloads the model from HuggingFace (HF). If you reveal the full model name you used, it can be downloaded and tested independently of LM Studio.
GiteaMirror commented 2026-04-28 13:15:20 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 20, 2024):

I also noticed that the threads on LM studio is set to 12 and context size is set to 4096. but in ollama it's higher \Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe --model C:\Users\Administrator.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 --ctx-size 8192 --batch-size 512 --verbose --threads 64 --no-mmap --parallel 4 --port 53194"

<!-- gh-comment-id:2488843168 --> @wixyskywriter commented on GitHub (Nov 20, 2024): I also noticed that the threads on LM studio is set to 12 and context size is set to 4096. but in ollama it's higher \Users\Administrator\AppData\Local\Programs\Ollama\lib\ollama\runners\cpu_avx2\ollama_llama_server.exe --model C:\Users\Administrator\.ollama\models\blobs\sha256-8261ac060034230a22c37e7387b15a45ead5043e6cc2f3fcf30eb960ade19054 --ctx-size 8192 --batch-size 512 --verbose --threads 64 --no-mmap --parallel 4 --port 53194"
GiteaMirror commented 2026-04-28 13:15:20 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 20, 2024):

Context size is higher because OLLAMA_NUM_PARALLEL is unset and ollama is using a default value of 4. Default context size for ollama is 2048, so a total of 8192 bytes (4*2048) for context is allocated.

<!-- gh-comment-id:2488849394 --> @rick-github commented on GitHub (Nov 20, 2024): Context size is higher because `OLLAMA_NUM_PARALLEL` is unset and ollama is using a default value of 4. Default context size for ollama is 2048, so a total of 8192 bytes (4*2048) for context is allocated.
GiteaMirror commented 2026-04-28 13:15:21 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 20, 2024):

Here is the model i used in LM studio, then i used to create the custom model in ollama. https://huggingface.co/lmstudio-community/Phi-3.1-mini-128k-instruct-GGUF

<!-- gh-comment-id:2489134271 --> @wixyskywriter commented on GitHub (Nov 20, 2024): Here is the model i used in LM studio, then i used to create the custom model in ollama. https://huggingface.co/lmstudio-community/Phi-3.1-mini-128k-instruct-GGUF
GiteaMirror commented 2026-04-28 13:15:22 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 20, 2024):

What Modelfile did you use to create the model?

<!-- gh-comment-id:2489233976 --> @rick-github commented on GitHub (Nov 20, 2024): What Modelfile did you use to create the model?
GiteaMirror commented 2026-04-28 13:15:23 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 20, 2024):

i downloaded the same model from ollama using pull. then created the model files from ollama show --modelfile phi3 >> Modelfile.
changed FROM to reference the GGUF file
Then used ollama create -f Modelfile

<!-- gh-comment-id:2489264187 --> @wixyskywriter commented on GitHub (Nov 20, 2024): i downloaded the same model from ollama using pull. then created the model files from ollama show --modelfile phi3 >> Modelfile. changed FROM to reference the GGUF file Then used ollama create -f Modelfile
GiteaMirror commented 2026-04-28 13:15:23 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 21, 2024):

I created a Modelfile with the following parameters to match the ones in LM Studio on my machine (i7-13700, 16 cores, 8/8 P/E, 96GB):

PARAMETER num_gpu 0
PARAMETER num_ctx 4096
PARAMETER num_thread 9

Some simple tests show comparable performance:

model tokens/s
lmstudio-community/Phi3-3.1-mini-128k-instruct-GGUF 21.72
ollama/Phi3-3.1-mini-128k-instruct 21.83

So the model seems fine. Unfortunately the debug logs you've added don't highlight a problem.

<!-- gh-comment-id:2489842239 --> @rick-github commented on GitHub (Nov 21, 2024): I created a Modelfile with the following parameters to match the ones in LM Studio on my machine (i7-13700, 16 cores, 8/8 P/E, 96GB): ```modelfile PARAMETER num_gpu 0 PARAMETER num_ctx 4096 PARAMETER num_thread 9 ``` Some simple tests show comparable performance: |model|tokens/s| |---|--- |lmstudio-community/Phi3-3.1-mini-128k-instruct-GGUF|21.72| |ollama/Phi3-3.1-mini-128k-instruct|21.83| So the model seems fine. Unfortunately the debug logs you've added don't highlight a problem.
GiteaMirror commented 2026-04-28 13:15:24 -05:00
Author
Owner
Copy Link

@wixyskywriter commented on GitHub (Nov 24, 2024):

changed the parameter as recommended and now i have almost the same performance. thanks

<!-- gh-comment-id:2495876704 --> @wixyskywriter commented on GitHub (Nov 24, 2024): changed the parameter as recommended and now i have almost the same performance. thanks
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#49869
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?