github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 21:25:46 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #3894] I have tested 4-5 phi-3-128K-Instruct models from different providers with different quants, all GGUF files, none are runnable with ollama #28171

New Issue
Open
opened 2026-04-22 06:01:22 -05:00 by GiteaMirror · 5 comments
GiteaMirror commented 2026-04-22 06:01:22 -05:00
Owner
Copy Link

Originally created by @phalexo on GitHub (Apr 24, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/3894

What is the issue?

ollama can import them, but not run them.

(Pythogora) developer@ai:~/PROJECTS/autogen$ ~/ollama/ollama run phi-3-mini-128k-instruct.Q6_K
Error: llama runner process no longer running: 1 error:failed to create context with model '/home/developer/.ollama/models/blobs/sha256-78f928e77e2470c7c09b151ff978bc348ba18ccde0991d03fe34f16fb9471460'

This how the model file looks like.

FROM /opt/data/PrunaAI/Phi-3-mini-128k-instruct-GGUF-Imatrix-smashed/Phi-3-mini-128k-instruct.Q6_K.gguf

TEMPLATE """
{{- if .First}}
<|system|>
{{ .System}}<|end|>
{{- end}}
<|user|>
{{ .Prompt}}<|end|>
<|assistant|>
"""

PARAMETER num_ctx 128000
PARAMETER temperature 0.2
PARAMETER num_gpu 100

PARAMETER stop <|end|>
PARAMETER stop <|endoftext|>

SYSTEM """You are a helpful AI which can plan, program, and test, analyze and debug."""

OS

Linux

GPU

Nvidia

CPU

Intel

Ollama version

latest from source.

Originally created by @phalexo on GitHub (Apr 24, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/3894 ### What is the issue? ollama can import them, but not run them. (Pythogora) developer@ai:~/PROJECTS/autogen$ ~/ollama/ollama run phi-3-mini-128k-instruct.Q6_K Error: llama runner process no longer running: 1 error:failed to create context with model '/home/developer/.ollama/models/blobs/sha256-78f928e77e2470c7c09b151ff978bc348ba18ccde0991d03fe34f16fb9471460' This how the model file looks like. FROM /opt/data/PrunaAI/Phi-3-mini-128k-instruct-GGUF-Imatrix-smashed/Phi-3-mini-128k-instruct.Q6_K.gguf TEMPLATE """ {{- if .First}} <|system|> {{ .System}}<|end|> {{- end}} <|user|> {{ .Prompt}}<|end|> <|assistant|> """ PARAMETER num_ctx 128000 PARAMETER temperature 0.2 PARAMETER num_gpu 100 PARAMETER stop <|end|> PARAMETER stop <|endoftext|> SYSTEM """You are a helpful AI which can plan, program, and test, analyze and debug.""" ### OS Linux ### GPU Nvidia ### CPU Intel ### Ollama version latest from source.
GiteaMirror added the bug label 2026-04-22 06:01:22 -05:00
GiteaMirror commented 2026-04-22 06:01:24 -05:00
Author
Owner
Copy Link

@hak8or commented on GitHub (Apr 25, 2024):

You are likely running into ollama using an older llama.cpp submodule, which lacks the following merges (the submodule is using a 1 week old version, these merges happened yesterday-ish) to get basic Phi-3 support;

and therefore will need an update similar to this;

Note though that this issue on llama.cpp has been re-opened specifically for 128k context though;

<!-- gh-comment-id:2076352145 --> @hak8or commented on GitHub (Apr 25, 2024): You are likely running into ollama using an older llama.cpp submodule, which lacks the following merges (the submodule is using a 1 week old version, these merges happened yesterday-ish) to get basic Phi-3 support; - https://github.com/ggerganov/llama.cpp/pull/6857 - https://github.com/ggerganov/llama.cpp/pull/6852 and therefore will need an update similar to this; - https://github.com/ollama/ollama/pull/3665 Note though that this issue on llama.cpp has been re-opened specifically for 128k context though; - https://github.com/ggerganov/llama.cpp/issues/6849
GiteaMirror commented 2026-04-22 06:01:24 -05:00
Author
Owner
Copy Link

@phalexo commented on GitHub (Apr 25, 2024):

This makes the most sense. Llama.cpp is able to load at least one of the
models I tried. Is there an easy way to pull that specific commit?

On Thu, Apr 25, 2024, 12:51 AM hak8or @.***> wrote:

You are likely running into ollama using an older llama.cpp submodule,
which lacks the following merges (the submodule is using a 1 week old
version, these merges happened yesterday-ish) to get basic Phi-3 support;

and therefore will need an update similar to this;

Note though that this issue on llama.cpp has been re-opened specifically
for 128k context though;

—
Reply to this email directly, view it on GitHub
https://github.com/ollama/ollama/issues/3894#issuecomment-2076352145,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABDD3ZI2F2DOOOJ6BVTECQ3Y7CDURAVCNFSM6AAAAABGX2WNBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGM2TEMJUGU
.
You are receiving this because you authored the thread.Message ID:
@.***>

<!-- gh-comment-id:2076918659 --> @phalexo commented on GitHub (Apr 25, 2024): This makes the most sense. Llama.cpp is able to load at least one of the models I tried. Is there an easy way to pull that specific commit? On Thu, Apr 25, 2024, 12:51 AM hak8or ***@***.***> wrote: > You are likely running into ollama using an older llama.cpp submodule, > which lacks the following merges (the submodule is using a 1 week old > version, these merges happened yesterday-ish) to get basic Phi-3 support; > > - ggerganov/llama.cpp#6857 > <https://github.com/ggerganov/llama.cpp/pull/6857> > - ggerganov/llama.cpp#6852 > <https://github.com/ggerganov/llama.cpp/pull/6852> > > and therefore will need an update similar to this; > > - #3665 <https://github.com/ollama/ollama/pull/3665> > > Note though that this issue on llama.cpp has been re-opened specifically > for 128k context though; > > - ggerganov/llama.cpp#6849 > <https://github.com/ggerganov/llama.cpp/issues/6849> > > — > Reply to this email directly, view it on GitHub > <https://github.com/ollama/ollama/issues/3894#issuecomment-2076352145>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABDD3ZI2F2DOOOJ6BVTECQ3Y7CDURAVCNFSM6AAAAABGX2WNBCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGM2TEMJUGU> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
GiteaMirror commented 2026-04-22 06:01:25 -05:00
Author
Owner
Copy Link

@phalexo commented on GitHub (Apr 25, 2024):

Temporary work-around is to set the context to 60000. Not as good as 128K, but better than 4K.

This appears to work for ollama with my four(4) 12.2GiB Titan GPUs. Others may have to play with their context sizes to match their hardware.

<!-- gh-comment-id:2077773704 --> @phalexo commented on GitHub (Apr 25, 2024): Temporary work-around is to set the context to 60000. Not as good as 128K, but better than 4K. This appears to work for ollama with my four(4) 12.2GiB Titan GPUs. Others may have to play with their context sizes to match their hardware.
GiteaMirror commented 2026-04-22 06:01:26 -05:00
Author
Owner
Copy Link

@TimeLordRaps commented on GitHub (Apr 26, 2024):

The <60k context for using pruneAI's phi-3-mini-128k gguf works through ollama run, but trying to use it through langchain's ChatOllama still produces gibberish.

self.model_name = ChatOllama(
            model=phi3-128k,
            temperature=1,
            num_ctx=32768,
            //num_keep=4,
            stop=[
                "<|user|>",
                "<|assistant|>"
                "<|system|>"
                "<|end|>"
                "<|endoftext|>"
            ]
        )

Here is the modelfile I used:
FROM "path\to\Phi-3-mini-128k-instruct.Q4_K_M.gguf"
TEMPLATE """{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>
"""
PARAMETER stop "<|user|>"
PARAMETER stop "<|assistant|>"
PARAMETER stop "<|system|>"
PARAMETER stop "<|end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER num_keep 4
PARAMETER num_ctx 32768

Any explanation or is there something I'm missing?

<!-- gh-comment-id:2078549944 --> @TimeLordRaps commented on GitHub (Apr 26, 2024): The <60k context for using pruneAI's phi-3-mini-128k gguf works through ollama run, but trying to use it through langchain's ChatOllama still produces gibberish. ``` self.model_name = ChatOllama( model=phi3-128k, temperature=1, num_ctx=32768, //num_keep=4, stop=[ "<|user|>", "<|assistant|>" "<|system|>" "<|end|>" "<|endoftext|>" ] ) ``` Here is the modelfile I used: FROM "path\to\Phi-3-mini-128k-instruct.Q4_K_M.gguf" TEMPLATE """{{ if .System }}<|system|> {{ .System }}<|end|> {{ end }}{{ if .Prompt }}<|user|> {{ .Prompt }}<|end|> {{ end }}<|assistant|> {{ .Response }}<|end|> """ PARAMETER stop "<|user|>" PARAMETER stop "<|assistant|>" PARAMETER stop "<|system|>" PARAMETER stop "<|end|>" PARAMETER stop "<|endoftext|>" PARAMETER num_keep 4 PARAMETER num_ctx 32768 Any explanation or is there something I'm missing?
GiteaMirror commented 2026-04-22 06:01:28 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (May 1, 2024):

Initial support for phi3 was added in 0.1.32, and conversion should be working in 0.1.33. Please give the latest RC a try and let us know if you're still having problems.

https://github.com/ollama/ollama/releases

<!-- gh-comment-id:2089208329 --> @dhiltgen commented on GitHub (May 1, 2024): Initial support for phi3 was added in 0.1.32, and conversion should be working in 0.1.33. Please give the latest RC a try and let us know if you're still having problems. https://github.com/ollama/ollama/releases
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#28171
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?