github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 05:05:37 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1964] Self-extend support #1132

New Issue
Open
opened 2026-04-12 10:52:30 -05:00 by GiteaMirror · 7 comments
GiteaMirror commented 2026-04-12 10:52:30 -05:00
Owner
Copy Link

Originally created by @coder543 on GitHub (Jan 12, 2024).
Original GitHub issue: https://github.com/ollama/ollama/issues/1964

I’m not sure what all would be involved, but something that’s making waves is “self extend”, where it seems to be possible to make models work at larger context sizes than what they were originally designed for.

In a hypothetical outcome, it would be amazing if models were automatically self-extended when the requested context is larger than the trained context.

Some relevant links:

https://www.reddit.com/r/LocalLLaMA/comments/194mmki/selfextend_works_for_phi2_now_looks_good/

https://github.com/ggerganov/llama.cpp/pull/4889

Originally created by @coder543 on GitHub (Jan 12, 2024). Original GitHub issue: https://github.com/ollama/ollama/issues/1964 I’m not sure what all would be involved, but something that’s making waves is “self extend”, where it seems to be possible to make models work at larger context sizes than what they were originally designed for. In a hypothetical outcome, it would be amazing if models were automatically self-extended when the requested context is larger than the trained context. Some relevant links: https://www.reddit.com/r/LocalLLaMA/comments/194mmki/selfextend_works_for_phi2_now_looks_good/ https://github.com/ggerganov/llama.cpp/pull/4889
GiteaMirror added the feature request label 2026-04-12 10:52:30 -05:00
GiteaMirror commented 2026-04-12 10:52:31 -05:00
Author
Owner
Copy Link

@blackrabbit17 commented on GitHub (Jan 12, 2024):

+1 for me too

<!-- gh-comment-id:1889986292 --> @blackrabbit17 commented on GitHub (Jan 12, 2024): +1 for me too
GiteaMirror commented 2026-04-12 10:52:32 -05:00
Author
Owner
Copy Link

@marklysze commented on GitHub (Jan 16, 2024):

+1 for me, would love to get more with Phi-2

<!-- gh-comment-id:1892943264 --> @marklysze commented on GitHub (Jan 16, 2024): +1 for me, would love to get more with Phi-2
GiteaMirror commented 2026-04-12 10:52:32 -05:00
Author
Owner
Copy Link

@cognitivetech commented on GitHub (Jan 31, 2024):

https://github.com/ggerganov/llama.cpp/pull/4963 seems support is in llama.cpp main and server

<!-- gh-comment-id:1919375802 --> @cognitivetech commented on GitHub (Jan 31, 2024): https://github.com/ggerganov/llama.cpp/pull/4963 seems support is in llama.cpp main and server
GiteaMirror commented 2026-04-12 10:52:32 -05:00
Author
Owner
Copy Link

@cognitivetech commented on GitHub (Feb 4, 2024):

according to latest release notes, (marking this commit 72b12c3be7) llama.cpp is bumped to b1999 which is from last week, where selfextend support was added 3 weeks ago. So it seems the foundation for support exists.

So the question will it pass a parameter set in my model-file? or does each parameter require specific coding?

here its described in more detail: https://github.com/ggerganov/llama.cpp/issues/4886#issuecomment-1890465266

First, you set -c to the context that you want to achieve - let's say -c 8192.

Next, given that the original training context of the model is T (let's assume T = 2048), you want to set G >= 8192 / T, so in this case: --grp-attn-n 4 or --grp-attn-n 8.

The --grp-attn-w corresponds to W from the paper. I think the authors generally used 512, but I think you can go up to T/2 - so in this case --grp-attn-w 1024.

Additionally, G has to be multiple of W

  1. According to transformers docs on huggingface mistral 0.1 was trained on 8k context length.
  2. According to the paper 0.2 also was trained on 8192 context

Have a look here at the implementation of selfextend for mistral 0.1
we get the following parameters:

        g_size=2,  # Group size for SelfExtend attention
        w_size=1024,  # Window size for SelfExtend attention

ChatGPT Says:

According to the provided reasoning, you can calculate the context size using the formula:
Context Size = G x T

  • ( G ) is the group size (g_size),
  • ( T ) is the original training context size.
    In this case, ( G = 2 ) and ( T = 8192 ), so the calculated context size would be:
    Context Size = 2 x 8192 = 16384
    Therefore, with g_size=2 and a model trained on an 8192-token context window, the resulting context size would be 16384 tokens.
<!-- gh-comment-id:1925662224 --> @cognitivetech commented on GitHub (Feb 4, 2024): according to latest release notes, (marking this commit https://github.com/ollama/ollama/commit/72b12c3be7f7d8b2e0d1fb703e6d6973caff6493) llama.cpp is bumped to [b1999](https://github.com/ggerganov/llama.cpp/releases/tag/b1999) which is from last week, where selfextend support was added 3 weeks ago. So it seems the foundation for support exists. So the question will it pass a [parameter](https://github.com/ollama/ollama/blob/main/docs/modelfile.md#parameter) set in my model-file? or does each parameter require specific coding? here its described in more detail: https://github.com/ggerganov/llama.cpp/issues/4886#issuecomment-1890465266 > First, you set -c to the context that you want to achieve - let's say -c 8192. > > Next, given that the original training context of the model is T (let's assume T = 2048), you want to set G >= 8192 / T, so in this case: --grp-attn-n 4 or --grp-attn-n 8. > > The --grp-attn-w corresponds to W from the paper. I think the authors generally used 512, but I think you can go up to T/2 - so in this case --grp-attn-w 1024. > > Additionally, G has to be multiple of W 1. According to [transformers docs on huggingface](https://huggingface.co/docs/transformers/en/model_doc/mistral) mistral 0.1 was trained on 8k context length. 2. According to [the paper](https://arxiv.org/pdf/2310.06825.pdf) 0.2 also was trained on 8192 context Have a look here at the implementation of [selfextend for mistral 0.1](https://github.com/sdan/selfextend/blob/master/configuration_mistral.py) we get the following parameters: ``` g_size=2, # Group size for SelfExtend attention w_size=1024, # Window size for SelfExtend attention ``` ChatGPT Says: > According to the provided reasoning, you can calculate the context size using the formula: > Context Size = G x T > - ( G ) is the group size (`g_size`), > - ( T ) is the original training context size. > In this case, ( G = 2 ) and ( T = 8192 ), so the calculated context size would be: > Context Size = 2 x 8192 = 16384 > Therefore, with `g_size=2` and a model trained on an 8192-token context window, **the resulting context size would be 16384 tokens.**
GiteaMirror commented 2026-04-12 10:52:32 -05:00
Author
Owner
Copy Link

@cognitivetech commented on GitHub (Feb 4, 2024):

Furthermore it seems relatively trivial to add the required parameters, based on previous additions shown here: #276 Configurable Rope Frequency Parameters

<!-- gh-comment-id:1925719648 --> @cognitivetech commented on GitHub (Feb 4, 2024): Furthermore it seems relatively trivial to add the required parameters, based on previous additions shown here: [#276 Configurable Rope Frequency Parameters](https://github.com/ollama/ollama/pull/276/files)
GiteaMirror commented 2026-04-12 10:52:33 -05:00
Author
Owner
Copy Link

@ismaelc commented on GitHub (Feb 5, 2024):

Anyone with a Go environment who wants to create a PR for this? Thanks @cognitivetech for the references

<!-- gh-comment-id:1927699068 --> @ismaelc commented on GitHub (Feb 5, 2024): Anyone with a Go environment who wants to create a PR for this? Thanks @cognitivetech for the references
GiteaMirror commented 2026-04-12 10:52:33 -05:00
Author
Owner
Copy Link

@cognitivetech commented on GitHub (Feb 10, 2024):

Ok, so I did a little more digging. For one thing, those files have moved now, to here:
https://github.com/ollama/ollama/blob/main/api/types.go
https://github.com/ollama/ollama/blob/main/llm/llama.go

For another thing, there are two places where options are added in types.go.

// Options specfied in GenerateRequest, if you add a new option here add it to the API docs also
type Options struct {
	Runner

	// Predict options used at runtime
	NumKeep          int      `json:"num_keep,omitempty"`
	Seed             int      `json:"seed,omitempty"`
	NumPredict       int      `json:"num_predict,omitempty"`
	TopK             int      `json:"top_k,omitempty"`
	TopP             float32  `json:"top_p,omitempty"`
	TFSZ             float32  `json:"tfs_z,omitempty"`
	TypicalP         float32  `json:"typical_p,omitempty"`
	RepeatLastN      int      `json:"repeat_last_n,omitempty"`
	Temperature      float32  `json:"temperature,omitempty"`
	RepeatPenalty    float32  `json:"repeat_penalty,omitempty"`
	PresencePenalty  float32  `json:"presence_penalty,omitempty"`
	FrequencyPenalty float32  `json:"frequency_penalty,omitempty"`
	Mirostat         int      `json:"mirostat,omitempty"`
	MirostatTau      float32  `json:"mirostat_tau,omitempty"`
	MirostatEta      float32  `json:"mirostat_eta,omitempty"`
	PenalizeNewline  bool     `json:"penalize_newline,omitempty"`
	Stop             []string `json:"stop,omitempty"`
}

// Runner options which must be set when the model is loaded into memory
type Runner struct {
	UseNUMA            bool    `json:"numa,omitempty"`
	NumCtx             int     `json:"num_ctx,omitempty"`
	NumBatch           int     `json:"num_batch,omitempty"`
	NumGQA             int     `json:"num_gqa,omitempty"`
	NumGPU             int     `json:"num_gpu,omitempty"`
	MainGPU            int     `json:"main_gpu,omitempty"`
	LowVRAM            bool    `json:"low_vram,omitempty"`
	F16KV              bool    `json:"f16_kv,omitempty"`
	LogitsAll          bool    `json:"logits_all,omitempty"`
	VocabOnly          bool    `json:"vocab_only,omitempty"`
	UseMMap            bool    `json:"use_mmap,omitempty"`
	UseMLock           bool    `json:"use_mlock,omitempty"`
	EmbeddingOnly      bool    `json:"embedding_only,omitempty"`
	RopeFrequencyBase  float32 `json:"rope_frequency_base,omitempty"`
	RopeFrequencyScale float32 `json:"rope_frequency_scale,omitempty"`
	NumThread          int     `json:"num_thread,omitempty"`
}

https://github.com/sdan/selfextend/blob/master/configuration_mistral.py

This is the configuration class to store the configuration of a [MistralModel]. It is used to instantiate an Mistral model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Mistral-7B-v0.1 or Mistral-7B-Instruct-v0.1.

*emphasis mine

I think that means they are runner options set when the model is loaded into memory

<!-- gh-comment-id:1937041986 --> @cognitivetech commented on GitHub (Feb 10, 2024): Ok, so I did a little more digging. For one thing, those files have moved now, to here: https://github.com/ollama/ollama/blob/main/api/types.go https://github.com/ollama/ollama/blob/main/llm/llama.go For another thing, there are two places where options are added in `types.go`. ```golang // Options specfied in GenerateRequest, if you add a new option here add it to the API docs also type Options struct { Runner // Predict options used at runtime NumKeep int `json:"num_keep,omitempty"` Seed int `json:"seed,omitempty"` NumPredict int `json:"num_predict,omitempty"` TopK int `json:"top_k,omitempty"` TopP float32 `json:"top_p,omitempty"` TFSZ float32 `json:"tfs_z,omitempty"` TypicalP float32 `json:"typical_p,omitempty"` RepeatLastN int `json:"repeat_last_n,omitempty"` Temperature float32 `json:"temperature,omitempty"` RepeatPenalty float32 `json:"repeat_penalty,omitempty"` PresencePenalty float32 `json:"presence_penalty,omitempty"` FrequencyPenalty float32 `json:"frequency_penalty,omitempty"` Mirostat int `json:"mirostat,omitempty"` MirostatTau float32 `json:"mirostat_tau,omitempty"` MirostatEta float32 `json:"mirostat_eta,omitempty"` PenalizeNewline bool `json:"penalize_newline,omitempty"` Stop []string `json:"stop,omitempty"` } // Runner options which must be set when the model is loaded into memory type Runner struct { UseNUMA bool `json:"numa,omitempty"` NumCtx int `json:"num_ctx,omitempty"` NumBatch int `json:"num_batch,omitempty"` NumGQA int `json:"num_gqa,omitempty"` NumGPU int `json:"num_gpu,omitempty"` MainGPU int `json:"main_gpu,omitempty"` LowVRAM bool `json:"low_vram,omitempty"` F16KV bool `json:"f16_kv,omitempty"` LogitsAll bool `json:"logits_all,omitempty"` VocabOnly bool `json:"vocab_only,omitempty"` UseMMap bool `json:"use_mmap,omitempty"` UseMLock bool `json:"use_mlock,omitempty"` EmbeddingOnly bool `json:"embedding_only,omitempty"` RopeFrequencyBase float32 `json:"rope_frequency_base,omitempty"` RopeFrequencyScale float32 `json:"rope_frequency_scale,omitempty"` NumThread int `json:"num_thread,omitempty"` } ``` https://github.com/sdan/selfextend/blob/master/configuration_mistral.py > This is the configuration class to store the configuration of a [`MistralModel`]. **_It is used to instantiate an Mistral model_** according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Mistral-7B-v0.1 or Mistral-7B-Instruct-v0.1. *emphasis mine I think that means they are runner options set when the model is loaded into memory
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#1132
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?