github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-05-23 14:13:13 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #13167] download.go: no resumable download support — failed transfers restart from zero #8705

New Issue
Closed
opened 2026-04-12 21:28:45 -05:00 by GiteaMirror · 1 comment
GiteaMirror commented 2026-04-12 21:28:45 -05:00
Owner
Copy Link

Originally created by @dimzon on GitHub (Nov 20, 2025).
Original GitHub issue: https://github.com/ollama/ollama/issues/13167

The current implementation of pull in download.go does not support resumable downloads. When a download stalls or fails (e.g. due to network timeout, proxy interruption, or connection reset), retrying ollama pull <model> restarts the entire download from scratch, even if the majority of layers have already been partially or fully fetched.

This behavior is observed in:

  • Unstable network conditions
  • High-latency or metered connections
  • Proxy/tunnel setups (e.g. ssh -R + privoxy)
  • Large models (e.g. 7B+, GGUF, or quantized variants)

Example log:

time=2025-11-20T05:39:39.801+03:00 level=INFO source=download.go:374 msg="7462734796d6 part 11 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection."

After interruption and re-run:

pulling manifest
pulling 7462734796d6... 0 B / ? MB

→ No attempt to resume. No use of Range headers. No partial file reuse.

Root Cause

  • Downloaded layer data is stored in ~/.ollama/tmp/ but deleted on failure
  • No use of partial file persistence or checksum validation to resume
  • HTTP client does not send Range: bytes=... on retry
  • No retry logic with exponential backoff or connection resilience
  • No support for --retry, --timeout, --proxy, or --header flags

✅ Expected Behavior

ollama pull should:

  1. Persist partial downloads in ~/.ollama/tmp/ until completion
  2. On restart, verify existing partial data (via size or chunk hash)
  3. Use Range: bytes=N- HTTP header to resume from server
  4. Retry with backoff on transient errors (408, 429, 502, 503, connection reset)
  5. Support resumable transfers without full re-download

This is standard in:

  • docker pull (uses content-addressable layers, resumes on reconnect)
  • skopeo copy (resumable, retry-aware)
  • curl, wget, aria2c (support --continue-at, --retry)

🛠 Proposed Solution

  1. Modify download logic to:
    -• Keep .part files in ~/.ollama/tmp/ after failed download
    -• On retry, check Content-Length and send Range: bytes=<current_size>-
    -• Fall back to full download only if server doesn’t support 206 Partial Content

  2. Add retry resilience:

client := &http.Client{
    Transport: &http.Transport{
        MaxIdleConns:        10,
        IdleConnTimeout:     30 * time.Second,
        TLSHandshakeTimeout: 10 * time.Second,
    },
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
        return http.ErrUseLastResponse
    },
}

And wrap download loop with:

for attempt := 0; attempt < maxRetries; attempt  {
    if err := downloadChunk(url, offset); err != nil {
        time.Sleep(backoff(attempt))
        continue
    }
    break
}
  1. Optional CLI flags (future):
  • --retry <n>
  • --retry-delay <seconds>
  • --proxy <url>
  • --header <key: value>
  • --output <path>

🧪 Reproduction Steps

  1. Run: ollama pull aya:8b
  2. Interrupt download (Ctrl+C) after 30–50%
  3. Re-run: ollama pull aya:8b
  4. Observe: download starts from 0%, all data re-fetched

🧩 Environment

  • Ollama version: 0.1.26 (or latest)
  • OS: Linux x86_64
  • Network: unstable, high RTT, behind proxy
  • Model: aya:8b, llama3:8b, etc.
  • Cache dir: ~/.ollama/tmp/

📌 Impact

Without resumable downloads, ollama is not viable for:

  • Users with limited bandwidth
  • Remote/cloud setups over SSH tunnels
  • Automated workflows in unstable environments
  • Large model distribution in production

This is not a QoL improvement — it's a core reliability requirement.

Originally created by @dimzon on GitHub (Nov 20, 2025). Original GitHub issue: https://github.com/ollama/ollama/issues/13167 The current implementation of `pull` in `download.go` does **not support resumable downloads**. When a download stalls or fails (e.g. due to network timeout, proxy interruption, or connection reset), retrying `ollama pull <model>` **restarts the entire download from scratch**, even if the majority of layers have already been partially or fully fetched. This behavior is observed in: - Unstable network conditions - High-latency or metered connections - Proxy/tunnel setups (e.g. `ssh -R` + `privoxy`) - Large models (e.g. 7B+, GGUF, or quantized variants) ## Example log: ``` time=2025-11-20T05:39:39.801+03:00 level=INFO source=download.go:374 msg="7462734796d6 part 11 stalled; retrying. If this persists, press ctrl-c to exit, then 'ollama pull' to find a faster connection." ``` After interruption and re-run: ``` pulling manifest pulling 7462734796d6... 0 B / ? MB ``` → **No attempt to resume. No use of `Range` headers. No partial file reuse.** ### Root Cause - Downloaded layer data is stored in `~/.ollama/tmp/` but **deleted on failure** - No use of **partial file persistence** or **checksum validation** to resume - HTTP client does not send `Range: bytes=...` on retry - No retry logic with exponential backoff or connection resilience - No support for `--retry`, `--timeout`, `--proxy`, or `--header` flags ## ✅ Expected Behavior `ollama pull` should: 1. **Persist partial downloads** in `~/.ollama/tmp/` until completion 2. On restart, **verify existing partial data** (via size or chunk hash) 3. Use `Range: bytes=N-` HTTP header to **resume from server** 4. Retry with backoff on transient errors (408, 429, 502, 503, connection reset) 5. Support resumable transfers **without full re-download** This is standard in: - `docker pull` (uses content-addressable layers, resumes on reconnect) - `skopeo copy` (resumable, retry-aware) - `curl`, `wget`, `aria2c` (support `--continue-at`, `--retry`) ## 🛠 Proposed Solution 1. **Modify download logic** to: -• Keep .part files in `~/.ollama/tmp/` after failed download -• On retry, check `Content-Length` and send `Range: bytes=<current_size>-` -• Fall back to full download only if server doesn’t support `206 Partial Content` 2. **Add retry resilience**: ```go client := &http.Client{ Transport: &http.Transport{ MaxIdleConns: 10, IdleConnTimeout: 30 * time.Second, TLSHandshakeTimeout: 10 * time.Second, }, CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, } ``` And wrap download loop with: ```go for attempt := 0; attempt < maxRetries; attempt { if err := downloadChunk(url, offset); err != nil { time.Sleep(backoff(attempt)) continue } break } ``` 3. **Optional CLI flags** (future): - `--retry <n>` - `--retry-delay <seconds>` - `--proxy <url>` - `--header <key: value>` - `--output <path>` ## 🧪 Reproduction Steps 1. Run: `ollama pull aya:8b` 2. Interrupt download (Ctrl+C) after 30–50% 3. Re-run: `ollama pull aya:8b` 4. Observe: download starts from 0%, all data re-fetched ## 🧩 Environment - Ollama version: `0.1.26` (or latest) - OS: Linux x86_64 - Network: unstable, high RTT, behind proxy - Model: `aya:8b`, `llama3:8b`, etc. - Cache dir: `~/.ollama/tmp/` ## 📌 Impact Without resumable downloads, `ollama` is **not viable** for: - Users with limited bandwidth - Remote/cloud setups over SSH tunnels - Automated workflows in unstable environments - Large model distribution in production This is not a QoL improvement — it's a **core reliability requirement**.
GiteaMirror commented 2026-04-12 21:28:47 -05:00
Author
Owner
Copy Link

@rick-github commented on GitHub (Nov 20, 2025):

0.1.26 restarts downloads.

$ ./ollama-linux-amd64 pull aya:8b
pulling manifest 
pulling 495401f864a6...  25% ▕████████                        ▏ 1.2 GB/4.8 GB   79 MB/s     45s^C
$ ./ollama-linux-amd64 pull aya:8b
pulling manifest 
pulling 495401f864a6... 100% ▕████████████████████████████████▏ 4.8 GB                         
pulling f0624a2393a5... 100% ▕████████████████████████████████▏  13 KB                         
pulling 42499e38acdf... 100% ▕████████████████████████████████▏  270 B                         
pulling 36b9655abe6a... 100% ▕████████████████████████████████▏   81 B                         
pulling e054a6188627... 100% ▕████████████████████████████████▏  492 B                         
verifying sha256 digest 
writing manifest 
removing any unused layers 
success

If the server is restarted between pulls, the incomplete downloads will be purged if OLLAMA_NOPRUNE is not set.

0.1.26 is very old, upgrading ollama is recommended.

<!-- gh-comment-id:3556804156 --> @rick-github commented on GitHub (Nov 20, 2025): 0.1.26 restarts downloads. ```console $ ./ollama-linux-amd64 pull aya:8b pulling manifest pulling 495401f864a6... 25% ▕████████ ▏ 1.2 GB/4.8 GB 79 MB/s 45s^C $ ./ollama-linux-amd64 pull aya:8b pulling manifest pulling 495401f864a6... 100% ▕████████████████████████████████▏ 4.8 GB pulling f0624a2393a5... 100% ▕████████████████████████████████▏ 13 KB pulling 42499e38acdf... 100% ▕████████████████████████████████▏ 270 B pulling 36b9655abe6a... 100% ▕████████████████████████████████▏ 81 B pulling e054a6188627... 100% ▕████████████████████████████████▏ 492 B verifying sha256 digest writing manifest removing any unused layers success ``` If the server is restarted between pulls, the incomplete downloads will be purged if `OLLAMA_NOPRUNE` is not set. 0.1.26 is very old, upgrading ollama is recommended.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#8705
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?