github-starred/ollama
2
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-07-16 13:19:40 -05:00
Code Issues 16.9k Packages Projects Releases 122 Wiki Activity

[GH-ISSUE #1585] CUDA error 2 [...] out of memory when using mixtral:8x7b-instruct-v0.1-q3_K_M but not on bigger models #62911

New Issue
Closed
opened 2026-05-03 10:45:07 -05:00 by GiteaMirror · 11 comments
GiteaMirror commented 2026-05-03 10:45:07 -05:00
Owner
Copy Link

Originally created by @AlessandroSpallina on GitHub (Dec 18, 2023).
Original GitHub issue: https://github.com/ollama/ollama/issues/1585

Originally assigned to: @mxyng on GitHub.

Hi, I'm opening this issue because I noticed a weird behavior running ollama on docker with GPU support and trying different mixtral 8x7B sizes: I can easily do inference on my GPU with models like mixtral:8x7b-instruct-v0.1-q4_K_M but I see a memory failure when running smaller models like mixtral:8x7b-instruct-v0.1-q3_K_M.
I'm on Ubuntu 23.10, my GPU is a NVIDIA 3090.

My docker-compose.yml:

version: '3.7'

services:
  ollama:
    container_name: ollama_cat_dev
    image: ollama/ollama:0.1.16
    restart: unless-stopped
    volumes:
      - ./ollama:/root/.ollama
    expose:
      - 11434
    environment:
      - gpus=all
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

As said, when I try to use mixtral:8x7b-instruct-v0.1-q3_K_M I see "out of memory" issues and the inference is completely done on CPU. Here the log (attached because it was too long for github issues)
ollama.log

Here instead the log when I run a bigger model like mixtral:8x7b-instruct-v0.1-q4_K_M:

2023/12/18 20:09:56 llama.go:300: 23732 MB VRAM available, loading up to 22 GPU layers
2023/12/18 20:09:56 llama.go:436: starting llama runner
2023/12/18 20:09:56 llama.go:494: waiting for llama runner to start responding
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6
{"timestamp":1702930196,"level":"INFO","function":"main","line":2652,"message":"build info","build":441,"commit":"948ff13"}
{"timestamp":1702930196,"level":"INFO","function":"main","line":2655,"message":"system info","n_threads":16,"n_threads_batch":-1,"total_threads":32,"system_info":"AVX = 1 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "}
llama_model_loader: loaded meta data with 24 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256:59a66936ed54cfe28136f99a3ec5336f2b404bad0bc0f3a48123f87c677d8623 (version GGUF V3 (latest))
llama_model_loader: - tensor    0:                token_embd.weight q4_K     [  4096, 32000,     1,     1 ]
llama_model_loader: - tensor    1:          blk.0.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor    2:          blk.0.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor    3:            blk.0.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor    4:          blk.0.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor    5:          blk.0.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor    6:            blk.0.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor    7:          blk.0.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor    8:          blk.0.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor    9:            blk.0.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   10:          blk.0.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   11:          blk.0.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   12:            blk.0.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   13:          blk.0.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   14:          blk.0.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   15:            blk.0.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   16:          blk.0.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   17:          blk.0.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   18:            blk.0.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   19:          blk.0.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   20:          blk.0.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   21:            blk.0.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   22:          blk.0.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   23:          blk.0.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   24:            blk.0.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   25:        blk.0.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor   26:           blk.0.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   27:            blk.0.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   28:              blk.0.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor   29:         blk.0.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   30:              blk.0.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   31:              blk.0.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor   32:          blk.1.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   33:          blk.1.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   34:            blk.1.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   35:          blk.1.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   36:          blk.1.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   37:            blk.1.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   38:          blk.1.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   39:          blk.1.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   40:            blk.1.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   41:          blk.1.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   42:          blk.1.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   43:            blk.1.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   44:          blk.1.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   45:          blk.1.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   46:        blk.1.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor   47:              blk.1.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor   48:         blk.1.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   49:              blk.1.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   50:              blk.1.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor   51:            blk.1.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   52:          blk.1.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   53:          blk.1.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   54:            blk.1.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   55:          blk.1.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   56:          blk.1.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   57:            blk.1.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   58:          blk.1.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   59:          blk.1.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   60:            blk.1.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   61:           blk.1.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   62:            blk.1.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   63:          blk.2.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   64:          blk.2.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   65:            blk.2.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   66:          blk.2.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   67:          blk.2.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   68:            blk.2.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   69:          blk.2.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   70:          blk.2.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   71:            blk.2.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   72:          blk.2.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   73:          blk.2.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   74:            blk.2.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   75:          blk.2.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   76:          blk.2.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   77:            blk.2.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   78:          blk.2.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   79:          blk.2.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   80:            blk.2.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   81:          blk.2.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   82:          blk.2.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   83:            blk.2.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   84:          blk.2.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   85:          blk.2.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   86:            blk.2.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   87:        blk.2.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor   88:           blk.2.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   89:            blk.2.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor   90:              blk.2.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor   91:         blk.2.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   92:              blk.2.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor   93:              blk.2.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor   94:          blk.3.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   95:          blk.3.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   96:            blk.3.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   97:          blk.3.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor   98:          blk.3.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor   99:            blk.3.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  100:          blk.3.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  101:        blk.3.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  102:              blk.3.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  103:         blk.3.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  104:              blk.3.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  105:              blk.3.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  106:          blk.3.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  107:            blk.3.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  108:          blk.3.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  109:          blk.3.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  110:            blk.3.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  111:          blk.3.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  112:          blk.3.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  113:            blk.3.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  114:          blk.3.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  115:          blk.3.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  116:            blk.3.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  117:          blk.3.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  118:          blk.3.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  119:            blk.3.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  120:          blk.3.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  121:          blk.3.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  122:            blk.3.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  123:           blk.3.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  124:            blk.3.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  125:          blk.4.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  126:          blk.4.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  127:            blk.4.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  128:          blk.4.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  129:          blk.4.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  130:            blk.4.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  131:          blk.4.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  132:          blk.4.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  133:            blk.4.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  134:          blk.4.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  135:          blk.4.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  136:            blk.4.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  137:          blk.4.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  138:          blk.4.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  139:            blk.4.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  140:          blk.4.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  141:          blk.4.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  142:            blk.4.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  143:          blk.4.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  144:          blk.4.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  145:            blk.4.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  146:          blk.4.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  147:          blk.4.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  148:            blk.4.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  149:        blk.4.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  150:           blk.4.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  151:            blk.4.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  152:              blk.4.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  153:         blk.4.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  154:              blk.4.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  155:              blk.4.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  156:        blk.5.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  157:              blk.5.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  158:         blk.5.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  159:              blk.5.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  160:              blk.5.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  161:          blk.5.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  162:          blk.5.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  163:            blk.5.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  164:          blk.5.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  165:          blk.5.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  166:            blk.5.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  167:          blk.5.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  168:          blk.5.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  169:            blk.5.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  170:          blk.5.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  171:          blk.5.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  172:            blk.5.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  173:          blk.5.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  174:          blk.5.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  175:            blk.5.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  176:          blk.5.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  177:          blk.5.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  178:            blk.5.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  179:          blk.5.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  180:          blk.5.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  181:            blk.5.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  182:          blk.5.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  183:          blk.5.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  184:            blk.5.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  185:           blk.5.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  186:            blk.5.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  187:          blk.6.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  188:          blk.6.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  189:            blk.6.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  190:          blk.6.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  191:          blk.6.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  192:            blk.6.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  193:          blk.6.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  194:          blk.6.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  195:            blk.6.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  196:          blk.6.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  197:          blk.6.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  198:            blk.6.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  199:          blk.6.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  200:          blk.6.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  201:            blk.6.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  202:          blk.6.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  203:          blk.6.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  204:        blk.6.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  205:              blk.6.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  206:         blk.6.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  207:              blk.6.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  208:              blk.6.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  209:            blk.6.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  210:          blk.6.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  211:          blk.6.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  212:            blk.6.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  213:          blk.6.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  214:          blk.6.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  215:            blk.6.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  216:           blk.6.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  217:            blk.6.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  218:          blk.7.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  219:          blk.7.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  220:            blk.7.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  221:          blk.7.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  222:          blk.7.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  223:            blk.7.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  224:          blk.7.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  225:          blk.7.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  226:            blk.7.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  227:          blk.7.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  228:          blk.7.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  229:            blk.7.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  230:          blk.7.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  231:          blk.7.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  232:            blk.7.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  233:          blk.7.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  234:          blk.7.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  235:            blk.7.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  236:          blk.7.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  237:          blk.7.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  238:            blk.7.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  239:          blk.7.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  240:          blk.7.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  241:            blk.7.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  242:        blk.7.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  243:           blk.7.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  244:            blk.7.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  245:              blk.7.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  246:         blk.7.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  247:              blk.7.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  248:              blk.7.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  249:          blk.8.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  250:          blk.8.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  251:            blk.8.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  252:          blk.8.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  253:          blk.8.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  254:            blk.8.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  255:          blk.8.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  256:          blk.8.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  257:            blk.8.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  258:          blk.8.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  259:        blk.8.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  260:              blk.8.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  261:         blk.8.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  262:              blk.8.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  263:              blk.8.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  264:         blk.10.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  265:         blk.10.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  266:           blk.10.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  267:       blk.10.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  268:             blk.10.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  269:        blk.10.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  270:             blk.10.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  271:             blk.10.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  272:          blk.8.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  273:            blk.8.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  274:          blk.8.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  275:          blk.8.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  276:            blk.8.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  277:          blk.8.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  278:          blk.8.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  279:            blk.8.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  280:          blk.8.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  281:          blk.8.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  282:            blk.8.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  283:          blk.8.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  284:          blk.8.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  285:            blk.8.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  286:           blk.8.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  287:            blk.8.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  288:          blk.9.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  289:          blk.9.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  290:            blk.9.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  291:          blk.9.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  292:          blk.9.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  293:            blk.9.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  294:          blk.9.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  295:          blk.9.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  296:            blk.9.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  297:          blk.9.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  298:          blk.9.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  299:            blk.9.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  300:          blk.9.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  301:          blk.9.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  302:            blk.9.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  303:          blk.9.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  304:          blk.9.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  305:            blk.9.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  306:          blk.9.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  307:          blk.9.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  308:            blk.9.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  309:          blk.9.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  310:          blk.9.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  311:            blk.9.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  312:        blk.9.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  313:           blk.9.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  314:            blk.9.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  315:              blk.9.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  316:         blk.9.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  317:              blk.9.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  318:              blk.9.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  319:         blk.10.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  320:         blk.10.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  321:           blk.10.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  322:         blk.10.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  323:         blk.10.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  324:           blk.10.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  325:         blk.10.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  326:         blk.10.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  327:           blk.10.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  328:         blk.10.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  329:         blk.10.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  330:           blk.10.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  331:         blk.10.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  332:         blk.10.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  333:           blk.10.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  334:         blk.10.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  335:         blk.10.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  336:           blk.10.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  337:         blk.10.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  338:         blk.10.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  339:           blk.10.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  340:          blk.10.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  341:           blk.10.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  342:         blk.11.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  343:         blk.11.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  344:           blk.11.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  345:         blk.11.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  346:         blk.11.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  347:           blk.11.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  348:         blk.11.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  349:         blk.11.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  350:           blk.11.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  351:         blk.11.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  352:         blk.11.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  353:           blk.11.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  354:         blk.11.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  355:         blk.11.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  356:           blk.11.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  357:         blk.11.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  358:         blk.11.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  359:           blk.11.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  360:         blk.11.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  361:         blk.11.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  362:       blk.11.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  363:             blk.11.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  364:        blk.11.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  365:             blk.11.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  366:             blk.11.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  367:           blk.11.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  368:         blk.11.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  369:         blk.11.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  370:           blk.11.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  371:          blk.11.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  372:           blk.11.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  373:         blk.12.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  374:         blk.12.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  375:           blk.12.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  376:         blk.12.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  377:         blk.12.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  378:           blk.12.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  379:         blk.12.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  380:         blk.12.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  381:           blk.12.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  382:         blk.12.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  383:         blk.12.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  384:           blk.12.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  385:         blk.12.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  386:         blk.12.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  387:           blk.12.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  388:         blk.12.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  389:         blk.12.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  390:           blk.12.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  391:         blk.12.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  392:         blk.12.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  393:           blk.12.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  394:         blk.12.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  395:         blk.12.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  396:           blk.12.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  397:       blk.12.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  398:          blk.12.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  399:           blk.12.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  400:             blk.12.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  401:        blk.12.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  402:             blk.12.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  403:             blk.12.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  404:         blk.13.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  405:         blk.13.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  406:           blk.13.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  407:         blk.13.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  408:         blk.13.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  409:           blk.13.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  410:         blk.13.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  411:         blk.13.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  412:           blk.13.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  413:         blk.13.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  414:         blk.13.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  415:           blk.13.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  416:         blk.13.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  417:       blk.13.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  418:             blk.13.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  419:        blk.13.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  420:             blk.13.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  421:             blk.13.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  422:         blk.13.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  423:           blk.13.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  424:         blk.13.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  425:         blk.13.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  426:           blk.13.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  427:         blk.13.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  428:         blk.13.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  429:           blk.13.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  430:         blk.13.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  431:         blk.13.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  432:           blk.13.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  433:          blk.13.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  434:           blk.13.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  435:         blk.14.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  436:         blk.14.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  437:           blk.14.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  438:         blk.14.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  439:         blk.14.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  440:           blk.14.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  441:         blk.14.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  442:         blk.14.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  443:           blk.14.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  444:         blk.14.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  445:         blk.14.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  446:           blk.14.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  447:         blk.14.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  448:         blk.14.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  449:           blk.14.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  450:         blk.14.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  451:         blk.14.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  452:           blk.14.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  453:         blk.14.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  454:         blk.14.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  455:           blk.14.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  456:         blk.14.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  457:         blk.14.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  458:           blk.14.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  459:       blk.14.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  460:          blk.14.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  461:           blk.14.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  462:             blk.14.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  463:        blk.14.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  464:             blk.14.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  465:             blk.14.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  466:         blk.15.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  467:         blk.15.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  468:           blk.15.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  469:         blk.15.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  470:         blk.15.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  471:           blk.15.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  472:       blk.15.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  473:             blk.15.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  474:        blk.15.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  475:             blk.15.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  476:             blk.15.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  477:         blk.15.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  478:         blk.15.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  479:           blk.15.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  480:         blk.15.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  481:         blk.15.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  482:           blk.15.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  483:         blk.15.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  484:         blk.15.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  485:           blk.15.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  486:         blk.15.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  487:         blk.15.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  488:           blk.15.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  489:         blk.15.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  490:         blk.15.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  491:           blk.15.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  492:         blk.15.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  493:         blk.15.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  494:           blk.15.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  495:          blk.15.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  496:           blk.15.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  497:         blk.16.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  498:         blk.16.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  499:           blk.16.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  500:         blk.16.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  501:         blk.16.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  502:           blk.16.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  503:         blk.16.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  504:         blk.16.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  505:           blk.16.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  506:         blk.16.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  507:         blk.16.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  508:           blk.16.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  509:         blk.16.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  510:         blk.16.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  511:           blk.16.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  512:         blk.16.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  513:         blk.16.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  514:           blk.16.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  515:         blk.16.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  516:         blk.16.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  517:           blk.16.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  518:         blk.16.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  519:         blk.16.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  520:       blk.16.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  521:             blk.16.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  522:        blk.16.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  523:             blk.16.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  524:             blk.16.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  525:           blk.16.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  526:          blk.16.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  527:           blk.16.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  528:         blk.17.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  529:         blk.17.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  530:           blk.17.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  531:         blk.17.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  532:         blk.17.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  533:           blk.17.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  534:         blk.17.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  535:         blk.17.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  536:           blk.17.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  537:         blk.17.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  538:         blk.17.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  539:           blk.17.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  540:         blk.17.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  541:         blk.17.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  542:           blk.17.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  543:         blk.17.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  544:         blk.17.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  545:           blk.17.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  546:         blk.17.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  547:         blk.17.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  548:           blk.17.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  549:         blk.17.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  550:         blk.17.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  551:           blk.17.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  552:       blk.17.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  553:          blk.17.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  554:           blk.17.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  555:             blk.17.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  556:        blk.17.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  557:             blk.17.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  558:             blk.17.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  559:         blk.18.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  560:         blk.18.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  561:           blk.18.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  562:         blk.18.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  563:         blk.18.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  564:           blk.18.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  565:         blk.18.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  566:         blk.18.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  567:           blk.18.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  568:         blk.18.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  569:         blk.18.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  570:           blk.18.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  571:         blk.18.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  572:         blk.18.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  573:           blk.18.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  574:         blk.18.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  575:       blk.18.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  576:             blk.18.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  577:        blk.18.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  578:             blk.18.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  579:             blk.18.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  580:         blk.18.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  581:           blk.18.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  582:         blk.18.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  583:         blk.18.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  584:           blk.18.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  585:         blk.18.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  586:         blk.18.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  587:           blk.18.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  588:          blk.18.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  589:           blk.18.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  590:         blk.19.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  591:         blk.19.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  592:           blk.19.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  593:         blk.19.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  594:         blk.19.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  595:           blk.19.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  596:         blk.19.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  597:         blk.19.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  598:           blk.19.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  599:         blk.19.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  600:         blk.19.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  601:           blk.19.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  602:         blk.19.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  603:         blk.19.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  604:           blk.19.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  605:         blk.19.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  606:         blk.19.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  607:           blk.19.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  608:         blk.19.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  609:         blk.19.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  610:           blk.19.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  611:         blk.19.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  612:         blk.19.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  613:           blk.19.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  614:       blk.19.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  615:          blk.19.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  616:           blk.19.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  617:             blk.19.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  618:        blk.19.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  619:             blk.19.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  620:             blk.19.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  621:         blk.20.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  622:         blk.20.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  623:           blk.20.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  624:         blk.20.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  625:         blk.20.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  626:           blk.20.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  627:         blk.20.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  628:         blk.20.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  629:           blk.20.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  630:       blk.20.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  631:             blk.20.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  632:        blk.20.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  633:             blk.20.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  634:             blk.20.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  635:         blk.20.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  636:         blk.20.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  637:           blk.20.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  638:         blk.20.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  639:         blk.20.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  640:           blk.20.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  641:         blk.20.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  642:         blk.20.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  643:           blk.20.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  644:         blk.20.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  645:         blk.20.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  646:           blk.20.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  647:         blk.20.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  648:         blk.20.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  649:           blk.20.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  650:          blk.20.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  651:           blk.20.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  652:         blk.21.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  653:         blk.21.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  654:           blk.21.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  655:         blk.21.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  656:         blk.21.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  657:           blk.21.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  658:         blk.21.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  659:         blk.21.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  660:           blk.21.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  661:         blk.21.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  662:         blk.21.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  663:           blk.21.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  664:         blk.21.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  665:         blk.21.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  666:           blk.21.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  667:         blk.21.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  668:         blk.21.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  669:           blk.21.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  670:         blk.21.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  671:         blk.21.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  672:           blk.21.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  673:         blk.21.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  674:         blk.21.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  675:           blk.21.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  676:       blk.21.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  677:          blk.21.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  678:           blk.21.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  679:             blk.21.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  680:        blk.21.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  681:             blk.21.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  682:             blk.21.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  683:         blk.22.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  684:         blk.22.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  685:       blk.22.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  686:             blk.22.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  687:        blk.22.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  688:             blk.22.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  689:             blk.22.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  690:           blk.22.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  691:         blk.22.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  692:         blk.22.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  693:           blk.22.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  694:         blk.22.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  695:         blk.22.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  696:           blk.22.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  697:         blk.22.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  698:         blk.22.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  699:           blk.22.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  700:         blk.22.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  701:         blk.22.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  702:           blk.22.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  703:         blk.22.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  704:         blk.22.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  705:           blk.22.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  706:         blk.22.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  707:         blk.22.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  708:           blk.22.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  709:         blk.22.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  710:         blk.22.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  711:           blk.22.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  712:          blk.22.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  713:           blk.22.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  714:         blk.23.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  715:         blk.23.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  716:           blk.23.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  717:         blk.23.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  718:         blk.23.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  719:           blk.23.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  720:         blk.23.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  721:         blk.23.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  722:           blk.23.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  723:         blk.23.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  724:         blk.23.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  725:           blk.23.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  726:         blk.23.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  727:         blk.23.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  728:           blk.23.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  729:         blk.23.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  730:         blk.23.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  731:           blk.23.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  732:         blk.23.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  733:       blk.23.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  734:             blk.23.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  735:        blk.23.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  736:             blk.23.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  737:             blk.23.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  738:         blk.23.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  739:           blk.23.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  740:         blk.23.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  741:         blk.23.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  742:           blk.23.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  743:          blk.23.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  744:           blk.23.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  745:         blk.24.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  746:         blk.24.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  747:           blk.24.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  748:         blk.24.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  749:         blk.24.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  750:           blk.24.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  751:         blk.24.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  752:         blk.24.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  753:           blk.24.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  754:         blk.24.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  755:         blk.24.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  756:           blk.24.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  757:         blk.24.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  758:         blk.24.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  759:           blk.24.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  760:         blk.24.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  761:         blk.24.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  762:           blk.24.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  763:         blk.24.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  764:         blk.24.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  765:           blk.24.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  766:         blk.24.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  767:         blk.24.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  768:           blk.24.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  769:       blk.24.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  770:          blk.24.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  771:           blk.24.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  772:             blk.24.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  773:        blk.24.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  774:             blk.24.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  775:             blk.24.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  776:         blk.25.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  777:         blk.25.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  778:           blk.25.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  779:         blk.25.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  780:         blk.25.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  781:           blk.25.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  782:         blk.25.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  783:         blk.25.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  784:           blk.25.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  785:         blk.25.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  786:         blk.25.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  787:           blk.25.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  788:       blk.25.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  789:             blk.25.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  790:        blk.25.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  791:             blk.25.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  792:             blk.25.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  793:         blk.25.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  794:         blk.25.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  795:           blk.25.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  796:         blk.25.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  797:         blk.25.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  798:           blk.25.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  799:         blk.25.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  800:         blk.25.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  801:           blk.25.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  802:         blk.25.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  803:         blk.25.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  804:           blk.25.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  805:          blk.25.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  806:           blk.25.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  807:         blk.26.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  808:         blk.26.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  809:           blk.26.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  810:         blk.26.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  811:         blk.26.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  812:           blk.26.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  813:         blk.26.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  814:         blk.26.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  815:           blk.26.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  816:         blk.26.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  817:         blk.26.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  818:           blk.26.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  819:         blk.26.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  820:         blk.26.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  821:           blk.26.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  822:         blk.26.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  823:         blk.26.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  824:           blk.26.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  825:         blk.26.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  826:         blk.26.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  827:           blk.26.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  828:         blk.26.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  829:         blk.26.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  830:           blk.26.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  831:       blk.26.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  832:          blk.26.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  833:           blk.26.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  834:             blk.26.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  835:        blk.26.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  836:             blk.26.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  837:             blk.26.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  838:         blk.27.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  839:         blk.27.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  840:           blk.27.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  841:         blk.27.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  842:         blk.27.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  843:       blk.27.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  844:             blk.27.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  845:        blk.27.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  846:             blk.27.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  847:             blk.27.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  848:           blk.27.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  849:         blk.27.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  850:         blk.27.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  851:           blk.27.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  852:         blk.27.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  853:         blk.27.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  854:           blk.27.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  855:         blk.27.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  856:         blk.27.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  857:           blk.27.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  858:         blk.27.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  859:         blk.27.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  860:           blk.27.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  861:         blk.27.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  862:         blk.27.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  863:           blk.27.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  864:         blk.27.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  865:         blk.27.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  866:           blk.27.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  867:          blk.27.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  868:           blk.27.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  869:         blk.28.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  870:         blk.28.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  871:           blk.28.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  872:         blk.28.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  873:         blk.28.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  874:           blk.28.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  875:         blk.28.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  876:         blk.28.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  877:           blk.28.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  878:         blk.28.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  879:         blk.28.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  880:           blk.28.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  881:         blk.28.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  882:         blk.28.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  883:           blk.28.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  884:         blk.28.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  885:         blk.28.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  886:           blk.28.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  887:         blk.28.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  888:         blk.28.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  889:           blk.28.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  890:         blk.28.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  891:       blk.28.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  892:             blk.28.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  893:        blk.28.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  894:             blk.28.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  895:             blk.28.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  896:         blk.28.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  897:           blk.28.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  898:          blk.28.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  899:           blk.28.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  900:         blk.29.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  901:         blk.29.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  902:           blk.29.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  903:         blk.29.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  904:         blk.29.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  905:           blk.29.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  906:         blk.29.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  907:         blk.29.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  908:           blk.29.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  909:         blk.29.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  910:         blk.29.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  911:           blk.29.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  912:         blk.29.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  913:         blk.29.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  914:           blk.29.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  915:         blk.29.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  916:         blk.29.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  917:           blk.29.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  918:         blk.29.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  919:         blk.29.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  920:           blk.29.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  921:         blk.29.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  922:         blk.29.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  923:           blk.29.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  924:       blk.29.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  925:          blk.29.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  926:           blk.29.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  927:             blk.29.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  928:        blk.29.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  929:             blk.29.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  930:             blk.29.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  931:         blk.30.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  932:         blk.30.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  933:           blk.30.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  934:         blk.30.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  935:         blk.30.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  936:           blk.30.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  937:         blk.30.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  938:         blk.30.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  939:           blk.30.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  940:         blk.30.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  941:         blk.30.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  942:           blk.30.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  943:         blk.30.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  944:         blk.30.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  945:           blk.30.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  946:       blk.30.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  947:             blk.30.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  948:        blk.30.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  949:             blk.30.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  950:             blk.30.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  951:                    output.weight q6_K     [  4096, 32000,     1,     1 ]
llama_model_loader: - tensor  952:         blk.30.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  953:         blk.30.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  954:           blk.30.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  955:         blk.30.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  956:         blk.30.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  957:           blk.30.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  958:         blk.30.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  959:         blk.30.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  960:           blk.30.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  961:          blk.30.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  962:           blk.30.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  963:         blk.31.ffn_gate.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  964:         blk.31.ffn_down.0.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  965:           blk.31.ffn_up.0.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  966:         blk.31.ffn_gate.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  967:         blk.31.ffn_down.1.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  968:           blk.31.ffn_up.1.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  969:         blk.31.ffn_gate.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  970:         blk.31.ffn_down.2.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  971:           blk.31.ffn_up.2.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  972:         blk.31.ffn_gate.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  973:         blk.31.ffn_down.3.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  974:           blk.31.ffn_up.3.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  975:         blk.31.ffn_gate.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  976:         blk.31.ffn_down.4.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  977:           blk.31.ffn_up.4.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  978:         blk.31.ffn_gate.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  979:         blk.31.ffn_down.5.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  980:           blk.31.ffn_up.5.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  981:         blk.31.ffn_gate.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  982:         blk.31.ffn_down.6.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  983:           blk.31.ffn_up.6.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  984:         blk.31.ffn_gate.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  985:         blk.31.ffn_down.7.weight q4_K     [ 14336,  4096,     1,     1 ]
llama_model_loader: - tensor  986:           blk.31.ffn_up.7.weight q4_K     [  4096, 14336,     1,     1 ]
llama_model_loader: - tensor  987:       blk.31.ffn_gate_inp.weight f16      [  4096,     8,     1,     1 ]
llama_model_loader: - tensor  988:          blk.31.attn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  989:           blk.31.ffn_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: - tensor  990:             blk.31.attn_k.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  991:        blk.31.attn_output.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  992:             blk.31.attn_q.weight q4_K     [  4096,  4096,     1,     1 ]
llama_model_loader: - tensor  993:             blk.31.attn_v.weight q8_0     [  4096,  1024,     1,     1 ]
llama_model_loader: - tensor  994:               output_norm.weight f32      [  4096,     1,     1,     1 ]
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = mistralai
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:                         llama.expert_count u32              = 8
llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2
llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:                          general.file_type u32              = 15
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  20:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  21:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  22:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  23:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type  f16:   32 tensors
llama_model_loader: - type q8_0:   64 tensors
llama_model_loader: - type q4_K:  833 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 8
llm_load_print_meta: n_expert_used    = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = mostly Q4_K - Medium
llm_load_print_meta: model params     = 46.70 B
llm_load_print_meta: model size       = 24.62 GiB (4.53 BPW) 
llm_load_print_meta: general.name     = mistralai
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.39 MiB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required  = 7999.20 MiB
llm_load_tensors: offloading 22 repeating layers to GPU
llm_load_tensors: offloaded 22/33 layers to GPU
llm_load_tensors: VRAM used: 17217.06 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 32768
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: VRAM kv self = 2816.00 MB
llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
llama_build_graph: non-view tensors processed: 1124/1124
llama_new_context_with_model: compute buffer total size = 2167.35 MiB
llama_new_context_with_model: VRAM scratch buffer: 2164.04 MiB
llama_new_context_with_model: total VRAM used: 22197.10 MiB (model: 17217.06 MiB, context: 4980.04 MiB)
{"timestamp":1702930214,"level":"INFO","function":"main","line":3035,"message":"HTTP server listening","hostname":"127.0.0.1","port":59735}
{"timestamp":1702930214,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":51756,"status":200,"method":"HEAD","path":"/","params":{}}
2023/12/18 20:10:14 llama.go:508: llama runner started in 17.801696 seconds
2023/12/18 20:10:14 llama.go:577: loaded 0 images
{"timestamp":1702930227,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":51756,"status":200,"method":"POST","path":"/completion","params":{}}
{"timestamp":1702930227,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":46844,"status":200,"method":"POST","path":"/tokenize","params":{}}
[GIN] 2023/12/18 - 20:10:27 | 200 | 31.133913884s |      172.28.0.4 | POST     "/api/generate"
{"timestamp":1702930227,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":46844,"status":200,"method":"HEAD","path":"/","params":{}}
2023/12/18 20:10:27 llama.go:577: loaded 0 images
{"timestamp":1702930255,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":46844,"status":200,"method":"POST","path":"/completion","params":{}}
{"timestamp":1702930255,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":48110,"status":200,"method":"POST","path":"/tokenize","params":{}}
[GIN] 2023/12/18 - 20:10:55 | 200 | 27.681923952s |      172.28.0.4 | POST     "/api/generate"
2023/12/18 20:15:57 llama.go:451: signal: killed
2023/12/18 20:15:57 llama.go:525: llama runner stopped successfully
Originally created by @AlessandroSpallina on GitHub (Dec 18, 2023). Original GitHub issue: https://github.com/ollama/ollama/issues/1585 Originally assigned to: @mxyng on GitHub. Hi, I'm opening this issue because I noticed a weird behavior running ollama on docker with GPU support and trying different mixtral 8x7B sizes: I can easily do inference on my GPU with models like mixtral:8x7b-instruct-v0.1-q4_K_M but I see a memory failure when running smaller models like mixtral:8x7b-instruct-v0.1-q3_K_M. I'm on Ubuntu 23.10, my GPU is a NVIDIA 3090. My docker-compose.yml: ``` version: '3.7' services: ollama: container_name: ollama_cat_dev image: ollama/ollama:0.1.16 restart: unless-stopped volumes: - ./ollama:/root/.ollama expose: - 11434 environment: - gpus=all deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] ``` As said, when I try to use mixtral:8x7b-instruct-v0.1-q3_K_M I see "out of memory" issues and the inference is completely done on CPU. Here the log (attached because it was too long for github issues) [ollama.log](https://github.com/jmorganca/ollama/files/13708188/ollama.log) Here instead the log when I run a bigger model like mixtral:8x7b-instruct-v0.1-q4_K_M: ``` 2023/12/18 20:09:56 llama.go:300: 23732 MB VRAM available, loading up to 22 GPU layers 2023/12/18 20:09:56 llama.go:436: starting llama runner 2023/12/18 20:09:56 llama.go:494: waiting for llama runner to start responding ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes ggml_init_cublas: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6 {"timestamp":1702930196,"level":"INFO","function":"main","line":2652,"message":"build info","build":441,"commit":"948ff13"} {"timestamp":1702930196,"level":"INFO","function":"main","line":2655,"message":"system info","n_threads":16,"n_threads_batch":-1,"total_threads":32,"system_info":"AVX = 1 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 0 | ARM_FMA = 0 | F16C = 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | "} llama_model_loader: loaded meta data with 24 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256:59a66936ed54cfe28136f99a3ec5336f2b404bad0bc0f3a48123f87c677d8623 (version GGUF V3 (latest)) llama_model_loader: - tensor 0: token_embd.weight q4_K [ 4096, 32000, 1, 1 ] llama_model_loader: - tensor 1: blk.0.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 2: blk.0.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 3: blk.0.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 4: blk.0.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 5: blk.0.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 6: blk.0.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 7: blk.0.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 8: blk.0.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 9: blk.0.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 10: blk.0.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 11: blk.0.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 12: blk.0.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 13: blk.0.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 14: blk.0.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 15: blk.0.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 16: blk.0.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 17: blk.0.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 18: blk.0.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 19: blk.0.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 20: blk.0.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 21: blk.0.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 22: blk.0.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 23: blk.0.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 24: blk.0.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 25: blk.0.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 26: blk.0.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 27: blk.0.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 28: blk.0.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 29: blk.0.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 30: blk.0.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 31: blk.0.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 32: blk.1.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 33: blk.1.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 34: blk.1.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 35: blk.1.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 36: blk.1.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 37: blk.1.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 38: blk.1.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 39: blk.1.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 40: blk.1.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 41: blk.1.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 42: blk.1.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 43: blk.1.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 44: blk.1.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 45: blk.1.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 46: blk.1.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 47: blk.1.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 48: blk.1.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 49: blk.1.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 50: blk.1.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 51: blk.1.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 52: blk.1.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 53: blk.1.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 54: blk.1.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 55: blk.1.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 56: blk.1.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 57: blk.1.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 58: blk.1.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 59: blk.1.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 60: blk.1.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 61: blk.1.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 62: blk.1.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 63: blk.2.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 64: blk.2.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 65: blk.2.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 66: blk.2.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 67: blk.2.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 68: blk.2.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 69: blk.2.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 70: blk.2.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 71: blk.2.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 72: blk.2.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 73: blk.2.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 74: blk.2.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 75: blk.2.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 76: blk.2.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 77: blk.2.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 78: blk.2.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 79: blk.2.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 80: blk.2.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 81: blk.2.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 82: blk.2.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 83: blk.2.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 84: blk.2.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 85: blk.2.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 86: blk.2.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 87: blk.2.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 88: blk.2.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 89: blk.2.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 90: blk.2.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 91: blk.2.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 92: blk.2.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 93: blk.2.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 94: blk.3.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 95: blk.3.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 96: blk.3.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 97: blk.3.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 98: blk.3.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 99: blk.3.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 100: blk.3.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 101: blk.3.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 102: blk.3.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 103: blk.3.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 104: blk.3.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 105: blk.3.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 106: blk.3.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 107: blk.3.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 108: blk.3.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 109: blk.3.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 110: blk.3.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 111: blk.3.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 112: blk.3.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 113: blk.3.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 114: blk.3.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 115: blk.3.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 116: blk.3.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 117: blk.3.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 118: blk.3.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 119: blk.3.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 120: blk.3.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 121: blk.3.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 122: blk.3.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 123: blk.3.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 124: blk.3.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 125: blk.4.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 126: blk.4.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 127: blk.4.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 128: blk.4.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 129: blk.4.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 130: blk.4.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 131: blk.4.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 132: blk.4.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 133: blk.4.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 134: blk.4.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 135: blk.4.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 136: blk.4.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 137: blk.4.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 138: blk.4.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 139: blk.4.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 140: blk.4.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 141: blk.4.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 142: blk.4.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 143: blk.4.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 144: blk.4.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 145: blk.4.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 146: blk.4.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 147: blk.4.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 148: blk.4.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 149: blk.4.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 150: blk.4.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 151: blk.4.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 152: blk.4.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 153: blk.4.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 154: blk.4.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 155: blk.4.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 156: blk.5.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 157: blk.5.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 158: blk.5.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 159: blk.5.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 160: blk.5.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 161: blk.5.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 162: blk.5.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 163: blk.5.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 164: blk.5.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 165: blk.5.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 166: blk.5.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 167: blk.5.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 168: blk.5.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 169: blk.5.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 170: blk.5.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 171: blk.5.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 172: blk.5.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 173: blk.5.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 174: blk.5.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 175: blk.5.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 176: blk.5.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 177: blk.5.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 178: blk.5.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 179: blk.5.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 180: blk.5.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 181: blk.5.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 182: blk.5.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 183: blk.5.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 184: blk.5.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 185: blk.5.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 186: blk.5.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 187: blk.6.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 188: blk.6.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 189: blk.6.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 190: blk.6.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 191: blk.6.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 192: blk.6.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 193: blk.6.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 194: blk.6.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 195: blk.6.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 196: blk.6.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 197: blk.6.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 198: blk.6.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 199: blk.6.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 200: blk.6.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 201: blk.6.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 202: blk.6.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 203: blk.6.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 204: blk.6.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 205: blk.6.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 206: blk.6.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 207: blk.6.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 208: blk.6.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 209: blk.6.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 210: blk.6.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 211: blk.6.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 212: blk.6.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 213: blk.6.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 214: blk.6.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 215: blk.6.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 216: blk.6.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 217: blk.6.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 218: blk.7.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 219: blk.7.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 220: blk.7.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 221: blk.7.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 222: blk.7.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 223: blk.7.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 224: blk.7.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 225: blk.7.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 226: blk.7.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 227: blk.7.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 228: blk.7.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 229: blk.7.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 230: blk.7.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 231: blk.7.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 232: blk.7.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 233: blk.7.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 234: blk.7.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 235: blk.7.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 236: blk.7.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 237: blk.7.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 238: blk.7.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 239: blk.7.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 240: blk.7.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 241: blk.7.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 242: blk.7.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 243: blk.7.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 244: blk.7.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 245: blk.7.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 246: blk.7.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 247: blk.7.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 248: blk.7.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 249: blk.8.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 250: blk.8.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 251: blk.8.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 252: blk.8.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 253: blk.8.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 254: blk.8.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 255: blk.8.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 256: blk.8.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 257: blk.8.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 258: blk.8.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 259: blk.8.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 260: blk.8.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 261: blk.8.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 262: blk.8.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 263: blk.8.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 264: blk.10.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 265: blk.10.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 266: blk.10.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 267: blk.10.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 268: blk.10.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 269: blk.10.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 270: blk.10.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 271: blk.10.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 272: blk.8.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 273: blk.8.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 274: blk.8.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 275: blk.8.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 276: blk.8.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 277: blk.8.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 278: blk.8.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 279: blk.8.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 280: blk.8.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 281: blk.8.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 282: blk.8.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 283: blk.8.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 284: blk.8.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 285: blk.8.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 286: blk.8.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 287: blk.8.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 288: blk.9.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 289: blk.9.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 290: blk.9.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 291: blk.9.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 292: blk.9.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 293: blk.9.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 294: blk.9.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 295: blk.9.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 296: blk.9.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 297: blk.9.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 298: blk.9.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 299: blk.9.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 300: blk.9.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 301: blk.9.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 302: blk.9.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 303: blk.9.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 304: blk.9.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 305: blk.9.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 306: blk.9.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 307: blk.9.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 308: blk.9.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 309: blk.9.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 310: blk.9.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 311: blk.9.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 312: blk.9.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 313: blk.9.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 314: blk.9.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 315: blk.9.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 316: blk.9.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 317: blk.9.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 318: blk.9.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 319: blk.10.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 320: blk.10.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 321: blk.10.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 322: blk.10.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 323: blk.10.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 324: blk.10.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 325: blk.10.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 326: blk.10.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 327: blk.10.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 328: blk.10.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 329: blk.10.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 330: blk.10.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 331: blk.10.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 332: blk.10.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 333: blk.10.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 334: blk.10.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 335: blk.10.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 336: blk.10.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 337: blk.10.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 338: blk.10.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 339: blk.10.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 340: blk.10.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 341: blk.10.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 342: blk.11.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 343: blk.11.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 344: blk.11.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 345: blk.11.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 346: blk.11.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 347: blk.11.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 348: blk.11.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 349: blk.11.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 350: blk.11.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 351: blk.11.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 352: blk.11.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 353: blk.11.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 354: blk.11.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 355: blk.11.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 356: blk.11.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 357: blk.11.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 358: blk.11.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 359: blk.11.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 360: blk.11.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 361: blk.11.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 362: blk.11.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 363: blk.11.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 364: blk.11.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 365: blk.11.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 366: blk.11.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 367: blk.11.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 368: blk.11.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 369: blk.11.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 370: blk.11.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 371: blk.11.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 372: blk.11.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 373: blk.12.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 374: blk.12.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 375: blk.12.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 376: blk.12.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 377: blk.12.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 378: blk.12.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 379: blk.12.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 380: blk.12.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 381: blk.12.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 382: blk.12.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 383: blk.12.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 384: blk.12.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 385: blk.12.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 386: blk.12.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 387: blk.12.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 388: blk.12.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 389: blk.12.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 390: blk.12.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 391: blk.12.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 392: blk.12.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 393: blk.12.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 394: blk.12.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 395: blk.12.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 396: blk.12.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 397: blk.12.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 398: blk.12.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 399: blk.12.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 400: blk.12.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 401: blk.12.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 402: blk.12.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 403: blk.12.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 404: blk.13.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 405: blk.13.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 406: blk.13.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 407: blk.13.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 408: blk.13.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 409: blk.13.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 410: blk.13.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 411: blk.13.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 412: blk.13.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 413: blk.13.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 414: blk.13.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 415: blk.13.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 416: blk.13.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 417: blk.13.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 418: blk.13.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 419: blk.13.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 420: blk.13.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 421: blk.13.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 422: blk.13.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 423: blk.13.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 424: blk.13.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 425: blk.13.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 426: blk.13.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 427: blk.13.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 428: blk.13.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 429: blk.13.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 430: blk.13.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 431: blk.13.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 432: blk.13.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 433: blk.13.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 434: blk.13.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 435: blk.14.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 436: blk.14.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 437: blk.14.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 438: blk.14.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 439: blk.14.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 440: blk.14.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 441: blk.14.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 442: blk.14.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 443: blk.14.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 444: blk.14.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 445: blk.14.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 446: blk.14.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 447: blk.14.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 448: blk.14.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 449: blk.14.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 450: blk.14.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 451: blk.14.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 452: blk.14.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 453: blk.14.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 454: blk.14.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 455: blk.14.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 456: blk.14.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 457: blk.14.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 458: blk.14.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 459: blk.14.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 460: blk.14.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 461: blk.14.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 462: blk.14.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 463: blk.14.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 464: blk.14.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 465: blk.14.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 466: blk.15.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 467: blk.15.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 468: blk.15.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 469: blk.15.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 470: blk.15.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 471: blk.15.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 472: blk.15.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 473: blk.15.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 474: blk.15.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 475: blk.15.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 476: blk.15.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 477: blk.15.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 478: blk.15.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 479: blk.15.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 480: blk.15.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 481: blk.15.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 482: blk.15.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 483: blk.15.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 484: blk.15.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 485: blk.15.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 486: blk.15.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 487: blk.15.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 488: blk.15.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 489: blk.15.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 490: blk.15.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 491: blk.15.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 492: blk.15.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 493: blk.15.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 494: blk.15.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 495: blk.15.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 496: blk.15.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 497: blk.16.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 498: blk.16.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 499: blk.16.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 500: blk.16.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 501: blk.16.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 502: blk.16.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 503: blk.16.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 504: blk.16.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 505: blk.16.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 506: blk.16.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 507: blk.16.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 508: blk.16.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 509: blk.16.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 510: blk.16.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 511: blk.16.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 512: blk.16.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 513: blk.16.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 514: blk.16.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 515: blk.16.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 516: blk.16.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 517: blk.16.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 518: blk.16.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 519: blk.16.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 520: blk.16.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 521: blk.16.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 522: blk.16.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 523: blk.16.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 524: blk.16.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 525: blk.16.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 526: blk.16.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 527: blk.16.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 528: blk.17.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 529: blk.17.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 530: blk.17.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 531: blk.17.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 532: blk.17.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 533: blk.17.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 534: blk.17.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 535: blk.17.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 536: blk.17.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 537: blk.17.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 538: blk.17.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 539: blk.17.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 540: blk.17.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 541: blk.17.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 542: blk.17.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 543: blk.17.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 544: blk.17.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 545: blk.17.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 546: blk.17.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 547: blk.17.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 548: blk.17.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 549: blk.17.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 550: blk.17.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 551: blk.17.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 552: blk.17.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 553: blk.17.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 554: blk.17.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 555: blk.17.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 556: blk.17.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 557: blk.17.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 558: blk.17.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 559: blk.18.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 560: blk.18.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 561: blk.18.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 562: blk.18.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 563: blk.18.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 564: blk.18.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 565: blk.18.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 566: blk.18.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 567: blk.18.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 568: blk.18.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 569: blk.18.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 570: blk.18.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 571: blk.18.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 572: blk.18.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 573: blk.18.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 574: blk.18.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 575: blk.18.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 576: blk.18.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 577: blk.18.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 578: blk.18.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 579: blk.18.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 580: blk.18.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 581: blk.18.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 582: blk.18.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 583: blk.18.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 584: blk.18.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 585: blk.18.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 586: blk.18.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 587: blk.18.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 588: blk.18.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 589: blk.18.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 590: blk.19.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 591: blk.19.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 592: blk.19.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 593: blk.19.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 594: blk.19.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 595: blk.19.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 596: blk.19.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 597: blk.19.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 598: blk.19.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 599: blk.19.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 600: blk.19.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 601: blk.19.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 602: blk.19.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 603: blk.19.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 604: blk.19.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 605: blk.19.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 606: blk.19.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 607: blk.19.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 608: blk.19.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 609: blk.19.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 610: blk.19.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 611: blk.19.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 612: blk.19.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 613: blk.19.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 614: blk.19.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 615: blk.19.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 616: blk.19.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 617: blk.19.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 618: blk.19.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 619: blk.19.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 620: blk.19.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 621: blk.20.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 622: blk.20.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 623: blk.20.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 624: blk.20.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 625: blk.20.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 626: blk.20.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 627: blk.20.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 628: blk.20.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 629: blk.20.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 630: blk.20.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 631: blk.20.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 632: blk.20.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 633: blk.20.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 634: blk.20.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 635: blk.20.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 636: blk.20.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 637: blk.20.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 638: blk.20.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 639: blk.20.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 640: blk.20.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 641: blk.20.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 642: blk.20.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 643: blk.20.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 644: blk.20.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 645: blk.20.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 646: blk.20.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 647: blk.20.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 648: blk.20.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 649: blk.20.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 650: blk.20.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 651: blk.20.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 652: blk.21.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 653: blk.21.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 654: blk.21.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 655: blk.21.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 656: blk.21.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 657: blk.21.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 658: blk.21.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 659: blk.21.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 660: blk.21.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 661: blk.21.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 662: blk.21.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 663: blk.21.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 664: blk.21.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 665: blk.21.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 666: blk.21.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 667: blk.21.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 668: blk.21.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 669: blk.21.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 670: blk.21.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 671: blk.21.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 672: blk.21.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 673: blk.21.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 674: blk.21.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 675: blk.21.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 676: blk.21.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 677: blk.21.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 678: blk.21.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 679: blk.21.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 680: blk.21.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 681: blk.21.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 682: blk.21.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 683: blk.22.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 684: blk.22.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 685: blk.22.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 686: blk.22.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 687: blk.22.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 688: blk.22.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 689: blk.22.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 690: blk.22.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 691: blk.22.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 692: blk.22.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 693: blk.22.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 694: blk.22.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 695: blk.22.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 696: blk.22.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 697: blk.22.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 698: blk.22.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 699: blk.22.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 700: blk.22.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 701: blk.22.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 702: blk.22.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 703: blk.22.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 704: blk.22.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 705: blk.22.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 706: blk.22.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 707: blk.22.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 708: blk.22.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 709: blk.22.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 710: blk.22.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 711: blk.22.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 712: blk.22.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 713: blk.22.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 714: blk.23.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 715: blk.23.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 716: blk.23.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 717: blk.23.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 718: blk.23.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 719: blk.23.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 720: blk.23.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 721: blk.23.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 722: blk.23.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 723: blk.23.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 724: blk.23.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 725: blk.23.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 726: blk.23.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 727: blk.23.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 728: blk.23.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 729: blk.23.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 730: blk.23.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 731: blk.23.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 732: blk.23.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 733: blk.23.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 734: blk.23.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 735: blk.23.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 736: blk.23.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 737: blk.23.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 738: blk.23.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 739: blk.23.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 740: blk.23.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 741: blk.23.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 742: blk.23.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 743: blk.23.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 744: blk.23.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 745: blk.24.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 746: blk.24.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 747: blk.24.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 748: blk.24.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 749: blk.24.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 750: blk.24.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 751: blk.24.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 752: blk.24.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 753: blk.24.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 754: blk.24.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 755: blk.24.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 756: blk.24.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 757: blk.24.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 758: blk.24.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 759: blk.24.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 760: blk.24.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 761: blk.24.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 762: blk.24.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 763: blk.24.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 764: blk.24.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 765: blk.24.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 766: blk.24.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 767: blk.24.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 768: blk.24.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 769: blk.24.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 770: blk.24.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 771: blk.24.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 772: blk.24.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 773: blk.24.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 774: blk.24.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 775: blk.24.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 776: blk.25.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 777: blk.25.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 778: blk.25.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 779: blk.25.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 780: blk.25.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 781: blk.25.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 782: blk.25.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 783: blk.25.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 784: blk.25.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 785: blk.25.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 786: blk.25.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 787: blk.25.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 788: blk.25.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 789: blk.25.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 790: blk.25.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 791: blk.25.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 792: blk.25.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 793: blk.25.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 794: blk.25.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 795: blk.25.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 796: blk.25.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 797: blk.25.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 798: blk.25.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 799: blk.25.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 800: blk.25.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 801: blk.25.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 802: blk.25.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 803: blk.25.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 804: blk.25.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 805: blk.25.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 806: blk.25.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 807: blk.26.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 808: blk.26.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 809: blk.26.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 810: blk.26.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 811: blk.26.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 812: blk.26.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 813: blk.26.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 814: blk.26.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 815: blk.26.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 816: blk.26.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 817: blk.26.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 818: blk.26.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 819: blk.26.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 820: blk.26.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 821: blk.26.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 822: blk.26.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 823: blk.26.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 824: blk.26.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 825: blk.26.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 826: blk.26.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 827: blk.26.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 828: blk.26.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 829: blk.26.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 830: blk.26.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 831: blk.26.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 832: blk.26.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 833: blk.26.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 834: blk.26.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 835: blk.26.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 836: blk.26.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 837: blk.26.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 838: blk.27.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 839: blk.27.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 840: blk.27.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 841: blk.27.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 842: blk.27.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 843: blk.27.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 844: blk.27.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 845: blk.27.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 846: blk.27.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 847: blk.27.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 848: blk.27.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 849: blk.27.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 850: blk.27.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 851: blk.27.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 852: blk.27.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 853: blk.27.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 854: blk.27.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 855: blk.27.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 856: blk.27.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 857: blk.27.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 858: blk.27.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 859: blk.27.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 860: blk.27.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 861: blk.27.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 862: blk.27.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 863: blk.27.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 864: blk.27.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 865: blk.27.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 866: blk.27.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 867: blk.27.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 868: blk.27.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 869: blk.28.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 870: blk.28.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 871: blk.28.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 872: blk.28.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 873: blk.28.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 874: blk.28.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 875: blk.28.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 876: blk.28.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 877: blk.28.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 878: blk.28.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 879: blk.28.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 880: blk.28.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 881: blk.28.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 882: blk.28.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 883: blk.28.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 884: blk.28.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 885: blk.28.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 886: blk.28.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 887: blk.28.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 888: blk.28.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 889: blk.28.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 890: blk.28.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 891: blk.28.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 892: blk.28.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 893: blk.28.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 894: blk.28.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 895: blk.28.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 896: blk.28.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 897: blk.28.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 898: blk.28.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 899: blk.28.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 900: blk.29.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 901: blk.29.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 902: blk.29.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 903: blk.29.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 904: blk.29.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 905: blk.29.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 906: blk.29.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 907: blk.29.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 908: blk.29.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 909: blk.29.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 910: blk.29.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 911: blk.29.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 912: blk.29.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 913: blk.29.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 914: blk.29.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 915: blk.29.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 916: blk.29.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 917: blk.29.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 918: blk.29.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 919: blk.29.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 920: blk.29.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 921: blk.29.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 922: blk.29.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 923: blk.29.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 924: blk.29.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 925: blk.29.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 926: blk.29.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 927: blk.29.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 928: blk.29.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 929: blk.29.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 930: blk.29.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 931: blk.30.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 932: blk.30.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 933: blk.30.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 934: blk.30.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 935: blk.30.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 936: blk.30.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 937: blk.30.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 938: blk.30.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 939: blk.30.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 940: blk.30.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 941: blk.30.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 942: blk.30.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 943: blk.30.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 944: blk.30.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 945: blk.30.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 946: blk.30.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 947: blk.30.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 948: blk.30.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 949: blk.30.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 950: blk.30.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 951: output.weight q6_K [ 4096, 32000, 1, 1 ] llama_model_loader: - tensor 952: blk.30.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 953: blk.30.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 954: blk.30.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 955: blk.30.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 956: blk.30.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 957: blk.30.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 958: blk.30.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 959: blk.30.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 960: blk.30.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 961: blk.30.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 962: blk.30.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 963: blk.31.ffn_gate.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 964: blk.31.ffn_down.0.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 965: blk.31.ffn_up.0.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 966: blk.31.ffn_gate.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 967: blk.31.ffn_down.1.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 968: blk.31.ffn_up.1.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 969: blk.31.ffn_gate.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 970: blk.31.ffn_down.2.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 971: blk.31.ffn_up.2.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 972: blk.31.ffn_gate.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 973: blk.31.ffn_down.3.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 974: blk.31.ffn_up.3.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 975: blk.31.ffn_gate.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 976: blk.31.ffn_down.4.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 977: blk.31.ffn_up.4.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 978: blk.31.ffn_gate.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 979: blk.31.ffn_down.5.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 980: blk.31.ffn_up.5.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 981: blk.31.ffn_gate.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 982: blk.31.ffn_down.6.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 983: blk.31.ffn_up.6.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 984: blk.31.ffn_gate.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 985: blk.31.ffn_down.7.weight q4_K [ 14336, 4096, 1, 1 ] llama_model_loader: - tensor 986: blk.31.ffn_up.7.weight q4_K [ 4096, 14336, 1, 1 ] llama_model_loader: - tensor 987: blk.31.ffn_gate_inp.weight f16 [ 4096, 8, 1, 1 ] llama_model_loader: - tensor 988: blk.31.attn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 989: blk.31.ffn_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: - tensor 990: blk.31.attn_k.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 991: blk.31.attn_output.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 992: blk.31.attn_q.weight q4_K [ 4096, 4096, 1, 1 ] llama_model_loader: - tensor 993: blk.31.attn_v.weight q8_0 [ 4096, 1024, 1, 1 ] llama_model_loader: - tensor 994: output_norm.weight f32 [ 4096, 1, 1, 1 ] llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = mistralai llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.expert_count u32 = 8 llama_model_loader: - kv 10: llama.expert_used_count u32 = 2 llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 12: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: tokenizer.ggml.model str = llama llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 2 llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 23: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type f16: 32 tensors llama_model_loader: - type q8_0: 64 tensors llama_model_loader: - type q4_K: 833 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 259/32000 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 8 llm_load_print_meta: n_expert_used = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = mostly Q4_K - Medium llm_load_print_meta: model params = 46.70 B llm_load_print_meta: model size = 24.62 GiB (4.53 BPW) llm_load_print_meta: general.name = mistralai llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.39 MiB llm_load_tensors: using CUDA for GPU acceleration llm_load_tensors: mem required = 7999.20 MiB llm_load_tensors: offloading 22 repeating layers to GPU llm_load_tensors: offloaded 22/33 layers to GPU llm_load_tensors: VRAM used: 17217.06 MiB .................................................................................................... llama_new_context_with_model: n_ctx = 32768 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: VRAM kv self = 2816.00 MB llama_new_context_with_model: KV self size = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB llama_build_graph: non-view tensors processed: 1124/1124 llama_new_context_with_model: compute buffer total size = 2167.35 MiB llama_new_context_with_model: VRAM scratch buffer: 2164.04 MiB llama_new_context_with_model: total VRAM used: 22197.10 MiB (model: 17217.06 MiB, context: 4980.04 MiB) {"timestamp":1702930214,"level":"INFO","function":"main","line":3035,"message":"HTTP server listening","hostname":"127.0.0.1","port":59735} {"timestamp":1702930214,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":51756,"status":200,"method":"HEAD","path":"/","params":{}} 2023/12/18 20:10:14 llama.go:508: llama runner started in 17.801696 seconds 2023/12/18 20:10:14 llama.go:577: loaded 0 images {"timestamp":1702930227,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":51756,"status":200,"method":"POST","path":"/completion","params":{}} {"timestamp":1702930227,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":46844,"status":200,"method":"POST","path":"/tokenize","params":{}} [GIN] 2023/12/18 - 20:10:27 | 200 | 31.133913884s | 172.28.0.4 | POST "/api/generate" {"timestamp":1702930227,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":46844,"status":200,"method":"HEAD","path":"/","params":{}} 2023/12/18 20:10:27 llama.go:577: loaded 0 images {"timestamp":1702930255,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":46844,"status":200,"method":"POST","path":"/completion","params":{}} {"timestamp":1702930255,"level":"INFO","function":"log_server_request","line":2596,"message":"request","remote_addr":"127.0.0.1","remote_port":48110,"status":200,"method":"POST","path":"/tokenize","params":{}} [GIN] 2023/12/18 - 20:10:55 | 200 | 27.681923952s | 172.28.0.4 | POST "/api/generate" 2023/12/18 20:15:57 llama.go:451: signal: killed 2023/12/18 20:15:57 llama.go:525: llama runner stopped successfully ```
GiteaMirror added the questionbugnvidia labels 2026-05-03 10:45:09 -05:00
GiteaMirror commented 2026-05-03 10:45:13 -05:00
Author
Owner
Copy Link

@Malnes commented on GitHub (Dec 21, 2023):

same

<!-- gh-comment-id:1866119249 --> @Malnes commented on GitHub (Dec 21, 2023): same
GiteaMirror commented 2026-05-03 10:45:13 -05:00
Author
Owner
Copy Link

@seanmavley commented on GitHub (Dec 22, 2023):

Update:

To build from source, include -DLLAMA_CUDA_FORCE_MMQ=on on the line in the file below. It should be the generate_linux.go file.

fabf2f3467/llm/llama.cpp/generate_linux.go (L24C141-L24C141)

Experiencing the same error.

Even building from source, I have no idea how to specify explicitly to turn on the GGML_CUDA_FORCE_MMQ

<!-- gh-comment-id:1867613090 --> @seanmavley commented on GitHub (Dec 22, 2023): Update: To build from source, include `-DLLAMA_CUDA_FORCE_MMQ=on` on the line in the file below. It should be the `generate_linux.go` file. https://github.com/jmorganca/ollama/blob/fabf2f3467860b7cb181b1d691cd3ae303c6eee6/llm/llama.cpp/generate_linux.go#L24C141-L24C141 Experiencing the same error. Even building from source, I have no idea how to specify explicitly to turn on the `GGML_CUDA_FORCE_MMQ`
GiteaMirror commented 2026-05-03 10:45:15 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Jan 27, 2024):

@AlessandroSpallina we've refined out memory prediction logic over the past few weeks to try to do a better job estimating what we can load into the GPU. Can you give 0.1.22 a try and see if you still experience the out of memory crash?

<!-- gh-comment-id:1912893629 --> @dhiltgen commented on GitHub (Jan 27, 2024): @AlessandroSpallina we've refined out memory prediction logic over the past few weeks to try to do a better job estimating what we can load into the GPU. Can you give 0.1.22 a try and see if you still experience the out of memory crash?
GiteaMirror commented 2026-05-03 10:45:17 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Feb 1, 2024):

Please re-open if you're still seeing the out-of-memory crash on 0.1.22 or newer.

<!-- gh-comment-id:1922464943 --> @dhiltgen commented on GitHub (Feb 1, 2024): Please re-open if you're still seeing the out-of-memory crash on 0.1.22 or newer.
GiteaMirror commented 2026-05-03 10:45:18 -05:00
Author
Owner
Copy Link

@m0wer commented on GitHub (Feb 2, 2024):

Please re-open if you're still seeing the out-of-memory crash on 0.1.22 or newer.

Still happens. It's not about this model in particular, just a matter of luck. But it isn't hard to make it crash.

<!-- gh-comment-id:1923138349 --> @m0wer commented on GitHub (Feb 2, 2024): > Please re-open if you're still seeing the out-of-memory crash on 0.1.22 or newer. Still happens. It's not about this model in particular, just a matter of luck. But it isn't hard to make it crash.
GiteaMirror commented 2026-05-03 10:45:19 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Feb 2, 2024):

@m0wer can you share an updated server log and which model you're running?

<!-- gh-comment-id:1924718085 --> @dhiltgen commented on GitHub (Feb 2, 2024): @m0wer can you share an updated server log and which model you're running?
GiteaMirror commented 2026-05-03 10:45:19 -05:00
Author
Owner
Copy Link

@m0wer commented on GitHub (Feb 17, 2024):

@m0wer can you share an updated server log and which model you're running?

Sure! With 0.1.24 and two GPUs of different memery sizes (RTX 2060 GB and RTX 3090 24 GB) I get:

time=2024-02-17T16:20:59.897Z level=INFO source=payload_common.go:145 msg="Dynamic LLM libraries [rocm_v5 cuda_v11 cpu rocm_v6 cpu_avx cpu_avx2]"
time=2024-02-17T16:20:59.897Z level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-02-17T16:20:59.897Z level=INFO source=gpu.go:242 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-02-17T16:20:59.898Z level=INFO source=gpu.go:288 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.545.23.08]"
time=2024-02-17T16:20:59.904Z level=INFO source=gpu.go:99 msg="Nvidia GPU detected"
time=2024-02-17T16:20:59.904Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-17T16:20:59.921Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 7.5"


time=2024-02-17T16:21:09.144Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-17T16:21:09.144Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 7.5"
time=2024-02-17T16:21:09.144Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-02-17T16:21:09.144Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 7.5"
time=2024-02-17T16:21:09.144Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
loading library /tmp/ollama1092102558/cuda_v11/libext_server.so
time=2024-02-17T16:21:09.152Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama1092102558/cuda_v11/libext_server.so"
time=2024-02-17T16:21:09.152Z level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server"
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   yes
ggml_init_cublas: CUDA_USE_TENSOR_CORES: no
ggml_init_cublas: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes
llama_model_loader: loaded meta data with 24 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256:f4dc3c36a568eb77b903e97e170f9745ee407a2d058019137d62bf3f59329082 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = cognitivecomputations
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:                         llama.expert_count u32              = 8
llama_model_loader: - kv  10:                    llama.expert_used_count u32              = 2
llama_model_loader: - kv  11:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  12:                       llama.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:                          general.file_type u32              = 15
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,32002]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  16:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  18:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 32000
llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  23:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type  f16:   32 tensors
llama_model_loader: - type q8_0:   64 tensors
llama_model_loader: - type q4_K:  833 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens definition check successful ( 261/32002 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32002
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 8
llm_load_print_meta: n_expert_used    = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 46.70 B
llm_load_print_meta: model size       = 24.62 GiB (4.53 BPW)
llm_load_print_meta: general.name     = cognitivecomputations
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 32000 '<|im_end|>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    1.14 MiB
llm_load_tensors: offloading 31 repeating layers to GPU
llm_load_tensors: offloaded 31/33 layers to GPU
llm_load_tensors:        CPU buffer size = 25215.88 MiB
llm_load_tensors:      CUDA0 buffer size = 19564.84 MiB
llm_load_tensors:      CUDA1 buffer size =  4695.56 MiB
....................................................................................................
llama_new_context_with_model: n_ctx      = 8192
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:  CUDA_Host KV buffer size =    32.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   800.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   192.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:  CUDA_Host input buffer size   =    24.04 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   646.81 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =   646.81 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =   589.60 MiB
llama_new_context_with_model: graph splits (measure): 7
time=2024-02-17T16:21:23.487Z level=INFO source=dyn_ext_server.go:156 msg="Starting llama main loop"
CUDA error: out of memory
  current device: 1, in function ggml_cuda_pool_malloc_vmm at /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.cu:7756
  cuMemCreate(&handle, reserve_size, &prop, 0)
GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.cu:241: !"CUDA error"

It works well at first until the context gets large enough and then crashes. The GGUF is:

FROM dolphin-mixtral:8x7b-v2.7-q4_K_M
PARAMETER num_ctx 8192
<!-- gh-comment-id:1950249915 --> @m0wer commented on GitHub (Feb 17, 2024): > @m0wer can you share an updated server log and which model you're running? Sure! With 0.1.24 and two GPUs of different memery sizes (RTX 2060 GB and RTX 3090 24 GB) I get: ``` time=2024-02-17T16:20:59.897Z level=INFO source=payload_common.go:145 msg="Dynamic LLM libraries [rocm_v5 cuda_v11 cpu rocm_v6 cpu_avx cpu_avx2]" time=2024-02-17T16:20:59.897Z level=INFO source=gpu.go:94 msg="Detecting GPU type" time=2024-02-17T16:20:59.897Z level=INFO source=gpu.go:242 msg="Searching for GPU management library libnvidia-ml.so" time=2024-02-17T16:20:59.898Z level=INFO source=gpu.go:288 msg="Discovered GPU libraries: [/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.545.23.08]" time=2024-02-17T16:20:59.904Z level=INFO source=gpu.go:99 msg="Nvidia GPU detected" time=2024-02-17T16:20:59.904Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-17T16:20:59.921Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 7.5" time=2024-02-17T16:21:09.144Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-17T16:21:09.144Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 7.5" time=2024-02-17T16:21:09.144Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" time=2024-02-17T16:21:09.144Z level=INFO source=gpu.go:146 msg="CUDA Compute Capability detected: 7.5" time=2024-02-17T16:21:09.144Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2" loading library /tmp/ollama1092102558/cuda_v11/libext_server.so time=2024-02-17T16:21:09.152Z level=INFO source=dyn_ext_server.go:90 msg="Loading Dynamic llm server: /tmp/ollama1092102558/cuda_v11/libext_server.so" time=2024-02-17T16:21:09.152Z level=INFO source=dyn_ext_server.go:145 msg="Initializing llama server" ggml_init_cublas: GGML_CUDA_FORCE_MMQ: yes ggml_init_cublas: CUDA_USE_TENSOR_CORES: no ggml_init_cublas: found 2 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes Device 1: NVIDIA GeForce RTX 2060, compute capability 7.5, VMM: yes llama_model_loader: loaded meta data with 24 key-value pairs and 995 tensors from /root/.ollama/models/blobs/sha256:f4dc3c36a568eb77b903e97e170f9745ee407a2d058019137d62bf3f59329082 (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = llama llama_model_loader: - kv 1: general.name str = cognitivecomputations llama_model_loader: - kv 2: llama.context_length u32 = 32768 llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 llama_model_loader: - kv 4: llama.block_count u32 = 32 llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 llama_model_loader: - kv 9: llama.expert_count u32 = 8 llama_model_loader: - kv 10: llama.expert_used_count u32 = 2 llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 12: llama.rope.freq_base f32 = 1000000.000000 llama_model_loader: - kv 13: general.file_type u32 = 15 llama_model_loader: - kv 14: tokenizer.ggml.model str = llama llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32002] = ["<unk>", "<s>", "</s>", "<0x00>", "<... llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32002] = [0.000000, 0.000000, 0.000000, 0.0000... llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32002] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 32000 llama_model_loader: - kv 20: tokenizer.ggml.add_bos_token bool = true llama_model_loader: - kv 21: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 22: tokenizer.chat_template str = {% if not add_generation_prompt is de... llama_model_loader: - kv 23: general.quantization_version u32 = 2 llama_model_loader: - type f32: 65 tensors llama_model_loader: - type f16: 32 tensors llama_model_loader: - type q8_0: 64 tensors llama_model_loader: - type q4_K: 833 tensors llama_model_loader: - type q6_K: 1 tensors llm_load_vocab: special tokens definition check successful ( 261/32002 ). llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32002 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 32768 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 8 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_embd_head_k = 128 llm_load_print_meta: n_embd_head_v = 128 llm_load_print_meta: n_gqa = 4 llm_load_print_meta: n_embd_k_gqa = 1024 llm_load_print_meta: n_embd_v_gqa = 1024 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: n_ff = 14336 llm_load_print_meta: n_expert = 8 llm_load_print_meta: n_expert_used = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 1000000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_yarn_orig_ctx = 32768 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = Q4_K - Medium llm_load_print_meta: model params = 46.70 B llm_load_print_meta: model size = 24.62 GiB (4.53 BPW) llm_load_print_meta: general.name = cognitivecomputations llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 32000 '<|im_end|>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 1.14 MiB llm_load_tensors: offloading 31 repeating layers to GPU llm_load_tensors: offloaded 31/33 layers to GPU llm_load_tensors: CPU buffer size = 25215.88 MiB llm_load_tensors: CUDA0 buffer size = 19564.84 MiB llm_load_tensors: CUDA1 buffer size = 4695.56 MiB .................................................................................................... llama_new_context_with_model: n_ctx = 8192 llama_new_context_with_model: freq_base = 1000000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA_Host KV buffer size = 32.00 MiB llama_kv_cache_init: CUDA0 KV buffer size = 800.00 MiB llama_kv_cache_init: CUDA1 KV buffer size = 192.00 MiB llama_new_context_with_model: KV self size = 1024.00 MiB, K (f16): 512.00 MiB, V (f16): 512.00 MiB llama_new_context_with_model: CUDA_Host input buffer size = 24.04 MiB llama_new_context_with_model: CUDA0 compute buffer size = 646.81 MiB llama_new_context_with_model: CUDA1 compute buffer size = 646.81 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 589.60 MiB llama_new_context_with_model: graph splits (measure): 7 time=2024-02-17T16:21:23.487Z level=INFO source=dyn_ext_server.go:156 msg="Starting llama main loop" CUDA error: out of memory current device: 1, in function ggml_cuda_pool_malloc_vmm at /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.cu:7756 cuMemCreate(&handle, reserve_size, &prop, 0) GGML_ASSERT: /go/src/github.com/jmorganca/ollama/llm/llama.cpp/ggml-cuda.cu:241: !"CUDA error" ``` It works well at first until the context gets large enough and then crashes. The GGUF is: ```gguf FROM dolphin-mixtral:8x7b-v2.7-q4_K_M PARAMETER num_ctx 8192 ```
GiteaMirror commented 2026-05-03 10:45:19 -05:00
Author
Owner
Copy Link

@mchiang0610 commented on GitHub (Mar 11, 2024):

hi @m0wer, really sorry about this. Are you still seeing this with the latest 0.1.28?

<!-- gh-comment-id:1989153328 --> @mchiang0610 commented on GitHub (Mar 11, 2024): hi @m0wer, really sorry about this. Are you still seeing this with the latest 0.1.28?
GiteaMirror commented 2026-05-03 10:45:21 -05:00
Author
Owner
Copy Link

@dhiltgen commented on GitHub (Mar 11, 2024):

Unfortunately it looks like our memory prediction algorithm didn't work correctly for this setup and we attempted to load too many layers into the GPUs and it ran out of VRAM. We're continuing to improve our calculations to avoid this.

In the next release (0.1.29) we'll be adding a new setting that can allow you to set a lower VRAM setting to workaround this type of crash until we get the prediction logic fixed. OLLAMA_MAX_VRAM=<bytes> For example, you could start with 23G and experiment until you find a setting that loads the as many layers as possible without hitting the OOM crash. OLLAMA_MAX_VRAM=24696061952

<!-- gh-comment-id:1989307831 --> @dhiltgen commented on GitHub (Mar 11, 2024): Unfortunately it looks like our memory prediction algorithm didn't work correctly for this setup and we attempted to load too many layers into the GPUs and it ran out of VRAM. We're continuing to improve our calculations to avoid this. In the next release (0.1.29) we'll be adding a new setting that can allow you to set a lower VRAM setting to workaround this type of crash until we get the prediction logic fixed. `OLLAMA_MAX_VRAM=<bytes>` For example, you could start with 23G and experiment until you find a setting that loads the as many layers as possible without hitting the OOM crash. `OLLAMA_MAX_VRAM=24696061952`
GiteaMirror commented 2026-05-03 10:45:25 -05:00
Author
Owner
Copy Link

@m0wer commented on GitHub (Mar 22, 2024):

hi @m0wer, really sorry about this. Are you still seeing this with the latest 0.1.28?

Looks better now, with 0.1.29 it's not crashing.

<!-- gh-comment-id:2014601094 --> @m0wer commented on GitHub (Mar 22, 2024): > hi @m0wer, really sorry about this. Are you still seeing this with the latest 0.1.28? Looks better now, with 0.1.29 it's not crashing.
GiteaMirror commented 2026-05-03 10:45:28 -05:00
Author
Owner
Copy Link

@jmorganca commented on GitHub (May 10, 2024):

Great! Will close this for now but please do feel free to chime in to other OOM error issues

<!-- gh-comment-id:2103642942 --> @jmorganca commented on GitHub (May 10, 2024): Great! Will close this for now but please do feel free to chime in to other OOM error issues
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
amd
api
app
bug
build
cli
cloud
compatibility
context-length
create
docker
documentation
embeddings
feature request
feedback wanted
good first issue
gpt-oss
gpu
harmony
help wanted
image
install
intel
js
launch
linux
macos
memory
mlx
model
needs more info
networking
nvidia
ollama.com
performance
pull-request
Mirrored from GitHub Pull Request
 python
question
registry
rendering
thinking
tools
top
vulkan
windows
wsl
No Label bug nvidia question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/ollama#62911
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?