* Add decision model integration tests and benchmarks
Support external JSONL corpora, quality scoring, concurrent requests, vision, and System One timing metrics.
* Refine decision tests and benchmarks with cache checks, external endpoints, and consistent response validation and timing
* bench: use code prompts for prompt-token runs
Replace the synthetic word-list prompt with packed HumanEval Python prompts so speculative draft models see code-like continuations.
Keep the benchmark on the chat path, and calibrate -prompt-tokens with untimed chat probes that read PromptEvalCount instead of relying on a separate token-count API.
Each request still gets a nonce prefix to avoid prefix-cache hits while timed epochs measure the chat-rendered workload.
* review comments
* use different windows for each epoch
* add OpenAI API support
* address comments
New features:
- Warmup phase to eliminate cold-start outliers
- time-to-first-token measured in each epoch
- VRAM/memory tracking to identify CPU spillover
- Controlled prompt length
- Defaults to 6 epochs and 200 tokens max
Benchstat fixes:
- ns/request instead of ns/op — non-standard unit created a separate group instead of grouping with timing metrics
- Token count as the N field — benchstat interprets N as iteration count for statistical weighting, not as a token count
This change adds a basic benchmarking test framework for Ollama which can
be used to determine the prefill, eval, load duration, and total duration
for running a given model or models.
Daniel HiltgenandGitHub
Eloi TorrentsandGitHub
Julia ScheafferandGitHub
Patrick DevineandGitHub