Daniel HiltgenandGitHub 16b4376aea bench: use HumanEval patch prompts (#17480) ...

* bench: use code prompts for prompt-token runs

Replace the synthetic word-list prompt with packed HumanEval Python prompts so speculative draft models see code-like continuations.

Keep the benchmark on the chat path, and calibrate -prompt-tokens with untimed chat probes that read PromptEvalCount instead of relying on a separate token-count API.

Each request still gets a nonce prefix to avoid prefix-cache hits while timed epochs measure the chat-rendered workload.

* review comments

* use different windows for each epoch

* add OpenAI API support

* address comments

2026-09-26 07:29:54 -07:00

..

HumanEval.jsonl

…

LICENSE

…