mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-22 12:12:39 -05:00
Vijay Janapa Reddi
cb0ae4082f
Run quartodoc build against the refactored package (engine/ extracted from core/, infra->infrastructure): regenerate the API reference so every stub points at the new paths. Adds engine.*/infrastructure.* stubs, removes the 41 stale core.solver.* / core.engine.* / core.scenarios.* / core.config.* / core.evaluation.* / infra*.qmd orphans, and refreshes the package pages (core now primitives-only) + index. Verified: all 47 documented symbols resolve in the new package; every index link resolves; zero stale mlsysim.core.<engine-mod> / mlsysim.infra references anywhere in docs/. (objects.json inventory is gitignored -- regenerated at build.)
39 lines
1.0 KiB
Plaintext
39 lines
1.0 KiB
Plaintext
# engine.solver.ServingCapacityModel { #mlsysim.engine.solver.ServingCapacityModel }
|
|
|
|
```python
|
|
engine.solver.ServingCapacityModel()
|
|
```
|
|
|
|
Sizes an LLM serving deployment from a QPS and tail-latency target.
|
|
|
|
The model deliberately composes existing first-order pieces: ``ServingModel``
|
|
for TTFT/ITL, ``ContinuousBatchingModel`` for per-replica token capacity,
|
|
and ``TailLatencyModel`` for queueing pressure. It is a capacity planner,
|
|
not a request-level scheduler.
|
|
|
|
## Methods
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [solve](#mlsysim.engine.solver.ServingCapacityModel.solve) | Return the minimum replica count that satisfies the target P99. |
|
|
|
|
### solve { #mlsysim.engine.solver.ServingCapacityModel.solve }
|
|
|
|
```python
|
|
engine.solver.ServingCapacityModel.solve(
|
|
model,
|
|
hardware,
|
|
qps,
|
|
target_p99_latency_ms,
|
|
seq_len=2048,
|
|
output_tokens=128,
|
|
max_batch_size=32,
|
|
precision='fp16',
|
|
efficiency=0.5,
|
|
max_replicas=1024,
|
|
service_time_cv=1.0,
|
|
)
|
|
```
|
|
|
|
Return the minimum replica count that satisfies the target P99.
|