mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-17 08:28:07 -05:00
42 lines
2.3 KiB
Plaintext
42 lines
2.3 KiB
Plaintext
# solvers.TailLatencyModel { #mlsysim.solvers.TailLatencyModel }
|
|
|
|
```python
|
|
solvers.TailLatencyModel()
|
|
```
|
|
|
|
Analyzes queueing delays and P99 tail latency for deployed inference models.
|
|
|
|
Models inference servers as M/M/c queues to determine if the deployment
|
|
can sustain the target arrival rate while meeting strict SLA latency bounds.
|
|
|
|
Literature Source:
|
|
1. Dean & Barroso (2013), "The Tail at Scale."
|
|
|
|
## Methods
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [solve](#mlsysim.solvers.TailLatencyModel.solve) | Solves for P50 and P99 tail latencies under variable load. |
|
|
|
|
### solve { #mlsysim.solvers.TailLatencyModel.solve }
|
|
|
|
```python
|
|
solvers.TailLatencyModel.solve(
|
|
arrival_rate_qps,
|
|
service_latency_ms,
|
|
num_replicas=1,
|
|
service_time_cv=1.0,
|
|
)
|
|
```
|
|
|
|
Solves for P50 and P99 tail latencies under variable load.
|
|
|
|
#### Parameters {.doc-section .doc-section-parameters}
|
|
|
|
| Name | Type | Description | Default |
|
|
|--------------------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
|
|
| arrival_rate_qps | float | Request arrival rate in queries per second. | _required_ |
|
|
| service_latency_ms | float | Mean service time per request in milliseconds. | _required_ |
|
|
| num_replicas | int | Number of server replicas (c in M/M/c). | `1` |
|
|
| service_time_cv | float | Coefficient of variation of service time (default 1.0 = exponential). When CV != 1, applies Kingman's M/G/1 correction factor (cv^2 + 1) / 2 to queue wait times, approximating M/G/c behavior. | `1.0` |
|