mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-16 23:24:55 -05:00
40 lines
1.1 KiB
Plaintext
40 lines
1.1 KiB
Plaintext
# solvers.ContinuousBatchingModel { #mlsysim.solvers.ContinuousBatchingModel }
|
|
|
|
```python
|
|
solvers.ContinuousBatchingModel()
|
|
```
|
|
|
|
Analyzes production LLM serving with Continuous Batching and PagedAttention.
|
|
|
|
Traditional static batching suffers from severe memory fragmentation and
|
|
padding waste. This model simulates the throughput improvements achieved by
|
|
iteration-level scheduling and non-contiguous KV cache allocation.
|
|
|
|
Literature Source:
|
|
1. Kwon et al. (2023), "Efficient Memory Management for Large Language Model
|
|
Serving with PagedAttention."
|
|
2. Yu et al. (2022), "ORCA: A Distributed Serving System for
|
|
Transformer-Based Generative Models."
|
|
|
|
## Methods
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [solve](#mlsysim.solvers.ContinuousBatchingModel.solve) | Calculates continuous batching throughput and PagedAttention memory. |
|
|
|
|
### solve { #mlsysim.solvers.ContinuousBatchingModel.solve }
|
|
|
|
```python
|
|
solvers.ContinuousBatchingModel.solve(
|
|
model,
|
|
hardware,
|
|
seq_len,
|
|
max_batch_size=1,
|
|
page_size=16,
|
|
precision='fp16',
|
|
efficiency=0.5,
|
|
)
|
|
```
|
|
|
|
Calculates continuous batching throughput and PagedAttention memory.
|