mirror of
https://github.com/harvard-edge/cs249r_book.git
synced 2026-07-21 17:57:24 -05:00
Vijay Janapa Reddi
cb0ae4082f
Run quartodoc build against the refactored package (engine/ extracted from core/, infra->infrastructure): regenerate the API reference so every stub points at the new paths. Adds engine.*/infrastructure.* stubs, removes the 41 stale core.solver.* / core.engine.* / core.scenarios.* / core.config.* / core.evaluation.* / infra*.qmd orphans, and refreshes the package pages (core now primitives-only) + index. Verified: all 47 documented symbols resolve in the new package; every index link resolves; zero stale mlsysim.core.<engine-mod> / mlsysim.infra references anywhere in docs/. (objects.json inventory is gitignored -- regenerated at build.)
58 lines
2.4 KiB
Plaintext
58 lines
2.4 KiB
Plaintext
# engine.solver.InferenceScalingModel { #mlsysim.engine.solver.InferenceScalingModel }
|
|
|
|
```python
|
|
engine.solver.InferenceScalingModel()
|
|
```
|
|
|
|
Models inference-time compute scaling (Wall 12: Reasoning/CoT Cost).
|
|
|
|
This model quantifies the cost of 'System-2 thinking' — inference-time
|
|
compute scaling via chain-of-thought (CoT) reasoning, where the model
|
|
generates K intermediate reasoning steps before producing the final answer.
|
|
Each step incurs the full cost of autoregressive decoding.
|
|
|
|
Literature Source:
|
|
1. Wei et al. (2022), "Chain-of-Thought Prompting Elicits Reasoning in
|
|
Large Language Models."
|
|
2. Snell et al. (2024), "Scaling LLM Test-Time Compute Optimally Can Be
|
|
More Effective Than Scaling Model Parameters."
|
|
3. OpenAI (2024), "Learning to Reason with LLMs." (o1 reasoning model.)
|
|
|
|
## Methods
|
|
|
|
| Name | Description |
|
|
| --- | --- |
|
|
| [solve](#mlsysim.engine.solver.InferenceScalingModel.solve) | Solves for inference-time reasoning cost. |
|
|
|
|
### solve { #mlsysim.engine.solver.InferenceScalingModel.solve }
|
|
|
|
```python
|
|
engine.solver.InferenceScalingModel.solve(
|
|
model,
|
|
hardware,
|
|
reasoning_steps=8,
|
|
context_length=2048,
|
|
precision='fp16',
|
|
efficiency=0.5,
|
|
)
|
|
```
|
|
|
|
Solves for inference-time reasoning cost.
|
|
|
|
#### Parameters {.doc-section .doc-section-parameters}
|
|
|
|
| Name | Type | Description | Default |
|
|
|-----------------|---------------------|------------------------------------------------------|------------|
|
|
| model | TransformerWorkload | The language model used for reasoning. | _required_ |
|
|
| hardware | HardwareNode | The target hardware node. | _required_ |
|
|
| reasoning_steps | int | Number of reasoning steps K (each generates tokens). | `8` |
|
|
| context_length | int | Input context length in tokens. | `2048` |
|
|
| precision | str | Numerical precision. | `'fp16'` |
|
|
| efficiency | float | Compute efficiency factor (0.0 to 1.0). | `0.5` |
|
|
|
|
#### Returns {.doc-section .doc-section-returns}
|
|
|
|
| Name | Type | Description |
|
|
|--------|------------------|---------------------------------------------------------|
|
|
| | Dict\[str, Any\] | Total reasoning time, cost per query, and token counts. |
|