docs(frontiers): update chapter with 2025 AI developments

- Add callout note about rapidly evolving field and newer models
- Update scaling hypothesis with inference time compute (o1/o3 models)
- Update compound AI section with Gemini 2.0, Claude 3.5, ChatGPT capabilities
- Add Mamba-2 and hybrid architectures (Jamba) to state space models section

quarto/contents/core/frontiers/frontiers.qmd

@@ -44,6 +44,10 @@ Machine learning systems operate in a rapidly evolving technological landscape w
 
 :::
 
+::: {.callout-note title="A Rapidly Evolving Field"}
+AI capabilities advance at extraordinary pace. Since this chapter was written, new models (GPT-4o, Claude 3.5, Gemini 2.0, DeepSeek, and OpenAI's o1/o3 reasoning models) have pushed boundaries further. The o1 and o3 models demonstrate that explicit reasoning chains and extended inference time computation can dramatically improve complex problem solving, representing a shift from pure scaling toward inference time optimization. While specific benchmarks and model names will continue to evolve, the systems engineering principles, architectural patterns, and fundamental challenges discussed here remain durable. Focus on understanding the underlying engineering trade-offs rather than memorizing current state of the art metrics.
+:::
+
 ## From Specialized AI to General Intelligence {#sec-agi-systems-specialized-ai-general-intelligence-2f0a}
 
 When tasked with planning a complex, multi-day project, ChatGPT generates plausible sounding plans that often contain logical flaws. When asked to recall details from previous conversations, it fails due to lack of persistent memory. When required to explain why a particular solution works through first principles reasoning, it reproduces learned patterns rather than demonstrating genuine comprehension. These failures represent not simple bugs but fundamental architectural limitations. Contemporary models lack persistent memory, causal reasoning, and planning capabilities, the very attributes that define general intelligence.
@@ -84,6 +88,8 @@ Contemporary AGI research divides into four competing paradigms, each offering d
 
 The scaling hypothesis, championed by OpenAI and Anthropic, posits that AGI will emerge through continued scaling of transformer architectures [@kaplan2020scaling]. This approach extrapolates from observed scaling laws that reveal consistent, predictable relationships between model performance and three key factors: parameter count N, dataset size D, and compute budget C. Empirically, test loss follows power law relationships: L(N) ∝ N^(-α) for parameters, L(D) ∝ D^(-β) for data, and L(C) ∝ C^(-γ) for compute, where α ≈ 0.076, β ≈ 0.095, and γ ≈ 0.050 [@kaplan2020scaling]. These smooth, predictable curves suggest that each 10× increase in parameters yields measurable capability improvements across diverse tasks, from language understanding to reasoning and code generation.
 
+Recent developments have expanded the scaling hypothesis beyond training time compute to include inference time compute. OpenAI's o1 and o3 reasoning models demonstrate that allowing models to "think longer" during inference through explicit chain of thought reasoning and search over solution paths can dramatically improve performance on complex reasoning tasks. This suggests a new scaling dimension: rather than solely investing compute in larger models, allocating compute to extended inference enables models to tackle problems requiring multi-step reasoning, planning, and self-verification. The systems implications are significant, as inference time scaling requires different infrastructure optimizations than training time scaling.
+
 The extrapolation becomes striking when projected to AGI scale. If these scaling laws continue, AGI training would require approximately 2.5 × 10²⁶ FLOPs[^fn-agi-compute-requirements], a 250× increase over GPT-4's estimated compute budget. This represents not merely quantitative scaling but a qualitative bet: that sufficient scale will induce emergent capabilities like robust reasoning, planning, and knowledge integration that current models lack.
 
 [^fn-agi-compute-requirements]: **AGI Compute Extrapolation**: Based on Chinchilla scaling laws, AGI might require 2.5 × 10²⁶ FLOPs (250× GPT-4's compute). Alternative estimates using biological baselines suggest 6.3 × 10²³ operations. At current H100 efficiency: 175,000 GPUs for one year, 122 MW power consumption, $52 billion total cost including infrastructure. These projections assume no architectural advances; actual requirements could differ by orders of magnitude.
@@ -138,7 +144,7 @@ The organizational analogy illuminates this architecture. A single, monolithic A
 
 The compound approach offers five key advantages over monolithic models. First, modularity enables components to update independently without full system retraining. When OpenAI improves code interpretation, they swap that module without touching the language model, similar to upgrading a graphics card without replacing the entire computer. Second, specialization allows each component to optimize for its specific task. A dedicated retrieval system using vector databases outperforms a language model attempting to memorize all knowledge, just as specialized ASICs outperform general purpose CPUs for particular computations. Third, interpretability emerges from traceable decision paths through component interactions. When a system makes an error, engineers can identify whether retrieval, reasoning, or generation failed, which remains impossible with opaque end to end models. Fourth, scalability permits new capabilities to integrate without architectural overhauls. Adding voice recognition or robotic control becomes a matter of adding modules rather than retraining trillion parameter models. Fifth, safety benefits from multiple specialized validators constraining outputs at each stage. A toxicity filter checks generated text, a factuality verifier validates claims, and a safety monitor prevents harmful actions. This creates layered defense rather than relying on a single model to behave correctly.
 
-These advantages explain why every major AI lab now pursues compound architectures. Google's Gemini combines separate encoders for text, images, and audio. Anthropic's Claude integrates constitutional AI components for self-improvement. The engineering principles established throughout this textbook, from distributed systems to workflow orchestration, now converge to enable these compound systems.
+These advantages explain why every major AI lab now pursues compound architectures. Google's Gemini 2.0 combines multimodal understanding with native tool use and agentic capabilities. Anthropic's Claude 3.5 integrates constitutional AI components, computer use capabilities, and extended context windows enabling sophisticated multi-step workflows. OpenAI's ChatGPT orchestrates plugins, code execution, image generation, and web browsing through unified interfaces. The rapid evolution of these systems, from single-purpose assistants to multi-capable agents, demonstrates that compound architecture adoption accelerates as capabilities mature. The engineering principles established throughout this textbook, from distributed systems to workflow orchestration, now converge to enable these compound systems.
 
 ## Building Blocks for Compound Intelligence {#sec-agi-systems-building-blocks-compound-intelligence-7a34}
 
@@ -630,7 +636,7 @@ where x ∈ ℝ is the input token, h ∈ ℝᵈ is the hidden state, y ∈ ℝ
 
 The technical breakthrough enabling competitive performance came from selective state spaces where the recurrence parameters themselves depend on the input: Āₜ = f_A(xₜ), B̄ₜ = f_B(xₜ), making the state transition input-dependent rather than fixed. This selectivity allows the model to dynamically adjust which information to remember or forget based on current input content. When processing "The trophy doesn't fit in the suitcase because it's too big," the model can selectively maintain "trophy" in state while discarding less relevant words, with the selection driven by learned input-dependent gating similar to LSTM forget gates but within the state space framework. This approach resembles maintaining a running summary that adapts its compression strategy based on content importance rather than blindly summarizing everything equally.
 
-Models like Mamba [@gu2023mamba], RWKV [@peng2023rwkv], and Liquid Time-constant Networks [@hasani2020liquid] demonstrate that this approach can match transformer performance on many tasks while scaling linearly rather than quadratically with sequence length. Using selective state spaces with input-dependent parameters, Mamba achieves 5× better throughput on long sequences (100K+ tokens) compared to transformers. Mamba-7B matches transformer-7B performance on text while using 5× less memory for 100K token sequences. RWKV combines the efficient inference of RNNs with the parallelizable training of transformers, while Liquid Time-constant Networks adapt their dynamics based on input, showing particular promise for time-series and continuous control tasks.
+Models like Mamba [@gu2023mamba], RWKV [@peng2023rwkv], and Liquid Time-constant Networks [@hasani2020liquid] demonstrate that this approach can match transformer performance on many tasks while scaling linearly rather than quadratically with sequence length. Using selective state spaces with input-dependent parameters, Mamba achieves 5× better throughput on long sequences (100K+ tokens) compared to transformers. Mamba-7B matches transformer-7B performance on text while using 5× less memory for 100K token sequences. Subsequent developments including Mamba-2 have further improved both efficiency and quality, while hybrid architectures combining state space layers with attention (as in Jamba) suggest that the future may involve complementary mechanisms rather than wholesale architectural replacement. RWKV combines the efficient inference of RNNs with the parallelizable training of transformers, while Liquid Time-constant Networks adapt their dynamics based on input, showing particular promise for time-series and continuous control tasks.
 
 Systems engineering implications are significant. Linear scaling enables processing book-length contexts, multi-hour conversations, or entire codebases within single model calls. This requires rethinking data loading strategies (handling MB-scale inputs), memory management (streaming rather than batch processing), and distributed inference patterns optimized for sequential processing rather than parallel attention.