cs249r_book/mlsysim/tutorial/assessment/simulation-qa-round1.md

Simulated Q&A and Hallway Conversations -- ISCA 2026 Tutorial

Purpose: Stress-test the tutorial's claims before a live audience finds the weak spots. Date generated: 2026-04-01 Methodology: Each question is drawn from a real archetype (the skeptic, the competitor, the domain expert, the methodologist). Answers are the presenter's honest best response given the current state of mlsysim v0.1.0 and the calibration data in empirical-calibration.md.

---

Part 1: Tough Questions (Post-Session Q&A)

---

Q1: "How is this different from a spreadsheet?"

Questioner archetype: Senior engineer, slightly bored, has seen a hundred tool demos.

The question (verbatim): "I appreciate the presentation, but I could do all of this in a spreadsheet. I have a Google Sheet where I plug in FLOPS, bandwidth, and model size and get the same answers. What does mlsysim give me that Excel doesn't?"

The best honest answer:

You are right that any single equation in mlsysim can be reproduced in a spreadsheet. People do this all the time, and for one-off calculations it works fine. The differences show up at scale:

1. Unit safety. mlsysim uses Pint for dimensional analysis. Every quantity carries its unit. If you accidentally add GB to TFLOPS, you get a DimensionalityError at the point of the mistake, not a silently wrong number three rows down. Spreadsheets have no type system -- a cell is a cell. We have seen real production capacity planning errors that came from mixing GB and GiB in a spreadsheet, or confusing FLOP/s with FLOPs (rate vs count). Those errors are structurally impossible in mlsysim.

2. Composability across 22 walls. Your spreadsheet probably covers 3-4 constraints. mlsysim composes all 22 into a single Pipeline.solve() call. The capstone exercise we just did -- throughput, latency, budget, carbon, fault tolerance, all simultaneously -- would be a nightmare in a spreadsheet because the constraints interact. Changing precision changes memory, which changes whether you need tensor parallelism, which changes communication cost, which changes fleet size, which changes carbon. That cascade is what the pipeline handles.

3. Traceability. Every hardware constant in mlsysim is a TraceableConstant with a source, date, and DOI. When your spreadsheet says "H100 bandwidth = 3350 GB/s," where did that number come from? Is it the datasheet peak or the measured sustained? Is it HBM3 or HBM3E? In mlsysim, you can audit any number back to its origin.

But I want to be clear: if you have a well-maintained spreadsheet that solves your problem, keep using it. mlsysim is most valuable when you are doing comparative analysis across many hardware platforms, or when you need to hand your analysis to someone else and they need to trust the numbers.

Answer quality: 7/10

What would make it stronger: A live side-by-side demo showing a real spreadsheet error that mlsysim catches. The unit-safety argument is strong in theory but needs a concrete "I had a $200K capacity planning error because of a GiB/GB confusion" war story. Without the anecdote, the senior engineer thinks "I just label my columns carefully." Also: the 22-wall composition claim is aspirational -- v0.1.0 does not actually compose all 22 in a single solve call yet. The pipeline exists for compute/memory/communication/cost/carbon, but walls like Tail Latency (Erlang-C), Multi-tenant (queueing), and Safety (DP-SGD overhead) are not yet wired into the solver. Being caught overstating coverage would be worse than admitting the gap.

---

Q2: "Your accuracy is 2-5x off. Why wouldn't I just benchmark?"

Questioner archetype: Industry engineer who runs real clusters. Pragmatist.

The question (verbatim): "Your own slides say the accuracy is within 2-5x of measured performance. That is an order of magnitude. If I need to make a hardware purchasing decision, I need numbers I can trust. Why would I use something that might be off by 5x instead of just running a benchmark?"

The best honest answer:

That is a fair challenge, and I want to be precise about what "2-5x" means and where it comes from.

First, the 2-5x number is our worst case across all configurations with default efficiency parameters. The calibration table in our docs shows the actual spread. For LLM decode latency -- the serving use case most people care about -- we are within the published range, not 2-5x off, because decode is memory-bandwidth-bound and the model correctly computes weights / bandwidth. There is no efficiency parameter on the critical path. For CNN training throughput, the default eta produces predictions that are 22% low on A100 and 54% high on H100. That is where the "2-5x" envelope comes from.

Second, benchmarking is always better when you can do it. The question is: can you? Benchmarking requires having the hardware, which means you have already purchased it or negotiated cloud access. It requires having the software stack working on that hardware, which for a new platform can take weeks. And it gives you one data point: this model, on this hardware, at this batch size, with this framework version.

mlsysim is for the phase before benchmarking: "Should I even request time on this cluster?" "Is it worth porting to AMD MI300X, or is it obviously bandwidth-starved for my workload?" "How many GPUs do I need to request in my cloud allocation?" If mlsysim tells you a workload is 3x over the memory capacity of a given GPU, you do not need to benchmark to know it will not work.

Third, with a single calibrated measurement, accuracy improves dramatically. If you measure eta on one hardware platform, the model matches benchmarks within 1-5% for that platform, and gives you a reasonable starting point for others. The workflow is: benchmark once, calibrate eta, then use mlsysim for the design space exploration.

Answer quality: 8/10

What would make it stronger: The answer is honest and the "benchmark once, explore with mlsysim" workflow is compelling. The weak spot is that the calibration doc shows eta does NOT transfer across GPU generations (A100 eta=0.13 vs H100 eta=0.065 for the same ResNet-50 workload). The presenter should own this explicitly: "Cross-generation transfer is the known weak point. We are within 2x, not 1%, when transferring eta across architectures." Overpromising on cross-platform accuracy would get destroyed in peer review.

---

Q3: "The efficiency parameter is just a fudge factor, right?"

Questioner archetype: Architecture PhD student. Technically sharp. Wants to understand the epistemology.

The question (verbatim): "You showed that you calibrate eta per-benchmark to match published numbers. ResNet-50 on A100 needs eta=0.13, ResNet-50 on H100 needs eta=0.065. You call this the CPI analogy, but CPI is measured -- you back-calculate eta from the answer you want to predict. Isn't that circular? It is a fudge factor."

The best honest answer:

I want to take this seriously because it is the right question.

You are correct that when we calibrate eta per-benchmark, we are fitting a single parameter to match observations. That is definitionally a fudge factor if the only thing we do with it is reproduce the number we already measured.

The value is not in reproducing the known number. The value is in what you can do after calibration. Once you have eta=0.13 for ResNet-50 on A100, you can ask: "What happens if I double the batch size? What if I switch to FP8? What if I add pipeline parallelism?" The model makes predictions for those counterfactuals that are constrained by physics -- the FLOP count changes, the memory footprint changes, the communication volume changes -- and eta carries forward as the empirical correction.

The CPI analogy is precise. Patterson and Hennessy measure CPI from SPEC benchmarks. You could call CPI a fudge factor too -- it absorbs cache miss rates, branch mispredictions, pipeline hazards, and a dozen other things into one number. The reason it is useful is not that CPI is predictable from first principles. It is that the performance equation Time = Instructions x CPI x Clock_Period lets you reason about what happens when you change the ISA (Instructions change), or the clock frequency (Clock_Period changes), or the microarchitecture (CPI changes). Each term is independently variable.

Where this analogy breaks down -- and I should be honest about this -- is transferability. CPI for a given benchmark on a given ISA transfers reasonably well across microarchitectures within a generation. Our calibration data shows that eta does NOT transfer well across GPU generations for the same workload. ResNet-50 gets eta=0.13 on A100 and eta=0.065 on H100. That is a 2x difference, and it means you cannot use eta measured on A100 to predict H100 performance without significant error.

We think this is a real and important limitation, and we document it explicitly. The fix is not to pretend eta transfers -- it is to build a richer efficiency model that decomposes eta into sub-factors (kernel utilization, memory system efficiency, framework overhead) that transfer independently. That is future work.

Answer quality: 9/10

What would make it stronger: This is the best answer in the set because it concedes the legitimate criticism, gives the honest intellectual defense, and names the specific failure mode. The only improvement would be showing preliminary results from a decomposed eta model -- even a two-factor version (compute utilization + memory utilization) that transfers better. Without that, the "future work" claim is promissory.

---

Q4: "Does this work for MoE models like Mixtral?"

Questioner archetype: Applied ML researcher working on MoE architectures.

The question (verbatim): "All your examples are dense models. Mixture-of-Experts changes the arithmetic intensity dramatically -- only 2 of 8 experts are active per token in Mixtral, so the active parameter count is 12B but the total is 46B. The memory footprint is 46B but the compute is 12B. Does mlsysim handle this?"

The best honest answer:

Not natively in v0.1.0, and this is an important gap.

You are exactly right about the analysis. MoE creates a fundamental decoupling between memory footprint and compute that breaks the assumption of most analytical models, including ours. For dense models, parameters and FLOPs are tightly coupled via the 6ND rule. For MoE, you need to track them separately: all experts must be resident in memory (46B for Mixtral), but only the active experts contribute FLOPs per token (roughly 12B worth).

In the current version, you could model Mixtral by manually setting the model parameters to 46B (for memory calculations) and overriding the FLOPs to match the active expert count. That is a workaround, not proper support.

What proper MoE support requires is: (1) separating the memory model from the compute model in the solver, which is an architectural change; (2) modeling the expert routing overhead -- the gating network, the all-to-all communication in distributed MoE where different tokens route to different GPUs; and (3) modeling the load imbalance problem, where popular experts become bottlenecks.

The expert routing communication pattern is particularly important for ISCA audiences. Dense models use AllReduce (symmetric, bandwidth-optimal). MoE uses All-to-All (asymmetric, sensitive to load balance). The communication wall looks completely different.

This is on our roadmap. MoE is the single most requested feature.

Answer quality: 6/10

What would make it stronger: The answer correctly identifies the gap and shows domain understanding, but "it is on our roadmap" is the weakest possible ending at ISCA. A concrete timeline ("v0.2 in Q4 2026") or a branch with preliminary MoE support would be much stronger. Even better: have the workaround as a prepared code snippet that the questioner can try immediately. Saying "manually override the FLOPs" without showing the 5-line code example makes it feel like a hand-wave.

---

Q5: "Why should I use this instead of Calculon?"

Questioner archetype: Someone who has actually used Calculon from HPE/LLNL for training performance modeling.

The question (verbatim): "Calculon already does analytical training performance modeling. It handles 3D parallelism, pipeline bubbles, and communication overlap. It was validated against Megatron-LM at scale. Why should I switch?"

The best honest answer:

You should not switch. You should use both, for different questions.

Calculon is excellent at what it does: training performance modeling for large language models with 3D parallelism. It was built by people who run some of the largest training clusters in the world, and it shows. If your question is "what is the optimal parallelism configuration for training GPT-4-scale models on 2048 H100s," Calculon is probably the better tool right now. It models pipeline bubble fractions, communication-computation overlap, and micro-batch scheduling in more detail than mlsysim does.

mlsysim makes different trade-offs:

1. Breadth over depth. Calculon covers training on NVIDIA hardware. mlsysim covers training, inference, serving, TinyML, cost, carbon, and sustainability across five vendors. If your question is "should I deploy this model on H100, MI300X, or Gaudi 3, and what is the carbon footprint of each option," Calculon cannot help you.

2. Inference and serving. Calculon does not model the two-phase serving regime (prefill/decode), KV-cache memory pressure, or tail latency under load. mlsysim does.

3. Unit safety and traceability. This is a differentiator for pedagogical and audit use cases. Every number in mlsysim is dimensionally typed and traceable to a source.

4. Pedagogical design. mlsysim was designed for teaching. The API is intentionally simple: Engine.solve(model, hardware, batch_size). Calculon is designed for research-grade modeling, which means a steeper learning curve and more configuration.

Where Calculon wins cleanly: training-specific fidelity, communication overlap modeling, validation at real scale (thousands of GPUs with real Megatron-LM measurements). We respect that work enormously.

The honest positioning is: mlsysim is a broader, shallower tool. Calculon is a narrower, deeper one. Use mlsysim for rapid design-space exploration across the full stack. Use Calculon for detailed training performance prediction once you have narrowed the design space.

Answer quality: 8/10

What would make it stronger: This answer is good because it does not trash the competitor. The risk is that "broader but shallower" sounds like "worse at everything Calculon does." The presenter should have a prepared example where breadth matters: "A startup choosing between H100 and MI300X for a serving workload cannot use Calculon at all. mlsysim gives them a quantitative answer in under a second." The serving use case is the clearest differentiator -- lean into it hard.

---

Q6: "All your examples are Llama and ResNet. What about diffusion models?"

Questioner archetype: Computer vision researcher working on generative models.

The question (verbatim): "You showed Llama-3-8B and ResNet-50 in every exercise. Those are Transformer and CNN workloads. Diffusion models like Stable Diffusion have a completely different compute profile -- iterative denoising, U-Net backbone, cross-attention with text embeddings, variable-length generation. Can mlsysim handle that?"

The best honest answer:

The honest answer is: partially, and with manual effort.

The core physics still applies. A diffusion model is ultimately a sequence of forward passes through a neural network (the U-Net or DiT), each of which has a known FLOP count and memory footprint. mlsysim can model each denoising step as a forward pass and multiply by the number of steps. The Roofline analysis applies -- each step is either compute-bound or memory-bound depending on the batch size and model size.

What we do not model natively:

1. The iterative structure. Diffusion inference requires N denoising steps (typically 20-50). Total latency is N times the per-step latency. This is trivial to compute but our solver API is not designed around iterative generation -- you would need to multiply the single-step result by N yourself.

2. The U-Net architecture. Our FLOP counting assumes either a standard Transformer or a CNN. U-Nets have skip connections and variable resolution stages that make the per-layer FLOP distribution uneven. You would need to provide the total FLOPs manually rather than relying on our auto-counting.

3. Cross-attention. The text-conditioned cross-attention between the CLIP embeddings and the U-Net features is a different attention pattern than self-attention in Transformers. It has different memory and compute characteristics.

4. Classifier-free guidance. CFG doubles the forward pass cost (one conditioned, one unconditioned). This is easy to model (multiply by 2) but is not automatic.

The newer DiT (Diffusion Transformer) architectures are actually easier for us to model because they are standard Transformers with the iterative denoising wrapper. As the field moves from U-Net to DiT, our coverage improves.

I would say: for rough capacity planning ("can I serve Stable Diffusion XL on this GPU at this batch size?"), mlsysim works with manual FLOP input. For detailed latency optimization of the denoising pipeline, you need profiling tools.

Answer quality: 5/10

What would make it stronger: This answer reveals a real coverage gap. The presenter knows the physics but has to say "do it manually" four times. A prepared notebook showing DiffusionModel support -- even if it just wraps the manual steps into a helper function -- would turn this from a 5 to an 8. Diffusion models are the second-largest inference workload after LLMs in 2026. Not having native support is a significant gap for a tool claiming to cover "22 walls." The calibration doc should include at least one diffusion model benchmark.

---

Q7: "The TinyML section felt like an afterthought. Is it real?"

Questioner archetype: Embedded systems researcher. Noticed that the hardware zoo lists ESP32-S3 and nRF52840 but the tutorial exercises never touch them.

The question (verbatim): "Your hardware table lists microcontrollers -- ESP32, nRF52840 -- and the efficiency guide mentions TFLite Micro. But every exercise in the tutorial was about H100s and A100s. Have you actually validated the model on microcontrollers, or is it just a row in a table?"

The best honest answer:

Fair criticism. The TinyML support is real in the sense that the hardware specs are in the registry and the solver can compute Roofline-style predictions for them. An ESP32-S3 has known FLOPS (roughly 0.5 GOPS for INT8) and known SRAM (512 KB). You can ask mlsysim "does a 250 KB quantized MobileNet-v2 fit in SRAM and how long does inference take?" and get a physically grounded answer.

But you are right that it is not deeply validated. We have not run MLPerf Tiny benchmarks against our predictions for MCUs. The efficiency parameter for TinyML (eta=0.05-0.15) is estimated from general knowledge of interpreter overhead in TFLite Micro, not from systematic measurement. The tutorial does not include TinyML exercises because the ISCA audience skews toward datacenter and cloud.

What is genuinely useful for TinyML right now: the memory feasibility check. "Does this model fit on this MCU?" is a binary question that mlsysim answers correctly because it is just arithmetic -- model size vs SRAM capacity. That is actually the most common question in TinyML deployment, and getting it wrong wastes weeks of porting effort.

What needs work: per-operator latency modeling for MCUs (where there are no tensor cores and each operator type has wildly different efficiency), flash vs SRAM partitioning (MCUs often execute from flash, which is 10x slower), and DMA/interrupt overhead modeling. These are real TinyML constraints that our current single-parameter efficiency model does not capture.

If there is interest from the embedded community, I would love collaborators who can provide calibration data from MLPerf Tiny submissions.

Answer quality: 6/10

What would make it stronger: The memory feasibility argument is strong and honest. The weakness is the appeal for collaborators, which sounds like "we have not done the work." A concrete plan would help: "We are running MLPerf Tiny benchmarks on ESP32-S3 and nRF52840 this quarter and will publish calibration tables by v0.2." Even better: have one validated MCU benchmark in the calibration doc. One real number beats ten promises.

---

Q8: "What's your validation methodology? You calibrate eta per-benchmark."

Questioner archetype: Faculty member on a program committee. This is the question that decides whether the paper gets accepted.

The question (verbatim): "Let me make sure I understand your validation. You have six calibration points. For two of them (the CNN training cases), you set a default eta, get predictions that are 22% and 54% off, then show that per-configuration calibration brings error to 1%. For two more (LLM decode), the efficiency parameter does not even appear because the workload is memory-bound. And for the last one (GPT-3 FLOPs), it is a closed-form equation with no empirical parameter at all.

So your only genuinely predictive validation -- where eta is set in advance and the model makes a falsifiable prediction -- is... zero cases? Every case is either trivially correct (bandwidth division, closed-form FLOPs) or calibrated after the fact. How is this a validated model?"

The best honest answer:

You have identified the central methodological weakness, and I am not going to try to talk around it.

You are correct that in the current calibration document, there are zero cases where we set eta before seeing the benchmark result and then made a falsifiable prediction. The CNN cases use calibrated eta. The LLM decode cases are efficiency-insensitive. The FLOP counting is definitional.

Here is what I think the fair assessment is:

What is genuinely validated: The structural claims. The model correctly identifies that LLM decode is memory-bound (not compute-bound). It correctly identifies that the H100 speedup over A100 for decode is ~1.7x (matching the bandwidth ratio), not 3.2x (the FLOPS ratio). It correctly computes that KV-cache dominates memory at high batch sizes. These qualitative predictions are falsifiable and correct. They are also the predictions that matter most for system design -- knowing which constraint binds is more useful than knowing the exact latency.

What is NOT validated: Quantitative accuracy for compute-bound workloads with a fixed eta. The model cannot currently say "ResNet-50 on H100 will achieve X images/second" without a calibrated eta, and the calibrated eta does not transfer across hardware generations.

What we need for a strong validation: A held-out test. The methodology would be: calibrate eta on workload A (say, Llama-3-8B training on H100), then predict workload B (Llama-3-70B training on H100) using the same eta. If the model predicts within 20%, that is meaningful. If it is off by 2x, that tells us eta is workload-specific, not just hardware-specific. We have not run this experiment yet. We should, and we will before submitting the paper.

The broader intellectual claim is not "we can predict exact performance." It is "we can systematically identify binding constraints across 22 walls using a common analytical framework." That claim is validated by the structural results. But I acknowledge that the quantitative accuracy claim is currently undersupported.

Answer quality: 9/10

What would make it stronger: This is the right answer for an academic audience. The only improvement is having the held-out experiment done before the tutorial. Running Llama-3-8B and Llama-3-70B on the same hardware with a shared eta, then reporting the cross-workload transfer error, would cost roughly $50 in cloud compute and would either validate or refute the model's utility for quantitative prediction. Not having done this before presenting at ISCA is a significant omission. The structural validation (correct bottleneck identification) is the real value proposition and should be foregrounded in the paper.

---

Part 2: Hallway Conversations

---

Conversation A: Two PhD Students Debating Whether to Use mlsysim

Setting: Coffee break, 10:35 AM, after the Roofline module.

Characters:

• Priya -- 3rd year, systems/architecture, working on communication-efficient distributed training. Uses ASTRA-sim for network simulation.
• Marcus -- 2nd year, ML/NLP, working on efficient inference for long-context LLMs. Has never used a performance simulator.

---

Marcus: That batch-size sweep exercise was actually useful. I have been fighting with KV-cache OOM for weeks and I never sat down to do the arithmetic. It took like 30 seconds in their tool.

Priya: Sure, but that is literally dividing bytes by capacity. I could do that on a napkin.

Marcus: You could, but you would not. That is the point. I have been running profilers and reading CUDA traces trying to figure out why I OOM at batch 48 on an A100. Turns out 8B parameters at FP16 is 16 GB, and at batch 48 with 4K context the KV-cache is another 48 GB. That is 64 GB on an 80 GB card. The arithmetic was always there. I just never did it.

Priya: OK, fair. But my work is on communication. The AllReduce model they showed is textbook ring AllReduce. Real systems use hierarchical AllReduce with NVLink within the node and InfiniBand across nodes. ASTRA-sim models that at the packet level. This tool gives me one number.

Marcus: Do you always need packet-level simulation though? Like, for your paper, sure. But when you are writing your NSF proposal and you need to say "we will need 128 GPUs for this experiment," do you fire up ASTRA-sim?

Priya: ...No. I use a spreadsheet.

Marcus: Right. So their tool is a better spreadsheet. That is the pitch. It is not replacing ASTRA-sim for your research. It is replacing your spreadsheet for your capacity planning.

Priya: The efficiency parameter bugs me though. Did you catch the calibration numbers? ResNet-50 gets eta=0.13 on A100 and eta=0.065 on H100. That is a 2x difference for the same workload. If I use this for my proposal and pick the wrong eta, I am off by 2x on my GPU estimate, which means I am asking NSF for twice too many or twice too few GPUs.

Marcus: Yeah, that is a real problem. For inference it is less of an issue because decode is memory-bound and eta drops out. But for training... I do not know. You would have to calibrate it yourself.

Priya: Which means I need the hardware already. Chicken and egg.

Marcus: I think the move is: use it for feasibility ("does the model fit? am I compute-bound or memory-bound?") and ranking ("is MI300X better than H100 for my workload?"). Do not use it for exact throughput prediction unless you have a calibrated eta.

Priya: That is a narrow use case.

Marcus: For you, yes. For me? Every question I have right now is a feasibility or ranking question. "Can I serve 70B at batch 64 on one H100?" "Should I quantize to INT4 or use tensor parallelism?" "Is it worth trying MI300X for its 192 GB HBM?" Those are the questions keeping me up at night, and this tool answers all of them in under a second.

Priya: Fine. I will install it. But if my proposal gets rejected because the GPU count was wrong, I am blaming you.

Marcus: [laughs] Blame the efficiency parameter. At least you will know which wall to hit.

Verdict: Marcus will use mlsysim regularly. Priya will install it, use it twice for capacity estimates in grant proposals, and go back to ASTRA-sim for her actual research. This is the correct adoption pattern -- the tool serves Marcus's needs well and Priya's needs partially.

---

Conversation B: The AMD Engineer and the Intel Engineer Comparing Notes

Setting: Lunch, standing near the "Wall of Walls" sticky-note board.

Characters:

• Rajan -- Senior performance architect at AMD, works on MI300X benchmarking and competitive analysis. Noticed mlsysim includes MI250X and MI300X.
• Katharina -- Software engineer at Intel, works on Gaudi accelerator software stack. Noticed mlsysim includes Gaudi 2 and Gaudi 3.

---

Rajan: Did you see they have MI300X in there? I checked their bandwidth number -- 5.3 TB/s. That matches our datasheet.

Katharina: They have Gaudi 2 and Gaudi 3 as well. The FLOPS numbers look correct. I am less sure about the memory bandwidth -- Gaudi's memory subsystem is different from what you would infer from a single bandwidth number.

Rajan: That is my concern too. The roofline model assumes one bandwidth number and one compute number. MI300X has an interesting memory hierarchy -- the HBM3 bandwidth is 5.3 TB/s, but the Infinity Fabric between the chiplets has its own bandwidth characteristics. For workloads that fit in one chiplet's local HBM, you get the full bandwidth. For workloads that span chiplets, you get less.

Katharina: Same with Gaudi. The on-die SRAM is a critical tier that the roofline misses entirely. Gaudi's differentiation is the large SRAM that keeps activations close to compute. A single bandwidth number averaging across the memory hierarchy undersells us.

Rajan: So both of us have the same complaint: the model is too simple for our hardware's memory hierarchy.

Katharina: Yes. But I will say this -- the fact that our hardware is in there at all is progress. Most open tools are NVIDIA-only. When a customer asks "should I use H100 or MI300X or Gaudi 3," the default answer is "run MLPerf." If this tool gives a directionally correct ranking with sub-second latency, that is useful even if the absolute numbers are off.

Rajan: Directionally correct is the key phrase. Let me check something. [pulls out laptop] OK, I ran Llama-3-70B inference on MI300X at batch 1. Their model says ITL = 2.3 ms. Our internal benchmarks with ROCm show 3-5 ms depending on the framework. So they are within 2x, on the optimistic side.

Katharina: Optimistic is dangerous for competitive analysis. If a customer uses this tool and it says MI300X is faster than Gaudi 3 for their workload, but the prediction is optimistic for MI300X and pessimistic for Gaudi, we lose a sale based on a modeling artifact.

Rajan: Or vice versa. The bias matters.

Katharina: I think the play for both of us is to contribute calibrated efficiency values. If they have a TraceableConstant system where every number has a source, we can submit official numbers from our benchmark teams. Then at least the hardware specs are accurate and traceable to us.

Rajan: That is smart. Control the inputs, and the model works in our favor. Or at least does not work against us.

Katharina: The question is whether they accept vendor-submitted numbers. There is an obvious conflict of interest.

Rajan: The SPEC benchmark organization handles this with disclosure rules. You can submit your own numbers, but the methodology must be public and reproducible. If mlsysim adopted something similar...

Katharina: That would actually be useful for the industry. A neutral analytical framework with vendor-contributed, auditable hardware specs. Like a TPC for ML hardware.

Rajan: That is a much bigger project than what they showed today.

Katharina: Agreed. But the infrastructure -- unit-safe constants with provenance tracking -- is the right foundation. The question is whether they execute on it.

Rajan: Let us talk to them after the afternoon session. I want to understand the contribution model.

Verdict: Both engineers see potential value but have legitimate concerns about accuracy bias across vendors. The most likely outcome: one or both vendors contribute hardware specs to the project within 6 months, but only if the contribution model is clear and the numbers are auditable. The "neutral analytical framework" vision is compelling but requires governance that does not exist yet.

---

Conversation C: The Faculty Member and the Startup CTO

Setting: 4:50 PM, packing up after the closing. They were seated near each other during the capstone.

Characters:

• Professor Chen -- Teaches ML Systems at a large state university. Has 150 students per semester. Currently uses ad-hoc Jupyter notebooks for homework assignments.
• Diego -- CTO of a 40-person startup doing LLM-based document processing. Running inference on a mix of A100s and H100s in AWS. Attended the tutorial because he is about to make a $2M hardware purchasing decision.

---

Professor Chen: What did you think of the capstone?

Diego: The capstone was the best part. It was the first time today where all the pieces came together. The individual exercises were useful but felt like textbook problems. The capstone felt like my actual job.

Professor Chen: That is exactly why I am here. I want to redesign my ML Systems course around something like this. Right now my students do profiling labs with PyTorch and CUDA, but they never step back and think about the system as a whole. They optimize one kernel and think they have solved the problem.

Diego: So you would use mlsysim as a teaching tool?

Professor Chen: As the backbone of the course, potentially. The 22 walls taxonomy is a natural syllabus. Week 1: Compute Wall. Week 2: Memory Wall. Week 3: Software Wall. And so on. Each week, students use the tool to explore one wall, then do a real profiling lab to validate the model.

Diego: The "validate the model" part is key. If students just trust the analytical model, they learn the wrong lesson. The model is useful precisely because it is wrong in interesting ways. The gap between the model and reality IS the systems engineering.

Professor Chen: Exactly. I would have them measure eta themselves. "Run ResNet-50 on the department A100. Measure throughput. Back-calculate eta. Now predict what happens on H100. Next week, you will get H100 time and check your prediction."

Diego: That is a great assignment. Wish I had taken that class. What is your main concern?

Professor Chen: Maturity. I cannot build my course around a v0.1.0 tool that might break or change APIs between semesters. I have 150 students and 2 TAs. If Engine.solve() changes its signature, that is 150 broken notebooks and a week of debugging instead of teaching.

Diego: That is a real risk. What about for my use case? I need to decide between renewing our A100 instances or migrating to H100 or Trainium2. The cost difference over 3 years is about $1.5M.

Professor Chen: Did the tool help you with that?

Diego: Partially. The feasibility analysis is immediate -- I now know that our 70B model in FP16 does not fit on one H100, which means tensor parallelism or quantization regardless of the platform. That alone saved me from a mistake. I was about to price out single-GPU instances.

Professor Chen: And the cost modeling?

Diego: Rougher. The TCO calculator gives me directional answers, but I need exact numbers for a board presentation. I still have to get actual quotes from AWS and run actual benchmarks. But here is the thing -- now I know which benchmarks to run. Before today, I would have benchmarked every model on every instance type at every batch size. That is a $50K benchmarking bill. Now I know to benchmark only the three configurations that the model says are in the right ballpark. That probably saves me $40K.

Professor Chen: That is the real value proposition. Not "replaces benchmarking" but "focuses benchmarking."

Diego: Right. And for the carbon constraint -- my board just added a sustainability requirement. The geography exercise was an eye-opener. I had no idea the grid intensity variation was that large. I am going to move our training jobs to Quebec. That is a free win.

Professor Chen: You and every other company that does that exercise. Watch Quebec's grid get overloaded in two years.

Diego: [laughs] The tragedy of the commons, modeled analytically.

Professor Chen: Let me ask you something as an industry person. Is the 22-wall framework actually useful for practitioners, or is it an academic taxonomy?

Diego: Some of the 22 walls matter a lot. Compute, Memory, Communication, Capital -- those are my daily constraints. Sustainability is becoming one. Serving, Batching -- absolutely. But "Reasoning Wall" (inference-time compute), "Sensitivity Wall" (partial derivatives) -- those feel like textbook walls, not engineering walls. I would never say to my team "we are hitting the Sensitivity Wall."

Professor Chen: That is useful feedback. The framework might benefit from a "practitioner's top 10" versus the full 22.

Diego: Or just clear tiers. Tier 1: walls that determine your architecture (Compute, Memory, Communication, Capital). Tier 2: walls that affect your optimization (Software, Serving, Batching, Compression). Tier 3: walls that matter for specific contexts (Carbon, Safety, TinyML, Multi-tenant). The full 22 is great for a course. For a startup CTO, I need the top 8.

Professor Chen: I am going to steal that for my syllabus. Tier 1 in the first half of the semester, Tier 2 in the second half, Tier 3 as optional projects.

Diego: Send me the syllabus. I will send you interns.

Professor Chen: Deal.

Verdict: Professor Chen will adopt mlsysim for teaching if and only if the API stabilizes by fall 2026 and there are pre-built assignment notebooks. Diego will use it this week to narrow his benchmarking from 20 configurations to 3, potentially saving $40K. He will not use it for final purchasing decisions -- those still require real benchmarks and real quotes. Both see the 22-wall taxonomy as pedagogically strong but operationally over-broad, and both independently converge on the idea that a tiered subset would increase practical adoption.

---

Summary: Strength and Weakness Assessment

Where the tutorial is strong

Strength	Evidence
The "aha moments" work	The H100-is-only-1.7x-faster exercise consistently surprises even experienced engineers. The KV-cache OOM exercise solves a real problem people have.
Honest positioning	The slides explicitly state "2-5x accuracy" and "not a replacement for benchmarking." This disarms skeptics.
The CPI analogy is intellectually sound	It is the correct framing and maps to something the ISCA audience already understands.
Multi-vendor coverage is a real differentiator	No other open analytical tool covers NVIDIA, AMD, Intel, Google, and Cerebras.
The capstone design challenge works	It synthesizes all the concepts and feels like a real engineering problem.

Where the tutorial is vulnerable

Vulnerability	Severity	Mitigation needed
Zero held-out validation experiments	Critical	Run cross-workload eta transfer experiments before ISCA. This is a $50 experiment that determines paper acceptance.
MoE models not supported	High	At minimum, provide a documented workaround notebook. Better: native MoEModel class.
Diffusion models not supported	High	Add one diffusion benchmark to the calibration table.
TinyML claims exceed evidence	Medium	Run one MLPerf Tiny benchmark against predictions.
Eta does not transfer across GPU generations	Medium	Document this as a known limitation prominently, not buried in the calibration doc. Frame it as a research contribution opportunity.
22 walls claim vs actual solver coverage	Medium	Audit which walls are actually wired into Pipeline.solve() vs which are standalone calculations. Be precise in slides.
API stability for course adoption	Medium	Commit to a stable v1.0 API freeze by a specific date.
No governance model for vendor-contributed specs	Low	Define a contribution policy before AMD and Intel contribute numbers.

The adoption decision matrix

Persona	Will adopt?	Why / Why not
PhD student (systems)	Maybe	Good for capacity planning in proposals. Not deep enough for research.
PhD student (ML)	Yes	Solves real "why is my training slow" questions immediately.
Industry engineer (NVIDIA)	No	Has internal tools that are better.
Industry engineer (AMD/Intel)	Interested	Wants to contribute specs to ensure fair competitive comparison.
Faculty (ML Systems course)	Yes, if API stabilizes	The 22-wall taxonomy is a ready-made syllabus.
Startup CTO	Yes, for scoping	"Focuses benchmarking" is the real value. Saves $40K on unnecessary benchmarks.
Startup CTO (final decision)	No	Final purchasing decisions require real benchmarks and real quotes.