Module 19 - Benchmarking #510

New Issue

GiteaMirror · 2026-03-22T15:44:50-05:00

GiteaMirror commented

2026-03-22 15:44:50 -05:00

Originally created by @ngbolin on GitHub (Feb 22, 2026).

Hello,

As I was looking at the notes on Benchmarking, the following graph seems counterintuitive to me. Since the x-axis refers to latency, shouldn't the lines connecting A, B and C have a positive slope to capture the trade-off?

Originally created by @ngbolin on GitHub (Feb 22, 2026). Hello, As I was looking at the notes on Benchmarking, the following graph seems counterintuitive to me. Since the x-axis refers to latency, shouldn't the lines connecting A, B and C have a positive slope to capture the trade-off? <img width="520" height="222" alt="Image" src="https://github.com/user-attachments/assets/eff7ee85-0b95-4789-ac19-df984153a57d" />

GiteaMirror added the area: tinytorch type: question labels 2026-03-22 15:44:50 -05:00

GiteaMirror closed this issue

2026-03-22 15:44:50 -05:00

GiteaMirror commented

2026-03-22 15:44:51 -05:00

@ngbolin commented on GitHub (Feb 22, 2026):

In addition, there are 4 benchmark tasks listed in the notes, but writeups for only 3 were provided. Furthermore, anomaly detection falls within binary classification, so it should be included in the case where 2 random variables between 0 and 1 are generated i.e. np.rand(2).

@ngbolin commented on GitHub (Feb 22, 2026): In addition, there are 4 benchmark tasks listed in the notes, but writeups for only 3 were provided. Furthermore, anomaly detection falls within binary classification, so it should be included in the case where 2 random variables between 0 and 1 are generated i.e. np.rand(2). <img width="331" height="325" alt="Image" src="https://github.com/user-attachments/assets/0fb3448c-f941-4175-89f1-a225c3ca60ec" /> <img width="853" height="293" alt="Image" src="https://github.com/user-attachments/assets/50b3466b-51d9-4453-b9c2-9356304aced3" />

GiteaMirror commented

2026-03-22 15:44:51 -05:00

@profvjreddi commented on GitHub (Feb 22, 2026):

Thanks @ngbolin for the feedback. Here’s what I changed:

Pareto diagram slope (lines 163–171)

You’re correct that with latency on the x-axis and accuracy on the y-axis, the Pareto frontier should have a positive slope. I swapped the points so C (low latency, low accuracy) is on the left and A (high latency, high accuracy) on the right.

Fourth benchmark writeup

I added the missing Image Classification task to the “Standard Benchmark Tasks” section.

Anomaly detection as binary classification

Anomaly detection is binary (anomaly/normal), so I updated it to use np.random.rand(2) like keyword spotting and visual wake words.

TinyTorch model compatibility

I fixed the benchmark harness so it passes TinyTorch Tensors instead of numpy arrays and transposes image inputs from HWC to CHW for Conv2d. All four benchmarks now run with TinyTorch models.

Honestly, these need more testing. 😓

I was originally considering importing MLPerf Tiny to run the real benchmarks instead of this simplified implementation. If you have suggestions for validation or integration with the official suite, I’d appreciate them.

Changes are in feature/tinytorch-core (commits 335bad8cd through 55007e1f6).

@all-contributors please add @ngbolin as a contributor for ✍️ Doc,Code in TinyTorch

@profvjreddi commented on GitHub (Feb 22, 2026): Thanks @ngbolin for the feedback. Here’s what I changed: 1. Pareto diagram slope (lines 163–171) You’re correct that with latency on the x-axis and accuracy on the y-axis, the Pareto frontier should have a positive slope. I swapped the points so C (low latency, low accuracy) is on the left and A (high latency, high accuracy) on the right. 2. Fourth benchmark writeup I added the missing Image Classification task to the “Standard Benchmark Tasks” section. 3. Anomaly detection as binary classification Anomaly detection is binary (anomaly/normal), so I updated it to use np.random.rand(2) like keyword spotting and visual wake words. 4. TinyTorch model compatibility I fixed the benchmark harness so it passes TinyTorch Tensors instead of numpy arrays and transposes image inputs from HWC to CHW for Conv2d. All four benchmarks now run with TinyTorch models. Honestly, these need more testing. 😓 I was originally considering importing MLPerf Tiny to run the real benchmarks instead of this simplified implementation. If you have suggestions for validation or integration with the official suite, I’d appreciate them. Changes are in feature/tinytorch-core (commits 335bad8cd through 55007e1f6). @all-contributors please add @ngbolin as a contributor for ✍️ Doc,Code in TinyTorch

GiteaMirror commented

2026-03-22 15:44:51 -05:00

@github-actions[bot] commented on GitHub (Feb 22, 2026):

I've added @ngbolin as a contributor to tinytorch! 🎉

Recognized for: doc
Project: tinytorch (explicitly mentioned in comment)
Based on: @all-contributors please add @ngbolin as a contributor for ✍️ Doc,Code in TinyTorch

The contributor list has been updated in:

tinytorch/.all-contributorsrc
tinytorch/README.md
Main README.md

We love recognizing our contributors! ❤️

@github-actions[bot] commented on GitHub (Feb 22, 2026): I've added @ngbolin as a contributor to **tinytorch**! :tada: **Recognized for:** doc **Project:** tinytorch (explicitly mentioned in comment) **Based on:** @all-contributors please add @ngbolin as a contributor for ✍️ Doc,Code in TinyTorch The contributor list has been updated in: - `tinytorch/.all-contributorsrc` - `tinytorch/README.md` - Main `README.md` We love recognizing our contributors! :heart:

GiteaMirror referenced this issue

2026-04-11 07:50:29 -05:00

[GH-ISSUE #510] Create a reading guide for learners #1464

GiteaMirror referenced this issue

2026-04-19 12:06:02 -05:00

[GH-ISSUE #510] Create a reading guide for learners #4084

GiteaMirror referenced this issue

2026-04-21 21:23:26 -05:00

[GH-ISSUE #510] Create a reading guide for learners #5426

Sign in to join this conversation.