TinyTorch/docs/instructor/ta-guide.md

Teaching Assistant Guide for TinyTorch

Complete guide for TAs supporting TinyTorch courses, covering common student errors, debugging strategies, and effective support techniques.

🎯 TA Preparation

Critical Modules for Deep Familiarity

TAs should develop deep familiarity with modules where students commonly struggle:

1. Module 05: Autograd - Most conceptually challenging
2. Module 09: CNNs (Spatial) - Complex nested loops and memory patterns
3. Module 13: Transformers - Attention mechanisms and scaling

Preparation Process

1. Complete modules yourself - Implement all three critical modules
2. Introduce bugs intentionally - Understand common error patterns
3. Practice debugging - Work through error scenarios
4. Review student submissions - Familiarize yourself with common mistakes

🐛 Common Student Errors

Module 05: Autograd

Error 1: Gradient Shape Mismatches

Symptom: ValueError: shapes don't match for gradient Common Cause: Incorrect gradient accumulation or shape handling Debugging Strategy:

• Check gradient shapes match parameter shapes
• Verify gradient accumulation logic
• Look for broadcasting issues

Example:

# Wrong: Gradient shape mismatch
param.grad = grad  # grad might be wrong shape

# Right: Ensure shapes match
assert grad.shape == param.shape
param.grad = grad

Error 2: Disconnected Computational Graph

Symptom: Gradients are None or zero Common Cause: Operations not tracked in computational graph Debugging Strategy:

• Verify requires_grad=True on input tensors
• Check that operations create new Tensor objects
• Ensure backward() is called on leaf nodes

Example:

# Wrong: Graph disconnected
x = Tensor([1, 2, 3])  # requires_grad=False by default
y = x * 2
y.backward()  # No gradients!

# Right: Enable gradient tracking
x = Tensor([1, 2, 3], requires_grad=True)
y = x * 2
y.backward()  # Gradients flow correctly

Error 3: Broadcasting Failures

Symptom: Shape errors during backward pass Common Cause: Incorrect handling of broadcasted operations Debugging Strategy:

• Understand NumPy broadcasting rules
• Check gradient accumulation for broadcasted dimensions
• Verify gradient shapes match original tensor shapes

Module 09: CNNs (Spatial)

Error 1: Index Out of Bounds

Symptom: IndexError in convolution loops Common Cause: Incorrect padding or stride calculations Debugging Strategy:

• Verify output shape calculations
• Check padding logic
• Test with small examples first

Error 2: Memory Issues

Symptom: Out of memory errors Common Cause: Creating unnecessary intermediate arrays Debugging Strategy:

• Profile memory usage
• Look for unnecessary copies
• Optimize loop structure

Module 13: Transformers

Error 1: Attention Scaling Issues

Symptom: Attention weights don't sum to 1 Common Cause: Missing softmax or incorrect scaling Debugging Strategy:

• Verify softmax is applied
• Check scaling factor (1/sqrt(d_k))
• Test attention weights sum to 1

Error 2: Positional Encoding Errors

Symptom: Model doesn't learn positional information Common Cause: Incorrect positional encoding implementation Debugging Strategy:

• Verify sinusoidal patterns
• Check encoding is added correctly
• Test with simple sequences

🔧 Debugging Strategies

Structured Debugging Questions

When students ask for help, guide them with questions rather than giving answers:

1. What error message are you seeing?

   • Read the full traceback
   • Identify the specific line causing the error

2. What did you expect to happen?

   • Clarify their mental model
   • Identify misconceptions

3. What actually happened?

   • Compare expected vs actual
   • Look for patterns

4. What have you tried?

   • Avoid repeating failed approaches
   • Build on their attempts

5. Can you test with a simpler case?

   • Reduce complexity
   • Isolate the problem

Productive vs Unproductive Struggle

Productive Struggle (encourage):

• Trying different approaches
• Making incremental progress
• Understanding error messages
• Passing additional tests over time

Unproductive Frustration (intervene):

• Repeated identical errors
• Random code changes
• Unable to articulate the problem
• No progress after 30+ minutes

When to Provide Scaffolding

Offer scaffolding modules when students reach unproductive frustration:

• Before Autograd: Numerical gradient checking module
• Before Tensor Autograd: Scalar autograd module
• Before CNNs: Simple 1D convolution exercises

📊 Office Hour Patterns

Expected Demand Spikes

Module 05 (Autograd): Highest demand

• Schedule additional TA capacity
• Pre-record debugging walkthroughs
• Create FAQ document

Module 09 (CNNs): High demand

• Focus on memory profiling
• Loop optimization strategies
• Padding/stride calculations

Module 13 (Transformers): Moderate-high demand

• Attention mechanism debugging
• Positional encoding issues
• Scaling problems

Support Channels

1. Synchronous: Office hours, lab sessions
2. Asynchronous: Discussion forums, email
3. Self-service: Common errors documentation, FAQ

🎓 Grading Support

Manual Review Focus Areas

While NBGrader automates 70-80% of assessment, focus manual review on:

1. Code Clarity and Design Choices

   • Is code readable?
   • Are design decisions justified?
   • Is the implementation clean?

2. Edge Case Handling

   • Does code handle edge cases?
   • Are there appropriate checks?
   • Is error handling present?

3. Computational Complexity Analysis

   • Do students understand complexity?
   • Can they analyze their code?
   • Do they recognize bottlenecks?

4. Memory Profiling Insights

   • Do students understand memory usage?
   • Can they identify memory issues?
   • Do they optimize appropriately?

Grading Rubrics

See INSTRUCTOR.md for detailed grading rubrics for:

• ML Systems Thinking questions
• Code quality assessment
• Systems analysis evaluation

💡 Teaching Tips

1. Encourage Exploration

• Let students try different approaches
• Support learning from mistakes
• Celebrate incremental progress

2. Connect to Production

• Reference PyTorch equivalents
• Discuss real-world debugging scenarios
• Share production war stories

3. Make Systems Visible

• Profile memory usage together
• Analyze computational complexity
• Visualize computational graphs

4. Build Confidence

• Acknowledge when students are on the right track
• Validate their understanding
• Provide encouragement during struggle

📚 Resources

• INSTRUCTOR.md: Complete instructor guide with grading rubrics
• Common Errors: This document (expanded as needed)
• Module Documentation: Each module's ABOUT.md file
• Student Forums: Community discussion areas

🔄 Continuous Improvement

Feedback Collection

• Track common errors in office hours
• Document new error patterns
• Update this guide regularly
• Share insights with instructor team

TA Training

• Regular TA meetings
• Share debugging strategies
• Review student submissions together
• Practice debugging sessions

---

Last Updated: November 2024
For Questions: See INSTRUCTOR.md or contact course instructor