cs249r_book/tinytorch/tests/cli/README.md

CLI Test Suite

Comprehensive test suite for the TinyTorch CLI (tito). Ensures all commands are properly registered, documented, and executable.

Test Files

1. test_cli_registry.py - Command Registration Tests

Validates the command registry in tito/main.py:

• TestCLIRegistry: Core registry validation

  • All commands inherit from BaseCommand
  • All commands have descriptions (>10 chars)
  • All commands implement execute() and add_arguments()
  • Parser creation succeeds
  • No duplicate command names
  • Help text accessible for all commands

• TestCommandFiles: File system consistency

  • All registered commands have corresponding files
  • No orphaned command files (files without registration)
  • Detects commands that need cleanup

• TestEpilogDocumentation: Help text consistency

  • Epilog mentions all registered command groups
  • No outdated references

2. test_cli_execution.py - Command Execution Tests

Smoke tests for actual command execution:

• TestCommandExecution: Basic execution

  • Bare tito command shows welcome screen
  • tito -h shows help
  • tito --version works
  • All 18 commands can show help (tito <cmd> -h)
  • All subcommands can show help

• TestCommandGrouping: Discoverability

  • Student-facing commands visible in welcome screen
  • Developer commands documented in help

• TestErrorMessages: Error handling

  • Invalid commands show helpful errors
  • Missing subcommands show help

3. test_cli_help_consistency.py - Help Text Quality

Ensures help text is consistent and complete:

• TestHelpConsistency: Formatting consistency

  • All command helps mention 'tito'
  • Bare tito vs tito -h serve different purposes
  • No references to removed commands

• TestWelcomeScreen: Welcome screen quality

  • Shows Quick Start section
  • Organizes commands into groups
  • Includes example commands

• TestCommandDocumentation: Documentation completeness

  • All registered commands documented
  • Checkpoint command specifically validated

Running the Tests

Run all CLI tests:

pytest tests/cli/ -v

Run specific test file:

pytest tests/cli/test_cli_registry.py -v

Run with output:

pytest tests/cli/ -v -s

What These Tests Catch

✅ Prevents

1. Orphaned Commands: Command files that aren't registered
2. Missing Documentation: Commands without descriptions
3. Broken Help: Commands that crash when showing help
4. Inconsistent UX: Different help formats across commands
5. Stale References: Help text mentioning removed commands

✅ Validates

1. Registration: All commands in TinyTorchCLI.commands dict
2. Implementation: All commands inherit from BaseCommand
3. Documentation: All commands have clear descriptions
4. Execution: All commands can run without crashing
5. Discoverability: Key commands visible to users

Test Coverage

52 tests covering:

• 18 registered commands
• 8 subcommand groups
• Registry validation
• Execution smoke tests
• Help text consistency
• Welcome screen UX

How Modern CLIs Handle Testing

These tests follow industry best practices from tools like:

1. Click (Python): Command registration validation
2. Git: Comprehensive help text testing
3. Docker: Smoke tests for each command
4. kubectl: Subcommand hierarchy validation

Key patterns used:

• Registry-based validation: Single source of truth in code
• Smoke tests: Don't test functionality, just "does it run?"
• Help text parsing: Ensure documentation stays current
• Snapshot testing: Compare outputs (could be added later)

Maintenance

When adding a new command:

1. Add to TinyTorchCLI.commands dict in tito/main.py
2. Create command file in tito/commands/
3. Add to epilog if it's a major command group
4. Tests will automatically validate it!

When removing a command:

1. Remove from TinyTorchCLI.commands dict
2. Delete command file OR add to known_internal in tests
3. Update epilog/welcome screen

Future Enhancements

Consider adding:

• Snapshot tests: Save known-good help output, compare changes
• Integration tests: Test actual command workflows
• Performance tests: Ensure CLI startup is fast
• Completion tests: Validate shell completion scripts
• Config tests: Test config file parsing and validation