Tren⚡Torch

Most people learn ML frameworks by importing them. We built ours by hand, then made it hit harder.

Quick Start · The Vision · 20 Modules · Milestones

[!NOTE]

This is our implementation of TinyTorch (Harvard CS249r), rebuilt in our own style and pushed further. Same bones, more muscle.

Quick Start

# macOS / Linux
git clone https://github.com/TrenTorch/TrenTorch.git
cd TrenTorch/TrenTorch_CLI
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -e .
tren setup
tren

# Windows (PowerShell)
git clone https://github.com/TrenTorch/TrenTorch.git
cd TrenTorch\TrenTorch_CLI
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt
pip install -e .
tren setup
tren

tren setup will ask to add tren to your PATH: say yes, and every terminal after that just needs tren, from any directory, with no activating first.

TrenTorch-Web

The same from-scratch, module-by-module curriculum, running in the browser: no local install, no tren setup, sign in and start solving. Lives at trentorch.com, built with SvelteKit.

cd TrenTorch_Web
npm install
npm run dev -- --open

npm run check   # svelte-check (types)
npm run lint     # prettier + eslint
npm run test     # vitest
npm run build    # production build

Why TrenTorch?

Be the best engineer yourself !

We wanted to know. So we built one. Then we didn't stop at "good enough."

The Bricks 🧱

TinyTorch teaches the fundamentals. TrenTorch takes those same bricks and adds the reps: cleaner internals, sharper performance instincts, and an implementation pushed past the original spec wherever we saw the chance.

Small enough to read in one sitting - every op traceable back to raw NumPy
Big enough to actually flex - the real architecture real frameworks run on
Ours - rebuilt, refactored, and hardened in our own hands

No black boxes. No import torch. Just the machinery, exposed.

What You'll Build

A complete ML framework, built from zero. No single finish line, a set of missions you clear on the way there:

🎯 Mission: Image - we teach Image

Real computer vision on standard benchmarks
Conv2d, pooling, and CNNs, built entirely from scratch on NumPy
Performance that holds its own against the frameworks it's built to demystify

🎯 Mission: NLP - we teach NLP

Tokenization, embeddings, and multi-head attention, hand-rolled
The groundwork every language model stands on

🎯 Mission: LLM - we teach LLM

Full GPT-style transformer blocks, not a wrapper around someone else's
Real self-attention, real language generation

🎯 Mission: Inference - we teach Inference

Profiling, quantization, and acceleration, so your model doesn't just train, it runs
KV-cache and memoization for the speed that production demands

🎯 Mission: LLMOps - we teach LLMOps

Modern optimizers with learning rate scheduling: SGD, Adam, AdamW, Lion, Muon
Competitive benchmarking and the capstone that ties it all together

Zero PyTorch. Zero TensorFlow. Every line is ours.

Current Status

Ready	In Progress	Coming Soon
✅ All 20 modules implemented	🔧 Documentation polish	📅 Community leaderboard
✅ Module, CLI, integration, and milestone tests	🔧 Edge case hardening	📅 More milestone exercises
✅ `tren` CLI for workflows	🔧 Performance tuning passes	📅 More milestones beyond MLPerf
✅ Historical milestone scripts

Want to explore the code? Browse the repository structure.

Adventurous? Local installation works, but bring a spotter. See the setup notes on the wiki's Getting Started page.

🏗 20 Progressive Modules

Build your framework through four progressive parts:

Part	Modules	What You Build
I. Foundations	01-08	Tensors, activations, layers, losses, dataloader, autograd, optimizers, training
II. Vision	09	Conv2d, CNNs for image classification
III. Language	10-13	Tokenization, embeddings, attention, transformers
IV. Optimization	14-20	Profiling, quantization, compression, acceleration, memoization, benchmarking, capstone

Each module asks one question: "Can I build this from scratch, and can I build it well?"

🏆 Historical Milestones

As you progress, you unlock recreations of landmark ML achievements, run on YOUR framework:

Year	Milestone	Your Achievement
1958	Perceptron	Binary classification with gradient descent
1969	XOR Crisis	Multi-layer networks solve non-linear problems
1986	Backpropagation	Multi-layer network training
1998	CNN Revolution	Image classification with convolutions
2017	Transformer Era	Language generation with self-attention
2018+	MLPerf	Production-ready optimization

Not toy demos. Historically significant ML achievements, rebuilt with a framework we wrote ourselves.

Learning Philosophy

# Most courses:
import torch
model.fit(X, y)  # magic happens somewhere else
TrenTorch:
You implement every component
You measure memory usage
You optimize performance
You own every layer of the stack

Why build your own framework?

Deep understanding - know exactly what loss.backward() does, because you wrote it
Systems thinking - memory, compute, and scaling stop being abstractions
Debugging at any depth - fix problems at the model level or the tensor level
Production instincts - the same patterns real ML systems run on

Repository Structure

TrenTorch/
├── TrenTorch_CLI/               # The tren CLI and framework curriculum
│   ├── data/
│   │   ├── src/                     # 💻 Curriculum source (edit here)
│   │   │   ├── 01_tensor/               # Module 01: Tensor operations from scratch
│   │   │   ├── 02_activations/          # Module 02: ReLU, Softmax activations
│   │   │   ├── 03_layers/               # Module 03: Linear layers, Module system
│   │   │   ├── 04_losses/               # Module 04: MSE, CrossEntropy losses
│   │   │   ├── 05_dataloader/           # Module 05: Efficient data pipelines
│   │   │   ├── 06_autograd/             # Module 06: Automatic differentiation
│   │   │   ├── 07_optimizers/           # Module 07: SGD, Adam optimizers
│   │   │   ├── 08_training/             # Module 08: Complete training loops
│   │   │   ├── 09_convolutions/         # Module 09: Conv2d, MaxPool2d, CNNs
│   │   │   ├── 10_tokenization/         # Module 10: Text processing
│   │   │   ├── 11_embeddings/           # Module 11: Token & positional embeddings
│   │   │   ├── 12_attention/            # Module 12: Multi-head attention
│   │   │   ├── 13_transformers/         # Module 13: Complete transformer blocks
│   │   │   ├── 14_profiling/            # Module 14: Performance analysis
│   │   │   ├── 15_quantization/         # Module 15: Model compression (precision reduction)
│   │   │   ├── 16_compression/          # Module 16: Pruning & distillation
│   │   │   ├── 17_acceleration/         # Module 17: Hardware optimization
│   │   │   ├── 18_memoization/          # Module 18: KV-cache/memoization
│   │   │   ├── 19_benchmarking/         # Module 19: Performance measurement
│   │   │   └── 20_capstone/             # Module 20: Complete ML systems
│   │   │
│   │   ├── modules/                 # 📓 Generated notebooks (learn here, stub-only)
│   │   │   └── ...                      # (20 module directories)
│   │   ├── solutions/                # 🔒 Reference implementations (maintainer/CI-only)
│   │   ├── datasets/                 # 🗂️ Curated training data (tinydigits, tinytalks)
│   │   ├── milestones/                # 🏆 Historical ML evolution - prove what you built
│   │   │   ├── 01_1958_perceptron/
│   │   │   ├── 02_1969_xor/
│   │   │   ├── 03_1986_mlp/
│   │   │   ├── 04_1998_cnn/
│   │   │   ├── 05_2017_transformer/
│   │   │   └── 06_2018_mlperf/
│   │   │
│   │   └── trentorch/                # 📦 Generated package (import from here)
│   │       ├── core/                     # Core ML components
│   │       └── ...                       # The framework you built
│   │
│   ├── platforms/
│   │   ├── cli/                     # 🎛️ The tren CLI itself
│   │   │   ├── main.py                  # Entry point
│   │   │   ├── core/                     # Shared plumbing: config, console, theme, runtime
│   │   │   ├── commands/                  # Genuinely shared code only: base.py, export_utils.py, jupyter.py
│   │   │   ├── cli_platform/              # The CLI's own bootstrap: setup, system, package, dev tooling
│   │   │   ├── processes/                  # The student-facing workflow: module_workflow, milestone, benchmark, olympics, convert
│   │   │   └── tests/                       # The CLI's own test suite
│   │   └── dev_tools/                # Maintainer scripts (release, fresh-install verification)
│   │
│   ├── user_data/                  # 🗃️ Your own progress, milestones, benchmarks (not committed)
│   │
│   └── tests/                      # ✅ Integration, e2e, and environment tests
│
├── TrenTorch_Web/                # The browser-based version (SvelteKit, trentorch.com)
│   ├── platform/                     # Routes, components, assets
│   ├── processes/                    # IDE content, POTD, auth, progress tracking
│   ├── data/app_data/                # Curriculum content compiled for the browser
│   └── supabase/migrations/          # Auth + user-data schema and RLS policies
│
└── .github/                     # CI/CD shared across both -- see below

CLI key workflow: TrenTorch_CLI/data/src/.py → TrenTorch_CLI/data/modules/.ipynb (you solve it) → TrenTorch_CLI/data/trentorch/*.py

Credit Where It's Due

TrenTorch is our implementation, built on the curriculum and foundation of TinyTorch, created by Prof. Vijay Janapa Reddi and the ML Systems Book community at Harvard University. Full respect to the original project, we just wanted to take it further.

Related educational frameworks worth knowing about:

tinygrad - George Hotz's minimalist framework
micrograd - Andrej Karpathy's tiny autograd
MiniTorch - Cornell's educational framework

We're addicted to making great software that runs (we're a bit of perfectionists ourselves) and is useful to humanity. This is our shot at making TinyTorch, but adding more stuff that is currently experimental and quite famous.

Team Engineers

Recomputed nightly from real issue/PR activity via .github/workflows/update-contributors.yml. Want to show up here? Open an issue, or get a PR merged: the first-contribution bot will say hello on your first PR, and this grid picks you up on the next nightly run after it merges. A closed-without-merging PR doesn't count.

maanas1234 _Maintainer _{Catches bugs, builds solutions and ships products} _{Issues: 11 · PRs: 12}	Aadityansha _Maintainer _{Reducing CPU stalls, one commit at a time.} _{Issues: 0 · PRs: 3}	Rocky _{Principal Maintainer} _{IIT Guwahati, Debugs autograd for fun, ships before sunrise.} _{Issues: 12 · PRs: 144}	Shivtej Gaikwad _Maintainer _{IIT Guwahati. Shows up, ships, moves on to the next thing.} _{Issues: 0 · PRs: 4}
JashT14 _{Spots bugs, corrects them and contributes} _{Issues: 4 · PRs: 3}	MahekPatel-2403 _{Spots bugs, corrects them and contributes} _{Issues: 0 · PRs: 1}	pushkarkumarvats _{Spots bugs, corrects them and contributes} _{Issues: 0 · PRs: 1}