LLM architecture from scratch

Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.

Implemented

| Model | Model Size | Modality | Normalization | Positional Encoding | Attention | Mixture of Experts | | --- | --- | --- | --- | --- | --- | --- | | GPT-2 XL | 1.5B | Text | LayerNorm | Learned Absolute | Multi-Head Self-Attention (causal) | ❌ | | Llama3-8B | 8B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ | | Llama 3.2-3B | 3B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ | | OlMo 2 7B | 7B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ | | Phi-4 | 14B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ | | DeepSeek v3 | 671B | Text | RMSNorm | RoPE | Multi-head Latent Attention | ✅ | | Gemma 3 | 27B | Text + Image | RMSNorm | RoPE | Sliding window + GQA | ❌ | | xLSTM | 7B | Text | RMSNorm | RoPE | mLSTM | ❌ | | Mistral 3.1 Small | 24B | Text + Image | RMSNorm | RoPE | Sliding window + GQA | ❌ | | Llama 4 Maverick | 400B | Text | RMSNorm | RoPE + NoPE | GQA | ✅ | | Qwen3 | 32B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ | | Qwen3 (235B-A22B) | 235B A22B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ✅ | | SmolLM3 | 3B | Text | RMSNorm | RoPE + NoPE | Grouped-Query Attention (causal) | ❌ | | Kimi K2 | 1T | Text | RMSNorm | RoPE | Multi-head Latent Attention | ✅ | | GPT OSS | 120B | Text | RMSNorm | YaRN RoPE | Grouped-Query Attention (causal) | ✅ |

In progress

• GLM 4.5 (355B)
• Grok 2.5 (270B)
• Qwen3 Next (80B-A3B)