EduArtha Interactive Books
Understanding AI: From Zero to nanoCodeGPT
An interactive journey building a multi-language coding AI from scratch — featuring real discussions, deep math, and hard-won debugging stories.
Based on Andrej Karpathy's nanoGPT • Built through 4 real conversation threads
The Vision & Architecture
What is nanoGPT? It is a minimal implementation of GPT-2 by Andrej Karpathy, consisting of roughly 300 lines of model code and 300 lines of training code. But can we extend it to generate code in Python, Java, C, and C++?
Yes. To build nanoCodeGPT, we applied key architectural modifications:
- Language Tags: Prepend
<|python|>to teach syntax separation. - RoPE: Rotary Positional Embeddings, enabling better length extrapolation.
- FIM: Fill-in-the-Middle tokens for code completion.
LayerNorm → Multi-Head Attention
LayerNorm → MLP (Feed Forward)
Yes! We built nanoCodeGPT with RoPE, Language Tags, and FIM (Fill-In-the-Middle) support.
The CausalSelfAttention block — the absolute heart. Everything else supports Attention.
The Heart of AI (Self-Attention)
The core of any Transformer is the Self-Attention mechanism. It is fundamentally a mathematical transformation using Queries (Q), Keys (K), and Values (V).
Think of it as a cocktail party: a word creates a Query ("I need a noun"), surrounding words broadcast a Key ("I am a noun"), and when they match, the Value (the underlying meaning) is transferred.
Python# The core Attention Formula: Softmax( (Q * K^T) / sqrt(d) ) * V
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
Why is_causal=True?
Without it, the model cheats by looking at future words. The Causal Mask mathematically applies negative infinity (-inf) to future connections, forcing the model to guess the next token!
The 2017 paper proved that attention alone (without recurrence or convolution) is sufficient for state-of-the-art NLP performance.