AI

It's autocomplete: a language model guesses the next word, adds it, and guesses again. Here a real one runs in your browser, and every number about its work comes from its real weights (pictures and reference figures are labelled). Your prompt becomes numbers, falls through every layer of knobs, and lands in the token cloud, where the model's options light up and a dice roll picks one. A version for kids and classrooms · Code on GitHub.

NewWatch an AI agent work · or make it go rogue

The picture in one minute

A machine with millions of glued knobs, and a cloud of words. Open it, then watch the real thing below.
  1. Each word gets a spot in a cloudEach word is looked up and becomes its own list of numbers: its address in the token cloud. Words used alike sit close together (it comes from how a word is used, not how it sounds).On the board: tokens and the embed strips
  2. Volume knobs, set at the factory and gluedRows of knobs turn parts of the list up or down and mix them. Training set every knob; nothing you type turns them.On the board: the dials in every layer
  3. Before each row, it listens backEach word listens to the earlier words and blends in the ones that matter. The only step whose mixing depends on what you type.On the board: the pink arcs and the bars on the context tape
  4. A wall of lights, one per wordAt the end, every word in the dictionary gets a light that glows by how well the final list of numbers matches it.On the board: the unembed, then the token cloud
  5. A dice weighted by the brightest lightsUsually the brightest wins; sometimes a dimmer one does. The temperature slider sets how loaded the dice is.On the board: choose
  6. Add the word, and go againThe new word joins the line and the whole thing runs again for the next one. That is all it ever does.On the board: repeat
How close are two words in the cloud?

Compares the two words' rows in the loaded model's embedding table (1 = the same direction, 0 = unrelated). Try two / too, red / read, cat / dog.

Where the picture bends

A real layer does more than turn parts up or down: it mixes every part with every other part, and it has bends that plain knobs cannot make. Listening back is not magic: it is worked out from the same glued knobs and the words themselves. And the wall of lights is one more set of knobs: one row of numbers per word, compared with the final list of numbers.

1Choose a model

2Write the start of a sentence

3Watch it pick the next words

    Token cloud

    0.7

    This model comes with no suggested setting; 0.7 is this page's choice. At 0 the top token always wins.

    What am I looking at?

    Step 1 of 10 Type
    Memory and speed

    Step 1 of 10

    Type

    Show the math

    Parameters

    What the parameters are

    Point at a block to see what it is; click it to look closer.

    The family tree

    How these models relate, from the 2017 Transformer to Mamba

    Every model on this page does the same job: turn tokens into balls of numbers, mix information between the tokens, let each token work on its own ball, and score every possible next token. They differ in how word order gets in, how tokens mix, and a few smaller choices. Four of the designs below run on this page with their real weights; two are here because the others grew out of them or answer them.

    Transformer
    2017    		GPT-2
    2019    		FNet
    2021    		GPT-NeoX · Pythia
    2022–23    		Llama-style
    2023–25    		Mamba
    2023
    Herenot on the board (translation)DistilGPT-2not on the board (no next-token output)Pythia-14MTinyStories 15M, SmolLM2 chatMamba-130M
    Jobtranslation: an encoder reads the sentence, a decoder writes the translationpredict the next token (the decoder half only)understand text (an encoder that sees both directions); does not writepredict the next tokenpredict the next tokenpredict the next token
    Word orderfixed sine waves added to each token's rowa learned position table added to each token's rowlearned positionsrotation (RoPE) of a quarter of each head's query and keyrotation (RoPE) of whole headsnone added: order comes from reading tokens one after another
    Tokens mix byattention (8 heads)attentiona Fourier transform: no knobs at allattentionattention (SmolLM2: heads share keys and values)a fixed-size memory per layer, written and read by each token
    Each token alonefeed-forward, ReLUfeed-forward, GELUfeed-forwardfeed-forward, GELU, side by side with attentionfeed-forward, SwiGLU (gate × up)no separate block: a gate inside the memory block
    RescalingLayerNorm after each stepLayerNorm before each stepLayerNormLayerNorm beforeRMSNorm beforeRMSNorm before
    Output tablethe embedding table reusedthe embedding table reused(no next-token output)a separate tableeither: Meta's LLaMA keeps a separate one; SmolLM2 and TinyStories reuse the embedding tablethe embedding table reused

    Attention Is All You Need (2017)

    Vaswani and others at Google introduced the Transformer for machine translation: an encoder of 6 layers reads the source sentence, a decoder of 6 layers writes the translation, and both are built from attention (8 heads, 512 numbers per token) and small feed-forward blocks, with no recurrence at all. Word order came from fixed sine-wave patterns added to each token's row (they reported that learned position rows worked about as well, which is what GPT-2 later used). Each step was followed by a LayerNorm, and the embedding table was reused as the output table. It is a translation model that needs a source sentence to work from, so it does not fit this board; GPT-style models keep only the decoder half, without its link to an encoder.

    GPT-2 (2019) → DistilGPT-2

    OpenAI's GPT-2 is a stack of decoder layers trained only to predict the next token. It moved each LayerNorm to the start of its step and added one at the end, and it adds a learned position row to every token at the start. DistilGPT-2 is Hugging Face's 6-layer, 82-million-knob version, trained to imitate the smallest GPT-2 (124 million knobs) on OpenWebText, an open copy of GPT-2's training text (knowledge distillation, as for DistilBERT). On this page: the "+p" under each embedding strip, and no rotation in the layer view.

    FNet (2021): mixing without attention

    Lee-Thorp and others at Google replaced every attention block in a BERT-style encoder with a plain two-dimensional Fourier transform (across the tokens and across each token's numbers, keeping the real part), which has no knobs at all. It reached 92–97% of BERT's accuracy on the GLUE benchmark while training 80% faster on GPUs and 70% faster on TPUs at 512 tokens. FNet is an encoder: it looks both ways and fills in or classifies text, it does not predict the next word, so it cannot run on this board. Its lesson carries over: tokens have to mix, but the mixing does not have to be attention. Mamba is another answer to that.

    GPT-NeoX and Pythia (2022–23)

    EleutherAI's GPT-NeoX design rotates only the first quarter of each head's query and key (rotary embeddings, RoPE, from Su and others, 2021; rotating only a quarter follows GPT-J), computes attention and the feed-forward block side by side and adds both (also from GPT-J; each side gets its own LayerNorm, which the GPT-NeoX paper says was an oversight that made no difference), and keeps a separate output table. Pythia is a family of these models trained on the same public data in the same order, with 154 checkpoints each, so that training itself can be studied; Pythia-14M was added in November 2023 with the same data, order and checkpoints. On this page: one station per layer on the board, both changes added at once, and the real training checkpoints in the training view.

    Llama-style models (2023–25)

    Meta's LLaMA settled the recipe most open models now use: RMSNorm before each step (a simpler rescaling), rotation of whole heads, and the SwiGLU feed-forward (a gate and an up grid multiplied together). SmolLM2 (Hugging Face, released in late 2024) follows it, with grouped-query attention (several query heads share one key and value head, which shrinks the KV cache) and a tied output table; TinyStories 15M was trained by Andrej Karpathy (llama2.c) on the TinyStories dataset of simple stories (Eldan and Li, 2023).

    Mamba (2023): a memory instead of attention

    Gu and Dao's Mamba has no attention and no feed-forward block. Each layer keeps a fixed-size memory per channel; for every token it decides how much to keep and how much to replace (the step size), writes the token in, and reads it back, so a token learns about earlier ones only through that memory. Its cost grows in line with the text, not with its square, and nothing grows per token. On this page: the memory boxes on the board and the step-by-step memory update in the layer view.

    Papers

    Training

    How the knobs get set: training

    A real model learning: Pythia-14M at its training checkpoints

    EleutherAI published 154 snapshots of one training run of Pythia-14M, from step 0 (random knobs) to step 143,000. Each step read a batch of about 2 million tokens of the Pile, about 300 billion tokens in all. Load a few snapshots (28 MB each) and watch the same model, on the same prompt, go from noise to sentences: its loss on a fixed test sentence falls, and the same 24 knobs (dials) turn into place.

    Loss here is the average surprise at each next token of: “The capital of France is Paris. The sun rises in the east and sets in the west. Water is made of hydrogen and oxygen.” (lower is better; a model that guesses at random over the 50,304 tokens scores about 10.8). The continuation is the most likely next token each time. Everything comes from the real checkpoints, run in your browser.

    The same loop, tiny enough to watch every knob

    How wrong its guesses are (loss)

    What it writes with these knobs

    Point at a knob to see what it means.

    At full size

    One neuron: multiply, then add

    Scores into chances: softmax

      Embedding explorer