A fragment of text the model reads and writes — typically 3–4 characters. Phi-3's vocabulary holds 32,064 tokens, so every step picks one of those.

The 3,072-dimensional vector flowing through the model. Every layer reads from it and adds back to it — information accumulates rather than being replaced.

A 96-dim projection that decides which past tokens to look at. Phi-3 has 32 heads × 32 layers = 1,024 heads. Each specializes in a different kind of pattern.

Feed-forward network. Expands residual 3,072 → 8,192, applies SiLU gating, projects back. Most compute happens here. Thought to store the model's facts.

Saved keys and values from every past token per layer, so attention doesn't recompute them. Paged memory — grows one slot per token, per layer.

Turns raw logits into probabilities: exponentiate, then normalize to sum to 1. Peaky softmax = confident; flat = guessing.

Storing weights as 4-bit integers with fp16 scales. ~4× smaller than fp16 with negligible quality loss on Phi-3. The reason this fits on a consumer GPU.

One WebGPU kernel invocation. Phi-3 takes 292 dispatches per output token in the headless engine path; the visualizer adds ~56 more for live attention-score readback and the 8-layer logit-lens probes. Every one is visible in the 3D scene as its tensor lights up.

A peek at the model's "current guess" at every layer, not just the last. If you took the residual after layer L and ran the final lm_head on it, what token would pop out? Watching L0 → L31 shows the answer crystallize with depth — early layers guess generically, later layers lock in the real answer.

Deliberately switching off part of the model to see what it was doing — like unplugging a wire to find what stops working. Shift-click attention heads in the 3D scene (they turn amber), then hit Run ablated: the model answers your prompt twice — once normally (Baseline) and once with those heads zeroed (Ablated). If the answers differ, those heads mattered for this prompt.

Phi-3 has 1,024 heads and is very redundant, so switching off just one or two usually changes nothing — it routes around them. Identical answers are a real finding, not a bug. To see the output actually collapse, ablate the heads that matter (see Sweep) or a whole component.

The discovery tool for ablation. Instead of guessing which heads matter, a sweep runs your prompt 33 times on one layer — a baseline, then once per head with just that head switched off — and measures how much each one changed the answer. The result is the 32-cell strip, colored by impact: cyan = this head didn't matter, amber → red = it changed the answer a lot. Click a red cell to mark that head, then Run ablated on the important ones.

It's 33 full generations back-to-back, so it takes ~60 s and spins up your GPU.