ANIMATIX DEEP DIVE · COMPILATION-FIRST EXPLAINER

The Transformer Architecture, Animated

7 interactive figures
WebGPU 60 FPS vector engine
pkg-slim runtime (~1.6 MB brotli)
Zero video buffering

Every frontier language model is powered by one core machine repeated in a stack: the transformer block. Dissect the architecture step by step below. Every figure is a declarative .amx scene compiled live in your browser and rendered at 60 FPS via WebGPU.

Fig 00 // Full Architecture Blueprint · Feedback Loop

pkg-slim · 1.6MB 60 FPS WEBGPU

Architecture Blueprint · Autoregressive Feedback

The Transformer Block at a Glance

Tokens climb through repeated self-attention and feed-forward sublayers (× N). The winning token from the LM head appends back onto the input row — that feedback loop is autoregressive generation.

overview.amx ↗

Depth: 12 Blocks Coordinate Width: 768 Float32 Vocabulary: 50,257 Tokens Bypass: Add & Norm Residuals

Token Entry Discrete sub-word tokens enter along the bottom baseline.

Coordinate Injection Lookup table extrudes embedding and positional coordinates.

Attention & FFN Context mixes across words, then transforms each coordinate row.

Autoregressive Loop Predicted next token appends back to input sequence.

Fig 01 // Sub-Word Fracture & Coordinate Extrusion

pkg-slim · 1.6MB 60 FPS WEBGPU

01 · Vector Extrusion

From Text to High-Dimensional Coordinates

Neural networks compute on geometry, not strings. The input string fractures into sub-word tokens via Byte-Pair Encoding, each extruding a 768-dimensional coordinate row from a learned lookup matrix.

tokens.amx ↗

Tokenizer: Byte-Pair Encoding d_model: 768 Dimensions Embedding Table: 38.6M Dedicated Weights

Raw Character Stream Unbroken string received by the model tokenizer.

Sub-Word Fracture Text splits into statistically optimal BPE token chips.

Vector Lookup Each chip retrieves its learned float32 coordinates.

Inspect Scene Code (.amx)

// Declarative token chips & coordinate extrusion
in_row: Row, at: (520, 190), gap: 18, align: "center" {
  c1: LabeledBox, label: "The", width: 120, height: 56, text_size: 17
  c2: LabeledBox, label: "animal", width: 120, height: 56, text_size: 17
  c3: LabeledBox, label: "didn't", width: 120, height: 56, text_size: 17
  c4: LabeledBox, label: "cross", width: 120, height: 56, text_size: 17
}

Fig 02 // Harmonic Frequency Spectrum

pkg-slim · 1.6MB 60 FPS WEBGPU

02 · Temporal Fingerprints

Injecting the Arrow of Time

Self-attention is permutation-invariant — it treats tokens as an unordered bag of words. Word order is injected by adding a unique harmonic frequency fingerprint to each position.

positional.amx ↗

Period Doubling: 3 → 6 → 12 → 24 Type: Sinusoidal Harmonic Basis Operation: Element-Wise Vector Addition

Coordinate Matrix Rows represent dimensions; columns represent sequence positions 0–7.

Frequency Doubling Lower rows halve frequency, transitioning from rapid oscillation to smooth gradients.

Fingerprint Extraction Cursor isolates column 3 composite vector added to token 3.

Fig 03 // Scaled Dot-Product Attention Spotlight

pkg-slim · 1.6MB 60 FPS WEBGPU

03 · Contextual Routing

How Words Look Back: Self-Attention

Token "it" projects a Query probing preceding Keys. Dot-product affinities are scaled by 1/√d_k and passed through Softmax to form attention weights.

attention.amx ↗

Formula: softmax(QKᵀ / √d_k) Key Width (d_k): 64 Float32 Focal Winner: "animal" (0.62)

Query Emitted Token "it" casts a search vector across all prior tokens.

Dot-Product Scoring Key vectors compute relevance affinities across the sentence.

Softmax Normalization Raw scores normalize into probability weights summing to 1.0.

Fig 04 // Multi-Head Parallel Subspaces

pkg-slim · 1.6MB 60 FPS WEBGPU

04 · Parallel Subspaces

Tracking Multiple Relationships in Parallel

A single attention head can only focus on one relationship at a time. Splitting coordinates into 12 parallel heads allows simultaneous tracking of syntax, coreference, and grammatical dependency.

multihead.amx ↗

Head Count (h): 12 Heads Subspace Width: 64 Coordinates / Head Recombination: Concat + Linear Projection W_o

Subspace Projection The 768-dim state projects into 12 specialized 64-dim subspaces.

Specialized Focus Head A tracks subject-action, Head B tracks patient-action.

Linear Synthesis All heads concatenate and project back into d_model.

Fig 05 // Feed-Forward Associative Memory & Residual Bypass

pkg-slim · 1.6MB 60 FPS WEBGPU

05 · Associative Memory

Nonlinear Memory Recall in Higher Dimensions

While attention routes information between tokens, the FFN processes each token alone. Expanding coordinates 4× creates a rich associative memory space for factual recall.

feedforward.amx ↗

Dimensional Expansion: 768 → 3072 → 768 Activation: GELU Nonlinearity Residual Highway: x + FFN(LN(x))

Residual Highway The identity stream carries the unbroken token vector forward.

4× Dimensional Widen Expands to 3072 features to activate associative memory keys.

Contract & Add Back Projected delta adds back directly into the residual highway.

Fig 06 // The Full Layer Stack & Logit Projection

pkg-slim · 1.6MB 60 FPS WEBGPU

06 · Deep Assembly Line

Refining Representation Across N Layers

Raw coordinates enter at block 1. Layer by layer, representation refines from low-level surface grammar to high-level abstract reasoning, culminating in vocabulary logit scoring at the LM head.

pipeline.amx ↗

Repetition: N = 12 Layers (GPT-2 Small) Total Parameters: 124 Million Vocabulary Projection: 768 → 50,257 Logits

Pipeline Entry Token embedding enters the base of the layer stack.

Iterative Refinement Blocks 1 through N successively contextualize meaning.

Logit Scoring LM head scores every token in the vocabulary.

How This Walkthrough Works

There are zero pre-rendered video files on this page. Every figure is a declarative .amx file compiled live in your browser and rendered at 60 FPS through WebGPU.

<script type="module" src="../../embed/amx-player.js" data-runtime-base="../../pkg-slim"></script>
<amx-player src="scenes/attention.amx" autoplay loop controls hold="1.5" aspect="16:9"></amx-player>

Because these scenes use vector geometries and typography without Typst or SVG dependencies, they run on the lightweight pkg-slim runtime profile (~1.6 MB brotli-compressed).