learning machines

01 / tokens

What is a token?

How text goes in, and the next piece comes out.

A text-generating language model works with pieces called tokens. A piece might be a word, part of a word, punctuation or a space. To produce the next piece, the model lets the last piece listen to everything before it.

Interactive teaching illustration · runs in your browser, no live model

00 / tokens

Start with something small.

Real tokenization / cl100k_base · tiktoken 0.14.0

The tokenizer splits the text, then looks up an ID for each piece. The IDs are labels in a vocabulary, not measurements of meaning.

Here, · marks a space. A token is not always a word: some words split into several pieces.

TEXT SO FAR

The cat sat on the

WhoDid what

unhappiness

  1. unID 359
  2. hID 71
  3. appinessID 67391
Real cl100k_base split · one word, 3 pieces.

Same word, different token: The = 791 vs ·the = 279. Case and a leading space each change the ID.

Where these IDs come from

Not from this page: no tokenizer runs in your browser. The IDs were produced once, offline, by regenerate_fixture.py in the open-source learning-machines repository, using tiktoken 0.14.0 with the cl100k_base encoding. It writes fixture.mjs, which this page serves unchanged and checks against the integrity manifest.

ids = encoding.encode(text)
pieces = [encoding.decode([token]) for token in ids]
assert encoding.decode(ids) == text   # every example must round-trip

Reproduce the fixture yourself (downloads the tokenizer vocabulary, not a model):

git clone https://github.com/rsteth/learning-machines
cd learning-machines
python3 -m venv .venv && .venv/bin/pip install tiktoken==0.14.0
.venv/bin/python experiments/tokens/regenerate_fixture.py
git diff --exit-code -- experiments/tokens/fixture.mjs   # no output = identical

Pinned to commit 14939b3. Add a string to the generator and re-sync, and it appears in the picker above.

01 / a card for each piece

Each piece gets a card.

Each ID looks up a row of numbers that was set during training. Think of the row as a card of notes: what the piece means on its own, before it has read anything around it.

·catID 8415

What is stored · six numbers

0.45
-0.45
-0.45
0.45
0
0.45

What can be read from it · eight notes

a thing comes next
0
on top of something
0
sitting
0
sleeping
0
a cat
1
royalty
0
low, cosy spot
0.7overlap
grand seat
0

The numbers are an address, not a list of ideas.

No single number on the card means “cat”. An idea is a direction. Writing a note pushes the card that way; reading a note checks how far along that direction the card sits.

A space has more directions than it has numbers, so this toy fits eight notes into six numbers. Directions that sit close together overlap, which is why the card for ·cat already reads a little like “low, cosy spot” before anything has written it.

The eight notes and their labels are ours. A real model learns thousands of directions and labels none of them.

02 / listening to the pieces before

Every earlier word gets a say.

Whatever comes next will be produced from the last piece. On its own, ·the knows almost nothing: its card says only “a thing comes next”. So it listens back.

A Listen

The last piece hears every earlier piece, some more than others: thicker arc, larger share. It copies what it hears onto its own card. It cannot hear pieces that come later.

28%34%34%5%The·cat·sat·on·the
Thicker arc = larger share of what is heard.
Cover a word to see what goes missing

B Work out

Then the card is worked over on its own. Some notes can only be written when two things are on the card together: a cat and sitting make a low, cosy spot. Neither word carries that alone.

THE CARD OF THE LAST PIECE · ·the

Starts witha thing comes nextits own note · 1
Hearsa thing comes nextfrom The · 0.14a catfrom ·cat · 1.02sittingfrom ·sat · 1.02on top of somethingfrom ·on · 0.83
Works outlow, cosy spotworked out · 1.25grand seatneeds royalty together with sitting

a cat 1.02 + sitting 1.02 + sleeping 0 = 2.03 · over the bar of 1.2 writes 1.25 of “low, cosy spot”

royalty 0 + sitting 1.02 = 1.02 · under the bar of 1.2 writes nothing

Inspect the exact numbers

Every row below is computed from the one above it. Shares are a softmax of hand-written match scores; a share times 3 is the amount copied.

The card of ·the, step by step
StepShareWritesSix stored numbers afterwards
starts as ·the—a thing comes next 1[0, 0.447, 0.447, 0.447, 0.447, 0.447]
hears ·on27.72%on top of something 0.8315[0.372, 0.447, 0.819, 0.075, 0.075, 0.819]
hears ·sat33.85%sitting 1.0155[0.826, 0.901, 0.819, 0.53, -0.379, 0.365]
hears ·cat33.85%a cat 1.0155[1.28, 0.447, 0.365, 0.984, -0.379, 0.819]
hears The4.58%a thing comes next 0.1374[1.28, 0.509, 0.426, 1.045, -0.317, 0.881]
works out low, cosy spot—low, cosy spot 1.2466[2.284, 0.118, 0.371, 1.658, -0.373, 0.769]

The finished card

2.28
0.12
0.37
1.66
-0.37
0.77
a thing comes next
1.14
on top of something
0.96
sitting
1.64
sleeping
0.62
a cat
1.89
royalty
0
low, cosy spot
2.55
grand seat
0.98

Both steps add to the card; neither erases it. A real model repeats them through dozens of layers, and every piece listens, not only the last.

A hand-authored example, not a trained model. How well each word matches is written by hand; in a model it comes from comparing a query from the listener with a key from each earlier piece, and what gets copied is that piece’s value. One round is shown, and the listener’s own position is left out of the shares. A covered word is still listened to but offers nothing, so the others do not get louder.

03 / choosing what comes next

One more piece, not a whole reply.

Put the card's journey in space. Each thing it hears moves it along that note's direction, and working out moves it again. Where it ends up decides what comes next.

start·mat·sofa·floor·bed·throne

Where the card ends up · furthest along the direction of ·mat (60%)

WhoDid what
  • heard from ·on
  • heard from the verb
  • heard from who
  • worked out from both
  • the other “who”
  • each candidate's direction

A three-number view of the six stored numbers. In a model, listening is one move; here it is drawn as one leg per word, nearest word first. Swap who is in the sentence and the same start bends to a different end.

Score every candidate.

Each candidate has its own direction. Its score is how far the finished card lies along it. Because the card is a sum of legs, every score splits into what each leg contributed.

·mat
6.60
·sofa
5.81
·floor
3.25
·bed
4.11
·throne
3.68
all other tokens
3.40
  • its own note
  • heard from ·on
  • heard from the verb
  • heard from who
  • heard from The
  • worked out from both
  • fixed stand-in

Scores become probabilities that share one strip. Raising one candidate squeezes the others.

Next-token probabilities · one strip = 100%

·mat 60% · ·sofa 27% · ·floor 2% · ·bed 5% · ·throne 3% · all other tokens 2%

Choose a token.

Always taking the most likely piece is called greedy decoding. A sampler draws from the strip instead, so another run could choose differently.

Cycle 1 · greedy would take ·mat.

A probability says how likely a piece is to come next, not how true it is. “All other tokens” stands in for the rest of a real vocabulary; if it is chosen, this toy stops instead of inventing a piece.

04 / returning the text

Append. Repeat. Stop.

The chosen piece joins the text and gets a fresh card of its own. Then everything runs again to produce one more piece.

The cat sat on the

Choose a piece in section 03 and it is appended here.

Cards on the table: 5. Every one has to be kept for the next piece to listen to.

Generation continues until a stop token, a configured limit or another stopping rule. Text can appear a piece at a time, or arrive as a finished response.

This example generates at most two pieces. The second is scripted punctuation, not computed.

Text. Pieces. Cards. Listening. Scores. Text again.

Next: what does the model need to keep?

Every earlier card had to stay on the table to be listened to. Attention and memory asks how much of that a model really has to keep.

Sources, assumptions & limits

Scope: a simplified decoder-only language model. This is a teaching illustration, not live model inference.

Token IDs and decoded pieces come from tiktoken 0.14.0, cl100k_base. The example is not a Qwen tokenizer demonstration. Notes, directions, match scores, work-out rules and candidate scores are hand-authored; every number shown is then computed from the step before it.

The model produces raw vocabulary scores (logits). Softmax turns scores into probabilities; sampling settings can adjust the distribution. “All other tokens” groups the rest of the vocabulary with a fixed stand-in score.

Implementations commonly cache earlier computation rather than rerunning the whole prompt. Decoding may buffer incomplete byte sequences. Stop tokens need not appear as visible text.

tiktoken tokenizer · Generation overview · Decoding strategies

Token experiment source at the pinned revision · Exact source commit · Integrity manifest · MIT license