Token

A token is a unit into which a model’s input or output is divided for processing.

Plan for this page

Understand one concept in plain language and connect it to nearby ideas.

You will leave with: A token can be a word, word piece, punctuation mark, byte, or another learned unit. The tokenizer maps content to token identifiers before the model processes it, so token counts do not equal word counts.
Time: 3–8 min
Before you begin: No prerequisite. Keep one real example in mind.
Do this now: Read the explanation, then compare the example with the common misunderstanding.

Prepared next stepRun the context experiment

TokenA token is a unit into which a model’s input or output is divided for processing.

Large language model
Embedding
Context window

Plain explanation

A token can be a word, word piece, punctuation mark, byte, or another learned unit. The tokenizer maps content to token identifiers before the model processes it, so token counts do not equal word counts.

Example

A tokenizer might split “unbelievable” into several pieces while treating a common short word as one token.

Why it matters

Tokens affect context limits, generation steps, latency, and often pricing. Different languages and writing systems can require different token counts for similar meaning.

Common misunderstanding

Misconception: one token always equals one word. Correction: token boundaries vary by model, text, and language.

Related concepts

Follow relationships, not only the alphabet

Large language modelA large language model, or LLM, is a parameter-rich model trained on language sequences to predict and generate tokens in context.EmbeddingAn embedding is a numerical vector that represents an item so useful relationships can be measured or learned.Context windowA context window is the limited amount of tokenized information a model can consider together during one generation or inference.

Run the context experiment

Sources and scope

What supports this object

Machine Learning GlossaryGoogle for Developers

Defines tokens and tokenization; exact token boundaries depend on the model’s tokenizer and language.

Open source

Loading page content...

…