The Core Idea: An LLM does not receive a sentence as a sequence of human-readable words. Before the model can process text, a tokenizer converts the text into smaller units called tokens, and those tokens are mapped to numerical token IDs. This conversion is the first major step between human language and the mathematical operations performed by the model.
- Beginner: What Is Tokenization?
Tokenization is the process of breaking text into smaller pieces called tokens. The tokenizer then maps those pieces to integer IDs from a vocabulary that the model understands.
A token is not necessarily a complete word. Depending on the tokenizer, a token may represent a whole word, part of a word, punctuation, whitespace-related information, or another frequently occurring text pattern.
Human Language → Model Input
Splits the text according to the tokenizer's vocabulary and rules.
The exact IDs depend on the tokenizer and vocabulary.
The input written by a human or produced by another system.
The pieces of text selected by the tokenizer.
Integer identifiers used to identify entries in the tokenizer vocabulary.
What does a tokenizer primarily do?
A) Convert text into tokens and token IDs▼
B) Train all of the model's neural-network weights▼
- Beginner: Words Are Not the Same as Tokens
One of the most common beginner assumptions is that one word equals one token. Modern LLM tokenizers generally do not work that way.
Each character can become a separate unit. This gives flexible coverage but can create very long sequences.
Each word can be treated as a unit. This is intuitive but creates a large vocabulary and struggles with rare words.
Words can be represented using smaller recurring pieces. This provides a practical balance between vocabulary size and sequence length.
Modern language models commonly use subword-based or subword-inspired tokenization. As a result, a familiar word may be one token while another word may be split into several tokens.
A Conceptual Example
This is a conceptual illustration. The exact split depends on the tokenizer and vocabulary.
Subword tokenization lets a tokenizer reuse common pieces across many words. A rare word does not necessarily need a completely new vocabulary entry; it can often be represented using smaller pieces that already exist in the vocabulary.
- Beginner: Vocabulary and Token IDs
A tokenizer has a vocabulary: a collection of token entries that the tokenizer knows how to map to integer IDs. The model later uses those IDs as the entry point for learned token representations.
| Token | Token ID | Purpose |
|---|---|---|
| hello | ... | A vocabulary entry representing a token piece. |
| world | ... | Another token entry with its own integer identifier. |
| . | ... | Punctuation can also be represented as a token. |
The exact numeric IDs are model and tokenizer specific. You should not memorize a particular ID. What matters is the pipeline:
What does a token ID represent?
A) An integer identifier for a token in the vocabulary▼
B) A numerical score showing how important the token is▼
- Beginner: Subword Tokenization
Subword tokenization is a practical compromise between character-level and word-level tokenization. Instead of requiring every possible word to appear in the vocabulary, the tokenizer can represent many words using reusable pieces.
A common word may be represented efficiently as one token or a small number of tokens.
A rare or unusual word can be divided into smaller pieces so the model can still represent it using known vocabulary entries.
This is especially useful for technical terminology and newly formed words. The tokenizer does not need a separate vocabulary entry for every possible word combination.
Why Subwords Help
Common Pieces
Reusable Vocabulary
Many Possible Words
- Intermediate: BPE, WordPiece, and SentencePiece
Modern tokenizers use algorithms that construct useful vocabulary pieces from training data. Three names you will frequently encounter are Byte Pair Encoding (BPE), WordPiece, and SentencePiece.
Builds a vocabulary by repeatedly combining frequently occurring symbol or subword pairs according to the tokenizer's training procedure.
Builds subword vocabularies using a likelihood-based training objective and is associated with models such as BERT.
A tokenizer framework that can train subword models directly from raw text without requiring whitespace-based pre-tokenization.
These names should not be treated as interchangeable. They describe different algorithms or tokenizer frameworks, and real model tokenizers can include additional preprocessing, normalization, byte-level behavior, special tokens, or model-specific rules.
Conceptual Tokenizer Construction
- Intermediate: Special Tokens
Tokenizers can contain special tokens that are not ordinary pieces of natural language. These tokens help the model represent boundaries, padding, unknown values, or other control information.
- Intermediate: Tokenization and Context Length
LLMs process a finite sequence of tokens at a time. This available input capacity is commonly described as the model's context window or context length.
Context Window
The model sees a sequence of tokens, not a fixed number of English words.
Two prompts contain the same number of words. Can they use different numbers of tokens?
A) Yes, because token boundaries depend on the tokenizer▼
B) No, one word always equals one token▼
- Intermediate: Tokenization Across Languages & Symbols
Tokenization is not equally efficient for every language or every type of content. Vocabulary coverage, script, training data, normalization, and tokenizer design can all affect how text is divided.
Different languages can require different numbers of tokens for similar semantic content. Script and vocabulary coverage strongly influence token efficiency.
Brackets, operators, identifiers, indentation-related patterns, and common code fragments can all influence token boundaries.
Numbers, punctuation, URLs, emojis, and unusual character sequences can tokenize differently from ordinary natural-language words.
- Intermediate: Tokenization in Chat Models
A chat interface may look like it sends a simple conversation to the model, but the underlying system still needs to convert that conversation into a token sequence. Modern instruction-tuned models often use chat templates to format roles and messages before tokenization.
Conversation → Model Input
Formatted conversation → Tokenizer → Token IDs
This explains why two chat models can receive the same visible messages but internally construct different token sequences. Their special tokens, role formatting, separators, and chat templates may differ.
When using an instruction-tuned model, the tokenizer and chat template are part of the model's expected interface. Passing the wrong format can change the model input and affect output quality.
- Advanced Preview: Tokenization Trade-Offs
Tokenization looks like preprocessing, but its design affects the entire LLM system. Vocabulary size, token sequence length, multilingual coverage, memory usage, context capacity, and inference cost are connected to the tokenizer.
- • More complete pieces can be represented directly.
- • May reduce token counts for some inputs.
- • Increases vocabulary-related parameters.
- • Does not automatically improve every language.
- • Uses fewer vocabulary entries.
- • Can represent text using more pieces.
- • May increase sequence length.
- • Can still provide strong coverage through reusable subwords.
A tokenizer tries to balance vocabulary size, sequence length, and coverage. There is no single tokenization scheme that is universally optimal for every language, domain, and model.
Why can tokenizer design affect the cost and context capacity of an LLM application?
A) Because the model processes token sequences, not raw word counts▼
B) Because token IDs directly determine the model's intelligence▼
- Code: Tokenizing Text with Python
Now let's turn the concepts into code. In practice, you normally use the tokenizer associated with the model you are using. The example below uses Hugging Face Transformers to inspect how a sentence becomes tokens and token IDs.
pip install transformersfrom transformers import AutoTokenizer # Load a tokenizer associated with a pretrained model.tokenizer = AutoTokenizer.from_pretrained("gpt2") text = "Large language models are useful." # Convert text into token IDs.token_ids = tokenizer.encode( text, add_special_tokens=False) # Convert IDs back into readable token pieces.tokens = tokenizer.convert_ids_to_tokens(token_ids) print("Tokens:")print(tokens) print("\nToken IDs:")print(token_ids) print("\nNumber of tokens:", len(token_ids))Tokens:['Large', 'Ġlanguage', 'Ġmodels', 'Ġare', 'Ġuseful', '.'] Token IDs:[model-specific integer IDs] Number of tokens: 6What to notice: The exact token IDs depend on the tokenizer. The important pipeline is text → tokens → token IDs.
Before using a real model tokenizer, it can be useful to see the basic idea with a deliberately simple example. This is not how production LLM tokenizers work; it is only a learning aid.
text = "hello world" vocabulary = { "hello": 0, "world": 1,} tokens = text.split()token_ids = [vocabulary[token] for token in tokens] print("Tokens:", tokens)print("Token IDs:", token_ids)Tokens: ['hello', 'world']Token IDs: [0, 1]What to notice: The toy tokenizer looks up each word in its vocabulary and replaces it with the corresponding integer ID.
- Visual Explanation: From Text to the Transformer
Tokenization is only the beginning of the LLM pipeline. Once text has been converted into token IDs, those IDs can be used to retrieve learned token representations. The resulting vectors are then processed by the Transformer layers.
How raw text becomes a numerical sequence that an LLM can process.
Embeddings: how token IDs become learned numerical vectors.
The exact tokenization behavior is model-specific. Always use the tokenizer associated with the model.
- Beginner: Key Terms to Remember
Before moving to Embeddings, make sure these foundational terms are clear. They will appear repeatedly throughout the rest of the LLM course.
| Term | Meaning |
|---|---|
| Token | A unit produced by a tokenizer and represented by a token ID. |
| Vocabulary | The collection of token entries known by a tokenizer. |
| Token ID | An integer identifier for a token vocabulary entry. |
| Subword | A reusable piece of a word or text sequence used by many modern tokenizers. |
| Special Token | A tokenizer entry used for control or structural information rather than ordinary text. |
| Context Window | The amount of tokenized context a model can process in a particular request. |
| Chat Template | A model-specific format that converts structured conversation messages into the sequence expected by the tokenizer and model. |
You now understand how an LLM turns human language into model-readable tokens.
You should now be comfortable with tokenization, token IDs, vocabulary, subword tokenization, BPE, WordPiece, SentencePiece, special tokens, context length, chat templates, and the relationship between tokenization and the rest of the LLM pipeline.