Tokens, Tokenisation and Context Windows
Models do not read text exactly as humans do. They process token IDs, and the amount of context available to a model is finite.
Text becomes tokens
A tokenizer converts input text into tokens and maps those tokens to numerical IDs. Common words may map to one token, while unusual words, names, punctuation or mixed-language text may split into several tokens.
This is why character count and token count are not interchangeable. The exact tokenisation depends on the model's tokenizer.
The context window
The context window is the amount of tokenised information a model can use in a single interaction or request. It may include system instructions, user messages, retrieved documents, tool results and the conversation history.
When an application approaches its context limit, it may need to truncate, summarise or selectively retrieve earlier information. Long conversations can therefore lose detail even when the interface still shows the full history.
Why security teams care
- Hidden instructions and untrusted text still consume context.
- Unicode and formatting differences can change how text is tokenised or interpreted.
- Large retrieved documents can crowd out important instructions.
- A context limit is a reliability constraint, not an access-control boundary.
Sign in to access the lab and homework
The lesson explanation is free to read. Create a free learner account to use the practical exercise, submit homework and track your progress.
CREATE FREE ACCOUNT →