← COURSE CURRICULUM
CNSLT ACADEMY / UNDERSTAND THE MODEL

Tokens, Tokenisation and Context Windows

14 MIN•BEGINNER•PUBLIC LESSON

Models do not read text exactly as humans do. They process token IDs, and the amount of context available to a model is finite.

01 / CONCEPT

Text becomes tokens

A tokenizer converts input text into tokens and maps those tokens to numerical IDs. Common words may map to one token, while unusual words, names, punctuation or mixed-language text may split into several tokens.

This is why character count and token count are not interchangeable. The exact tokenisation depends on the model's tokenizer.

02 / CONCEPT

The context window

The context window is the amount of tokenised information a model can use in a single interaction or request. It may include system instructions, user messages, retrieved documents, tool results and the conversation history.

When an application approaches its context limit, it may need to truncate, summarise or selectively retrieve earlier information. Long conversations can therefore lose detail even when the interface still shows the full history.

03 / CONCEPT

Why security teams care

  • Hidden instructions and untrusted text still consume context.
  • Unicode and formatting differences can change how text is tokenised or interpreted.
  • Large retrieved documents can crowd out important instructions.
  • A context limit is a reliability constraint, not an access-control boundary.
PRACTICAL WORK

Sign in to access the lab and homework

The lesson explanation is free to read. Create a free learner account to use the practical exercise, submit homework and track your progress.

CREATE FREE ACCOUNT →
← PREVIOUS LESSONNEXT LESSON →