Embeddings and Attention: How Context Gets Represented
Embeddings represent tokens as numerical vectors. Attention helps a transformer weigh relationships between tokens when building contextual representations.
Embeddings are coordinates, not definitions
An embedding maps a token or other item into a vector of numbers. During training, these representations are adjusted so that useful patterns can be captured by the network.
In embedding-based search, semantically related pieces of text can be close in vector space even when they do not share the same keywords. Similarity is useful, but it is not the same as truth or relevance in every context.
Attention connects context
Self-attention allows each token representation to incorporate information from other tokens in the sequence. The mechanism computes how strongly positions should influence one another, using learned query, key and value projections.
This helps the model handle relationships across a sentence or longer context. It does not mean the model consciously focuses on information the way a person does.
A practical distinction
- Embeddings are useful for similarity search and retrieval.
- Attention is part of the transformer's internal processing.
- Neither mechanism independently verifies facts.
- Retrieval systems need good chunking, ranking and source checks.
Sign in to access the lab and homework
The lesson explanation is free to read. Create a free learner account to use the practical exercise, submit homework and track your progress.
CREATE FREE ACCOUNT →