Generative AI & LLM Engineering · Chapter 4 · ~18 min · Hands-on Visuals
Embeddings: Meaning as Points on a Map
Computers only understand numbers, but keyword search is blind to synonyms and human intent. Embeddings solve this by turning any word, sentence, or document into a list of coordinates where semantic meaning becomes geometry.
- Turn text into multi-dimensional vectors
- See concepts cluster on an interactive map
- Perform semantic algebra: King - Man + Woman
- Measure similarity with Cosine Angle vs Keywords
1Why computers struggle with meaning
If you search for "How to fix a leaky faucet", a traditional keyword search scans for the exact letters faucet. If an article says "Repairing a dripping bathroom tap", keyword search misses it completely!
An embedding model converts text into a list of numbers called a vector. Sentences with similar meanings are given coordinates close together in vector space, regardless of the words used.
Why does semantic search find 'automobile' when you type 'car'?
2The 2D Meaning Map: Watch concepts cluster
Explore the map below. Notice how animals cluster in one neighborhood, royal titles in another, and programming languages in another. Click any point to inspect its coordinates and nearest semantic neighbors.
Words with similar context naturally cluster together. Click any point to inspect neighbors.
Cat
Independent domestic feline
Nearest Semantic NeighborsSimilarity
[X, Y]. Real embedding models use 1,536 or 3,072 dimensions! Each dimension captures an abstract aspect like formality, sentiment, domain, or tense.What determines whether two words are close together in vector space?
3An approximate vector arithmetic example
Because embeddings are geometric coordinates, you can explore arithmetic patterns between concepts. In some embedding spaces, the direction from "Man" to "Woman" roughly aligns with a gender-related direction. Adding it to "King" may put "Queen" nearby, but the result depends on the model and embedding space; it is not an exact or universal rule.
Directions in embedding space encode real human concepts.
| Concept | Royalty | Masculine | Feminine | Power |
|---|---|---|---|---|
| King | 0.95 | 0.92 | 0.05 | 0.90 |
| - Man | -0.05 | -0.90 | -0.05 | -0.20 |
| + Woman | +0.05 | +0.05 | +0.92 | +0.20 |
| Result Vector | 0.95 | 0.07 | 0.92 | 0.90 |
How it works
Subtracting 'Man' strips away the masculine direction; adding 'Woman' redirects the royal concept to the feminine coordinate.Nearest Vector Matches
Which result is commonly used as an illustrative analogy for embedding arithmetic?
4Measuring Closeness: Cosine Similarity
How do we know if two vectors are similar? We don't measure the straight-line distance, because long documents would naturally have longer vectors. Instead, we measure the angle between them using Cosine Similarity.
Cosine similarity ignores the length of the text (whether it is 3 words or 300 words) and measures only the angle between their meaning vectors.
Cosine scores are model- and dataset-dependent; there is no universal cutoff for a strong or irrelevant match. Calibrate thresholds using representative queries and labeled results.
Identical meaning or exact paraphrase.
Strong semantic relationship (ideal for search).
Completely orthogonal and unrelated topics.
Why is Cosine Similarity preferred over Euclidean distance for text?
5Semantic Search in action & Chunking
Toggle between Keyword Search and Vector Search below. See how vector embeddings understand what the user is actually trying to solve.
See why traditional keyword matching fails where vector embeddings excel.
10 science-backed tips for combating insomnia & falling asleep fast
Establish a cool bedroom, avoid evening caffeine, and maintain a fixed bedtime.
Choosing the best ergonomic mattress and pillows for neck pain
Orthopedic memory foam helps spine alignment and deep rest.
Night sleep schedule of nocturnal animals in North America
Owls and bats hunt at night and sleep during the day when temperatures rise.
How to configure dark mode for night browsing on iPhone and Android
Reduce eye strain by turning on night shift and scheduling dark mode.
Notice how keyword search ranks nocturnal animal articles higher just because of the word "night"! Semantic search understands intent, not just spelling.
When preparing documents for vector search (the foundation of RAG), you must divide text into chunks (the demo uses 300–500 tokens as one starting point). The best size depends on your documents and retrieval task; slide below to see how chunk size can matter:
An embedding vector has a fixed number of dimensions. You cannot squeeze infinite knowledge into 1,536 numbers without losing precision.
Example Starting Point (300–500 tokens with 10% overlap)
16 Vectors generatedIn this illustrative example, each paragraph or section keeps a focused vector. The example query matches the cancellation chunk; actual retrieval quality depends on the embedding model, text, and search setup.
6Chapter challenge
Embeddings & Vectors Mastery
Answer all four questions correctly to complete Chapter 4.
1What is a text embedding?
2What does a Cosine Similarity score of 0.92 between two sentences indicate?
3Why might we split a long manual into smaller chunks before embedding it?
4Which of these is the greatest superpower of semantic vector search over keyword search?
7Recap
- Embeddings: Turn text into coordinate vectors of 384, 1,536, or 3,072 numbers.
- Clustering: Words with related meaning end up in the same geometric neighborhood.
- Vector Arithmetic: Some embedding spaces capture approximate relationships such as
King - Man + Woman ≈ Queen; results vary by model. - Cosine Similarity: Compares vector directions (1 means same direction; 0 means orthogonal for nonzero vectors). Score interpretation depends on the embedding space and task.
- Chunking: Chunk size is a design choice; 300–500 tokens is one starting point to evaluate against your documents and retrieval task.
Solve the challenge above to complete the chapter, or