Introduction
In the last post, we covered the big picture: what embeddings are, why Transformers replaced RNNs, and how encoder and decoder blocks fit together. This post goes one level deeper into the two ideas that actually make a Transformer work: positional encoding (how the model knows word order) and self-attention (how the model figures out what each word means in context). We’ll build both up from the actual problem they solve, work through the real formulas with numbers, and — since a lot of us think in database terms — tie each step back to a DBA analogy.
What you’ll learn:
- Why Transformers need positional encoding at all (and why sin/cos, specifically)
- How to actually compute a positional encoding vector by hand
- Why “self-attention” needs three separate vectors — Query, Key, and Value — instead of just one
- How to walk through the full self-attention calculation on a real example (“money bank grows”)
- Why this whole approach scales better than naive word-similarity lookups
Prerequisites
- [ ] Read Part 1 of this series (embeddings, Transformer overview, encoder/decoder split)
- [ ] Comfortable with the idea that a word becomes a vector of numbers (an embedding)
- [ ] Basic familiarity with dot products (multiplying and summing two vectors) — we’ll walk through it anyway
Part 1: Positional Encoding
The Problem Statement
A Transformer processes all tokens in parallel — not one at a time like an RNN. That parallelism is what makes it fast, but it comes with a side effect: without extra help, the model has no idea what order the words came in.
Consider:
- “I love AI”
- “AI love I”
Same three words. Completely different meaning. If the model only sees the embeddings for “I”, “love”, “AI” with no notion of sequence, both sentences look identical to it. The classic newspaper-headline version of this problem: “Dog bites man” vs. “man bites dog” — same words, opposite (and very different) story.
DBA analogy: this is exactly the problem with an unordered result set. If you run a query without an ORDER BY, the database is free to return rows in any order — the data is the same, but any process relying on sequence (like a running total, or a time-series chart) breaks. Positional encoding is your ORDER BY for token meaning: it stamps a sequence signal onto the data before anything else happens.
The Solution: Positional Encoding (PE)
Positional Encoding is a technique used to inject information about the position (order) of tokens into their embeddings, so the model can understand sequence order — even though it processes everything in parallel.
Instead of appending a fixed sequence number (which would work but scale badly and be “discrete,” not continuous), the original Transformer paper uses sine and cosine waves:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Where:
- pos = the position of the word in the sentence (0, 1, 2, 3, …)
- d_model = the dimensionality of the embedding vector
- i = an index that ranges from
0to(d_model / 2) − 1, alternating between sine (even positions) and cosine (odd positions)
Why sine and cosine specifically? Both functions output smooth, bounded values between -1 and +1, and each position gets a unique combination of wave values across all the dimensions — like a fingerprint for “where am I in this sentence,” without ever producing an unbounded number that could destabilize training.
Working the Math by Hand
Let’s say d_model = 6 (a tiny toy example) and we want the positional encoding for two words: “Great” at position 0, and “India” at position 1.
Since d_model = 6, i ranges over {0, 1, 2} (three sine/cosine pairs, filling all 6 dimensions).
For “Great” (pos = 0):
i=0: PE(0,0) = sin(0 / 10000^0) = sin(0) = 0
PE(0,1) = cos(0 / 10000^0) = cos(0) = 1
i=1: PE(0,2) = sin(0 / 10000^(2/6)) = 0
PE(0,3) = cos(0 / 10000^(2/6)) = 1
i=2: PE(0,4) = sin(0 / 10000^(4/6)) = 0
PE(0,5) = cos(0 / 10000^(4/6)) = 1
→ PE1 = [0, 1, 0, 1, 0, 1]
For “India” (pos = 1):
i=0: PE(1,0) = sin(1 / 10000^0) = sin(1) ≈ 0.84
PE(1,1) = cos(1 / 10000^0) = cos(1) ≈ 0.54
i=1: PE(1,2) = sin(1 / 10000^(1/3)) ≈ 0.04
PE(1,3) = cos(1 / 10000^(1/3)) ≈ 0.99
i=2: PE(1,4) = sin(1 / 10000^(2/3)) ≈ 0.01
PE(1,5) = cos(1 / 10000^(2/3)) ≈ 0.99
→ PE2 ≈ [0.84, 0.54, 0.04, 0.99, 0.01, 0.99]
Notice: every position gets a distinct vector, and nearby positions produce similar (but not identical) vectors — which is exactly what lets the model infer relative distance between words later.
Addition, Not Concatenation
Here’s a subtlety that trips people up: the positional vector isn’t appended to the word embedding — it’s added, element by element, to the existing embedding, and the result is the same dimensionality as before.
Why not concatenate (stick them side by side, doubling the dimension)? Because that would multiply the computational cost of every downstream self-attention and neural network layer. Addition keeps the same vector size while still injecting position information into every dimension.
DBA analogy: concatenation is like adding a whole new column to every row in a huge table — every subsequent join and aggregation now scans more bytes. Addition is more like updating existing column values in place — the “shape” of your table doesn’t change, but the values now silently carry extra information (sequence) baked in.
Part 2: Self-Attention — The Core Idea
The Problem: Word2Vec’s Vectors Never Change
Recall from Part 1: Word2Vec gives every word one fixed vector, no matter the sentence. So “bank” in “money bank grows” and “bank” in “river bank flows” get the exact same embedding — which is wrong, because the two “banks” mean completely different things.
Self-Attention = the relationship of a word with every other word in the same sentence, used to build a new, contextual version of that word’s embedding.
Why Not Just “Similarity”? — The Naive Approach and Its Drawback
A first instinct might be: just compute how similar (via dot product) each word’s embedding is to every other word’s embedding, and blend them by similarity. For the sentence “money bank grows”, that would look like:
bank = w1 × money + w2 × bank + w3 × grows
where each w comes from comparing “bank”‘s embedding to each word’s embedding via a dot product, then normalizing with softmax.
The problem: this reuses the exact same vector for three different roles at once — comparing itself to others, being compared against, and being the thing that gets blended in. That’s confusing and limiting for the model to learn from, and it caps how expressive the mechanism can be.
The fix: transform each word’s embedding into three separate, purpose-built vectors using three separate learnable weight matrices:
- Query (Q) — “what am I looking for?”
- Key (K) — “what do I contain, that others might be looking for?”
- Value (V) — “what information do I actually contribute, once matched?”
DBA analogy: this maps almost exactly onto a dictionary/hash-map lookup: D = {a: 1, b: 2, c: 3}. When you do D[a], a is your query, the stored a inside the dictionary is the key, and 1 is the value returned. Self-attention just does this lookup softly — instead of an exact match, every query compares against every key with a similarity score, and the output is a weighted blend of all the values, weighted by how well each key matched.
Walking Through the Full Calculation
Take our sentence: money, bank, grows → embeddings e_money, e_bank, e_grow (each already has positional encoding added in).
Step 1 — Project into Q, K, V. Each embedding is multiplied by three separate learnable weight matrices (Wq, Wk, Wv) — the same three matrices are used for every word in the sentence:
Q_money = e_money × Wq K_money = e_money × Wk V_money = e_money × Wv
Q_bank = e_bank × Wq K_bank = e_bank × Wk V_bank = e_bank × Wv
Q_grow = e_grow × Wq K_grow = e_grow × Wk V_grow = e_grow × Wv
If the embedding is 512-dimensional, a common choice is to project down to 64-dimensional Q, K, and V vectors (more on why in the multi-head attention post coming next).
Step 2 — Score every word against every other word. For “bank”, compute the dot product of Q_bank against every word’s K vector:
score(bank, money) = Q_bank · K_money
score(bank, bank) = Q_bank · K_bank
score(bank, grow) = Q_bank · K_grow
A dot product is just: multiply corresponding elements, sum them up → a single scalar “similarity” number. Do this for every word against every other word, and you get a full score matrix.
Step 3 — Scale. Divide every score by √d_k (the square root of the key dimension, e.g. √64 = 8). This keeps the numbers in a stable range so the next step doesn’t produce extreme, unstable gradients during training.
Step 4 — Softmax. Convert the scaled scores into weights that sum to 1:
w1 = e^s1 / (e^s1 + e^s2 + e^s3)
w2 = e^s2 / (e^s1 + e^s2 + e^s3)
w3 = e^s3 / (e^s1 + e^s2 + e^s3)
Step 5 — Weighted sum of Values. Multiply each softmax weight by its corresponding Value vector, and sum them up:
contextual_bank = w1 × V_money + w2 × V_bank + w3 × V_grow
The result — contextual_bank — is a brand-new vector for “bank” that has absorbed relevant information from “money” and “grows.” This is the dynamic embedding we promised in Part 1: same word, different vector, depending on what’s around it.
All of this collapses into one formula from the original paper:
Attention(Q, K, V) = softmax( (Q · Kᵀ) / √d_k ) · V
DBA analogy for the scaling + softmax step: think of the raw dot-product scores like raw cost estimates from a query optimizer for several candidate execution plans — they’re on arbitrary, unnormalized scales. Softmax is like converting those costs into a normalized probability distribution (“70% likely this is the best plan, 20% this one, 10% this one”) so they can be combined into a single sensible answer instead of picking just one winner-takes-all path.
Why Q, K, V Are Learnable, Not Fixed
The weight matrices Wq, Wk, Wv start out as random values and get updated during training via backpropagation — just like every other weight in a neural network. Over time, the model learns what kind of question a good Query should ask, and what kind of information a good Key/Value should expose, purely from data — nobody hand-designs these.
DBA analogy: this is like an optimizer’s statistics getting refreshed over time — a fresh database has generic default statistics, but as more query workload runs against it, the optimizer “learns” better cardinality estimates for your actual data patterns. Wq, Wk, Wv are the model’s equivalent of continuously refined statistics — except here, “training” is the refresh process.
One More Analogy: The Same Data, Transformed for Different Purposes
Here’s a nice real-world parallel from the notes: imagine your own resume/profile data (name, DOB, address, phone, job history). That same underlying information gets transformed differently depending on where it’s going:
- Transformed into a LinkedIn profile (name, DOB, address, photo)
- Transformed for a job search (skills, experience)
- Transformed for job matching (relevant skills only: Gen AI, Data Science, ML)
Same source data, three different “views,” each optimized for its consumer. That’s conceptually what Wq, Wk, and Wv do to the same embedding — one matrix reshapes it into “what to look for” (Query), another into “what I offer to be found by” (Key), and another into “what I actually hand over once matched” (Value).
Common Points of Confusion
“Why can’t we just reuse the same vector for Q, K, and V?” Because a word playing all three roles with an identical vector limits what the model can express — it can’t simultaneously optimize “how I search,” “how I get found,” and “what I actually contribute” if they’re forced to be the same number. Separate learnable projections give the model three independent degrees of freedom to work with.
“Isn’t a plain sequence number simpler than sine/cosine for positional encoding?” A discrete integer (1, 2, 3, …) is unbounded and doesn’t generalize well to unseen sequence lengths, and directly feeding raw integers into a neural network can destabilize training. Sine/cosine values stay bounded between -1 and +1 and naturally encode relative distance between positions.
Key Takeaways
✅ Transformers process tokens in parallel, so they need positional encoding to recover word order — it’s added to, not concatenated with, the embedding ✅ Positional encoding uses sine (even dimensions) and cosine (odd dimensions) waves so every position gets a unique, bounded, generalizable signal ✅ Self-attention’s job is to compute a contextual embedding for each word based on its relationship to every other word in the sentence ✅ Self-attention needs three separate vectors per word — Query, Key, and Value — produced by three separate learnable weight matrices ✅ The core formula is softmax((Q·Kᵀ)/√d_k) · V — score, scale, normalize, then blend ✅ Q, K, V weight matrices are learned during training, not hand-designed
References
- Attention Is All You Need (Transformer paper) — https://arxiv.org/pdf/1706.03762
- The Illustrated Transformer (blog) — https://jalammar.github.io/illustrated-transformer/
- Sentence Transformers — https://huggingface.co/sentence-transformers
Found this helpful? Share it with your team! Questions? Drop them in the comments below.