Introduction
If you’ve ever tried to learn how large language models (LLMs) work, you’ve probably hit a wall of jargon — embeddings, tokens, self-attention, encoders, decoders — thrown at you all at once with no map of how they fit together. In this post, I’ll walk through the actual evolution of these ideas, from plain neural networks to the modern Transformer architecture, and explain why each piece exists. Since a lot of us come from a database background, I’ll lean on DBA analogies (indexing, query plans, joins, context-aware lookups) wherever they make the concept click faster.
What you’ll learn:
- Why “embeddings” are just numbers that carry meaning — and why that matters
- How we got from RNNs to Transformers (and why each step was a fix for the previous one’s weakness)
- What self-attention actually does, with worked examples
- The difference between encoder models (BERT) and decoder models (GPT)
- How to pick a good embedding model for your own projects
Prerequisites
Before you dive in, it helps to have:
- [ ] A basic idea of what a neural network is (input layer → hidden layer → output layer)
- [ ] Familiarity with the general concept of NLP tasks (classification, summarization, translation)
- [ ] No coding required for this post — it’s conceptual
It All Starts With One Problem: Computers Don’t Understand Words
Neural networks only understand numbers. So the very first challenge in NLP was: how do you turn a word like "bank" into a number that a neural network can process?
The earliest answer was Word2Vec — a neural network trained not to be used directly, but so that its hidden layer could be extracted as a numeric representation of a word. That hidden layer output is what we call an embedding: a vector of numbers that captures some of the meaning of the input.
DBA analogy: think of an embedding like a composite index key built from multiple columns. On its own it’s just a set of numbers, but it’s constructed so that “similar” rows end up with “similar” keys — which is exactly what lets a similarity search work instead of a full table scan.
Embeddings aren’t limited to single words. The same idea extends to:
- Word-level embeddings
- Sentence-level embeddings
- Paragraph/document-level embeddings
- Image embeddings
- Audio embeddings
- Video embeddings
Whatever the input type, the goal is the same: convert data into meaningful numbers.
From Encoding to Embeddings: A Quick Comparison
Not all numeric representations are equal. Here’s how the three generations stack up:
| Feature | Encoding (One-hot / TF-IDF) | Neural Network Embedding (Word2Vec) | Transformer Embedding (OpenAI, Gemini, Sentence Transformers) |
|---|---|---|---|
| Meaning capture | No semantics | Some semantics | Deep contextual semantics |
| Context awareness | None | Limited (window-based) | Full context (bidirectional) |
| Vector type | Sparse | Dense | Dense |
| Same word, same embedding? | Always the same vector | Always the same vector | Different per context |
| Performance | Weak | Good | Best |
The key upgrade from Word2Vec to Transformer embeddings is context awareness — and that’s worth its own section, because it’s the single biggest idea in this whole field.
DBA analogy: one-hot/TF-IDF encoding is like a raw LIKE '%keyword%' search — no understanding, just presence/absence. Word2Vec is like a static lookup table that never changes regardless of context. A Transformer embedding is like a query optimizer that re-evaluates the execution plan based on the actual data distribution at runtime — same query, different plan, depending on context.
Why Context Matters: The “Bank” Problem
Consider two sentences:
- S1: “Money bank grows”
- S2: “River bank flows”
With Word2Vec, the word “bank” gets the exact same vector in both sentences — because Word2Vec doesn’t look at the surrounding context deeply enough to distinguish “financial institution” from “riverbank.” That’s a real limitation.
This is where self-attention comes in:
Self-attention = a word looking at other other words in the same sentence to understand its own meaning.
In S1, “bank” pulls meaning from “money” and “grows” → financial institution. In S2, “bank” pulls meaning from “river” and “flows” → geographic feature.
The same mechanism resolves other classic ambiguities too — like “Apple” the company vs. “apple” the fruit, disambiguated purely by which other words appear nearby in the sentence.
DBA analogy: this is like a correlated subquery instead of a static lookup. Every row (word) doesn’t just get a fixed value — it re-evaluates itself against the other rows in the same result set before producing its final value. That’s exactly why Transformer embeddings are called dynamic embeddings: same word, different vector, depending on context — unlike Word2Vec’s static, always-the-same vector.
The Timeline: How We Actually Got to Transformers
Understanding why the Transformer architecture looks the way it does is easier once you see the sequence of fixes that led to it:
- RNN (Recurrent Neural Network) — processes sequence data (like text) step by step, one token at a time.
- 2014–15: LSTM / GRU — “gated” versions of RNNs designed to fix RNN’s short-term memory problem, giving it long-short term memory.
- 2015: Encoder-Decoder model — introduced the concept of an encoder (reads the input) and a decoder (produces the output), the foundation of machine translation.
- 2016: Encoder-Decoder with Attention — added an attention mechanism so the decoder could “look back” at relevant parts of the input instead of relying on a single compressed context vector.
- 2017–18: ULMFiT — introduced the pretrain-then-fine-tune pattern that’s now standard in NLP: train a large language model on huge amounts of data first, then fine-tune it for a specific task.
- 2017–18: The Transformer (“Attention Is All You Need”) — replaced recurrence entirely with self-attention, allowing the model to process all tokens in parallel instead of sequentially.
DBA analogy: RNNs are like processing a transaction log sequentially, row by row, in order — slow, and by the time you’re at row 10,000, you’ve forgotten what happened at row 1 (the “vanishing gradient” problem). Transformers are like a parallel, distributed query engine that reads the whole dataset at once and can relate any row to any other row directly, regardless of distance — no information gets lost due to sequence length.
RNN/LSTM vs Transformer at a Glance
| Feature | RNN/LSTM | Transformer |
|---|---|---|
| Processing style | Sequential (step by step) | Parallel (all tokens at once) |
| Speed | Slow | Fast |
| Long-term dependency | Limited (vanishing gradient issue) | Very strong |
| Architecture | Gates: input, forget, output | Attention-based (self-attention) |
This parallelism is also why Transformers scale so well with huge datasets — you can throw enormous compute at them because there’s no step-by-step bottleneck. That scalability is a big part of why we went from small task-specific models to today’s massive large language models (LLMs) trained on huge, diverse datasets.
Inside the Transformer: What Actually Happens to Your Text
Here’s the pipeline your text goes through inside a Transformer, from raw sentence to output:
- Tokenization — break the input into tokens (e.g.,
"Sunny","is","a","mentor") - Embedding — convert each token into a dense numeric vector (Word2Vec-style starting point)
- Positional Encoding (PE) — since the model processes tokens in parallel (not sequentially), it needs an explicit signal for word order. Positional encoding injects that “this token is 3rd in the sentence” information into the embedding.
- Self-Attention (Multi-Head Attention) — each token looks at every other token and re-weighs its own meaning based on context, using Query (Q), Key (K), and Value (V) matrices.
- Add & Layer Normalization — stabilizes and normalizes the output at each step (a residual connection + normalization).
- Feed Forward Network (FFNN) — a standard neural network layer applied on top of the attention output.
- Repeat — the whole encoder block is stacked N times (commonly 6) for progressively richer representations.
DBA analogy for positional encoding: because a Transformer processes all tokens in parallel (unlike an RNN which naturally processes in order), it loses the “row order” that a sequential scan would give it for free. Positional encoding is like explicitly storing a sequence_number column so that even a parallel, unordered scan can reconstruct correct ordering afterward.
Encoder vs Decoder — Two Halves, Two Jobs
The original Transformer paper has two stacked halves:
Encoder side (reads and understands input):
- Input sequence → Input embedding → Positional encoding
- Self-attention (multi-head)
- Add & Layer Normalization
- Feed Forward Network
- Residual connection + normalization
- Repeat block N times
Decoder side (generates output, one token at a time):
- Output sequence (shifted right) → Output embedding → Positional encoding
- Masked multi-head attention (can only see previous tokens, not future ones)
- Add & Layer Normalization
- Cross-attention — this is where the decoder looks back at the encoder’s output
- Add & Layer Normalization
- Feed Forward Network
- Linear layer → Softmax → final output probabilities
- Repeat block N times
DBA analogy: think of the encoder as a fully-materialized view built once over your entire input dataset — every row can see every other row. The decoder is more like a cursor-based process generating output row by row, where each new row can only reference rows already committed (masked attention = “you can’t reference a row that doesn’t exist yet”), plus it can always JOIN back against that materialized view (cross-attention).
This split is also why two very different families of models exist today:
- BERT (Google, 2018) uses only the encoder half, trained with Masked Language Modeling (MLM) — hide a word in the middle of a sentence and predict it using context from both directions. Great for understanding tasks (classification, embeddings).
- GPT (OpenAI) uses only the decoder half, trained with autoregressive, next-token prediction — given everything so far, predict the next token. Great for generation tasks (chat, text generation, summarization).
Choosing the Right Embedding Model
If you’re building anything with embeddings (search, RAG, recommendation), here’s the practical checklist from the notes:
1. Embedding quality (most important) Check benchmark leaderboards before trusting marketing claims:
- MTEB leaderboard
- BEIR benchmark (arxiv.org/abs/2104.08663)
2. Dimensionality
- 384 dimensions → lightweight
- 768 dimensions → balanced
- 1536 dimensions → high quality, but heavier
Higher dimensions generally mean better semantic representation, but at the cost of more storage, more memory usage, and slower search — the classic space/speed tradeoff.
3. Cost
- Closed-source (OpenAI, Gemini embeddings) → easy to use, high quality, but you pay per API call
- Open-source (e.g., all-MiniLM, Sentence Transformers) → free, but you carry the infrastructure and scaling cost yourself
4. Domain suitability General-purpose models (all-MiniLM, OpenAI embeddings, Gemini embeddings) work well for broad use cases, but a domain-specific fine-tuned model can outperform them for narrow, specialized data.
DBA analogy: picking an embedding model is a lot like picking an index strategy — more dimensions is like a wider composite index (better selectivity, more storage and maintenance overhead); a hosted API embedding is like a managed cloud database (less ops work, ongoing cost); a self-hosted open-source model is like running your own instance (free license, but you own the capacity planning).
Key Takeaways
✅ Embeddings turn any type of data (text, image, audio, video) into meaningful numbers a model can work with ✅ Word2Vec embeddings are static (same word, same vector); Transformer embeddings are dynamic (same word, different vector, depending on context) ✅ Self-attention is what makes context-awareness possible — every word looks at every other word before deciding its own meaning ✅ Transformers replaced RNN/LSTM’s sequential processing with parallel processing, enabling massive scale ✅ Encoder-only models (BERT) are built for understanding; decoder-only models (GPT) are built for generation ✅ When choosing an embedding model, weigh quality (MTEB/BEIR benchmarks), dimensionality, cost, and domain fit — not just brand name
What’s Next
This post is part of the GenAI Foundations series:
| # | Post | Status |
|---|---|---|
| 1 | What is Generative AI? Complete Beginner Guide | ⬜ Coming |
| 2 | How Transformers and Embeddings Actually Work | 📍 You are here |
| 3 | ChatGPT vs Claude vs Gemini — Honest Comparison | ⬜ Coming next week |
👉 Next up: Embeddings Explained — From Text to Vectors (Series 3)
References
- Attention Is All You Need (Transformer paper) — https://arxiv.org/pdf/1706.03762
- Sequence to Sequence Learning (Encoder-Decoder paper) — https://arxiv.org/pdf/1409.3215
- Neural Machine Translation by Jointly Learning to Align and Translate (Encoder-Decoder with Attention) — https://arxiv.org/pdf/1409.0473
- Universal Language Model Fine-tuning (ULMFiT) — https://arxiv.org/pdf/1801.06146
- The Illustrated Transformer (blog) — https://jalammar.github.io/illustrated-transformer/
- BEIR Benchmark — https://arxiv.org/abs/2104.08663
- Sentence Transformers — https://huggingface.co/sentence-transformers
Found this helpful? Share it with your team! Questions? Drop them in the comments below.