🤖 GenAI from Scratch — Post 6 of 24 | Module 1: NLP Foundations

← Post 5: Word2Vec and Embeddings Explained Post 7: Transformers and Attention — The SOTA Era →

In the last post you saw what Word2Vec does — it turns words into dense vectors where KING − MAN + WOMAN ≈ QUEEN. But we treated the model as a black box.

Today we open the box. By the end of this post you’ll understand word2vec CBOW and Skip-gram training from raw text all the way to the final embedding matrix — including the single most important insight in this whole module:

The embeddings ARE the hidden layer weights. Word2Vec is a neural network trained on a fake prediction task — and after training, we throw the prediction away and keep the weights.

What you’ll learn:

A 5-minute neural network crash course (perceptron → MLP → forward/backward propagation)
How vocabulary and window size turn raw text into training pairs
CBOW vs Skip-gram data preparation, step by step, with the exact class examples
The full Word2Vec architecture walkthrough: 5×3 and 3×5 weight matrices
Why BoW/TF-IDF dimension = vocab size, but Word2Vec dimension = hidden layer size
Training your own custom Word2Vec with gensim — validated in VS Code

📋 Table of Contents

The Story So Far — Encoding vs Embedding
Neural Network Crash Course (Perceptron → MLP)
Forward Propagation, Loss, Backpropagation, Epochs
Step 1: Build the Vocabulary
Step 2: Window Size — The Context Hyperparameter
Step 3: CBOW vs Skip-gram Data Preparation
Step 4: Inside the Network — Where Embeddings Live
Dimensions: Vocab Size vs Hidden Layer Size
Hands-On: Train Your Own Word2Vec with gensim
Word2Vec Pros and Cons (Interview Favourite)
From Static to Dynamic — Why Transformers Won
Common Errors and Fixes
Key Takeaways

Prerequisites

✅ Read Post 3 — OHE, BoW and TF-IDF (we build directly on One-Hot Encoding)
✅ Read Post 5 — Word2Vec and Embeddings intro (KING−MAN+WOMAN)
✅ Python 3.11 with uv installed (setup from Post 1)
✅ VS Code or any editor to run the code samples

Lab Environment

Component	Version/Details
Operating System	Windows 11 / Ubuntu 22.04
Python	3.11 (CPython, managed by uv)
gensim	4.3.x
Editor	VS Code

1. The Story So Far — Encoding vs Embedding

Quick recap of the four techniques we’ve covered, in one line each. This exact framing came up in class as an interview answer:

Technique	Answers the question	Type
One-Hot Encoding	“Is the word present or not?” (0 or 1)	Encoding
Bag of Words	“How many times does the word appear?” (count)	Encoding
TF-IDF	“How important is the word in this document?” (count + importance)	Encoding
Embeddings (Word2Vec →)	“What is the meaning of the text in context?”	Embedding

📌 Interview one-liner from class

OHE, BoW, TF-IDF and BM25 are encoding — direct calculations, no training. Embeddings come from a trained model (a neural network or transformer) and carry semantic meaning. Word2Vec (Google, 2013) is the bridge between the two worlds.

Remember the failure case that motivates everything today:

Sentence 1: "I like this movie"
Sentence 2: "I love this film"

OHE / BoW / TF-IDF:   "like" ≠ "love",  "movie" ≠ "film"   → unrelated
Embeddings:           "like" ≈ "love",  "movie" ≈ "film"   → model understands similarity

2. Neural Network Crash Course (Perceptron → MLP)

Word2Vec is a neural network, so we need just enough NN theory to read its architecture diagram. Class 06 used a home-buying prediction example.

The Single Perceptron

Imagine predicting whether someone buys a home — a binary classification (Y/N) from three features:

F1 (Size)	F2 (Location)	F3 (Bedrooms)	Output (Y/N)
1250 sq ft	City center	3	Y
1140 sq ft	Outskirts	2	N

A single perceptron takes the inputs, multiplies each by a weight (w), adds a bias (b), sums everything (Σ), and passes the result through an activation function:

Output = Act( W·x + b )

I/P ──w──▶ [ Σ | Act ] ──▶ O/P
             ▲
             b (bias)

From Perceptron to Neural Network

Stack perceptrons into layers and you get a neural network. Every NN has exactly three kinds of layers:

Input layer — receives your features
Hidden layer(s) — learn intermediate representations
Output layer — produces the prediction

A network with multiple hidden layers is a Multi-Layer Perceptron (MLP). Keep this in mind — because Word2Vec’s most surprising design choice is that it has just one hidden layer.

3. Forward Propagation, Loss, Backpropagation, Epochs

Training any neural network — including Word2Vec — is a loop of four ideas from the class whiteboard:

Forward Propagation (FP):   FP = Act(Wx + b), layer by layer → prediction
Loss:                       Loss = Output(Prediction) − Actual
Backward Propagation (BP):  BP → Optimizer → Gradient Descent → adjust the weights
1 Epoch:                    FP + BP over the data = 1 epoch

Weights → initialised with RANDOM values
Learning → epoch after epoch → Loss ↓↓↓↓ → BEST POSSIBLE WEIGHTS

🗄️ DBA Analogy — Training = Query Tuning Iterations

Random initial weights are like a fresh database with no statistics — the optimizer’s first execution plan is a guess. Each epoch is like gathering stats and re-running the workload: the plan (weights) gets adjusted until response time (loss) stops improving. “Best possible weights” = your stable, tuned execution plan.

Hold on to the phrase “best possible weights”. In Word2Vec, those final weights are not a means to an end — they are literally the product we ship.

4. Step 1: Build the Vocabulary

Word2Vec training is a 4-step data preparation pipeline. Step 1 is the same as OHE/BoW: extract the unique words (a set) from the data.

Data:  "I love AI and I love coding"

Vocab → unique words (set of words)
Vocab = { "I", "love", "AI", "and", "coding" }     # 5 unique words

One subtle detail from class — vocabulary is case-sensitive by default:

Data:  "Sunny love AI sunny love coding"

Vocab = { "Sunny", "love", "AI", "sunny", "coding" }   # 5 words!
# "Sunny" (capital S) and "sunny" (small s) are DIFFERENT tokens
# Same for "Apple" vs "apple" — this is why lowercasing is a
# standard preprocessing step before training

5. Step 2: Window Size — The Context Hyperparameter

Window size = how many neighbours we consider around the word. It defines what “context” means for training.

Window Size = 1

Sentence: "I love deep learning" — each word looks 1 position left and 1 position right:

Context (Input)	Target (Output)
love	I
I, deep	love
love, learning	deep
deep	learning

Window Size = 2

Same sentence, but now each word looks 2 positions in each direction — more context per pair:

Context (Input)	Target (Output)
love, deep	I
I, deep, learning	love
I, love, learning	deep
love, deep	learning

⚠️ Window size is a hyperparameter — experiment!

Typical values to try: 5, 10, 15, 20… But beware the trade-off from class: window size ↑↑↑ → context ↑↑↑ → but if it goes very, very high → NOISE. Distant words stop being meaningful context and start polluting the training pairs.

6. Step 3: CBOW vs Skip-gram Data Preparation

Here’s the key reframe from Class 06: CBOW and Skip-gram are not two different models — they are two ways of preparing the training data for the same neural network.

                       Word2Vec
                (based on data preparation)
                    /              \
              CBOW                  Skip-gram
    (Continuous Bag of Words)
    Context → predict TARGET      Target → predict CONTEXT

Class example sentence: "Follow Sunny for Gen AI", window size = 1.

CBOW — Context In, Target Out

Input (Context)	Output (Target)
Sunny	Follow
Follow, for	Sunny
Sunny, Gen	for
for, AI	Gen
Gen	AI

Skip-gram — Target In, Context Out

Input (Target)	Output (Context)
Follow	Sunny
Sunny	Follow, for
for	Sunny, Gen
Gen	for, AI
AI	Gen

Notice the tables are mirror images — same pairs, flipped direction. This is supervised learning: we manufactured X (input) and Y (output) columns from raw, unstructured text. No human labelling required.

The Full 4-Step Data Preparation Pipeline

Data: "Follow Sunny for Gen AI"

Step 1 → Vocab:        { AI, Follow, for, Gen, Sunny }        # sorted set
Step 2 → One-Hot each word (5-dim vectors, one per vocab word):
           AI     → [1 0 0 0 0]
           Follow → [0 1 0 0 0]
           for    → [0 0 1 0 0]
           Gen    → [0 0 0 1 0]
           Sunny  → [0 0 0 0 1]
Step 3 → Window size = 1
Step 4 → Build (X, Y) training pairs (CBOW or Skip-gram)

💡 Full-circle moment: One-Hot Encoding — the “obsolete” technique from Post 3 — is exactly how words enter the Word2Vec network. OHE isn’t dead; it moved backstage.

7. Step 4: Inside the Network — Where Embeddings Live

Now the whiteboard diagram that ties everything together. Training pair (CBOW): context ["Follow", "for"] → target "Sunny". Vocab size = 5, hidden layer = 3 neurons.

INPUT (OHE, 5-dim)        HIDDEN (3 neurons)        OUTPUT (5-dim, softmax)

Follow [0 1 0 0 0] ──┐
                     ├──[ W1: 5×3 ]──▶ (h1 h2 h3) ──[ W2: 3×5 ]──▶ [p1 p2 p3 p4 p5]
for    [0 0 1 0 0] ──┘        ▲                            │
                              │                            ▼
                        EMBEDDING                 Predicted word vs
                      (weight matrix)             Expected word: "Sunny"
                                                          │
   ◀──────────── Backpropagation: adjust weights ◀── Loss ┘

FP + BP, epoch after epoch → loss ↓ → BEST POSSIBLE WEIGHTS
                                       = the EMBEDDING VECTORS

Walk through one forward pass:

Each context word enters as a one-hot vector (1×5)
It’s multiplied by the input weight matrix W1 (5×3) — pure matrix multiplication (vector × matrix)
The hidden layer holds a 3-dim representation
Hidden output is multiplied by W2 (3×5) to produce 5 scores — one per vocab word
The predicted word is compared with the expected word (“Sunny”) → loss
Backpropagation adjusts both weight matrices; repeat until loss is minimised

🔑 THE key insight of Class 06

After training, we don’t care about the prediction at all. We keep W1, the input weight matrix. Because each word’s one-hot vector has a single 1, multiplying it by W1 simply selects one row of the matrix. That row is the word’s embedding. The 5×3 weight matrix IS the embedding table: 5 words × 3 dimensions. The prediction task was just a training excuse.

And this is why Word2Vec is fast: it’s a shallow model — one hidden layer, simple architecture. Google’s production model was exactly this, scaled up: 1 hidden layer with 300 neurons.

8. Dimensions: Vocab Size vs Hidden Layer Size

This comparison from the class notes clears up a confusion almost everyone has:

Technique	Vector dimension =	Controlled by
BoW / TF-IDF	Vocabulary size (number of features)	Your data — grows with vocab (10k, 50k, 100k…)
Word2Vec	Hidden layer size	You — it’s a hyperparameter (3, 10, 100, 300, 500, 1000…)

Google’s pretrained Word2Vec chose 300 dimensions — every word becomes a 300-feature vector regardless of whether the vocabulary has 5 words or 3 million. That’s the escape from the high-dimensionality curse of BoW/TF-IDF: dense, compact, fixed-size vectors.

9. Hands-On: Train Your Own Word2Vec with gensim

Two practical routes from class: use Google’s pretrained model (Post 5) or do custom training on your own data. Today: custom training, where you control every hyperparameter we just learned.

Setup

# In your project folder (uv setup from Post 1)
uv init word2vec-lab
cd word2vec-lab
uv add gensim

[SCREENSHOT: VS Code terminal showing uv add gensim completing successfully]

File: `train_word2vec.py`

"""
Custom Word2Vec training — Class 06 concepts in code.
Every hyperparameter below maps to something from the whiteboard.
"""
from gensim.models import Word2Vec

# ── Step 1: Data (already tokenised, lowercased to avoid the
#            "Sunny" vs "sunny" case-sensitivity trap) ────────────
sentences = [
    ["follow", "sunny", "for", "gen", "ai"],
    ["i", "love", "ai", "and", "i", "love", "coding"],
    ["i", "love", "deep", "learning"],
    ["sunny", "teach", "ai"],
    ["people", "watch", "movie"],
    ["people", "like", "movie"],
    ["ai", "is", "the", "future"],
]

# ── Step 2: Train — the 4-step pipeline happens inside ────────────
model = Word2Vec(
    sentences   = sentences,
    vector_size = 3,      # HIDDEN LAYER SIZE = embedding dims (Google: 300)
    window      = 1,      # WINDOW SIZE from the class tables
    min_count   = 1,      # keep every word in the vocab
    sg          = 0,      # 0 = CBOW (context→target), 1 = Skip-gram
    epochs      = 200,    # FP + BP loops — loss ↓ each epoch
    seed        = 42,
)

# ── Step 3: Inspect the vocabulary (Step 1 of the pipeline) ──────
print("Vocab size:", len(model.wv.key_to_index))
print("Vocab:", sorted(model.wv.key_to_index))

# ── Step 4: THE EMBEDDING MATRIX = hidden layer weights ──────────
print("\nEmbedding matrix shape:", model.wv.vectors.shape)
# → (vocab_size, 3)  — exactly the W1 matrix from the diagram!

# ── Step 5: One word = one ROW of that matrix ────────────────────
print("\nVector for 'ai':", model.wv["ai"].round(3))
print("Vector for 'sunny':", model.wv["sunny"].round(3))

# ── Step 6: Semantic neighbours ──────────────────────────────────
print("\nMost similar to 'ai':")
for word, score in model.wv.most_similar("ai", topn=3):
    print(f"  {word:10s}  {score:.3f}")

Run it

uv run train_word2vec.py

# Expected output (values vary — weights start RANDOM, remember?):
Vocab size: 20
Vocab: ['ai', 'and', 'coding', 'deep', 'follow', 'for', 'future', ...]

Embedding matrix shape: (20, 3)

Vector for 'ai': [ 0.217 -0.041  0.294]
Vector for 'sunny': [-0.166  0.089  0.312]

Most similar to 'ai':
  love        0.912
  sunny       0.847
  learning    0.803

[SCREENSHOT: VS Code output showing embedding matrix shape (20, 3) and similarity results]

Experiment Like the Class Told You To

# Try Skip-gram instead of CBOW:
sg = 1

# Try a bigger hidden layer (more expressive, needs more data):
vector_size = 100

# Try a wider window (more context — but remember: too high = noise):
window = 5

With a 7-sentence toy corpus, results are rough — that’s the honest lesson. Word2Vec’s quality comes from scale. Google trained on ~100 billion words of Google News to get its famous 300-dim vectors.

10. Word2Vec Pros and Cons (Interview Favourite)

✅ Pros

Efficient & fast training — shallow model, no deep layers (1 hidden layer!)
Semantic understanding — similar meaning words → close vectors (king ≈ queen, car ≈ vehicle). Captures similarity AND relationships
Vector arithmetic — king − man + woman = queen, demonstrable in code
Pretrained embeddings available — Google News vectors (300d), easy to reuse
General-purpose — usable in NLP pipelines, ML models, clustering, and search

❌ Cons

Context-independent (static) — same word = same vector, always. bank (river) = bank (finance). The vector will not change according to the sentence context
Fixed embeddings — trained once, frozen forever
No OOV handling — a new word like “ChatGPT” (born after 2013) → unknown to the Google model
Cannot handle subwords — running, runner, runs are treated as three unrelated words; no morphology understanding
Compared to BERT/GPT — no attention, no deep context, no dynamic embeddings

🗄️ DBA Analogy — Static vs Dynamic Embeddings

A Word2Vec embedding is like a materialized view built once and never refreshed — great snapshot, but it can’t reflect what’s happening in the current query. Transformer embeddings are like a live view — recomputed per sentence, so “bank” gets a different vector next to “river” than next to “finance”.

11. From Static to Dynamic — Why Transformers Won

The Class 06 roadmap positions everything we’ve learned:

1. OHE          ─┐
2. BoW           ├── Encoding (no training, direct calculation)
3. TF-IDF       ─┘
4. Word2Vec     ──── Classical embedding (Google 2013, shallow NN)
                     ↳ CBOW & Skip-gram = interview concept
                     ↳ Custom training  = practical skill (done today ✅)
5. Transformers ──── SOTA (state of the art)
                     ↳ HF Sentence Transformers, OpenAI embeddings,
                       Gemini embeddings — the models behind RAG

Two more facts from class worth remembering:

Transformers still use embeddings internally — every token gets an embedding as the first step. Word2Vec’s core idea never died; it became layer zero of every LLM.
LLM vs MMLLM — LLMs like Llama are text-to-text; Multi-Modal LLMs handle image-to-text and text-to-image. All of them start by embedding their input.

Next post: the transformer era — attention, contextual embeddings, and calling real embedding models (Hugging Face free tier, OpenAI, Gemini) from Python.

12. Common Errors and Fixes

Error 1: gensim fails to install or import (NumPy conflict)

Symptom:

ImportError: numpy.core.multiarray failed to import
# or during install:
error: Failed to build `gensim`

Cause: gensim compiled against an older NumPy; very new Python versions (3.12/3.13) may lack prebuilt wheels.

Fix: pin Python 3.11 (as our lab does) and let uv resolve compatible versions:

uv python pin 3.11
uv add "gensim>=4.3" "numpy<2"
uv run train_word2vec.py

Error 2: KeyError when looking up a word

Symptom:

KeyError: "Key 'chatgpt' not present in vocabulary"

Cause: This is the OOV problem from the cons list, live in your terminal — the word never appeared in training data (or appeared fewer than min_count times, or with different casing: “AI” vs “ai”).

Fix: guard your lookups and normalise case:

word = "ChatGPT".lower()
if word in model.wv:
    print(model.wv[word])
else:
    print(f"'{word}' is OOV — not in the training vocabulary")

Error 3: Every training run gives different vectors

Symptom: You rerun the script and ‘ai’ has a completely different vector.

Cause: Not a bug — weights are initialised randomly (Section 3!), and gensim uses multiple worker threads, which adds nondeterminism.

Fix: for reproducible demos, set seed=42 and workers=1. The relative geometry (which words are near which) is what matters, not the raw numbers.

Key Takeaways

✅ Word2Vec is a shallow neural network — one hidden layer — trained with the standard FP → Loss → BP → epoch loop starting from random weights.

✅ CBOW and Skip-gram are data-preparation strategies, not different models: CBOW predicts target from context; Skip-gram predicts context from target.

✅ Window size is the context hyperparameter — bigger window = more context, but too big = noise.

✅ The embeddings ARE the hidden layer weight matrix (W1) — the prediction task is thrown away after training; each word’s vector is one row of W1.

✅ BoW/TF-IDF dimension = vocab size (grows with data); Word2Vec dimension = hidden layer size (you choose it — Google chose 300).

🧠 Test Your Knowledge

20 questions on Word2Vec training, CBOW vs Skip-gram, window size and embeddingsTake the Free Quiz →

Instant feedback · Detailed explanations · Free

What’s Next in the Series

#	Post	Status
4	Word2Vec and Embeddings Explained (KING−MAN+WOMAN)	✅ Published
5	Word2Vec in Practice — Pretrained Models	✅ Published
6	Inside Word2Vec: CBOW, Skip-gram & Neural Network Training	📍 You are here
7	Transformers and Attention — The SOTA Era	⬜ Coming next week

👉 Next Post: Transformers and Attention — The SOTA Era →

References

Efficient Estimation of Word Representations in Vector Space — Mikolov et al., 2013
gensim Word2Vec documentation
Full-Stack GenAI Bootcamp v1.0 — Class 06 (11 April), Instructor: Sunny Savita

Found this helpful? Share it with your team!
Questions? Drop them in the comments below.
Part of the GenAI from Scratch series — published every Friday at gradeupnow.in/genai-blog/

🤖 GenAI from Scratch — Post 6 of 24 | Module 1: NLP Foundations | Class 06 (11 April)

Prerequisites

Lab Environment

1. The Story So Far — Encoding vs Embedding

2. Neural Network Crash Course (Perceptron → MLP)

The Single Perceptron

From Perceptron to Neural Network

3. Forward Propagation, Loss, Backpropagation, Epochs

4. Step 1: Build the Vocabulary

5. Step 2: Window Size — The Context Hyperparameter

Window Size = 1

Window Size = 2

6. Step 3: CBOW vs Skip-gram Data Preparation

CBOW — Context In, Target Out

Skip-gram — Target In, Context Out

The Full 4-Step Data Preparation Pipeline

7. Step 4: Inside the Network — Where Embeddings Live

8. Dimensions: Vocab Size vs Hidden Layer Size

9. Hands-On: Train Your Own Word2Vec with gensim

Setup

File: `train_word2vec.py`

Run it

Experiment Like the Class Told You To

10. Word2Vec Pros and Cons (Interview Favourite)

✅ Pros

❌ Cons

11. From Static to Dynamic — Why Transformers Won

12. Common Errors and Fixes

Error 1: gensim fails to install or import (NumPy conflict)

Error 2: KeyError when looking up a word

Error 3: Every training run gives different vectors

Key Takeaways

What’s Next in the Series

References

Leave a Comment Cancel Reply

Prerequisites

Lab Environment

1. The Story So Far — Encoding vs Embedding

2. Neural Network Crash Course (Perceptron → MLP)

The Single Perceptron

From Perceptron to Neural Network

3. Forward Propagation, Loss, Backpropagation, Epochs

4. Step 1: Build the Vocabulary

5. Step 2: Window Size — The Context Hyperparameter

Window Size = 1

Window Size = 2

6. Step 3: CBOW vs Skip-gram Data Preparation

CBOW — Context In, Target Out

Skip-gram — Target In, Context Out

The Full 4-Step Data Preparation Pipeline

7. Step 4: Inside the Network — Where Embeddings Live

8. Dimensions: Vocab Size vs Hidden Layer Size

9. Hands-On: Train Your Own Word2Vec with gensim

Setup

File: train_word2vec.py

Run it

Experiment Like the Class Told You To

10. Word2Vec Pros and Cons (Interview Favourite)

✅ Pros

❌ Cons

11. From Static to Dynamic — Why Transformers Won

12. Common Errors and Fixes

Error 1: gensim fails to install or import (NumPy conflict)

Error 2: KeyError when looking up a word

Error 3: Every training run gives different vectors

Key Takeaways

What’s Next in the Series

References

Leave a Comment Cancel Reply

File: `train_word2vec.py`