🤖 GenAI from Scratch — Post 7 of 24  |  Module 1: NLP Foundations  |  Class 07 (12 April)

← Post 6: Inside Word2Vec — CBOW, Skip-gram & NN Training Post 8: Sentence Transformers — The SOTA Era →

In Post 6 we opened the Word2Vec black box and saw the training loop on the whiteboard. Today we finish the story and run all of it for real. This is the most hands-on post in the series so far — two complete labs, both validated in VS Code:

1. Lab 1 — Custom training: train your own Word2Vec on two short stories with gensim, and watch it struggle (an honest, important lesson)
2. Lab 2 — Google’s pretrained model: load the famous Google News 300-dim vectors and prove king − man + woman ≈ queen with real numbers

What you’ll learn:

• Skip-gram data preparation and the one-pair-per-row trick
• What happens at prediction time — freezing the model and extracting embeddings
• The REAL CBOW vs Skip-gram difference (speed, accuracy, rare words) with the “king drank wine” example
• Full custom training pipeline: NLTK tokenising → gensim training → similarity queries
• Google News pretrained model: similarity, odd-one-out, vector arithmetic — with actual outputs
• Average Word2Vec — turning a whole sentence into one vector (the ancestor of sentence embeddings)

📋 Table of Contents

1. Skip-gram Data Preparation — One Pair Per Row
2. Training Walkthrough — Softmax, Loss, Global Minimum
3. Prediction Time — Freeze the Model, Keep the Weights
4. CBOW vs Skip-gram — The Real Difference
5. Lab 1: Custom Word2Vec Training with gensim
6. Lab 1 Results — Why a Small Corpus Fails (and Why That Matters)
7. Lab 2: Google News Pretrained Model (300d)
8. The KING − MAN + WOMAN Moment — For Real This Time
9. Average Word2Vec — From Word to Sentence Vectors
10. The Static Vector Wall — bank vs bank
11. Common Errors and Fixes
12. Your Assignment This Week
13. Key Takeaways

Prerequisites

• ✅ Read Post 6 — Inside Word2Vec (vocabulary, window size, CBOW/Skip-gram basics, the 5×3 architecture)
• ✅ Python 3.11 with uv (setup from Post 1)
• ✅ ~2 GB free disk space and a decent connection (the Google News model download is ~1.6 GB)

Lab Environment

Component	Version/Details
Operating System	Windows 11 / Ubuntu 22.04
Python	3.11 (CPython, managed by uv)
gensim	4.3.x
nltk	3.9.x
Pretrained model	word2vec-google-news-300 (~1.6 GB, Skip-gram)

1. Skip-gram Data Preparation — One Pair Per Row

Post 6 covered CBOW data prep in detail. Class 07 completed the picture with Skip-gram, using the same sentence: "Follow Sunny for Gen AI", window size = 1.

Skip-gram takes the target as input and predicts the context as output:

Input (Context word)	Output (Target)
Sunny	Follow, for
for	Sunny, Gen
Gen	for, AI

But here’s the practical detail from the class whiteboard — we can also represent this data one pair per row, and this is how it’s actually trained:

Input	Output
Sunny	follow
Sunny	for
for	Sunny
for	gen
gen	for
gen	AI

💡 Why this matters: one word with 2 context neighbours becomes 2 separate training samples. For every context word: separate prediction, separate loss. Skip-gram generates far more training pairs than CBOW from the same text — that’s exactly why it’s slower AND why it learns better representations. Hold this thought for Section 4.

2. Training Walkthrough — Softmax, Loss, Global Minimum

Let’s run one full training step exactly as drawn in class, for the CBOW pair (["Follow", "for"] → "Sunny"), vocab = 5, hidden = 3 neurons:

INPUT                    HIDDEN            OUTPUT (softmax probability)
Follow [0 1 0 0 0]──┐                      Predicted     Actual (Sunny)
                    ├─[5×3]──▶ (3) ──[3×5]──▶ [0.1]        [0]
for    [0 0 1 0 0]──┘                        [0.2]        [0]
                                             [0.05]       [0]
    vector × weight-matrix                   [0.15]       [0]
    = hidden output                          [0.5]  ◀──▶  [1]
    (no activation needed here)
                                    LOSS = Predicted − Actual
                                    Loss is HIGH at first → must go ↓↓↓

Key mechanics from the notes:

• Input layer — as many words as needed can pass in parallel; each word passes as a one-hot vector
• Hidden layer — there is only ONE hidden layer, but inside it we can have as many neurons as we want (it’s a hyperparameter: 3 for our demo, 300 for Google)
• vector × matrix = hidden output = the embedding — the input-to-hidden multiplication produces the word’s dense representation directly
• Softmax at the output converts raw scores into probabilities (formula: e^x / Σe^x) so we can compare against the one-hot actual
• Weights start random → each epoch = FP + BP → the optimizer walks the loss curve down to the global minimum → best possible weights with minimum loss

3. Prediction Time — Freeze the Model, Keep the Weights

Class 07 answered the question Post 6 left hanging: after training, how do we actually USE the model?

1. Training finishes → FREEZE the model (trained/learned weights)
2. A word comes in that we want to convert into a vector: "Sunny"
3. "Sunny" → one-hot vector → × trained weight matrix (dot product)
4. The hidden layer output IS the embedding
   → an INTERMEDIATE REPRESENTATION that is MEANINGFUL

Word ──▶ [ Frozen model ] ──▶ vector

🗄️ DBA Analogy — Frozen Model = Read-Only Standby

Training is like your primary database taking writes (weight updates every epoch). Once training ends, you open the model like a read-only standby — no more writes, just fast lookups. Every “query” (a word) returns its row from the learned weight matrix. That’s all inference is: an indexed read against frozen weights.

And one forward-looking note straight from the class: GPT and text-embedding models use this same concept — different angle, same idea. When you call a modern embedding API with “hi, how are you?”, a (much bigger) trained-and-frozen network converts your text into a meaningful vector you can use.

4. CBOW vs Skip-gram — The Real Difference

The interview-ready comparison table from Class 07:

Feature	CBOW	Skip-gram
Speed	Fast	Slow
Accuracy	Medium	High
Rare words	Weak	Strong
Data requirement	Less	More (relatively large data / research)
Input vectors	Multiple (averaged)	Single (direct)
Hidden layer	Average of contexts	Direct
Output	Single prediction	Multiple predictions

Why Skip-gram Wins on Rare Words — “The king drank wine”

This example from class is the clearest explanation of the difference you’ll find anywhere. Take the sentence "The king drank wine" and focus on the word “king”:

CBOW:
  (The, drank, wine) → king
  "king" is only an OUTPUT word
  → its embedding gets updated INDIRECTLY

Skip-gram:
  king → The
  king → drank        ◀── "king" is the INPUT three times
  king → wine
  → "king" is DIRECTLY trained, again and again
  → strong connection → stronger embedding

Think of a rare, information-dense term — the class used U-235 from the periodic table as the analogy. Rare elements matter precisely because they’re rare. If “U-235” appears only a handful of times in your corpus, CBOW barely touches its embedding (it’s just occasionally an output). Skip-gram puts it in the driver’s seat as an input every single time it appears, so even rare words get well-trained vectors.

One honest engineering footnote from the notes: conceptually Skip-gram performs a separate forward pass per context word, but in practice these operations are vectorized and batched on GPUs. Even so, Skip-gram remains slower than CBOW simply because it produces more training samples — more computation, no way around it.

📌 Interview one-liner

“CBOW and Skip-gram differ not only in data preparation but also in objective function, training dynamics, and learning behavior. CBOW predicts a target from averaged context — faster, but weaker on rare words. Skip-gram predicts multiple context words from a single target — more compute, but better representations, especially for rare words.” That’s a complete, senior-level answer.

5. Lab 1: Custom Word2Vec Training with gensim

Time to train our own model on real files — the exact pipeline from custom_word2vec_training.ipynb.

Setup

uv init word2vec-lab
cd word2vec-lab
uv add gensim nltk

# Create a data/ folder with story1.txt and story2.txt
# (any 2 short stories work — ours are about a boy named Arjun
#  learning AI, and farmers using technology)

File: custom_training.py

"""
Custom Word2Vec training — Lab 1 (Class 07).
Pipeline: raw .txt files → sentences → tokens → trained model
"""
import os
import gensim
import nltk
from nltk import sent_tokenize
from gensim.utils import simple_preprocess

nltk.download('punkt')
nltk.download('punkt_tab')

# ── Step 1: Read every file and build the corpus ─────────────────
folder_path = "./data"
story = []
for filename in os.listdir(folder_path):
    file_path = os.path.join(folder_path, filename)
    with open(file_path, encoding="unicode_escape") as f:
        corpus = f.read()
    raw_sent = sent_tokenize(corpus)          # split into sentences
    for sent in raw_sent:
        story.append(simple_preprocess(sent)) # lowercase + tokenize

print("Total sentences:", len(story))         # 44 in our two stories
print("First sentence:", story[0])

# ── Step 2: Define the model (every arg = a hyperparameter!) ─────
custom_model = gensim.models.Word2Vec(
    window      = 10,     # context neighbours each side
    min_count   = 1,      # keep every word (tiny corpus)
    vector_size = 150,    # hidden layer size = embedding dims
    # sg = 0 → CBOW (default) | sg = 1 → Skip-gram
)

# ── Step 3: Build vocabulary (Step 1 of the pipeline from Post 6)─
custom_model.build_vocab(story)
print("Vocab size:", len(custom_model.wv.index_to_key))   # 257
print("Corpus count:", custom_model.corpus_count)         # 44

# ── Step 4: Train — FP + BP × epochs ─────────────────────────────
custom_model.train(
    story,
    total_examples = custom_model.corpus_count,
    epochs         = 10,
)

# ── Step 5: Query the trained model ──────────────────────────────
print("\nVector for 'arjun' (first 5 dims):",
      custom_model.wv["arjun"][:5].round(4))
print("Dimensions:", len(custom_model.wv["arjun"]))       # 150

print("\nMost similar to 'ai':")
print(custom_model.wv.most_similar("ai"))

print("\nOdd one out of [arjun, engineer, programming, farmers]:")
print(custom_model.wv.doesnt_match(
    ["arjun", "engineer", "programming", "farmers"]))

print("\nSimilarity ai ↔ technology:",
      custom_model.wv.similarity("ai", "technology"))

uv run custom_training.py

6. Lab 1 Results — Why a Small Corpus Fails (and Why That Matters)

Here are the real outputs from our run — and they’re deliberately underwhelming:

Most similar to 'ai':
[('computer', 0.252), ('decision', 0.247), ('so', 0.240),
 ('the', 0.180), ('predictions', 0.150), ...]

Odd one out of [arjun, engineer, programming, farmers]:
'farmers'                          # ✓ actually correct!

Similarity ai ↔ technology:      -0.0436    # ✗ basically random
Similarity arjun ↔ engineer:     -0.0088    # ✗ basically random

Some green shoots — “ai” is closest to “computer” and “decision”, and the odd-one-out picked “farmers” correctly. But similarity scores near zero (or negative!) between obviously related words tell the truth:

⚠️ The honest lesson: neural networks and LLMs are DATA HUNGRY

44 sentences and 257 vocabulary words simply cannot teach a network what “technology” means. Google trained its model on millions of Google News articles. The scale IS the magic. Your custom model isn’t broken — it’s starving. This is the single most transferable insight to everything else in GenAI: model quality tracks data quantity and quality.

So what does a properly-fed Word2Vec look like? That’s Lab 2.

7. Lab 2: Google News Pretrained Model (300d)

The word2vec-google-news-300 model: trained on Google News (millions of articles), Skip-gram, hidden layer = 300 neurons → every word is a 300-dim vector. One line to download it via gensim’s downloader API:

File: pretrained_google.py

"""
Lab 2 — Google News pretrained Word2Vec (Class 07).
First run downloads ~1.6 GB, then it's cached locally.
"""
import gensim.downloader as api

model = api.load("word2vec-google-news-300")   # KeyedVectors

# ── Every word → 300 dimensions ─────────────────────────────────
print(len(model["sunny"]))                     # 300

# ── Semantic neighbours ─────────────────────────────────────────
print(model.most_similar("man"))
# [('woman', 0.766), ('boy', 0.682), ('teenager', 0.659), ...]

print(model.most_similar("cricket"))
# [('cricketing', 0.837), ('cricketers', 0.817),
#  ('Test_cricket', 0.809), ...]

# ── Similarity scores that finally make sense ───────────────────
print(model.similarity("man", "woman"))        # 0.766  ✓ related
print(model.similarity("man", "python"))       # 0.210  ✓ unrelated
print(model.similarity("man", "man"))          # 1.0    ✓ identical
print(model.similarity("python", "java"))      # 0.125

# ── Case sensitivity is real (remember Post 6!) ─────────────────
print(model.similarity("sunny", "Sunny"))      # 0.430 — different tokens!
print(model.most_similar("sunny"))
# [('balmy', 0.731), ('sunshine', 0.725), ('overcast', 0.692), ...]
# lowercase "sunny" = the WEATHER, not a person's name

# ── Odd one out ─────────────────────────────────────────────────
print(model.doesnt_match(["PHP", "JAVA", "DOG", "C++"]))   # 'DOG'

💡 Notice similarity("sunny", "Sunny") = 0.43 — the vocabulary case-sensitivity trap from Post 6, live in a production model. Lowercase “sunny” clusters with balmy, sunshine, overcast (weather!), while “Sunny” is a name. Same letters, different tokens, different meanings.

8. The KING − MAN + WOMAN Moment — For Real This Time

Posts 5 and 6 showed this analogy on the whiteboard with made-up feature values. Now we run it against 300 real learned dimensions:

vec = model["king"] - model["man"] + model["woman"]
print(model.most_similar([vec]))

# Actual output:
# [('king',   0.845),      ← the input word itself (expected)
#  ('queen',  0.730),      ← 🎯 THERE IT IS
#  ('monarch',0.645),
#  ('princess', 0.616),
#  ('crown_prince', 0.582),
#  ('prince', 0.578), ...]

Pure vector arithmetic on frozen weights recovers queen at 0.73 similarity — with monarch, princess and prince right behind. Nobody programmed royalty into this model. It read the news and learned the relationship. This is the moment the whole “words as vectors” idea stops being theory.

9. Average Word2Vec — From Word to Sentence Vectors

Word2Vec gives one vector per word. But RAG, semantic search, and classification need one vector per sentence or document. The classical answer — Average Word2Vec:

Recipe:
1. Create a vector for each word in the sentence
2. Take the AVERAGE of all vectors
   → a single vector for the entire sentence

import numpy as np

sentence = "I love machine learning"
words = sentence.lower().split()      # ['i','love','machine','learning']

word_vectors = [model[word] for word in words]   # four 300-dim vectors

sentence_vector = np.mean(word_vectors, axis=0)
print(sentence_vector.shape)          # (300,) — one vector, whole sentence

Simple, fast, and it works surprisingly well for basic tasks. Keep the mapping in your head:

Technique	Converts
Word2Vec	Every word → a vector (KeyedVectors)
Average Word2Vec	Every sentence → a vector
Sentence Transformers (next post)	Every sentence → a context-aware vector

10. The Static Vector Wall — bank vs bank

Average Word2Vec also exposes Word2Vec’s fatal flaw at sentence level. Consider:

Sentence 1: "I love to sit near river bank"
Sentence 2: "I always deposit money in bank"

Word2Vec: "bank" = the SAME static vector in both sentences
          e.g. bank = [1, 2, 3]  — frozen, context-blind

Averaging can't fix it: both sentence vectors are polluted
by the same wrong-for-one-of-them "bank" vector.

Word2Vec is a static vector technique — one word, one vector, forever. Understanding that “bank” should mean different things in different sentences requires a model that reads the whole sentence before assigning vectors. That is exactly what transformers do, and it’s why the class closed with: “this technique is old — we’ll see the SOTA (transformer-based) technique in the next class.”

11. Common Errors and Fixes

Error 1: KeyError — “Key ‘chatGPT’ not present in vocabulary”

Symptom:

model["chatGPT"]
KeyError: "Key 'chatGPT' not present in vocabulary"

Cause: the OOV problem, live. The Google News model was trained in 2013 — “ChatGPT” didn’t exist. Same error if you pass a whole phrase like model["i am sunny"]: the model only knows individual tokens, not sentences.

Fix: check membership first; for phrases, tokenize and use Average Word2Vec:

word = "chatGPT"
if word in model:
    vec = model[word]
else:
    print(f"'{word}' is OOV for this 2013-era model")

Error 2: NLTK LookupError — punkt not found

Symptom:

LookupError: Resource punkt_tab not found.
Please use the NLTK Downloader to obtain the resource

Cause: sent_tokenize needs NLTK’s tokenizer data, which isn’t installed with the package. Newer NLTK versions specifically need punkt_tab.

Fix: download just what you need (avoid nltk.download('all') — it pulls hundreds of MB):

import nltk
nltk.download('punkt')
nltk.download('punkt_tab')

Error 3: The Google model download hangs or fills your disk

Symptom: api.load("word2vec-google-news-300") appears frozen on first run, or fails with a disk-space or connection error.

Cause: it’s downloading ~1.6 GB to ~/gensim-data/. On slow connections this genuinely takes many minutes; the progress output is minimal.

Fix: be patient on first run (it’s cached afterwards), ensure 2+ GB free, and if you’re on a metered connection, run it once at home. To check the cache location:

import gensim.downloader as api
print(api.base_dir)     # where models are cached

12. Your Assignment This Week

Straight from the class notebook — make Lab 1 better:

1. Add more data — find 5–10 more short stories or articles, drop them in data/
2. Train for more epochs — try 50, 100, 200 and watch similarity scores change
3. Check model accuracy — do related words (ai ↔ technology) finally score above 0.3? 0.5?
4. Play with the embedding dimension — 50 vs 150 vs 300 on the same data
5. Try Skip-gram — add sg=1 and compare rare-word quality against CBOW

In short: play with all the hyperparameters. This experimentation instinct is the skill; the specific numbers are not.

Key Takeaways

✅ Skip-gram trains one pair per row — more samples than CBOW from the same text, which makes it slower but stronger, especially on rare words (“king” is directly trained as input three times in “The king drank wine”).

✅ Inference = frozen weights + a dot product: word → one-hot → × trained matrix → the embedding (a meaningful intermediate representation).

✅ Custom training on 44 sentences gives near-random similarities — neural networks are data hungry; Google’s edge is millions of news articles, not a fancier architecture.

✅ The pretrained Google News model (300d, Skip-gram) delivers the real thing: man↔woman = 0.766, and king − man + woman → queen (0.73) via pure vector arithmetic.

✅ Average Word2Vec turns sentences into vectors by mean-pooling word vectors — useful, but static vectors hit the “bank vs bank” wall, which is exactly the problem transformers solve next.

What’s Next in the Series

#	Post	Status
5	Word2Vec and Embeddings Explained	✅ Published
6	Inside Word2Vec: CBOW, Skip-gram & NN Training	✅ Published
7	Word2Vec Hands-On: Custom Training vs Google Pretrained	📍 You are here
8	Sentence Transformers — The SOTA Era	⬜ Coming next week

👉 Next Post: Sentence Transformers — The SOTA Era →

References

• gensim Word2Vec documentation
• gensim downloader API (pretrained models)
• Mikolov et al., 2013 — Efficient Estimation of Word Representations

---

Found this helpful? Share it with your team!
Questions? Drop them in the comments below.
Part of the GenAI from Scratch series — published every Friday at gradeupnow.in/genai-blog/