🤖 GenAI from Scratch — Post 8 of 24 | Module 1: NLP Foundations

← Post 7: Word2Vec Hands-On — Custom vs Pretrained Post 9: RAG Explained — Retrieval Augmented Generation →

Every technique so far has hit the same wall. One-Hot has no meaning. Bag of Words and TF-IDF count words but miss context. Word2Vec learns semantics but gives every word a single static vector — “bank” is the same whether you’re by a river or at an ATM.

This post crosses into the SOTA (State Of The Art) era: transformer-based embeddings. These are the models that actually power modern search, RAG, and every serious GenAI application today. And the best part — you don’t train them. You call them.

What you’ll build in this post (all validated in VS Code):

Open-source embeddings with sentence-transformers (all-MiniLM-L6-v2, 384-dim)
Closed-source embeddings via the OpenAI and Gemini APIs (up to 3072-dim)
Multimodal image embeddings with CLIP (text AND images in the same idea-space)
A working semantic search engine using cosine similarity, dot product and Euclidean distance
A keyword search baseline — so you can feel exactly why semantic search wins
A practical framework for choosing the right embedding model

📋 Table of Contents

The Transformer’s Two Children — LLMs vs Embedding Models
The Full Evolution: Encoding → NN → Transformer
Open-Source Embeddings with Sentence Transformers
Closed-Source: OpenAI & Gemini Embedding APIs
Multimodal: Embedding Images with CLIP
Semantic Search — Three Ways to Measure Similarity
Keyword Search vs Semantic Search — The Showdown
How to Choose the Right Embedding Model
Common Errors and Fixes
Key Takeaways

Prerequisites

✅ Read Post 7 — Word2Vec Hands-On (static vectors, the “bank vs bank” problem)
✅ Python 3.11 with uv (setup from Post 1)
✅ API keys if you want the closed-source parts: OpenAI, Gemini (Google AI Studio), and a Hugging Face token

Lab Environment

Component	Version/Details
Operating System	Windows 11 / Ubuntu 22.04
Python	3.11 (CPython, managed by uv)
sentence-transformers	3.x
langchain-openai / langchain-google-genai	latest
transformers + torch + Pillow	for CLIP image embeddings

1. The Transformer’s Two Children — LLMs vs Embedding Models

The 2017 transformer architecture (“Attention Is All You Need”) was originally built for machine translation. From it, two families of models emerged that matter to us:

Branch	Job	Examples
LLM	Text generation — predict the next token	GPT, Gemini, Llama
Embedding model	Convert data into meaningful numbers	Sentence Transformers, OpenAI/Gemini embeddings

This post is entirely about the second branch. An embedding model’s only job is to convert our data into meaningful numbers — and “our data” is broader than you might think:

Embedding model can encode:
  1. Word          4. Image
  2. Sentence      5. Audio
  3. Paragraph /   6. Video
     Document

SOTA → transformer → SOTA embedding model → ANY type of data
                                          → meaningful numbers

💡 The through-line of this entire series: every technique — OHE, BoW, TF-IDF, Word2Vec, transformers — has been trying to do one thing: turn data into numbers that carry meaning. Transformers are simply the best at it, because they read the whole input with self-attention before producing a vector.

2. The Full Evolution: Encoding → NN → Transformer

Here is the comparison table that ties all eight posts together. Pin this — it’s a complete interview answer on its own:

Feature	Encoding (OHE / TF-IDF)	NN Embedding (Word2Vec)	Transformer Embedding (OpenAI, Gemini, Sentence Transformer)
Meaning capture	No semantics	Some semantics	Deep contextual semantics
Context awareness	None	Limited (window-based)	Full context (bidirectional)
Vector type	Sparse	Dense	Dense
Same-word embedding	Same vector always	Same vector always	Different per context
Performance	Weak	Good	Best

The row that changes everything is “Different per context.” Transformers process the full sentence at once with attention, so “bank” gets a different vector next to “river” than next to “money”. That single capability is why the industry moved on from Word2Vec — and why it’s worth learning the API three ways below.

3. Open-Source Embeddings with Sentence Transformers

The easiest on-ramp is sentence-transformers — free, runs locally, no API key. The workhorse model is all-MiniLM-L6-v2, which produces 384-dimensional vectors.

Setup

uv init embeddings-lab
cd embeddings-lab
uv add sentence-transformers

File: `opensource_embeddings.py`

from sentence_transformers import SentenceTransformer

# First run downloads the model (~90 MB), then it's cached
model = SentenceTransformer("all-MiniLM-L6-v2")

# ── A single word ────────────────────────────────────────────
embedding = model.encode("machine")
print(len(embedding))          # 384
print(embedding.shape)         # (384,)

# ── A sentence — SAME 384 dimensions ─────────────────────────
sentence = "How to reduce heart disease risk?"
print(model.encode(sentence).shape)     # (384,)

# ── A whole paragraph — STILL 384 dimensions ─────────────────
paragraph = """
Machine learning is a field of artificial intelligence that focuses
on building systems that learn from data. It is widely used in
recommendation systems, image recognition, and NLP.
"""
print(model.encode(paragraph).shape)    # (384,)

🔑 The key observation

A word, a sentence, and an entire paragraph all come out as 384 numbers. Unlike TF-IDF (where dimension = vocabulary size and grows forever), a transformer embedding is a fixed-size, dense vector no matter how long the input. That fixed size is what makes vector databases and semantic search practical.

4. Closed-Source: OpenAI & Gemini Embedding APIs

Closed-source models trade infrastructure worries for an API bill: no downloads, no GPU, higher quality, higher dimensions. We’ll call them through LangChain’s tidy wrappers.

Setup & API keys

uv add langchain-openai langchain-google-genai

# Set keys as environment variables (never hard-code them!)
# Linux/macOS:
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."
# Windows PowerShell:
setx OPENAI_API_KEY "sk-..."
setx GOOGLE_API_KEY "..."

File: `api_embeddings.py`

import os
# Keys are read from the environment — keep them out of your code
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
os.environ["GOOGLE_API_KEY"] = os.getenv("GOOGLE_API_KEY")

# ── Gemini embeddings (3072 dims) ────────────────────────────
from langchain_google_genai import GoogleGenerativeAIEmbeddings
google_model = GoogleGenerativeAIEmbeddings(model="gemini-embedding-001")
g = google_model.embed_query("What is the capital of France?")
print("Gemini dims:", len(g))                 # 3072

# ── OpenAI large (3072 dims) ─────────────────────────────────
from langchain_openai import OpenAIEmbeddings
openai_large = OpenAIEmbeddings(model="text-embedding-3-large")
o_large = openai_large.embed_query("What is the capital of France?")
print("OpenAI large dims:", len(o_large))     # 3072

# ── OpenAI small (1536 dims — cheaper, lighter) ──────────────
openai_small = OpenAIEmbeddings(model="text-embedding-3-small")
o_small = openai_small.embed_query("What is the capital of France?")
print("OpenAI small dims:", len(o_small))     # 1536

Model	Dimensions	Source
all-MiniLM-L6-v2	384	Open (free, local)
OpenAI text-embedding-3-small	1536	Closed (API)
OpenAI text-embedding-3-large	3072	Closed (API)
Gemini gemini-embedding-001	3072	Closed (API)

💡 Notice all three query methods return a plain list of floats. That uniformity is the point — swap all-MiniLM for OpenAI or Gemini and the rest of your pipeline doesn’t change. You choose based on quality, cost, and dimensionality, not on rewriting code.

5. Multimodal: Embedding Images with CLIP

Embeddings aren’t just for text. Two transformer-based vision models matter here:

ViT (Vision Transformer) — applies the transformer idea to image patches
CLIP (Contrastive Language-Image Pretraining) — puts text and images in the same vector space, so you can compare a caption to a picture

File: `image_embeddings.py`

import torch
from PIL import Image
from transformers import CLIPModel, CLIPProcessor

# Load a sample image (any .png/.jpg works)
image = Image.open("./images/sample.png")

processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
model     = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")

# Preprocess → run through the vision encoder
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    image_embedding = model.get_image_features(**inputs)

print(image_embedding.shape)        # torch.Size([1, 512])
print(len(image_embedding[0]))      # 512 — the image is now a vector!

An image becomes a 512-dimensional vector — the same kind of object as a text embedding. This is the foundation for image search, “find similar photos”, and multimodal RAG. Same core idea as everything else in this series: data in, meaningful numbers out.

6. Semantic Search — Three Ways to Measure Similarity

Embeddings are only useful if you can compare them. Three standard measures, each with its own range:

Measure	Range	Better when
Cosine similarity	−1 to 1	Higher (angle between vectors)
Dot product	−∞ to +∞	Higher (depends on magnitude)
Euclidean distance (L2)	0 to +∞	Lower (straight-line distance)

File: `semantic_search.py`

import numpy as np
from langchain_openai import OpenAIEmbeddings

model = OpenAIEmbeddings(model="text-embedding-3-large")

# ── Three similarity functions ───────────────────────────────
def dot_product(a, b):
    return np.dot(a, b)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def euclidean_distance(a, b):
    a, b = np.array(a), np.array(b)
    return np.linalg.norm(a - b)

# ── The search problem ───────────────────────────────────────
query = "How to reduce heart disease risk?"
documents = [
    "Eating fiber reduces heart risk.",
    "Fruits and vegetables lower cardiovascular disease chances.",
    "Buying a new car improves driving comfort.",
    "Regular exercise improves heart health.",
]

# ── Embed the query and every document ───────────────────────
query_embedding = model.embed_query(query)
doc_embeddings  = model.embed_documents(documents)

# ── Score each document against the query ────────────────────
for doc, emb in zip(documents, doc_embeddings):
    print(f"\n{doc}")
    print(f"  cosine    : {cosine_similarity(query_embedding, emb):.4f}")
    print(f"  dot       : {dot_product(query_embedding, emb):.4f}")
    print(f"  euclidean : {euclidean_distance(query_embedding, emb):.4f}")

Real output

Eating fiber reduces heart risk.
  cosine    : 0.5985   ← highest ✓
Fruits and vegetables lower cardiovascular disease chances.
  cosine    : 0.5579   ← second — no shared words with the query!
Regular exercise improves heart health.
  cosine    : (heart-related, scores well)
Buying a new car improves driving comfort.
  cosine    : (lowest — unrelated) ✗

The magic is in the second result. “Fruits and vegetables lower cardiovascular disease chances” scores 0.56 against “How to reduce heart disease risk?” — even though they share almost no words. The model understands that cardiovascular ≈ heart and lower chances ≈ reduce risk. That’s semantic understanding, impossible with any counting technique.

7. Keyword Search vs Semantic Search — The Showdown

To feel the difference, let’s build the “old way” — keyword search, which just counts matching words.

from collections import Counter

def keyword_search(query, documents):
    query_words = query.lower().replace("?", "").split()
    results = []
    for doc in documents:
        doc_words = doc.lower().replace(".", "").split()
        doc_count = Counter(doc_words)
        # score = number of query words found in the document
        score = sum(doc_count[word] for word in query_words)
        results.append((doc, score))
    return sorted(results, key=lambda x: x[1], reverse=True)

print(keyword_search(query, documents))

# Output:
[('Eating fiber reduces heart risk.', 2),                          # 'heart','risk'
 ('Fruits and vegetables lower cardiovascular disease chances.', 1),  # 'disease'
 ('Regular exercise improves heart health.', 1),                   # 'heart'
 ('Buying a new car improves driving comfort.', 0)]

⚠️ Where keyword search breaks

The fruits-and-vegetables document is genuinely one of the best answers to the query — but keyword search gives it a score of just 1 (only “disease” matches), tying it with a generic exercise sentence. It has no idea that “cardiovascular” means “heart”. Semantic search ranked it a clear second at 0.56. Keyword search matches letters; semantic search matches meaning. This gap is the entire reason RAG uses embeddings.

8. How to Choose the Right Embedding Model

With dozens of models available, use these four criteria from the class notes.

1. Embedding quality (most important)

Check benchmark performance before anything else. The two references to know:

MTEB leaderboard — the standard ranking for embedding models (huggingface.co/spaces/mteb/leaderboard)
BEIR benchmark — retrieval-focused evaluation (arxiv.org/abs/2104.08663)

2. Dimensionality

Dimensions	Profile
384	Lightweight
768	Balanced
1536	High quality but heavy

Higher dimensions give better semantic representation — but cost more storage, more memory, and slower search. Bigger is not automatically better.

3. Cost

Closed source (e.g. OpenAI): easy, high quality, but you pay per API call
Open source (e.g. all-MiniLM): free to use, but you pay in infrastructure and scaling

4. Domain suitability

For general-purpose work, the reliable starting points are all-MiniLM, OpenAI embeddings, and Gemini embeddings. Specialised domains (legal, medical, code) may need a domain-tuned model — check the leaderboard filtered for your task.

📌 Practical default

Prototype with free all-MiniLM-L6-v2 (384-dim, local). If quality is the bottleneck, move to OpenAI text-embedding-3-small (1536-dim). Reserve the 3072-dim large models for when you’ve proven you need them — they double your storage for a modest quality gain.

9. Common Errors and Fixes

Error 1: AuthenticationError / missing API key

Symptom:

openai.AuthenticationError: No API key provided
# or:
google.api_core.exceptions.Unauthenticated: 401

Cause: the environment variable isn’t set in the process actually running your code (a new terminal, or your editor didn’t inherit the variable).

Fix: verify the key is visible before running, and prefer a .env file with python-dotenv:

uv add python-dotenv
# .env  →  OPENAI_API_KEY=sk-...
from dotenv import load_dotenv
load_dotenv()   # call this before creating any embeddings model

Error 2: TqdmWarning — IProgress not found

Symptom:

TqdmWarning: IProgress not found. Please update jupyter and ipywidgets.

Cause: a harmless progress-bar warning when importing sentence-transformers in a notebook. It does not break anything.

Fix: ignore it, or silence it by installing widgets: uv add ipywidgets.

Error 3: CLIP / transformers install is huge or fails on torch

Symptom: pip pulls gigabytes, or torch fails to resolve a wheel.

Cause: transformers + torch are large, and torch is picky about Python/CUDA versions.

Fix: pin Python 3.11 and install the CPU build if you don’t have a GPU:

uv python pin 3.11
uv add transformers pillow
uv add torch --index https://download.pytorch.org/whl/cpu

Key Takeaways

✅ Transformers spawned two families: LLMs (generate text) and embedding models (turn any data into meaningful numbers). This post is all about the second.

✅ The decisive advantage over Word2Vec is context: transformer embeddings are different per context, so “bank” finally gets the right vector for each sentence.

✅ You get the same fixed-size dense vector for a word, sentence, or paragraph — 384 dims (all-MiniLM), 1536/3072 dims (OpenAI), 3072 dims (Gemini), 512 dims (CLIP images).

✅ Semantic search (cosine / dot / Euclidean over embeddings) beats keyword search because it matches meaning, not letters — the fruits-and-vegetables result proves it.

✅ Choose a model by quality (MTEB/BEIR) → dimensionality → cost → domain. Prototype free with all-MiniLM; scale up only when you must.

What’s Next in the Series

#	Post	Status
6	Inside Word2Vec: CBOW, Skip-gram & NN Training	✅ Published
7	Word2Vec Hands-On: Custom vs Pretrained	✅ Published
8	SOTA Embeddings: Sentence Transformers, OpenAI & Gemini	📍 You are here
9	RAG Explained — Retrieval Augmented Generation	⬜ Coming next week

👉 Next Post: RAG Explained — Retrieval Augmented Generation →

Everything in this post — embeddings + semantic search — is exactly the retrieval engine inside a RAG system. Next week we assemble the full pipeline.

References

Sentence Transformers (Hugging Face)
MTEB Embedding Leaderboard
BEIR Benchmark paper
OpenAI Embeddings documentation
Gemini Embeddings documentation
Full-Stack GenAI Bootcamp v1.0 — Class 08 (18 April)

Found this helpful? Share it with your team!
Questions? Drop them in the comments below.
Part of the GenAI from Scratch series — published every Friday at gradeupnow.in/genai-blog/

🤖 GenAI from Scratch — Post 8 of 24 | Module 1: NLP Foundations | Class 08 (18 April)

Prerequisites

Lab Environment

1. The Transformer’s Two Children — LLMs vs Embedding Models

2. The Full Evolution: Encoding → NN → Transformer

3. Open-Source Embeddings with Sentence Transformers

Setup

File: `opensource_embeddings.py`

4. Closed-Source: OpenAI & Gemini Embedding APIs

Setup & API keys

File: `api_embeddings.py`

5. Multimodal: Embedding Images with CLIP

File: `image_embeddings.py`

6. Semantic Search — Three Ways to Measure Similarity

File: `semantic_search.py`

Real output

7. Keyword Search vs Semantic Search — The Showdown

8. How to Choose the Right Embedding Model

1. Embedding quality (most important)

2. Dimensionality

3. Cost

4. Domain suitability

9. Common Errors and Fixes

Error 1: AuthenticationError / missing API key

Error 2: TqdmWarning — IProgress not found

Error 3: CLIP / transformers install is huge or fails on torch

Key Takeaways

What’s Next in the Series

References

Leave a Comment Cancel Reply

Prerequisites

Lab Environment

1. The Transformer’s Two Children — LLMs vs Embedding Models

2. The Full Evolution: Encoding → NN → Transformer

3. Open-Source Embeddings with Sentence Transformers

Setup

File: opensource_embeddings.py

4. Closed-Source: OpenAI & Gemini Embedding APIs

Setup & API keys

File: api_embeddings.py

5. Multimodal: Embedding Images with CLIP

File: image_embeddings.py

6. Semantic Search — Three Ways to Measure Similarity

File: semantic_search.py

Real output

7. Keyword Search vs Semantic Search — The Showdown

8. How to Choose the Right Embedding Model

1. Embedding quality (most important)

2. Dimensionality

3. Cost

4. Domain suitability

9. Common Errors and Fixes

Error 1: AuthenticationError / missing API key

Error 2: TqdmWarning — IProgress not found

Error 3: CLIP / transformers install is huge or fails on torch

Key Takeaways

What’s Next in the Series

References

Leave a Comment Cancel Reply

File: `opensource_embeddings.py`

File: `api_embeddings.py`

File: `image_embeddings.py`

File: `semantic_search.py`