How to Access LLMs and Multimodal Models: SDKs vs. LangChain, Paid vs. Free Providers (Hands-On)

Introduction

In the last three posts we opened the hood: embeddings, self-attention, encoder/decoder families, and how LLMs are trained. Now for the practical question every learner hits next: how do I actually call one of these models from my code — without a big bill?

There are more options than most tutorials admit: paid APIs, free tiers, managed cloud platforms, and models you download and run yourself. This post maps that landscape, shows working LangChain code for each, and explains when to drop down to a provider’s native SDK instead. As always, we’ll use DBA analogies where they help.

What you’ll learn:

• The four ways to reach a model (paid API, free-tier API, managed cloud, your own machine)
• Why LangChain exists when every provider already ships an SDK
• How system and human messages work
• How to run text-to-text, image-to-text, audio-to-text, text-to-image, and text-to-audio calls
• How to run models locally with Ollama

---

Prerequisites

• [ ] Python 3.10+ and a Jupyter notebook or VS Code
• [ ] Read Parts 1–3 of this series (helpful, not mandatory)
• [ ] API keys for whichever providers you want to try (links in the References section)

Heads-up: model names, free-tier limits, and pricing change frequently. The model names below are the ones used in class; always check each provider’s current model list and limits before you build on them.

---

The Big Picture: Which Modality Do You Need?

Multimodal just means different input and output types. Here is the grid from class — and the first thing to know is that roughly 95% of the time you’ll use text-to-text, especially for RAG and agent apps:

Input → Output	Typical use	Needed often?
Text → Text	Chatbots, RAG, agents, code	Yes — the default
Text → Image	Image generation	Sometimes
Image → Text	OCR, captioning, visual Q&A	Sometimes (key in multimodal RAG)
Text → Audio	Voice assistants, narration	Sometimes
Audio → Text	Transcription	Sometimes
Text → Video / Video → Text	Generation / understanding	Rare
Image→Image, Audio→Audio	Editing, voice conversion	Not required for this course

---

Four Ways to Reach a Model

1. Paid APIs (OpenAI, Anthropic/Claude)

You buy credits, generate an API key, and call the provider’s servers. Easiest and highest quality, but not free — so not everyone can use them for practice.

2. Free-tier APIs (Groq, OpenRouter, Gemini, Hugging Face)

Great for learning. Groq and OpenRouter offer free access to selected models, Gemini has a free tier for several models (with rate limits), and Hugging Face offers an inference API with free access. Expect caps on tokens, requests, and which models you can use.

3. Managed cloud platforms (AWS Bedrock, Azure AI Foundry, Google Vertex AI, Hugging Face enterprise)

You skip the hassle of hosting a model yourself, but you authenticate with that cloud’s own credentials. These are the enterprise route.

4. Your own machine (Ollama, Hugging Face)

Download the model weights and run them locally. No per-token bill — but you carry the hardware and setup cost.

DBA analogy: this is the same decision you make about a database. Paid APIs are a fully managed service (RDS/Azure SQL) — pay more, ops-free. Free tiers are the developer sandbox. Managed cloud is your enterprise-contract option. Running Ollama locally is installing the engine on your own server — free license, but you own capacity planning.

The “Hub” Pattern

One idea from class makes this much less confusing — every ecosystem has a hub:

Hub	What lives there	What you pull to your machine
GitHub	Code repos	Code
Docker Hub	Container images	Images
Hugging Face Hub	Models and datasets	Model weights, datasets
Ollama Hub	Ready-to-run models	Models

DBA analogy: a hub is like a package repository or backup catalog — you pull an artifact from a central store to your local environment. Models are just another artifact.

---

SDK vs. LangChain: Why Not Just Use the Provider’s SDK?

Every provider has its own SDK — OpenAI’s, Anthropic’s, Google’s, Groq’s. You can call models directly with them. Compare these two calls from class:

# OpenAI SDK
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain RAG in simple words"}],
)
print(response.choices[0].message.content)

# Google SDK — different client, different method, different response shape
from google import genai
client = genai.Client()
response = client.models.generate_content(
    model="gemini-3-flash-preview",
    contents="Explain how AI works in a few words",
)
print(response.text)

Same job, two different APIs. Now the LangChain version — same shape for every provider:

llm = ChatOpenAI(model="gpt-4o-mini")        # or ChatGroq(...), ChatAnthropic(...), ChatGoogleGenerativeAI(...)
response = llm.invoke("Explain RAG in simple words")
print(response.content)

An SDK is mainly for calling one provider’s model. LangChain is for building complete LLM applications around models — with a unified interface.

Beyond the unified invoke() call, a real app also needs prompt templates, memory, document loaders, text splitters, embeddings, vector databases, retrievers, tools, agents, tracing, evaluation, and model switching. An SDK gives you the engine; LangChain gives you the rest of the car.

DBA analogy: provider SDKs are like vendor-specific database drivers, each with its own quirks. LangChain is the ODBC/JDBC layer — you write against one interface and swap the backend (OpenAI → Groq → Ollama) by changing one line. And just as ODBC doesn’t contain a database, LangChain doesn’t provide the model — you still generate a key from the respective platform (OpenAI, Claude, Gemini, OpenRouter) and LangChain just connects you.

A related clarification from class: LangChain, LangGraph, LangSmith, and LangFuse are different things. LangChain and LangGraph are open-source frameworks (for RAG and agents); LangSmith and LangFuse are platforms (tracing, evaluation). And platforms like Groq, OpenRouter, and Perplexity serve models but don’t necessarily own them.

---

How Messages Work

Every LLM call is: prompt (input message) → LLM → output (AI message). The input can be a single user message, or a system message plus a user message.

There are two ways to write them:

# 1. Tuple format — older, still works
messages = [
    ("system", "You are a helpful assistant that translates English to Hindi."),
    ("human", "I love programming."),
]

# 2. Message classes — the current LangChain style
from langchain_core.messages import SystemMessage, HumanMessage
messages = [
    SystemMessage(content="You're a helpful assistant"),
    HumanMessage(content="What happens when an unstoppable force meets an immovable object?"),
]

• SystemMessage sets the permanent role/behavior of the LLM for the active session.
• HumanMessage carries whatever the user wants to ask.

DBA analogy: the system message is like session-level settings (SET options, role, isolation level) that govern every statement that follows; the human message is the individual query.

---

Hands-On: Text → Text

Paid: OpenAI and Claude via LangChain

import os
from langchain_openai import ChatOpenAI

# OPENAI_API_KEY must be set in your environment
llm = ChatOpenAI(model="gpt-5-nano")
message = llm.invoke("What is the capital of India?")
print(message.content)

from langchain_anthropic import ChatAnthropic

# ANTHROPIC_API_KEY must be set in your environment
llm = ChatAnthropic(model="claude-haiku-4-5-20251001", temperature=0)
print(llm.invoke("What is the capital of India?").content)

Free: Groq

from langchain_groq import ChatGroq

llm = ChatGroq(model="qwen/qwen3-32b")
messages = [
    ("system", "You are a helpful assistant that translates English to Hindi. Translate the user sentence."),
    ("human", "I love programming."),
]
print(llm.invoke(messages).content)

Free: OpenRouter

OpenRouter is a gateway — one key, many models (including Claude). In class, the free access came with a cap on output tokens, so set max_tokens explicitly:

from langchain_openrouter import ChatOpenRouter

llm = ChatOpenRouter(model="anthropic/claude-sonnet-4.5", max_tokens=2048)
messages = [
    ("system", "You are a helpful assistant."),
    ("human", "Can you talk about climate change?"),
]
print(llm.invoke(messages).content)

Homework from class: browse the model catalog on OpenRouter (filter by text and image input) and try several models.

Free: Hugging Face

from langchain_huggingface import HuggingFaceEndpoint, ChatHuggingFace

llm = HuggingFaceEndpoint(
    repo_id="deepseek-ai/DeepSeek-V4-Pro",   # swap in any text-generation repo you have access to
    task="text-generation",
    max_new_tokens=512,
)
chat_model = ChatHuggingFace(llm=llm)
print(chat_model.invoke(messages).content)

Free tier: Google Gemini

from langchain_google_genai import ChatGoogleGenerativeAI

model = ChatGoogleGenerativeAI(model="gemini-2.5-flash-lite")
print(model.invoke([
    ("system", "You are a helpful assistant that translates English to French."),
    ("human", "I love programming."),
]).content)

---

Hands-On: Image → Text (Vision and OCR)

Computers process everything as binary. An image is a collection of pixels, so to send one inside a message you base64-encode it — converting the binary into text that can travel inside the request. The same trick works for audio and video.

import base64
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

b64 = encode_image("dogs.png")

message = HumanMessage(content=[
    {"type": "text", "text": "What is in this image?"},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
])

llm = ChatOpenAI(model="gpt-4o-mini")
print(llm.invoke([message]).content)

You can also pass a public image URL instead of base64 data. The same message structure works with Groq’s vision-capable models (class used meta-llama/llama-4-scout-17b-16e-instruct) and with Gemini.

Why this matters: OCR — pulling text out of images — used to be done with CNN-based models. Today transformer-based vision models (the ViT family from the last post) handle it, and this capability plays a key role in multimodal RAG pipelines.

---

Hands-On: Audio → Text, Text → Audio, Text → Image

Audio → Text with Gemini

import base64
from langchain_google_genai import ChatGoogleGenerativeAI
from langchain_core.messages import HumanMessage

with open("audio.mp3", "rb") as f:
    b64_audio = base64.b64encode(f.read()).decode("utf-8")

llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash")
message = HumanMessage(content=[
    {"type": "text", "text": "Transcribe this audio into English text."},
    {"type": "media", "mime_type": "audio/mp3", "data": b64_audio},
])
print(llm.invoke([message]).content)

Text → Image with OpenAI (via a tool)

llm = ChatOpenAI(model="gpt-5.4-mini")
llm_with_tools = llm.bind_tools([{"type": "image_generation", "quality": "low"}])

ai_message = llm_with_tools.invoke("Draw a picture of a cute fuzzy cat with an umbrella")
image = next(i for i in ai_message.content_blocks if i["type"] == "image")

import base64
from IPython.display import Image
Image(base64.b64decode(image["base64"]), width=200)

Text → Audio: when LangChain has no wrapper, drop to the SDK

For audio-to-audio, image-to-image, video-to-video, or text-to-audio/video, you often won’t find a LangChain wrapper yet. In that case, call the provider’s SDK directly — this is exactly when an SDK is the right tool:

from google import genai
from google.genai import types
import wave

client = genai.Client(api_key=GOOGLE_API_KEY)
response = client.models.generate_content(
    model="gemini-2.5-flash-preview-tts",
    contents="Hello students, today we are learning multimodal AI.",
    config=types.GenerateContentConfig(
        response_modalities=["AUDIO"],
        speech_config=types.SpeechConfig(
            voice_config=types.VoiceConfig(
                prebuilt_voice_config=types.PrebuiltVoiceConfig(voice_name="Kore")
            )
        ),
    ),
)

audio_data = response.candidates[0].content.parts[0].inline_data.data
with wave.open("output.wav", "wb") as wf:
    wf.setnchannels(1)
    wf.setsampwidth(2)
    wf.setframerate(24000)
    wf.writeframes(audio_data)

---

Running Models Locally with Ollama

Ollama downloads models from its hub to your machine and serves them locally. Core commands:

ollama pull llama3.1      # download a model
ollama run llama3.1       # chat in the terminal
ollama list               # models on disk
ollama ps                 # models currently loaded
ollama show llama3.1      # model details (add --modelfile to see its config)
ollama stop llama3.1      # unload from memory
ollama rm llama3.1        # delete from disk

Then call it through LangChain exactly like any other provider:

from langchain_ollama import ChatOllama
from langchain_core.messages import SystemMessage, HumanMessage

llm = ChatOllama(model="llama3.2:1b", temperature=0)   # fully local
response = llm.invoke([
    SystemMessage(content="You are a helpful AI assistant."),
    HumanMessage(content="Explain RAG in simple words"),
])
print(response.content)

Ollama also offers cloud-hosted models (a :cloud model tag, as used in class) where Ollama runs the model for you. The free version only exposes a limited set; more models need a paid plan. For vision, you can pull a multimodal model such as bakllava and send base64 images the same way as above.

---

Which Option Should You Choose?

Situation	Reasonable choice
Learning / prototyping on a budget	Groq, OpenRouter, Gemini free tier, Hugging Face
Production quality, minimal ops	OpenAI or Claude paid API
Enterprise with existing cloud contracts	Bedrock, Azure AI Foundry, Vertex AI
Privacy-sensitive or offline	Ollama (local)
Modality LangChain doesn’t wrap yet	Provider SDK directly

---

Key Takeaways

✅ Roughly 95% of GenAI app work is text-to-text, so master that first ✅ There are four ways to reach a model: paid API, free-tier API, managed cloud, or your own machine ✅ Provider SDKs call one vendor; LangChain gives a unified interface plus the building blocks for RAG and agents ✅ LangChain doesn’t provide models — you still bring a key from the actual provider ✅ System messages set session behavior; human messages carry the user’s input ✅ Images, audio, and video are sent as base64-encoded data inside a message ✅ When LangChain has no wrapper for a modality, use the provider SDK directly ✅ Free-tier limits and model names change often — verify before building

---

---

References

• Course repo — https://github.com/sunnysavita10/Full-Stack-GenAI-Bootcamp-1.0
• LangChain docs — https://docs.langchain.com/
• LangChain providers list — https://docs.langchain.com/oss/javascript/integrations/providers/all_providers
• OpenAI API docs and keys — https://platform.openai.com/docs
• Claude API docs — https://platform.claude.com/docs
• Gemini API docs and keys — https://ai.google.dev/gemini-api/docs
• OpenRouter keys and models — https://openrouter.ai/models
• Hugging Face tokens — https://huggingface.co/settings/tokens
• Ollama — https://ollama.com/download
• LLM leaderboards — https://artificialanalysis.ai/leaderboards/models and https://huggingface.co/spaces/lmarena-ai/arena-leaderboard

---

Found this helpful? Share it with your team! Questions? Drop them in the comments below.