Building LLM Applications
A hands-on tour of the tools you use to turn a raw language model into a real application β organized around the three problems you hit as your app grows up.
Too much glue code
Prompts, parsing, tools, retrieval β LangChain gives you reusable building blocks.
Logic isn't a straight line
Loops, branches, multiple agents β LangGraph gives you stateful orchestration.
It's a black box
Can't see or measure it β LangSmith gives you tracing & evaluation.
Problem 1 LangChain β building blocks
Calling a model directly is easy for one prompt. A real feature needs prompt templating, output parsing, tool-calling, memory, and β to ground answers in your own data β document loading, chunking, embedding, and retrieval. LangChain provides standard, composable pieces so you write that glue once and swap providers with a one-line change.
The LCEL pipe (|) composes these into a chain with streaming, batching, and async for free. Here's the same minimal chain across two providers:
# pip install langchain-anthropic
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
model = ChatAnthropic(model="claude-opus-4-8")
prompt = ChatPromptTemplate.from_template("Summarize in one sentence: {text}")
chain = prompt | model
print(chain.invoke({"text": "..."}).content)
# pip install langchain-openai
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
model = ChatOpenAI(model="gpt-4o")
prompt = ChatPromptTemplate.from_template("Summarize in one sentence: {text}")
chain = prompt | model
print(chain.invoke({"text": "..."}).content)
Notice only the import and class name changed β the chain is identical. That provider-agnostic interface is LangChain's core value.
2 Β· RAG β grounding answers in your data
Retrieval-Augmented Generation feeds the model only the chunks of your documents relevant to a question, so it answers from your data instead of guessing. The pipeline is a straight line β no loops β which makes it a perfect LangChain chain.
# load β split β embed β store β retrieve β grounded answer
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
docs = TextLoader("kb.txt").load()
chunks = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50).split_documents(docs)
store = FAISS.from_documents(chunks, HuggingFaceEmbeddings(
model_name="sentence-transformers/all-MiniLM-L6-v2"))
retriever = store.as_retriever(search_kwargs={"k": 4})
prompt = ChatPromptTemplate.from_template(
"Answer using ONLY this context:\n{context}\n\nQuestion: {question}")
model = ChatAnthropic(model="claude-opus-4-8", temperature=0)
chain = ({"context": retriever, "question": RunnablePassthrough()}
| prompt | model | StrOutputParser())
print(chain.invoke("What is our refund policy?"))
Problem 2 LangGraph β stateful orchestration
A chain is a one-way pipe. The moment your app must loop ("keep using tools until you can answer"), branch ("refund vs. escalate"), coordinate multiple agents, or pause for a human, you need a graph. LangGraph models your app as nodes (steps) connected by edges (control flow) that share a state object β and edges may form cycles.
The agent β tools cycle is the reasonβactβobserve loop a chain can't express.
from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition
from langchain_anthropic import ChatAnthropic
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage
@tool
def calculator(expression: str) -> str:
"""Evaluate arithmetic, e.g. '15 * 0.2'."""
return str(eval(expression, {"__builtins__": {}}))
model = ChatAnthropic(model="claude-opus-4-8").bind_tools([calculator])
class State(TypedDict):
messages: Annotated[list, add_messages]
def call_model(state): return {"messages": [model.invoke(state["messages"])]}
g = StateGraph(State)
g.add_node("agent", call_model)
g.add_node("tools", ToolNode([calculator]))
g.add_edge(START, "agent")
g.add_conditional_edges("agent", tools_condition) # branch: tool or done?
g.add_edge("tools", "agent") # loop back
graph = g.compile()
out = graph.invoke({"messages": [HumanMessage("What's 15% of 240?")]})
print(out["messages"][-1].content)
What does the graph buy you over a chain?
The loop lets the model call several tools in sequence, reacting to each result. The conditional edge decides at runtime whether to act or answer. Add a checkpointer and the run becomes durable, resumable, and human-in-the-loop-ready (pause before a tool, wait for approval, resume).
Problem 3 LangSmith β see & measure it
LLMs are non-deterministic. When an answer is wrong, was it bad retrieval, a weak prompt, or the model? LangSmith gives you a trace β a nested tree of every step with latency, tokens, and cost β plus evaluation to catch regressions before shipping. It auto-instruments LangChain & LangGraph with zero code change:
# the RAG chain and agent above are now fully traced
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="..."
What a trace looks like
Running the agent on "What's 15% of 240?" produces a run tree β the three LLM calls are the reasonβactβobserve loop made visible:
βΈ LangGraph run Β· agent 3.1s Β· 1,240 tok Β· $0.021 β
ββ agent Β· LLM call #1 0.9s Β· 320 tok Β· calls calculator('240*0.15')
ββ tools Β· calculator 0.02s Β· returns '36'
ββ agent Β· LLM call #2 0.8s Β· 360 tok Β· final answer β
Evaluation β did my change help?
from langsmith import Client, evaluate
client = Client()
client.create_dataset("qa") # add {question β reference answer} examples
def target(inputs): # the system under test (your RAG chain)
return {"answer": rag_chain.invoke(inputs["question"])}
def correct(outputs, reference_outputs): # a grader
return reference_outputs["answer"].lower() in outputs["answer"].lower()
evaluate(target, data="qa", evaluators=[correct], experiment_prefix="rag-v1")
# change a prompt β run "rag-v2" β compare scores side by side
5 Β· Vector databases β where chunks live
An embedding turns text into a vector positioned so similar meanings sit close together. A vector database stores those vectors + metadata and finds the nearest ones to a query β fast, over millions of items. That "semantic search" is the retrieval engine behind RAG.
| Database | Type | Best for |
|---|---|---|
| Chroma | OSS, embedded/server | Prototyping, local RAG |
| FAISS | OSS library, in-process | Static, read-heavy, research |
| Qdrant | OSS server (Rust) | Balanced prod, rich filters |
| Pinecone | Managed cloud | Hands-off production |
| pgvector | Postgres extension | Already on Postgres |
The storage flow never changes β only the client does. Compare storing the same chunks in two backends:
# pip install chromadb
import chromadb
from chromadb.utils import embedding_functions
client = chromadb.PersistentClient(path="./chroma_db")
ef = embedding_functions.SentenceTransformerEmbeddingFunction("all-MiniLM-L6-v2")
col = client.get_or_create_collection("policies", embedding_function=ef)
col.add( # store: Chroma embeds for you
ids=[f"chunk-{i}" for i in range(len(chunks))],
documents=chunks,
metadatas=[{"source": "policy.txt"} for _ in chunks])
print(col.query(query_texts=["Can I get money back?"], n_results=2)["documents"])
# pip install qdrant-client sentence-transformers
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("all-MiniLM-L6-v2")
c = QdrantClient(":memory:")
c.recreate_collection("policies",
vectors_config=VectorParams(size=384, distance=Distance.COSINE))
vecs = m.encode(chunks, normalize_embeddings=True)
c.upsert("policies", [ # store: id + vector + payload
PointStruct(id=i, vector=v.tolist(), payload={"text": chunks[i]})
for i, v in enumerate(vecs)])
q = m.encode(["Can I get money back?"], normalize_embeddings=True)[0].tolist()
print([h.payload["text"] for h in c.search("policies", q, limit=2)])
6 Β· create-llm-app β scaffold it all
Everything above is wired together by create-llm-app, a CLI that generates a runnable project. You pick a provider, an app type (chat / RAG / agent), and whether to enable LangSmith tracing β it emits main.py, requirements.txt, .env.example (loaded via python-dotenv), and more.
# install
pip install -e .
# interactive β prompts for provider, app type, tracing
create-llm-app new my-assistant
# or non-interactive with flags
create-llm-app new my-rag --provider anthropic --type rag --no-tracing
cd my-rag
pip install -r requirements.txt
cp .env.example .env # add your API key(s)
python main.py
Under the hood it maps your choices to a provider registry and renders Jinja2 templates β the same chat / RAG / agent code you saw in sections 1β3.
7 Β· Agentic patterns & techniques
Once a single agent isn't enough, you compose multiple β and there's a small catalogue of proven shapes for doing so. This repo has a runnable scaffold for each, following Google Cloud's Choose a design pattern for your agentic AI system, built on the same LangChain / LangGraph / LangSmith stack from this tutorial.
The doc's key axis is who decides the route:
Design patterns (Google Cloud)
| # | Pattern | When to use | Stack |
|---|---|---|---|
| 01 | Single-Agent | Moderate multi-step tasks, no delegation | LangGraph (prebuilt ReAct) |
| 02 | Sequential | Structured, repeatable pipelines | LangChain (LCEL) |
| 03 | Parallel | Independent sub-tasks; diverse perspectives at once | LangChain (RunnableParallel) |
| 04 | Loop | Iterative refinement with an exit condition | LangGraph (cycle) |
| 05 | Review & Critique | High accuracy / compliance validation | LangGraph (generator + critic) |
| 06 | Iterative Refinement | Output hard to nail in one shot; quality gate | LangGraph (self-score loop) |
| 07 | Coordinator | Adaptive routing of varied inputs | LangGraph (router) |
| 08 | Hierarchical | Ambiguous, open-ended problems | LangGraph (orchestratorβworker) |
| 09 | Swarm | Complex problems that benefit from debate | LangChain (multi-persona) |
| 10 | ReAct | Dynamic tasks needing tools + adaptation | LangChain (manual loop) |
| 11 | Human-in-the-Loop | High-stakes / safety / compliance approvals | LangGraph (interrupt) |
| 12 | Custom Logic | Mixed-pattern, bespoke branching | LangChain (plain Python) |
| 13 | RAG | Ground answers in your own data | LangChain + FAISS |
| 14 | RAG + CAG | Stable facts cached + volatile corpus retrieved | LangChain + FAISS + prompt caching |
Reasoning & agentic techniques
Prompting, reasoning, and retrieval techniques that make an individual agent smarter β complementary to the orchestration patterns above.
| # | Technique | What it adds | Stack |
|---|---|---|---|
| 15 | Chain-of-Thought | Reason step by step before answering | LangChain |
| 16 | Self-Consistency | Sample many CoT paths, majority-vote | LangChain |
| 17 | Tree of Thoughts | Branch, score, expand the best (search) | LangChain |
| 18 | Reflexion | Retry with accumulated self-feedback | LangGraph |
| 19 | Plan-and-Execute | Plan up front, then run each step | LangGraph |
| 20 | ReWOO | Plan all tool calls once, then solve | LangChain |
| 21 | Chain-of-Verification | Draft β verify β revise (anti-hallucination) | LangChain |
| 22 | Agentic RAG | Retrieve on demand as a tool | LangGraph + FAISS |
| 23 | Corrective RAG | Grade retrieval, fall back if weak | LangChain + FAISS |
| 24 | Self-RAG | Model gates its own retrieval + grounding | LangChain + FAISS |
| 25 | Memory Agent | Recall earlier turns (checkpointer) | LangGraph |
py_compile on Python 3.11β3.13). Each folder has a README, a runnable main.py, requirements.txt, and .env.example. Browse them: experiments/ on GitHub.8 Β· Full-stack example: a support agent
Everything in this tutorial, wired into one runnable app: a customer-support agent that answers questions grounded in a knowledge base and escalates to a human when it can't help. It combines all six pieces:
LangGraph
The agent β tools loop + memory across turns.
OpenAI
gpt-4o reasoning + text-embedding-3-small embeddings.
Chroma
Persistent vector database for the knowledge base.
RAG
search_kb retrieves grounding context on demand.
LangSmith
Traces every model + tool + retrieval step.
Retrieval is a tool the agent chooses to call (agentic RAG), and a sensitive request routes to escalate_to_human instead of a guess.
@tool
def search_kb(query: str) -> str:
"""Search the support KB β RAG over the Chroma vector store."""
docs = store.similarity_search(query, k=4)
return "\n\n".join(d.page_content for d in docs)
model = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools([search_kb, escalate_to_human])
builder = StateGraph(State)
builder.add_node("agent", agent) # gpt-4o reasons
builder.add_node("tools", ToolNode(TOOLS)) # search_kb (RAG) + escalate
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", tools_condition) # call a tool, or answer?
builder.add_edge("tools", "agent") # loop back with the result
graph = builder.compile(checkpointer=MemorySaver()) # memory across turns
pip install -r requirements.txt
cp .env.example .env # add your OPENAI_API_KEY
python ingest.py # load KB -> embed -> store in Chroma (once)
python agent.py # grounded answer, then an escalation
Complete source β copy each file
The whole example, ready to build. Hit Copy setup script and paste it into a terminal β it writes every file into a new support-agent/ folder. Or switch tabs and Copy file one at a time.
python-dotenv
langchain
langchain-core
langchain-openai
langchain-chroma
langchain-text-splitters
chromadb
langgraph
langsmith
# Required β OpenAI powers both the chat model and the embeddings
OPENAI_API_KEY=sk-...
# Optional β enable LangSmith tracing to see every step of the agent loop
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=your-langsmith-key
# Optional overrides (defaults shown)
CHAT_MODEL=gpt-4o
EMBED_MODEL=text-embedding-3-small
CHROMA_DIR=./chroma_db
"""Shared settings for the support agent (overridable via environment variables)."""
import os
# Where the Chroma vector store persists on disk.
CHROMA_DIR = os.getenv("CHROMA_DIR", "./chroma_db")
COLLECTION = os.getenv("CHROMA_COLLECTION", "support_kb")
# OpenAI models.
CHAT_MODEL = os.getenv("CHAT_MODEL", "gpt-4o")
EMBED_MODEL = os.getenv("EMBED_MODEL", "text-embedding-3-small")
# Source knowledge base (relative to this folder).
KB_PATH = os.getenv("KB_PATH", "data/knowledge_base.md")
"""Ingest step β load the knowledge base, chunk it, embed with OpenAI, and store
the vectors in a persistent Chroma database.
Run this ONCE before the agent (re-run to rebuild the index):
python ingest.py
"""
from pathlib import Path
from dotenv import load_dotenv
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
from config import CHROMA_DIR, COLLECTION, EMBED_MODEL, KB_PATH
load_dotenv() # OPENAI_API_KEY from .env
def build_index() -> int:
text = Path(KB_PATH).read_text(encoding="utf-8")
# Split into overlapping chunks so retrieval returns focused, self-contained
# passages while preserving context across boundaries.
splitter = RecursiveCharacterTextSplitter(chunk_size=600, chunk_overlap=80)
chunks = splitter.create_documents([text], metadatas=[{"source": KB_PATH}])
# Embed with OpenAI and persist to Chroma on disk.
store = Chroma(
collection_name=COLLECTION,
embedding_function=OpenAIEmbeddings(model=EMBED_MODEL),
persist_directory=CHROMA_DIR,
)
store.add_documents(chunks)
return len(chunks)
if __name__ == "__main__":
n = build_index()
print(f"Ingested {n} chunks into Chroma collection '{COLLECTION}' at {CHROMA_DIR}")
"""Customer-support agent.
Ties together the whole stack:
- LangGraph β the stateful agent loop (agent β tools) with conversation memory
- OpenAI β gpt-4o for reasoning, text-embedding-3-small for retrieval
- Chroma β the persistent vector database holding the knowledge base
- RAG β the `search_kb` tool retrieves grounding context on demand
- LangSmith β automatic tracing when LANGSMITH_TRACING=true
Run `python ingest.py` first to build the vector store, then `python agent.py`.
"""
from typing import Annotated, TypedDict
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage, SystemMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition
from langgraph.checkpoint.memory import MemorySaver
from config import CHROMA_DIR, COLLECTION, CHAT_MODEL, EMBED_MODEL
load_dotenv() # OPENAI_API_KEY, LANGSMITH_TRACING, LANGSMITH_API_KEY
# Connect to the vector store built by ingest.py.
_store = Chroma(
collection_name=COLLECTION,
embedding_function=OpenAIEmbeddings(model=EMBED_MODEL),
persist_directory=CHROMA_DIR,
)
@tool
def search_kb(query: str) -> str:
"""Search the customer-support knowledge base for relevant policy/help text."""
docs = _store.similarity_search(query, k=4) # RAG retrieval over Chroma
return "\n\n".join(d.page_content for d in docs) or "No relevant policy found."
@tool
def escalate_to_human(reason: str) -> str:
"""Escalate to a human agent when the KB can't answer or the issue is sensitive."""
return f"Escalated to a human specialist. Reason: {reason}"
TOOLS = [search_kb, escalate_to_human]
model = ChatOpenAI(model=CHAT_MODEL, temperature=0).bind_tools(TOOLS)
SYSTEM = SystemMessage(
"You are Acme's support assistant. Ground every policy answer by calling search_kb first. "
"If the knowledge base lacks the answer, or the request is sensitive (refunds over $500, "
"legal, account security), call escalate_to_human instead of guessing. "
"Cite the policy you relied on and keep replies concise."
)
class State(TypedDict):
messages: Annotated[list, add_messages]
def agent(state: State) -> State:
# The model reasons over the conversation and decides: answer, or call a tool.
return {"messages": [model.invoke([SYSTEM] + state["messages"])]}
builder = StateGraph(State)
builder.add_node("agent", agent)
builder.add_node("tools", ToolNode(TOOLS))
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", tools_condition) # agent -> tools, or -> END
builder.add_edge("tools", "agent") # loop back after a tool runs
# The checkpointer gives the agent short-term memory across turns of a thread.
graph = builder.compile(checkpointer=MemorySaver())
def ask(question: str, thread_id: str = "demo") -> str:
cfg = {"configurable": {"thread_id": thread_id}}
out = graph.invoke({"messages": [HumanMessage(question)]}, cfg)
return out["messages"][-1].content
if __name__ == "__main__":
# A grounded policy question (agent should call search_kb):
print("Q1:", ask("How many days do I have to return an item, and who pays return shipping?"))
print("-" * 60)
# A sensitive request on the same thread (agent should escalate):
print("Q2:", ask("I want a $2000 refund for a laptop that arrived damaged."))
# Acme Support Knowledge Base
## Returns & Refunds
- Items may be returned within **30 days** of delivery for a full refund.
- The **customer pays return shipping** unless the item arrived damaged or incorrect.
- Refunds are issued to the original payment method within **5β7 business days** of us receiving the return.
- **Shipping fees are non-refundable** except for damaged or incorrect items.
- Digital goods are **non-refundable** once downloaded.
- Refund requests over **$500** require manual review by a specialist.
## Shipping
- Standard shipping takes **3β5 business days**; express takes **1β2 business days**.
- Free standard shipping on orders over **$50**.
- We ship to the US, Canada, the UK, and the EU.
- Tracking numbers are emailed once an order ships.
## Orders
- Orders can be **modified or cancelled within 1 hour** of being placed.
- After 1 hour, an order enters fulfillment and can no longer be changed.
- To check order status, sign in and visit **Account β Orders**.
## Accounts & Security
- Reset your password from the sign-in page via **"Forgot password"**.
- We never ask for your password by email. Report suspicious messages to security@acme.example.
- Account deletion requests are handled by a specialist and require identity verification.
## Payments
- We accept Visa, Mastercard, American Express, and PayPal.
- Prices are in **USD**.
- Promo codes apply to the item subtotal only, not taxes or shipping.
9 Β· Knowledge check
Five quick questions. Pick an answer for each, then check your score.
Next steps
π§ Transformers
How LLMs work under the hood β tokens, embeddings, attention and sampling.
βοΈ Prompt engineering
Steer the model well β prompt anatomy, few-shot, structured output & context.
π LangChain guide
The full LangChain / LangGraph / LangSmith write-up with diagrams and comparisons.
ποΈ Vector databases
Deep dive on storing chunk data across Chroma, FAISS, Qdrant, Pinecone, pgvector.
π RAG vs CAG vs GraphRAG
Grounding techniques compared β when to retrieve, when to preload & cache, when to mix, and GraphRAG.
βοΈ LangChain vs LlamaIndex
Which framework? β LangChain's breadth vs LlamaIndex's data/RAG depth, and when to use each.
π οΈ create-llm-app
The CLI on GitHub β scaffold a runnable app in seconds.
π§ͺ Agentic experiments
25 runnable scaffolds β agentic design patterns plus reasoning techniques.
π€ Support agent
The full-stack example β LangGraph + OpenAI + Chroma RAG + LangSmith.
βοΈ AWS services guide
An architect's field guide to 40 core AWS services β the problem each solves and its limits (pairs with Bedrock).
π§° Agent Development Kits
Google ADK vs AWS Strands + Bedrock AgentCore vs LangGraph β problem, solution & limits for each.
π‘ IoT Γ Agents
A smart fridge that auto-orders groceries β AWS IoT Core wired to a Strands agent on Bedrock AgentCore.