RAG vs CAG vs GraphRAG β grounding LLMs in your data
A language model only knows its training data β not your private documents, and nothing that happened after its cutoff. Every technique here solves the same core problem, grounding, in a different way. The art is matching the method to your corpus's size, volatility and question type.
Retrieve at query time
π RAG
Fetch only the relevant chunks from a big, changing corpus and inject them into the prompt.
Preload & cache
π§ CAG
Put the whole (small, stable) corpus in context once and reuse the cached state for every query.
Reason over structure
πΈοΈ GraphRAG
Build a knowledge graph so the model can answer multi-hop and "big picture" questions.
One sentence each. RAG searches a library and hands you the right pages; CAG hands the model the whole (small) book and remembers it; GraphRAG first maps how everything connects, then answers.
At a glance Which technique? A decision map
Three questions β does it fit in context?, does it change often?, are questions relational? β route you to the right method. Follow the branches:
Reading it: the two RAG leaves aren't a mistake β RAG is the default workhorse whenever CAG (small & stable) and GraphRAG (relational / global) don't apply. The dashed band is the common production reality: most real systems mix.
Why The grounding problem
Ask a raw model about your company's refund policy and it will confidently make something up β a hallucination β because the answer isn't in its weights. Fine-tuning to teach it facts is slow, expensive, hard to update, and still blurs details. The pragmatic fix is to put the right facts in the prompt at inference time and tell the model to answer only from them.
That's the whole game. The techniques differ only in how the right facts get into the context:
your dataβget it into contextβmodel answers, groundedβcite sources
Three questions decide which method fits: How big is the corpus? (does it fit in a context window?) How often does it change? (stable vs. live) and What kind of questions? (pinpoint lookups vs. multi-hop or whole-corpus synthesis).
Solves Grounding over a large or constantly changing corpus that could never fit in a prompt. You embed documents into a vector store, and at query time retrieve only the handful of chunks semantically closest to the question β so the model sees just what's relevant, and the index can hold millions of documents and be updated continuously.
Limits The answer is only as good as retrieval β miss the right chunk and the model can't recover ("retrieval is the ceiling"). Chunking severs context across boundaries; purely semantic search struggles with exact keywords, and with multi-hop ("compare A and B across docs") or global ("what are the main themes?") questions. It also adds infrastructure (embeddings + a vector DB) and per-query retrieval latency.
Choose Big, dynamic knowledge bases; fact-lookup and "find the relevant passage" questions; when the corpus can't fit in context or changes too often to re-cache.
# Only the top-k relevant chunks reach the model β scales to millions of docs.
retriever = store.as_retriever(search_kwargs={"k": 4})
def ask(question: str) -> str:
chunks = retriever.invoke(question) # semantic search
context = "\n\n".join(c.page_content for c in chunks)
return llm.invoke(
f"Answer using ONLY this context:\n{context}\n\nQuestion: {question}"
)
Solves The failure modes of retrieval, when your corpus is small and stable enough to fit in a long context window. You load the entire knowledge base into the prompt once; the model's key/value state for that prefix is cached, so every later question reuses it with no retrieval step. No vector DB, no chunking, no "missed the right chunk" β and the model sees the full document, preserving context a chunk would lose.
Limits Hard-bounded by the context window β unusable for corpora larger than it (roughly hundreds of thousands of tokens today). Without caching you re-pay to process the big prefix on every call; even with caching, a large context adds cost/latency versus a few retrieved chunks. The cache goes stale when data changes β you must rebuild it β so it's wrong for fast-moving data.
Choose Small, stable corpora: a product manual, a policy handbook, an API spec, a single long contract β especially with repeated questions against the same fixed text.
preload whole KB β cacheβ³Q1Β·Q2Β·Q3 β¦βanswers (no retrieval)
cag.py β preload once, cache the prefix
# The whole handbook fits in context; prompt caching stores the KV state so# repeated questions skip re-processing it β CAG in ~15 lines.
import anthropic
client = anthropic.Anthropic()
KB = open("handbook.md").read() # small & stable β fits in context
def ask(question: str) -> str:
msg = client.messages.create(
model="claude-opus-4-8",
max_tokens=512,
system=[{
"type": "text",
"text": f"Answer only from this handbook:\n\n{KB}",
"cache_control": {"type": "ephemeral"}, # cache the big prefix
}],
messages=[{"role": "user", "content": question}],
)
return msg.content[0].text
CAG β long context + caching. It only became practical as context windows grew and providers added prompt caching. The "cache" is the model's KV state for the preloaded text β reused across questions, not recomputed.
RAG vs CAG β a direct comparison
RAG
CAG
Corpus size
Unbounded (millions of docs)
Must fit in the context window
Data volatility
Great β update the index anytime
Poor β rebuild the cache on change
Per-query latency
Retrieval step adds latency
None once cached; but large context to process
Main failure mode
Missed / wrong retrieval
Context-window overflow & staleness
Sees full context?
No β only the chunks
Yes β the whole document
Infrastructure
Embeddings + vector DB
Just the model (+ caching)
Best for
Large, dynamic, lookup-style
Small, stable, repeated Q&A
Rule of thumb. Does the knowledge fit in the context window and rarely change? Start with CAG β it's simpler and avoids retrieval error. Otherwise you need RAG. Many real systems end up combining both.
Technique 3 When to mix them (RAG + CAG)
The techniques aren't rivals β they cover different slices of your knowledge. A common production shape splits knowledge by stability:
Cache the always-relevant, slow-changing core β the style guide, glossary, core policies, product overview β so it's always in context with zero retrieval. Retrieve the large, fast-moving, or long-tail material β this week's tickets, the full product catalog, live docs β only when a question needs it.
Put in CAG (cached context)
Put in RAG (retrieved)
Small, stable, always-needed facts
Large, dynamic, or rarely-needed facts
Company handbook, glossary, tone guide
Ticket history, catalog, changelog, news
Fits in context; changes monthly
Won't fit; changes hourly
Runnable example. This repo's experiment 14 (RAG + CAG) wires exactly this: a cached stable prefix plus a retrieved volatile corpus.
Technique 4 GraphRAG β reasoning over structure
GraphRAG knowledge graph + community summaries
Solves The questions plain RAG is worst at: multi-hop reasoning ("how is person A connected to project C?") and global synthesis ("what are the major themes across all these reports?"). Instead of a flat pile of chunks, an LLM first extracts entities and relationships into a knowledge graph, clusters it into communities, and summarizes each β so retrieval can follow relationships and roll up whole-corpus answers.
Limits Building the graph is expensive and slow β it's an LLM pass over the entire corpus to extract entities/relations, costing far more to index than embedding chunks. It's more moving parts to maintain and keep fresh, and it's overkill for simple fact lookups where vector RAG already wins. Extraction quality drives everything.
Choose Connected, entity-rich corpora (research, investigations, org/knowledge bases) where questions span many documents or need a "big picture" no single chunk contains.
docsβextract entities & relationsβknowledge graphβcommunity summariesβlocal or global query
Local vs global GraphRAG queries
Local queries start from the entities named in the question and walk their neighborhood in the graph β great for "tell me everything about X and what it connects to." Global queries map-reduce over the pre-computed community summaries β great for "summarize the whole corpus," which flat top-k retrieval simply can't assemble. Microsoft's GraphRAG popularized this; lighter variants (e.g. LightRAG) cut indexing cost.
The wider family
"RAG" is really a toolbox of techniques that each fix a specific weakness. Most are complementary β you compose them.
Technique
Problem it targets
What it does
Hybrid search
Semantic search misses exact terms/IDs
Combine dense (embeddings) + sparse (BM25/keyword) retrieval, then fuse scores.
Reranking
Top-k has relevant docs buried low
A cross-encoder re-scores retrieved candidates for precision before they hit the prompt.
HyDE
Short queries embed poorly
Generate a hypothetical answer, embed that, and retrieve against it.
Query rewriting
Vague or multi-part questions
Rephrase/expand/decompose the query before retrieving.
Contextual Retrieval
Chunks lose their document context
Prepend an LLM-written situating blurb to each chunk before embedding (pairs with prompt caching).
Agentic RAG
One retrieval isn't enough
Give the agent retrieval as a tool it calls iteratively as it reasons. exp 22
Corrective RAG (CRAG)
Retrieved docs are low quality
Grade retrieval; if weak, fall back (e.g. web search) before answering. exp 23
Self-RAG
Retrieving when unneeded; ungrounded claims
Model decides whether to retrieve and critiques its own grounding. exp 24
GraphRAG
Multi-hop & global questions
Knowledge graph + community summaries (see above).
Multimodal RAG
Answers live in images/tables/charts
Embed and retrieve across modalities, not just text.
Layering order that works: hybrid retrieve β rerank β (optionally) grade & correct β generate with citations. Add agentic/self-RAG when a single pass isn't enough, and GraphRAG when questions are relational or global.
Decision cheat sheet
If your situation isβ¦
Reach forβ¦
Corpus fits in context & rarely changes; repeated Q&A
CAG
Large or frequently-updated corpus; fact lookups
RAG
A stable core + a big/volatile long tail
RAG + CAG (cache the core, retrieve the rest)
Questions span many docs or need a whole-corpus summary
GraphRAG
Exact IDs/keywords matter alongside meaning
RAG + hybrid search + reranking
Retrieval quality is unreliable
Corrective or Self-RAG
A task needs several retrieval rounds while reasoning
Agentic RAG
Start simple. Reach for the lightest method that answers your questions well β often CAG for small corpora, or plain RAG β and add hybrid/rerank/graph machinery only when you can measure that it helps. Evaluate retrieval and answers (see tracing & evaluation) before adding complexity.
Knowledge check
Six questions on matching the technique to the problem. Pick an answer for each, then check your score.
Next steps
𧬠Embedding models
How embeddings work β the vectors that make retrieval possible, and how to choose one.
π€ RAG in an app
The LLM apps tutorial β a full RAG chain, plus tracing & evaluation to measure retrieval quality.