← All tutorials
RAG vs CAG

RAG vs CAG vs GraphRAG β€” grounding LLMs in your data

A language model only knows its training data β€” not your private documents, and nothing that happened after its cutoff. Every technique here solves the same core problem, grounding, in a different way. The art is matching the method to your corpus's size, volatility and question type.

Retrieve at query time

πŸ”Ž RAG

Fetch only the relevant chunks from a big, changing corpus and inject them into the prompt.

Preload & cache

🧠 CAG

Put the whole (small, stable) corpus in context once and reuse the cached state for every query.

Reason over structure

πŸ•ΈοΈ GraphRAG

Build a knowledge graph so the model can answer multi-hop and "big picture" questions.

One sentence each. RAG searches a library and hands you the right pages; CAG hands the model the whole (small) book and remembers it; GraphRAG first maps how everything connects, then answers.

At a glance  Which technique? A decision map

Three questions β€” does it fit in context?, does it change often?, are questions relational? β€” route you to the right method. Follow the branches:

Decision map for choosing a grounding technique If the whole corpus fits in the context window: choose CAG if it rarely changes, or RAG if it changes frequently. If it does not fit: choose GraphRAG for multi-hop or whole-corpus questions, otherwise RAG. A stable core plus a volatile long tail means mixing CAG and RAG. fits βœ“ too big βœ• stable changes simple relational Your knowledge base Fits in the context window? Changes frequently? Multi-hop or whole-corpus? CAG small Β· stable RAG fits, but changing RAG big Β· lookups GraphRAG multi-hop Β· global Stable core AND a big / volatile long tail? β†’ Mix: CAG the core + RAG the rest
Reading it: the two RAG leaves aren't a mistake β€” RAG is the default workhorse whenever CAG (small & stable) and GraphRAG (relational / global) don't apply. The dashed band is the common production reality: most real systems mix.

Why  The grounding problem

Ask a raw model about your company's refund policy and it will confidently make something up β€” a hallucination β€” because the answer isn't in its weights. Fine-tuning to teach it facts is slow, expensive, hard to update, and still blurs details. The pragmatic fix is to put the right facts in the prompt at inference time and tell the model to answer only from them.

That's the whole game. The techniques differ only in how the right facts get into the context:

your data→ get it into context→ model answers, grounded→ cite sources

Three questions decide which method fits: How big is the corpus? (does it fit in a context window?) How often does it change? (stable vs. live) and What kind of questions? (pinpoint lookups vs. multi-hop or whole-corpus synthesis).

Technique 1  RAG β€” Retrieval-Augmented Generation

RAG retrieve relevant chunks, then generate

Solves Grounding over a large or constantly changing corpus that could never fit in a prompt. You embed documents into a vector store, and at query time retrieve only the handful of chunks semantically closest to the question β€” so the model sees just what's relevant, and the index can hold millions of documents and be updated continuously.

Limits The answer is only as good as retrieval β€” miss the right chunk and the model can't recover ("retrieval is the ceiling"). Chunking severs context across boundaries; purely semantic search struggles with exact keywords, and with multi-hop ("compare A and B across docs") or global ("what are the main themes?") questions. It also adds infrastructure (embeddings + a vector DB) and per-query retrieval latency.

Choose Big, dynamic knowledge bases; fact-lookup and "find the relevant passage" questions; when the corpus can't fit in context or changes too often to re-cache.

docs→ chunk→ embed→ vector store→ retrieve top-k→ answer
rag.py β€” retrieve then answer
# Only the top-k relevant chunks reach the model β€” scales to millions of docs.
retriever = store.as_retriever(search_kwargs={"k": 4})

def ask(question: str) -> str:
    chunks  = retriever.invoke(question)                 # semantic search
    context = "\n\n".join(c.page_content for c in chunks)
    return llm.invoke(
        f"Answer using ONLY this context:\n{context}\n\nQuestion: {question}"
    )
Where the chunks live & a full pipeline: see the RAG section of the LLM apps tutorial and the vector databases guide (Chroma, FAISS, Qdrant, Pinecone, pgvector).

Technique 2  CAG β€” Cache-Augmented Generation

CAG preload the whole corpus, cache the context

Solves The failure modes of retrieval, when your corpus is small and stable enough to fit in a long context window. You load the entire knowledge base into the prompt once; the model's key/value state for that prefix is cached, so every later question reuses it with no retrieval step. No vector DB, no chunking, no "missed the right chunk" β€” and the model sees the full document, preserving context a chunk would lose.

Limits Hard-bounded by the context window β€” unusable for corpora larger than it (roughly hundreds of thousands of tokens today). Without caching you re-pay to process the big prefix on every call; even with caching, a large context adds cost/latency versus a few retrieved chunks. The cache goes stale when data changes β€” you must rebuild it β€” so it's wrong for fast-moving data.

Choose Small, stable corpora: a product manual, a policy handbook, an API spec, a single long contract β€” especially with repeated questions against the same fixed text.

preload whole KB β†’ cache⟳ Q1Β· Q2Β· Q3 …→ answers (no retrieval)
cag.py β€” preload once, cache the prefix
# The whole handbook fits in context; prompt caching stores the KV state so
# repeated questions skip re-processing it β€” CAG in ~15 lines.
import anthropic
client = anthropic.Anthropic()

KB = open("handbook.md").read()          # small & stable β†’ fits in context

def ask(question: str) -> str:
    msg = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=512,
        system=[{
            "type": "text",
            "text": f"Answer only from this handbook:\n\n{KB}",
            "cache_control": {"type": "ephemeral"},   # cache the big prefix
        }],
        messages=[{"role": "user", "content": question}],
    )
    return msg.content[0].text
CAG β‰ˆ long context + caching. It only became practical as context windows grew and providers added prompt caching. The "cache" is the model's KV state for the preloaded text β€” reused across questions, not recomputed.

RAG vs CAG β€” a direct comparison

RAGCAG
Corpus sizeUnbounded (millions of docs)Must fit in the context window
Data volatilityGreat β€” update the index anytimePoor β€” rebuild the cache on change
Per-query latencyRetrieval step adds latencyNone once cached; but large context to process
Main failure modeMissed / wrong retrievalContext-window overflow & staleness
Sees full context?No β€” only the chunksYes β€” the whole document
InfrastructureEmbeddings + vector DBJust the model (+ caching)
Best forLarge, dynamic, lookup-styleSmall, stable, repeated Q&A
Rule of thumb. Does the knowledge fit in the context window and rarely change? Start with CAG β€” it's simpler and avoids retrieval error. Otherwise you need RAG. Many real systems end up combining both.

Technique 3  When to mix them (RAG + CAG)

The techniques aren't rivals β€” they cover different slices of your knowledge. A common production shape splits knowledge by stability:

CAG: stable core + RAG: volatile / long-tail β†’ grounded answer

Cache the always-relevant, slow-changing core β€” the style guide, glossary, core policies, product overview β€” so it's always in context with zero retrieval. Retrieve the large, fast-moving, or long-tail material β€” this week's tickets, the full product catalog, live docs β€” only when a question needs it.

Put in CAG (cached context)Put in RAG (retrieved)
Small, stable, always-needed factsLarge, dynamic, or rarely-needed facts
Company handbook, glossary, tone guideTicket history, catalog, changelog, news
Fits in context; changes monthlyWon't fit; changes hourly
Runnable example. This repo's experiment 14 (RAG + CAG) wires exactly this: a cached stable prefix plus a retrieved volatile corpus.

Technique 4  GraphRAG β€” reasoning over structure

GraphRAG knowledge graph + community summaries

Solves The questions plain RAG is worst at: multi-hop reasoning ("how is person A connected to project C?") and global synthesis ("what are the major themes across all these reports?"). Instead of a flat pile of chunks, an LLM first extracts entities and relationships into a knowledge graph, clusters it into communities, and summarizes each β€” so retrieval can follow relationships and roll up whole-corpus answers.

Limits Building the graph is expensive and slow β€” it's an LLM pass over the entire corpus to extract entities/relations, costing far more to index than embedding chunks. It's more moving parts to maintain and keep fresh, and it's overkill for simple fact lookups where vector RAG already wins. Extraction quality drives everything.

Choose Connected, entity-rich corpora (research, investigations, org/knowledge bases) where questions span many documents or need a "big picture" no single chunk contains.

docs→ extract entities & relations→ knowledge graph→ community summaries→ local or global query
Local vs global GraphRAG queries

Local queries start from the entities named in the question and walk their neighborhood in the graph β€” great for "tell me everything about X and what it connects to." Global queries map-reduce over the pre-computed community summaries β€” great for "summarize the whole corpus," which flat top-k retrieval simply can't assemble. Microsoft's GraphRAG popularized this; lighter variants (e.g. LightRAG) cut indexing cost.

The wider family

"RAG" is really a toolbox of techniques that each fix a specific weakness. Most are complementary β€” you compose them.

TechniqueProblem it targetsWhat it does
Hybrid searchSemantic search misses exact terms/IDsCombine dense (embeddings) + sparse (BM25/keyword) retrieval, then fuse scores.
RerankingTop-k has relevant docs buried lowA cross-encoder re-scores retrieved candidates for precision before they hit the prompt.
HyDEShort queries embed poorlyGenerate a hypothetical answer, embed that, and retrieve against it.
Query rewritingVague or multi-part questionsRephrase/expand/decompose the query before retrieving.
Contextual RetrievalChunks lose their document contextPrepend an LLM-written situating blurb to each chunk before embedding (pairs with prompt caching).
Agentic RAGOne retrieval isn't enoughGive the agent retrieval as a tool it calls iteratively as it reasons. exp 22
Corrective RAG (CRAG)Retrieved docs are low qualityGrade retrieval; if weak, fall back (e.g. web search) before answering. exp 23
Self-RAGRetrieving when unneeded; ungrounded claimsModel decides whether to retrieve and critiques its own grounding. exp 24
GraphRAGMulti-hop & global questionsKnowledge graph + community summaries (see above).
Multimodal RAGAnswers live in images/tables/chartsEmbed and retrieve across modalities, not just text.
Layering order that works: hybrid retrieve β†’ rerank β†’ (optionally) grade & correct β†’ generate with citations. Add agentic/self-RAG when a single pass isn't enough, and GraphRAG when questions are relational or global.

Decision cheat sheet

If your situation is…Reach for…
Corpus fits in context & rarely changes; repeated Q&ACAG
Large or frequently-updated corpus; fact lookupsRAG
A stable core + a big/volatile long tailRAG + CAG (cache the core, retrieve the rest)
Questions span many docs or need a whole-corpus summaryGraphRAG
Exact IDs/keywords matter alongside meaningRAG + hybrid search + reranking
Retrieval quality is unreliableCorrective or Self-RAG
A task needs several retrieval rounds while reasoningAgentic RAG
Start simple. Reach for the lightest method that answers your questions well β€” often CAG for small corpora, or plain RAG β€” and add hybrid/rerank/graph machinery only when you can measure that it helps. Evaluate retrieval and answers (see tracing & evaluation) before adding complexity.

Knowledge check

Six questions on matching the technique to the problem. Pick an answer for each, then check your score.

1. What core problem do RAG, CAG and GraphRAG all address?

2. Your knowledge base is a 40-page product manual that changes a few times a year, asked about all day. Best starting point?

3. What is RAG's main failure mode?

4. Why is CAG a poor fit for fast-changing data?

5. "What are the major themes across all 500 reports?" β€” which technique is built for this?

6. You have a small stable core plus a huge, hourly-changing long tail. Best design?

Next steps

🧬 Embedding models

How embeddings work β€” the vectors that make retrieval possible, and how to choose one.

πŸ€– RAG in an app

The LLM apps tutorial β€” a full RAG chain, plus tracing & evaluation to measure retrieval quality.

πŸ—„οΈ Vector databases

Where chunks live, compared β€” Chroma, FAISS, Qdrant, Pinecone, pgvector & more.

🎚️ RAG vs fine-tuning

When to retrieve vs tune vs prompt β€” RAG isn't always the answer; a decision guide.

πŸ§ͺ RAG experiments

Runnable scaffolds β€” RAG, RAG+CAG, Agentic, Corrective & Self-RAG.

πŸ““ All tutorials

The hub β€” LLM apps, agent kits, AWS services, and the IoT example.