← All tutorials
How LLMs Work

How LLMs & transformers work

Every large language model does one thing: predict the next token. Under the hood it's a transformer β€” text becomes tokens, tokens become vectors, attention layers mix in context, and the model outputs a probability for every possible next token. Sample one, append it, repeat. That loop, at scale, is everything.

The LLM forward pass Text is tokenized, embedded with position, passed through N transformer layers (attention plus feed-forward), producing logits that are turned into probabilities and sampled into the next token, which is appended and the process repeats. append token → repeat (autoregressive) Text→ tokenize Embed+ position Transformer ×Nattention + feed-forward Logits→ softmax Samplenext token
The big idea. An LLM isn't a database of facts or a reasoning engine β€” it's a very good next-token predictor trained on enormous text. Everything it appears to "know" or "reason" emerges from predicting plausible continuations.

Step 1  Tokens

Models don't see letters or words β€” they see tokens: common chunks of text (whole words, word-pieces, or characters) from a fixed vocabulary (~50k–200k entries). "tokenization" is roughly "β‰ˆ 4 characters" in English, so ~750 words β‰ˆ 1,000 tokens.

Why you care. The context window is measured in tokens (how much the model can see at once), and you're billed per token. Rare words, code and other languages tokenize into more tokens.

Step 2  Embeddings & position

Each token id is mapped to a learned vector (its embedding) β€” a point in high-dimensional space capturing meaning (see the embeddings guide). Because a plain transformer has no inherent sense of order, a positional encoding is added so the model knows where each token sits in the sequence.

Step 3 Β· the key idea  Attention

Self-attention is what made transformers work. For every token, the model asks: which other tokens should I pay attention to, to understand this one? It computes a query, key and value for each token; the queryΒ·key match decides how much each other token's value contributes. So "it" can attend to the noun it refers to, and a word's meaning is shaped by its whole context.

Multi-head attention runs several of these in parallel, each learning a different kind of relationship (syntax, coreference, topic…). This is the "Attention Is All You Need" mechanism.

In one line. Attention lets every token gather relevant context from the rest of the sequence β€” that's how the model handles long-range meaning instead of just local word order.

Step 4  The transformer block

One transformer layer = multi-head attention β†’ a feed-forward network β†’ with residual connections and normalization around each. Stack N of these (dozens to over a hundred) and you have the model. The count of all the numbers in these layers is the model's parameters (billions).

input vectors→ attention→ feed-forward→ richer vectors× N layers

Step 5  Predicting the next token

The final layer produces a score (logit) for every token in the vocabulary. Softmax turns those into probabilities, and the model samples one β€” then appends it and runs the whole pass again for the next token. This one-at-a-time generation is called autoregressive.

KnobEffect
TemperatureHigher = more random/creative; lower = more focused/deterministic.
Top-p / Top-kLimit sampling to the most likely tokens (nucleus sampling).
Max tokensHow many tokens to generate before stopping.

Step 6  How it's trained

πŸ“š Pretraining

Predict the next token across a huge text corpus. Errors adjust the parameters (weights) via back-propagation β€” for a very long time, on many GPUs.

🎯 Fine-tuning / RLHF

Then align it β€” instruction tuning and reinforcement learning from human feedback teach it to be helpful, follow instructions, and be safe.

The result is a fixed set of weights β€” the model. It has a knowledge cutoff (it only "knows" what was in its training data) and doesn't learn from your chats unless you add memory or retrieval.

So what  Why it matters for builders

πŸͺŸ Context window

Everything the model uses must fit in the token budget β€” prompt + retrieved data + output.

πŸ’Έ Tokens = cost & latency

You pay and wait per token. Concise prompts and outputs save both.

🎲 Hallucination

It predicts plausible tokens, not true ones β€” so ground it (RAG) and verify.

πŸ“… Knowledge cutoff

No live knowledge; use retrieval or tools for fresh/private facts.

The mental model that pays off: treat the LLM as a powerful next-token predictor with a fixed context window and no memory β€” then everything else in this repo (prompting, RAG, tools, agents, evals) is about feeding it the right context and checking its output.

Cheat sheet

QuestionShort answer
What does an LLM fundamentally do?Predict the next token, repeatedly (autoregressive).
What's a token?A chunk of text (~4 chars) from a fixed vocabulary; the unit of context & billing.
The key mechanism?Self-attention β€” each token gathers context from the others.
What are parameters?The billions of learned weights in the stacked transformer layers.
Why does it hallucinate?It outputs plausible tokens, not verified facts.
Two knobs to know?Temperature (randomness) and max tokens (length).

Knowledge check

Six questions on how LLMs work. Pick an answer for each, then check your score.

1. What does an LLM fundamentally do?

2. What is a token?

3. What is self-attention for?

4. What are a model's "parameters"?

5. Why do LLMs hallucinate?

6. What does raising the temperature do?

Next steps

🧬 Embedding models

Vectors in depth β€” the token embeddings this relies on.

✍️ Prompt & context engineering

Feed it well β€” get the most from the next-token predictor.

πŸ€– Build LLM apps

Put it to work β€” chains, RAG and agents on top of the model.

πŸ““ All tutorials

The hub β€” LLMs, agents, ML, cloud & more.