How LLMs & transformers work
Every large language model does one thing: predict the next token. Under the hood it's a transformer β text becomes tokens, tokens become vectors, attention layers mix in context, and the model outputs a probability for every possible next token. Sample one, append it, repeat. That loop, at scale, is everything.
Step 1 Tokens
Models don't see letters or words β they see tokens: common chunks of text (whole words, word-pieces, or characters) from a fixed vocabulary (~50kβ200k entries). "tokenization" is roughly "β 4 characters" in English, so ~750 words β 1,000 tokens.
Step 2 Embeddings & position
Each token id is mapped to a learned vector (its embedding) β a point in high-dimensional space capturing meaning (see the embeddings guide). Because a plain transformer has no inherent sense of order, a positional encoding is added so the model knows where each token sits in the sequence.
Step 3 Β· the key idea Attention
Self-attention is what made transformers work. For every token, the model asks: which other tokens should I pay attention to, to understand this one? It computes a query, key and value for each token; the queryΒ·key match decides how much each other token's value contributes. So "it" can attend to the noun it refers to, and a word's meaning is shaped by its whole context.
Multi-head attention runs several of these in parallel, each learning a different kind of relationship (syntax, coreference, topicβ¦). This is the "Attention Is All You Need" mechanism.
Step 4 The transformer block
One transformer layer = multi-head attention β a feed-forward network β with residual connections and normalization around each. Stack N of these (dozens to over a hundred) and you have the model. The count of all the numbers in these layers is the model's parameters (billions).
Step 5 Predicting the next token
The final layer produces a score (logit) for every token in the vocabulary. Softmax turns those into probabilities, and the model samples one β then appends it and runs the whole pass again for the next token. This one-at-a-time generation is called autoregressive.
| Knob | Effect |
|---|---|
| Temperature | Higher = more random/creative; lower = more focused/deterministic. |
| Top-p / Top-k | Limit sampling to the most likely tokens (nucleus sampling). |
| Max tokens | How many tokens to generate before stopping. |
Step 6 How it's trained
π Pretraining
Predict the next token across a huge text corpus. Errors adjust the parameters (weights) via back-propagation β for a very long time, on many GPUs.
π― Fine-tuning / RLHF
Then align it β instruction tuning and reinforcement learning from human feedback teach it to be helpful, follow instructions, and be safe.
The result is a fixed set of weights β the model. It has a knowledge cutoff (it only "knows" what was in its training data) and doesn't learn from your chats unless you add memory or retrieval.
So what Why it matters for builders
πͺ Context window
Everything the model uses must fit in the token budget β prompt + retrieved data + output.
πΈ Tokens = cost & latency
You pay and wait per token. Concise prompts and outputs save both.
π² Hallucination
It predicts plausible tokens, not true ones β so ground it (RAG) and verify.
π Knowledge cutoff
No live knowledge; use retrieval or tools for fresh/private facts.
Cheat sheet
| Question | Short answer |
|---|---|
| What does an LLM fundamentally do? | Predict the next token, repeatedly (autoregressive). |
| What's a token? | A chunk of text (~4 chars) from a fixed vocabulary; the unit of context & billing. |
| The key mechanism? | Self-attention β each token gathers context from the others. |
| What are parameters? | The billions of learned weights in the stacked transformer layers. |
| Why does it hallucinate? | It outputs plausible tokens, not verified facts. |
| Two knobs to know? | Temperature (randomness) and max tokens (length). |
Knowledge check
Six questions on how LLMs work. Pick an answer for each, then check your score.
Next steps
𧬠Embedding models
Vectors in depth β the token embeddings this relies on.
βοΈ Prompt & context engineering
Feed it well β get the most from the next-token predictor.
π€ Build LLM apps
Put it to work β chains, RAG and agents on top of the model.
π All tutorials
The hub β LLMs, agents, ML, cloud & more.