How LLMs work

Why this matters

Large language models (LLMs) are now part of many data science and ML engineering jobs: summarising documents, extracting fields from messy text, classifying tickets without training data, answering questions over company documents. To use them well you need a working model of what they actually do. Most surprises (a made-up citation, a different answer each time, a forgotten instruction, a surprising bill) follow directly from three ideas: tokens, the context window and sampling.

The core idea: predict the next token

An LLM is a neural network trained on a huge amount of text to do one thing: given a sequence of tokens, output a probability for every possible next token. To generate text it repeats that step: pick a token, append it to the input, predict again.

input:  "The capital of France is"
model:  " Paris" 0.92   " a" 0.03   " the" 0.02   ...
append " Paris", predict again: "." 0.71   "," 0.18 ...

Nothing in this loop looks facts up. The model produces text that is likely given its training data and the prompt. That is why it can write fluent, confident and wrong answers (often called hallucinations), and why grounding the model in real documents (next lesson) matters.

Tokens

Models read tokens, not words or characters. A tokenizer (usually byte-pair encoding) splits text into frequent chunks. One tokenizer might split a sentence like this (the exact pieces differ between models):

"Tokenization is unbelievable"  ->  ["Token", "ization", " is", " un", "believ", "able"]

Rules of thumb for English: one token is about four characters, or about three quarters of a word. Code, numbers and many non-English languages use more tokens per word. Tokens matter because:

• Pricing and limits are per token (input and output are often priced differently).
• Spelling and arithmetic are hard: the model sees "believ" as one unit, not letters, and numbers may be split oddly.
• Every tokenizer is different, so count with the right one (for example the tiktoken library for OpenAI models, or the tokenizer that ships with an open model on Hugging Face).

The transformer and attention

Modern LLMs are transformers. Each token becomes an embedding (as in the text lesson), and a stack of layers updates those vectors. The key operation is self-attention: each token computes how relevant every earlier token is and mixes in information from them. That is how "it" in "The trophy didn't fit in the case because it was too big" can draw on "trophy". Attention looks only backwards in a generative model, so each prediction depends on everything before it in the input.

Training happens in stages. Pretraining on large text collections gives the next-token skill. Instruction tuning and preference tuning (for example reinforcement learning from human feedback) teach the model to follow instructions, use a chat format and decline harmful requests.

The context window

The context window is the maximum number of tokens the model can consider at once: the system instructions, the conversation so far, any documents you paste in and the answer it is writing. Windows range from a few thousand to over a million tokens depending on the model. Consequences:

• The model has no memory between calls. A chat app resends the conversation every turn.
• Anything outside the window does not exist for the model. Long conversations get truncated or summarised.
• More context costs more and is slower, and models can miss details buried in the middle of very long inputs. Put the important instructions clearly at the start or end, and send only what is relevant.
• The model's built-in knowledge stops at its training cutoff. Recent facts must come from the context.

Sampling: why answers vary

After computing next-token probabilities, the model must choose one. Sampling settings control how:

• Temperature rescales the probabilities. Near 0, the top token almost always wins (focused and repeatable). Around 1, lower-probability tokens get picked more often (more varied, more creative, more errors).
• Top-p (nucleus sampling) samples only from the smallest set of tokens whose probabilities add up to p, for example 0.9, which cuts off the long tail of unlikely tokens.
• Max tokens caps the length of the output.

Worked example with three candidates:

probabilities at temperature 1:     "Paris" 0.70   "Lyon" 0.20   "banana" 0.10
at a low temperature (0.2):         "Paris" ~0.998 "Lyon" ~0.002 "banana" ~0.00006
with top-p = 0.85 (temperature 1):  keep Paris + Lyon (0.70 + 0.20 >= 0.85), drop banana

For extraction, classification and evaluation, use a low temperature. Even at temperature 0, outputs may vary slightly between runs because of hardware and batching, so never assume perfect determinism; test with several runs.

Using an LLM in an application

• Write a clear system prompt: the task, the output format, what to do when unsure ("say you don't know").
• Ask for structured output (JSON with named fields) when code will read the answer, and validate it.
• Give examples in the prompt (few-shot) for tricky formats.
• Evaluate on a fixed set of test inputs before and after every prompt or model change (lesson 3 of this module).
• Treat inputs from users or documents as untrusted: prompt injection ("ignore previous instructions...") can arrive inside a document.
• Know where data goes: hosted APIs send your text to a provider. Check your organisation's policy before sending personal or confidential data. Open-weight models (run locally with tools such as Ollama or Hugging Face) keep data in-house.

Common mistakes

• Treating output as fact. Verify claims, or ground the model in sources.
• Relying on the model for exact arithmetic or counting. Do the maths in code.
• High temperature for tasks with one right answer.
• Stuffing the whole knowledge base into the prompt instead of retrieving what is relevant.
• Changing prompts without re-running an evaluation set.

Quick recap

• An LLM predicts the next token from everything in its context, repeatedly.
• Tokens drive cost, limits and odd failures with spelling and numbers.
• The context window is all the model can see; it has no memory between calls and a training cutoff.
• Temperature and top-p trade consistency for variety; use low values for factual tasks.