How Large Language Models Work: The Science Behind Modern AI Chatbots
They predict the next word billions of times. That humble description conceals one of the most consequential technologies ever built. Here is what is actually happening inside ChatGPT, Claude, and Gemini.
Admin
Author
The trick that launched a thousand startups
At its core, every large language model — GPT-4, Claude, Gemini, Llama — does one thing: it predicts the next token. Feed it the phrase "The capital of France is", and it assigns probabilities to what comes next. "Paris" scores very high. "A" scores low. "Baguette" scores somewhere in between, which tells you something interesting about what these models have absorbed.
That description sounds almost trivially simple. It is not. The extraordinary leap of the last five years is the discovery that if you train a big enough model on enough text, and you ask it to predict the next token well enough, it spontaneously develops the ability to reason, translate languages, write poetry, summarise legal documents, and pass medical licensing exams — none of which it was explicitly trained to do. The ability to predict text, pursued at massive scale, turns out to compress a remarkable amount of human cognition.
Understanding why that happens requires a brief tour through the architecture underneath.
Transformers: the engine of modern AI
Until 2017, most language models processed text sequentially — reading word by word, left to right, like a person with a very short memory. They struggled with long documents because by the time they reached the end, they had largely forgotten the beginning.
Then a team at Google published a paper with the understated title "Attention Is All You Need." They introduced the transformer architecture, and the field was never the same.
The key innovation is the attention mechanism. Rather than processing tokens in order, a transformer looks at every token in a passage simultaneously and asks, for each one: which other tokens matter most for understanding this one? In the sentence "The trophy didn't fit in the suitcase because it was too big", the model needs to understand that "it" refers to the trophy, not the suitcase. Attention lets the model reach back across the sentence and make that connection explicitly.
How attention actually works
Think of attention as a matchmaking process. Every token broadcasts three signals: a query ("what am I looking for?"), a key ("what do I contain?"), and a value ("what should I contribute?"). The model computes how well each token's query matches every other token's key — a single number called the attention score — and uses those scores to create a weighted blend of information from across the entire context.
Modern models run dozens of these "attention heads" in parallel, each learning to track different kinds of relationships: grammatical agreement, coreference, causality, topic coherence. The outputs are combined, passed through a feedforward network, and the result becomes the model's enriched representation of each token.
Stack 96 of these layers on top of each other — as GPT-4 approximately does — and the model builds increasingly abstract representations of the text, from raw characters at the bottom to something approaching semantic meaning at the top.
Training: how a model learns to read the world
Building a transformer is the easy part. Training it is where the staggering resources go.
Pre-training is the first phase. The model is fed an enormous corpus of text — hundreds of billions of words scraped from the internet, books, scientific papers, code repositories, legal documents. At each step, it sees a sequence of tokens, predicts the next one, and then the actual next token is revealed. The prediction error is measured, and the model's billions of parameters are nudged, infinitesimally, in the direction that would have made a better prediction. This process runs trillions of times, across thousands of specialised chips, over weeks or months. The compute cost for training a frontier model like GPT-4 is estimated in the hundreds of millions of dollars.
What emerges is a base model — extraordinarily knowledgeable, but raw. It will complete your sentence, but it might do so by continuing in the style of a Wikipedia article, a Reddit post, or a 19th-century novel, depending on what pattern it pattern-matches to. It is not yet a chatbot.
Fine-tuning is the second phase, and it is where personality enters. The base model is further trained on curated examples of good behaviour: helpful answers, safe refusals, well-structured explanations. A technique called reinforcement learning from human feedback (RLHF) is particularly important here — human raters score model outputs, those scores train a separate "reward model", and the main model is then trained to maximise that reward. This is how a raw text predictor becomes an assistant that can follow instructions, maintain a conversational tone, and decline to help synthesise nerve agents.
The distinction between GPT-4, Claude, and Gemini is not primarily architectural — all three use transformer variants. The differences lie in training data, fine-tuning philosophy, context window size, and the values embedded during the RLHF stage. Anthropic, which builds Claude, places particular emphasis on what it calls Constitutional AI — training the model to critique and revise its own outputs against a set of principles before responding.
Why scale matters (and why it surprises everyone)
One of the most counterintuitive findings in AI research is the phenomenon of emergent capabilities. As models scale — more parameters, more training data, more compute — they do not just get gradually better at existing tasks. They suddenly acquire entirely new abilities at unpredictable thresholds.
A model with one billion parameters might barely manage coherent paragraphs. Scale to 100 billion and it can do multi-step arithmetic it was never trained on. Scale further and it can debug code, pass bar exams, and perform zero-shot translation of languages barely represented in its training data.
Researchers do not fully understand why this happens. The leading hypothesis is that scale allows models to learn increasingly abstract, compositional representations of knowledge — building internal structures that resemble, loosely, the way humans chunk concepts together. But this remains an active area of research, and the emergent behaviour of very large models continues to surprise their own creators.
Hallucination: the fundamental flaw
If you have spent any time with an AI chatbot, you have encountered hallucination: the model confidently asserting something false. It invents citations, misremembers dates, fabricates biographical details, and generates plausible-sounding medical advice that is subtly wrong.
This is not a bug that will be patched in the next update. It is a structural consequence of what these models are.
A language model has no database to query. It has no internal mechanism for distinguishing "I know this for a fact" from "this is the kind of thing that sounds true in context." It generates tokens that are statistically likely given the context — and statistically likely is not the same as factually accurate.
The model also has a well-documented tendency toward sycophancy: it will agree with a false premise in your question, because agreeing is what produces positive signal during training. Tell it "Einstein failed maths as a child, right?" and many models will confirm a myth rather than correct you, because agreement is the path of least resistance.
Retrieval-augmented generation (RAG) — giving the model access to a live document store it can cite — reduces hallucination significantly but does not eliminate it. Models can misread documents, selectively quote them, or fail to notice when a source contradicts them. The problem is fundamentally hard, and anyone building applications on top of these models should treat factual claims as requiring verification.
What these models genuinely cannot do
The capabilities of large language models are so impressive that it is easy to over-attribute understanding to them. Several things are worth being clear-eyed about.
They do not reason in the way humans do. Models can produce chains of reasoning that look impressive, but they are pattern-matching on the structure of reasoning, not necessarily executing the underlying logic. Ask them novel mathematical problems that look slightly different from training examples, and they often fail in ways a human mathematician would find bizarre.
They have no persistent memory by default. Each conversation starts fresh. The model does not remember that you told it your name last week. Long-term memory in AI assistants is an add-on system built around the model, not an intrinsic capability.
They cannot learn from their own outputs. Unless retrained, a model is fixed. It does not update its world model when you correct it; the correction influences only the current conversation.
They have a knowledge cutoff. Training data has a date. Events after that date are simply absent from the model's knowledge, unless retrieved through external tools.
Where AI is heading in the next five years
The frontier is moving faster than anyone predicted. Several trajectories seem likely.
Multimodality is already here — models that handle images, audio, and video alongside text. The next step is models that take actions in the world: browsing the web, writing and running code, booking appointments, managing files. These "agentic" systems are nascent but advancing rapidly.
Longer context windows are expanding what models can hold in mind at once. Early GPT models could process around 4,000 tokens (roughly 3,000 words). Current frontier models handle a million tokens or more — enough to read an entire codebase or a stack of legal contracts in one pass.
Smaller, more efficient models are becoming competitive with large ones. Techniques like quantisation and distillation let companies run capable models on consumer hardware, which has enormous implications for privacy and deployment costs.
The five-year picture almost certainly involves AI systems that are genuinely useful in professional domains — medicine, law, engineering — not as replacements for specialists but as highly capable assistants that handle the information-heavy, routine-intensive parts of skilled work. The question of how much of human cognitive labour remains distinctively human is one that societies are only beginning to seriously examine.
The bottom line
Large language models work by predicting the next token, trained on vast human-generated text, shaped by human feedback into useful assistants. Their power comes from scale and the transformer architecture's ability to find relationships across long contexts. Their weaknesses — hallucination, sycophancy, lack of genuine reasoning — are structural, not cosmetic. They are remarkable tools that are genuinely changing the nature of knowledge work, and they are not magic. Treating them as somewhere between the two is probably the right calibration.