On this page
What is a large language model?
A large language model (LLM) is an AI system trained on very large amounts of text to understand and generate human language. Its core job is simple: predict the next token in a sequence. Repeat that step thousands of times and the model writes an answer, a summary or working code.
If you only need the definition, Ailoitte’s short explainer on what an LLM is covers it in a minute. This page goes inside the model.
LLMs are built with deep learning, a branch of machine learning that uses neural networks with many layers. Almost every modern LLM uses the transformer architecture. Well-known model families include OpenAI’s GPT models, Anthropic’s Claude, Google’s Gemini and Meta’s Llama.
Adoption is why the anatomy of an LLM now matters to more than researchers. In McKinsey’s 2025 global survey on the state of AI, 88% of respondents said their organizations use AI, a rise of 10 percentage points from 2024. The survey ran in mid-2025 with almost 2,000 participants. Teams that understand how these models work make better choices about which model to use and how to test it.
The anatomy of an LLM: 5 core components
Every LLM is built from five core parts that work as a pipeline: a tokenizer, an embedding layer, attention layers, feedforward layers and an output layer. Text goes in at one end. A probability for every possible next token comes out at the other.
The 5 core components of an LLM and what each does
| Component | What It Does | Analogy |
|---|---|---|
| Tokenizer | Splits text into tokens and gives each an ID | Cutting a sentence into puzzle pieces |
| Embedding layer | Turns each token ID into a vector of numbers | Giving each piece map coordinates |
| Attention | Works out which tokens matter to each other | Reading the whole sentence for context |
| Feedforward layers | Refine each token’s meaning after attention | Polishing a rough draft |
| Output layer | Scores every possible next token | Ranking candidate next words |
Tokenizer
The tokenizer is the part of an LLM that splits text into tokens: whole words, word pieces or punctuation marks. The word “unbelievable” might become “un”, “believ” and “able”. Each token gets a number ID from a a fixed vocabulary, typically 128,000 to 256,000 entries in current models (early GPT models used about 50,000).
Tokens matter in practice because models count, price and limit everything in tokens. In English, one token averages roughly three-quarters of a word on OpenAI’s tokenizer, though the ratio varies by model.
Embedding layer
The embedding layer turns each token ID into a vector, a list of hundreds or thousands of numbers that captures meaning. Tokens used in similar ways end up with similar vectors, so “king” and “queen” sit close together. Each vector also carries the token’s position, so the model knows word order.
Embeddings also power search outside the LLM. Retrieval systems compare vectors to find passages that match a question. Retrieval-augmented generation is built on this, as covered later on this page.
Attention mechanism
The attention mechanism lets each token look at the other tokens and decide which ones matter most for its meaning. In “She gave the book to John because he asked for it,” attention links “he” to “John” and “it” to “the book.”
The original transformer paper, Attention Is All You Need, introduced this design in 2017.
Modern LLMs run many attention heads in parallel. Each head can track a different relationship, such as grammar, who did what, or which noun a pronoun refers to. Attention also separates meanings, so “bank” reads as a riverbank or a financial bank depending on the sentence.
Feedforward layers
Feedforward layers are small neural networks that process each token after attention has mixed in context. They refine the token’s representation and appear to store much of the model’s factual knowledge.
One attention step plus one feedforward step forms a transformer block. An LLM stacks dozens of these blocks, often 30 to more than 100. Early blocks handle spelling and grammar. Later blocks handle topic and intent.
Output layer and parameters
The output layer converts the final vector into a score for every token in the vocabulary, then turns those scores into probabilities. The model picks the next token from that list.
Parameters are the adjustable numbers, mainly weights, inside all of these layers. Small models have a few billion parameters. The largest frontier models are estimated to have hundreds of billions to around a trillion. Many large models activate only a fraction of these for each token.
Some newer models use a mixture-of-experts (MoE) design. In an MoE model, only a few specialist sub-networks run for each token. Mistral’s Mixtral and DeepSeek-V3 use this approach to make large models faster and cheaper to run.
How are LLMs trained?
LLMs are trained in stages. Pretraining teaches general language. Fine-tuning teaches the model to follow instructions. Preference training aligns its answers with what people find helpful and safe. Each stage uses less data than the one before, but shapes behavior more directly.
- Pretraining: the model reads a huge dataset of text and code and learns to predict the next token.
- Supervised fine-tuning: the model studies curated examples of instructions paired with good answers.
- Preference training: people or models rank answers, and the model learns to prefer the better ones.
- Reasoning training: newer models get reinforcement learning on problems with checkable answers, such as maths and code.
Pretraining on large text datasets
Pretraining is the stage where an LLM learns language by predicting the next token across a huge text dataset. Meta, for example, reported pretraining Llama 3 on more than 15 trillion tokens. No labels are needed, because the next word in real text is the answer key. This is called self-supervised learning.
Gradient descent is the method that improves the model during pretraining. After each prediction, the model measures how far it was from the right token; this gap is called the loss. Gradient descent then nudges the weights slightly in the direction that reduces the loss. Across the whole dataset, these small corrections add up to grammar, facts and reasoning patterns.
Supervised fine-tuning
Supervised fine-tuning trains a pretrained model on a smaller, labeled dataset of prompts and ideal responses. A base model only continues text. After fine-tuning on examples such as “Summarize this article in two sentences” paired with a good summary, the model treats a prompt as a task to complete.
Businesses use the same technique on their own data to adapt a model to a domain, a tone or an output format. Methods such as LoRA (low-rank adaptation) update only a small share of the weights, which cuts the cost of fine-tuning.
Reinforcement learning from human feedback (RLHF)
Reinforcement learning from human feedback (RLHF) works in three steps. Human reviewers rank several model answers, a reward model learns to predict those rankings, and the LLM is then tuned to produce answers the reward model scores highly.
OpenAI’s InstructGPT research (https://arxiv.org/abs/2203.02155) found that people preferred a much smaller model trained this way over a far larger model trained without it.
Newer methods such as direct preference optimization (DPO) learn from the same kind of rankings without a separate reward model. Many labs also use AI feedback, where a model judges answers against written principles.
Reasoning training
Reasoning training teaches a model to work through a problem step by step before it answers. Labs use reinforcement learning on tasks with verifiable answers, such as maths problems and code that must pass tests.
Reasoning models such as OpenAI’s o-series and DeepSeek-R1 spend extra compute thinking at answer time. This improves accuracy on hard problems but adds cost and latency.
How does an LLM generate text?
An LLM generates text one token at a time. It reads everything written so far, predicts a probability for every possible next token, picks one, adds it to the text, and repeats. This loop is called autoregressive generation.
- The tokenizer splits your prompt into tokens.
- The embedding layer turns each token into a vector.
- The transformer blocks apply attention and feedforward steps to build context.
- The output layer scores every token in the vocabulary.
- A sampling rule picks the next token from those scores.
- The new token is added, and the loop runs again until the answer ends.
For “The cat sat on the”, the model might give “mat” a high probability, “floor” a lower one and “piano” a very low one. Which token wins depends on the sampling settings.
Sampling settings: temperature, top-k and top-p
Temperature, top-k and top-p are the settings that control how predictable or varied an LLM’s output is. Most model APIs let developers set them per request.
- Temperature scales the probabilities: low values favor the likeliest tokens, and high values let the model take more chances.
- Top-k limits the choice to the k most likely tokens.
- Top-p limits the choice to the smallest set of tokens whose probabilities add up to p, such as 0.9.
- Maximum output length caps how many tokens the model can generate.
Use a low temperature for data extraction, classification and code. Use a higher one for brainstorming and creative copy.
Context window
The context window is the maximum number of tokens an LLM can handle in one request, including the prompt, any documents and the answer. Current flagship models from OpenAI, Anthropic and Google offer windows of about 1 million tokens; older or chat-optimized models are often smaller, around 128,000.
Text outside the context window is invisible to the model. Long inputs also raise cost and latency.
How do LLMs adapt to new tasks?
LLMs adapt to new tasks in four main ways, from cheapest to most involved: zero-shot prompting, few-shot prompting, retrieval-augmented generation and fine-tuning. Most teams start with prompts and move down the list only when results fall short.
Zero-shot learning means the model performs a task from instructions alone, with no examples. Few-shot learning means the prompt includes a few worked examples, and the model copies their pattern for new inputs. Writing these prompts well is a skill of its own, often called prompt engineering.
Retrieval-augmented generation, or RAG, gives the model relevant passages from your own documents at the moment it answers. The model can then use current, private facts it never saw in training.
Four ways to adapt an LLM to your task
| Method | What Changes | Best For |
|---|---|---|
| Zero-shot prompting | Only the instructions | Simple, common tasks |
| Few-shot prompting | Instructions plus examples | Fixed formats and styles |
| RAG | The context the model reads | Answers from private or fresh data |
| Fine-tuning | The model’s weights | Consistent tone, format or domain skill |
The trade-offs between grounding a model and training it are compared in Ailoitte’s analysis of RAG vs fine-tuning. For budgeting, see this breakdown of RAG system development cost.
What are the main types of LLMs?
LLMs are usually grouped three ways: by architecture, by training stage, and by size and access. Knowing the type tells you what a model is good at and how much work it needs before production.
Types of LLMs by architecture
The three transformer architectures differ in how the model reads and writes text.
- Decoder-only models read left to right and predict the next token, and most chat assistants, including the GPT and Llama families, use this design. Best for: chat, writing and code.
- Encoder-only models read the whole text at once to understand it, but they do not generate long text, and Google’s BERT is the best-known example. Best for: classification, search and sentiment analysis.
- Encoder-decoder models read the input with an encoder and write the output with a decoder, as Google’s T5 does. Best for: translation and summarization.
Types of LLMs by training stage
The training stage decides how well a model follows instructions out of the box.
- Base models have only been pretrained, so they continue text well but follow instructions poorly. Best for: teams that plan their own fine-tuning.
- Instruction-tuned models have been fine-tuned to follow requests such as “summarize this in two sentences.” Best for: single tasks inside an app.
- Chat or dialogue-tuned models are trained on multi-turn conversations, so they track context and tone across turns. Best for: assistants and customer support.
- Reasoning models are trained to work through problems before answering. Best for: maths, coding, planning and multi-step analysis.
A chat model answers questions, while an agent uses the model to take actions through tools. The difference is explained in this comparison of AI agent vs chatbot.
Types of LLMs by size and access
Frontier LLMs are the largest, most capable models, usually reached through an API. Small language models (SLMs) have far fewer parameters and can run on a single server or device, which lowers cost and keeps data in-house. Teams weighing that route can read about small language model development.
Access is the other split. Proprietary models such as GPT, Claude and Gemini are used through the provider’s API. Open-weight models such as Llama, Mistral, Qwen and DeepSeek publish their weights. Teams can host and fine-tune open-weight models on their own infrastructure, subject to each model’s license.
Where do LLMs fit in a product?
LLMs fit best where a product needs to read, write or reason over language at scale. Common uses include support assistants, search over internal documents, drafting and data extraction, coding help, and agents that complete multi-step tasks through tools and APIs.
The model is only one part of a working system. A production LLM feature also needs retrieval, guardrails, evaluation sets, monitoring and a plan for cost per request. Teams without that experience in-house often bring in an AI development company to scope the architecture before choosing a model.
When the goal is software that acts, not only answers, the LLM becomes the reasoning engine of an agent. Providers of AI agent development services.
What are the limitations of LLMs?
LLMs have four main limitations. They can hallucinate. They can repeat biases from their training data. Their reasoning is hard to inspect. They are costly to run and bounded by a knowledge cutoff. Each limitation can be managed, but none can be ignored in production.
Hallucinations
A hallucination is a confident LLM answer that is false or made up. LLMs predict likely text, not verified facts, so a fluent sentence can still be wrong. Three steps reduce the risk: ground answers in retrieved sources, ask the model to say when it does not know, and check outputs against test sets.
Bias
LLMs learn from human-written text, so they can absorb and repeat social biases related to gender, race and culture. Without testing, those patterns can lead to unfair or inappropriate outputs. The NIST AI Risk Management Framework gives organizations a structured way to find and manage risks like these.
Interpretability
Interpretability is the ability to explain why a model produced a given output. In traditional software, you can trace each decision. An LLM’s decisions are spread across billions of weights, so it behaves much like a black box.
Research is making progress. In Anthropic’s interpretability research on mapping concepts inside Claude, the team located millions of concepts inside Claude Sonnet, a production model.
Cost, speed and knowledge cutoff
Every token an LLM reads or writes costs compute, so long prompts and large models raise both the bill and the wait. A model also knows nothing after its training cutoff unless you supply fresh data through retrieval or tools. Choosing the smallest model that meets the quality bar is often the biggest cost saving.
Summing up
The anatomy of an LLM comes down to a few parts working in sequence: a tokenizer, embeddings, stacked transformer blocks of attention and feedforward layers, and an output layer that scores the next token. Pretraining, fine-tuning and preference training set the model’s weights. Knowing these parts helps you pick the right model type, adapt it with prompts, RAG or fine-tuning, and plan for its limits.









