Artificial Intelligence: A Conceptual Learner's Series
Part 11: Frontier models
Part 9 explained how a language model works. This part looks at the models at the leading edge, usually called frontier models: what sets them apart, why making them bigger keeps working, how they're built, what they can do now, and how people test and govern them. Specific models change every few months, so this part sticks to the ideas that last.
Concept 1: What makes a model "frontier"
In one line: A frontier model is one of the most capable general-purpose AI models at a given moment, trained with more data and computing power than almost anything before it.
Example: In the 2010s, a company that wanted to translate text, sort customer emails and summarize reports trained a separate model for each job. Today it can send all three jobs to one large model, along with questions about photos and requests to write code. A single general model, trained once at enormous cost and then used for thousands of tasks, is called a foundation model. The few most capable ones at any time make up the frontier.
Formally:
A foundation model is a large model trained on broad data that can be adapted to many tasks. Laws and policy documents often call it a general-purpose AI model.
Scale is measured in parameters (the numbers the model learns), training data (counted in tokens) and training compute, counted in floating-point operations, or FLOPs.
By some estimates, the compute used to train the largest models has grown roughly fourfold or more each year for over a decade.
The frontier moves. A frontier model today is ordinary within a year or two, as newer models overtake it and its abilities reach smaller, cheaper models.
Open-weight models publish their learned parameters for anyone to download and run. Closed models are available only through the developer's own apps and services.
Connections: Frontier models are the large language models of Part 9, Concept 1, built from transformers (Part 7, Concept 4) and taking the scale described in Part 7, Concept 5 much further.
Watch out:
"Frontier" describes a moment, not a model. Claims about what frontier models can't do go out of date quickly.
Size isn't the only lever. Better data, better training methods and more thinking time when answering (Concept 4) matter too.
Concept 2: Scaling laws
In one line: As training compute, data and model size grow, a model's prediction error falls along a smooth, predictable curve, which lets developers forecast a big model's performance from small experiments.
Example: A lab plans a training run that will cost tens of millions of dollars. Before committing, it trains small models at a thousandth, a hundredth and a tenth of the budget, and plots their error against compute on log-log axes. The points fall on a straight line. Extending the line predicts how the full-size model will do, and when the big run finishes, it lands close to the forecast.
Formally:
A scaling law says a model's loss (its average prediction error on text it wasn't trained on) falls as a power law in compute C: loss ≈ A × C−α. On log-log axes, a power law is a straight line. Similar laws hold for parameters and for data.
Training compute is roughly 6 × parameters × tokens FLOPs.
Compute-optimal training balances model size against data. A 2022 study, often called the Chinchilla study after the model it produced, found that for a fixed budget, parameters and training tokens should grow together, at roughly 20 tokens per parameter.
Lower loss tends to bring new abilities, but which abilities appear, and when, is much harder to predict than the loss itself. Some abilities seem to switch on suddenly, though part of that effect comes from how they're measured.
Connections: Fitting a line to data and checking it against new points is the core of supervised learning (Part 6, Concept 1). Loss is the quantity training minimizes (Part 7, Concept 2).
Watch out:
A trend isn't a law of nature. Scaling laws summarize past experiments and could bend. The supply of high-quality text, for one, is limited.
Lower loss isn't the same as more useful. A small improvement in prediction can mean a big change on some tasks and none on others.
Concept 3: How a frontier model is built
In one line: A frontier model is made in stages: pretraining on a huge collection of data to learn broad knowledge, post-training to make it helpful, honest and safe, then testing before release.
Example: Think of how a doctor is trained. Years of study build broad knowledge (pretraining). A residency, with experienced doctors correcting their work, teaches how to apply that knowledge with patients and when to decline a risky request (post-training). Board exams and supervised practice check readiness before they work alone (testing). A frontier model goes through the same three phases, compressed into months.
Formally:
Data: trillions of tokens of web text, books and code, plus images, audio and video for multimodal models, filtered for quality and with duplicates removed. Synthetic data generated by other models is a growing part of the mix.
Pretraining: next-token prediction (Part 9, Concept 1) over that data, run on thousands of specialized chips (GPUs or TPUs) for weeks to months. The result is a base model: knowledgeable, but not yet a good assistant.
Post-training: supervised fine-tuning on example conversations; reinforcement learning from human feedback (RLHF, Part 8, Concept 5); feedback from AI models guided by a written set of principles, as in Constitutional AI; and reinforcement learning on tasks with checkable answers, such as math problems and code with tests.
Distillation: training a smaller, cheaper model to imitate a larger one. This is one way frontier abilities spread to everyday models.
Pre-release testing: capability and safety evaluations, and red teaming, in which people deliberately try to make the model misbehave (Concept 5).
Connections: Pretraining is self-supervised learning (Part 9, Concept 1), post-training applies reinforcement learning (Part 8), and both run on deep learning (Part 7).
Watch out:
The base model and the assistant behave very differently. Much of what users experience as a model's judgment and manner comes from post-training.
Training data leaves fingerprints. Gaps, biases and errors in the data carry into the model, however large it is (Part 10, Concept 1).
Concept 4: What frontier models can do
In one line: Today's frontier models work with text, images and audio, read long documents, reason step by step before answering and use tools, but their skills are uneven in ways that can surprise people.
Example: You photograph a page of handwritten lab notes along with a messy spreadsheet, and ask whether the results support the hypothesis. A frontier model reads the handwriting, cross-checks the numbers, notices that one column mixes units, works through the statistics step by step and writes a clear summary. The same model might then stumble on a simple logic puzzle that a child would solve: skill in one area doesn't guarantee skill in a neighboring one.
Formally:
Multimodal models take in and produce several kinds of data, such as text, images, audio and video.
Long context: context windows have grown from a few thousand tokens to hundreds of thousands or more, enough to hold whole books or codebases.
Reasoning models spend extra computation when answering: they write out intermediate steps (a chain of thought), check and revise them, then give a final answer. Accuracy on hard problems tends to rise with this test-time compute, a second way to scale besides training compute.
Another way to spend test-time compute is to sample several answers and take a majority vote, a technique called self-consistency.
Tool use and agents: frontier models increasingly act through tools over many steps (Part 12).
Jagged capabilities: a model can perform at an expert level on one task and fail on a nearby task that seems easier.
Connections: Thinking before answering is search (Part 2) carried out in language. Voting over several samples is a form of the ensemble idea from Part 6: combining several imperfect answers cancels out some of their random errors.
Watch out:
Showing its work doesn't make a model right. A written chain of thought can look convincing and still be wrong, and it may not reflect how the model actually reached its answer.
Test-time compute costs time and money. A model that thinks for minutes is slower and pricier than one that answers at once, so it's worth using when the problem needs it.
Concept 5: Measuring and governing frontier models
In one line: Because frontier models are powerful and general, testing what they can do, including the harm they could do, matters as much as building them, and developers and governments have begun setting rules for it.
Example: Before releasing a new model, a lab tests whether it would give meaningful help to someone trying to make a biological weapon or break into computer systems. Outside experts, including government AI safety institutes, run tests of their own. The lab's published safety framework says that if a model crosses certain capability thresholds, stronger safeguards must be in place before release. The new model crosses one threshold, so the lab adds protections, such as filters that block dangerous requests, before launch.
Formally:
Benchmarks are standard test sets: exam questions, coding tasks, math competitions. They saturate as top models approach full marks, and they become contaminated when test questions leak into training data.
Preference comparisons: people compare two models' answers side by side, and the votes are pooled into rankings.
Dangerous-capability evaluations test for abilities that could cause serious harm, such as help with weapons, cyberattacks, or acting autonomously in ways that escape oversight.
Red teaming: specialists deliberately try to make the model misbehave or reveal capabilities that ordinary tests miss.
Frontier safety frameworks, published by leading developers, commit to specific evaluations and safeguards as capabilities cross set thresholds. Examples include Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework.
Regulation: the European Union's AI Act sets obligations for general-purpose AI models and presumes "systemic risk" for those trained with more than 1025 FLOPs.
Open versus closed release weighs research access and independence against the fact that once weights are public, a model can't be withdrawn or its use controlled.
Connections: This is honest measurement (Part 6, Concept 5) at the highest stakes, and the misuse and safety concerns of Part 10, Concepts 4 and 5, applied to the most capable systems.
Watch out:
A high score can hide memorization. If test questions leaked into training, the score measures recall, not skill.
Not finding a capability doesn't prove it's absent. An evaluation that fails to find a dangerous ability may simply have asked the wrong way.
Retention kit
Answer from memory, then check. Review at about 1 day, 3 days, 1 week, 2 weeks and 1 month.
What makes a model "frontier," and why is the label always temporary?
What does a scaling law predict, and why does it look like a straight line on log-log axes?
Name the stages of building a frontier model and what each one adds.
What is test-time compute? Give two ways a model can spend it.
Why can benchmark scores mislead? Name two other ways frontier models are evaluated.
Build exercise (1–2 hours): Measure test-time compute yourself. Collect 20 math word problems with known answers, and use any model API or an open-weight model on your own computer. For each problem, ask for the answer alone ("reply with only the number"), then ask again with "work through it step by step," then sample five answers at a higher temperature and take the most common one. Record each method's accuracy and how many tokens it used. Finally, write five new problems of your own, which can't be in any training data, and compare: does the model do as well on yours as on the well-known ones?
Python algorithms
Small simulations of the ideas in this part: fitting a scaling law on small runs and forecasting a run 100 times bigger (Concept 2), compute-optimal model sizing (Concept 2), majority voting as test-time compute (Concept 4), and how contamination inflates a benchmark score (Concept 5). The training runs are simulated, but the fitting and forecasting method is the real one. Runs with plain Python 3.
import math, random
from collections import Counter
# 1. Scaling laws: loss falls as a power law in training compute, loss = A * compute^(-alpha).
# Fit the law on small runs, then forecast a run 100x bigger than any you've tried.
def run_training(compute, rng):
"""Stand-in for a real training run: the true law plus a little noise."""
return 30 * compute ** -0.05 * (1 + rng.gauss(0, 0.005))
def fit_power_law(xs, ys):
lx, ly = [math.log10(x) for x in xs], [math.log10(y) for y in ys]
mx, my = sum(lx) / len(lx), sum(ly) / len(ly)
slope = (sum((a - mx) * (b - my) for a, b in zip(lx, ly))
/ sum((a - mx) ** 2 for a in lx)) # a straight line on log-log axes
return 10 ** (my - slope * mx), -slope # A, alpha
# 2. Compute-optimal sizing: training compute is about 6 x parameters x tokens,
# and a widely used rule of thumb trains on about 20 tokens per parameter.
def compute_optimal(flops, tokens_per_param=20):
params = math.sqrt(flops / (6 * tokens_per_param))
return params, tokens_per_param * params
# 3. Test-time compute: sample several answers and take a majority vote.
# One sample is right 40% of the time; each of four wrong answers gets 15%.
def majority_vote_accuracy(k, p_right=0.4, wrong=4, questions=2000, seed=0):
rng = random.Random(seed)
answers = ["right"] + [f"wrong{i}" for i in range(wrong)]
weights = [p_right] + [(1 - p_right) / wrong] * wrong
correct = 0
for _ in range(questions):
votes = Counter(rng.choices(answers, weights=weights, k=k))
top = max(votes.values())
correct += rng.choice([a for a, v in votes.items() if v == top]) == "right"
return correct / questions
# 4. Benchmark contamination: test questions that leaked into training data
# get memorized, inflating the public score above the model's real skill.
def benchmark_scores(skill=0.55, leaked=0.30, n=2000, seed=0):
rng = random.Random(seed)
public = sum(rng.random() < leaked or rng.random() < skill for _ in range(n)) / n
fresh = sum(rng.random() < skill for _ in range(n)) / n
return public, fresh
if __name__ == "__main__":
rng = random.Random(0)
small = [10 ** e for e in (18, 19, 20, 21)]
losses = [run_training(c, rng) for c in small]
a, alpha = fit_power_law(small, losses)
print("Scaling law fitted on four small runs:")
for c, l in zip(small, losses):
print(f" compute 1e{round(math.log10(c))} FLOPs -> loss {l:.2f}")
big = 1e23
print(f" fitted law: loss = {a:.1f} x compute^-{alpha:.3f}")
print(f" forecast at 1e23: {a * big ** -alpha:.2f} actual run: {run_training(big, rng):.2f}")
print("Compute-optimal model size (20 tokens per parameter):")
for flops in (1e21, 1e23, 1e25):
n, d = compute_optimal(flops)
print(f" 1e{round(math.log10(flops))} FLOPs -> {n / 1e9:,.1f}B parameters, {d / 1e9:,.0f}B tokens")
print("Majority vote over k sampled answers:")
for k in (1, 5, 25, 101):
print(f" k = {k:3d}: {majority_vote_accuracy(k):.0%} correct")
public, fresh = benchmark_scores()
print(f"Benchmark with 30% of questions leaked: public {public:.0%}, fresh questions {fresh:.0%}")Output when run:
Scaling law fitted on four small runs:
compute 1e18 FLOPs -> loss 3.79
compute 1e19 FLOPs -> loss 3.34
compute 1e20 FLOPs -> loss 2.99
compute 1e21 FLOPs -> loss 2.68
fitted law: loss = 30.3 x compute^-0.050
forecast at 1e23: 2.12 actual run: 2.11
Compute-optimal model size (20 tokens per parameter):
1e21 FLOPs -> 2.9B parameters, 58B tokens
1e23 FLOPs -> 28.9B parameters, 577B tokens
1e25 FLOPs -> 288.7B parameters, 5,774B tokens
Majority vote over k sampled answers:
k = 1: 39% correct
k = 5: 56% correct
k = 25: 90% correct
k = 101: 100% correct
Benchmark with 30% of questions leaked: public 68%, fresh questions 54%A straight-line fit through four cheap runs forecasts the expensive run almost exactly; that predictability is why labs are willing to commit huge budgets. At 1025 FLOPs, the point where the EU AI Act presumes systemic risk, a compute-optimal model would have around 290 billion parameters. Majority voting turns a model that's right 40% of the time into one that's almost always right, because the right answer is the single most likely one even though it isn't a majority. And the leaked benchmark reports 68% for a model whose real skill is about 55%.
Try it:
Change the exponent
-0.05inrun_trainingto-0.08and run again: the fit recovers the new exponent from the small runs alone.Call
majority_vote_accuracy(k, p_right=0.18). Now each wrong answer is slightly more likely than the right one, and voting makes the model worse, not better.Set
leaked=0.0inbenchmark_scoresand check that the public and fresh scores now roughly agree.