You Always Want the Smartest Model

The cheaper model’s mistakes are the ones you never see. That is the whole argument, and it is worth sitting with before you reach for the dropdown and pick the model that costs a fifth as much.

When a model fails on a task you understand cold, you catch it. You read the diff, you spot the off-by-one, you move on. But the reason you handed the task to a model is usually that it sits at or past the edge of your own competence. At that edge, a wrong answer and a right answer look identical: confident, well-formatted, internally plausible, nothing flagging it. So the failure mode that matters is not “the model is wrong” — it is “the model is wrong and you sign off anyway.” A weaker model raises the rate of exactly those undetectable errors, and you pay for them later, at a worse time, with interest.

This idea surfaced on Naval’s Full Episode: The AI Industrial Revolution, where the framing was blunt: intelligence is an unalloyed good, you always want more of it, and “when these models make a mistake you don’t know it.” Worth pressure-testing, because as a default it is mostly right and occasionally expensive.

The capability gap is real, and it is biggest exactly where you can’t see it #

Start with the part that is not in dispute. Frontier models are measurably better at hard problems than the tier below them, and the gap has been widening fast. The UK AI Security Institute’s 2025 Frontier AI Trends Report documents frontier systems moving from apprentice-level cyber-task performance in 2023 to expert level — the equivalent of ten-plus years of experience — by 2025, and outperforming PhD-level experts on open-ended chemistry and biology questions. Autonomous hour-long software tasks went from under 5% success in late 2023 to over 40% by mid-2025 (UK AISI). On real-world software engineering specifically, Stanford’s HAI AI Index reports AI going from solving 4.4% of SWE-bench tasks in 2023 to 71.7% in 2024 (Stanford HAI).

The cleaner data point for the smartest-model argument is self-correction. In a 2025 study of error detection in radiology reports, GPT-4o caught 88% of injected errors and GPT-4 caught 83% (PMC). The more capable model was not just better at the task — it was better at noticing when something was wrong. That is the property you are buying: not a higher score on the happy path, but a model more likely to flinch when the answer is bad.

And the flinch matters more than the average, because the average hides the tail. On SWE-bench Verified, Claude Haiku 4.5 scores 73.3% against Claude Sonnet 4.5 at 77.2% — a four-point delta — while Haiku runs at roughly a third of the per-token cost (Caylent). Four points sounds like a rounding error. But that delta is computed over the whole distribution, and the cases where a weaker model breaks are disproportionately the gnarly, ambiguous, tail-case ones — exactly where you can least verify the output and the errors compound silently. The scoreboard gap understates the gap where it hurts.

The leverage is what makes this asymmetric #

The reason intelligence reads as an unalloyed good is that the cost of the model is trivial next to the cost of the decision it feeds. One bad output that you don’t catch can trigger a real-world commitment, and the commitment is where the money is.

Air Canada is the textbook case. In February 2024 the airline’s chatbot — a weaker model — invented a bereavement-discount policy that did not exist. A customer relied on it, booked, and was refused the discount. A Canadian tribunal held the airline liable for what its bot fabricated (Responsible AI Labs). The error was undetectable to the customer, who saw authoritative-looking text, and undetected internally until a legal claim forced the issue. The legal fees and reputational damage dwarfed whatever the company saved by running a cheaper model. The same article catalogs Gauthier v. Goodyear (November 2024), where an attorney filed a brief full of fabricated citations — fictional case names, numbers, and quotations — and got sanctioned. Confident output, nothing flagging it, real-world cost.

These are not freak events. Responsible AI Labs counted 233 documented AI safety incidents in 2024, up 56% from 149 in 2023, with hallucinations the single largest failure category at 38% (Responsible AI Labs). The recurring root cause across the worst ones is the same: the error was not caught by the human relying on the output. When every interaction can put your name on something — a PR, a brief, a customer commitment — the asymmetry between a correct and a fabricated answer is enormous, and a few cents of inference is not where you economize.

There is also a structural reason the gap should keep widening, which makes “always reach for the best” a durable default. Epoch AI’s scaling analysis finds training compute growing roughly 4x annually as the dominant driver of capability, and projects that models trained at frontier scale by 2030 should exceed today’s frontier to the same degree today’s exceeds GPT-2 (Epoch AI). The smartest available model is a moving target, and it keeps moving up.

The honest counter-argument #

The default is right. The maximalist version of it is not, and a builder who ships should hold both.

“Smartest” is not the same as “biggest” or “most expensive.” The same Epoch analysis notes algorithmic progress contributing a 7.5x effective-compute gain independent of raw parameter count (Epoch AI), and the 2024–2025 shift toward inference-time compute — extended chain-of-thought, search at runtime — lets smaller base models punch well above their weight on specific problem types. The most intelligent response to a problem is sometimes a mid-sized model given room to think, not the largest checkpoint on the menu. Reach for the smartest result, which is not always the priciest line item.

Plenty of work genuinely is well-scoped enough not to need the frontier. IBM’s own analysis argues enterprises do not always need the largest model, and that for narrow, repetitive tasks smaller models can be the right call on cost and latency (IBM). Document routing, classification, simple summarization — workloads where errors are cheap and caught immediately by a downstream check — do not benefit from frontier reasoning enough to justify a multiple on cost. The smartest-model rule earns its keep on consequential, hard-to-verify decisions; it is overkill on rote, easy-to-verify ones. (I’d treat any specific “small models cover X% of enterprise work” figure with suspicion — the qualitative claim holds; the round numbers floating around it are not well sourced.)

Smarter does not mean safer, and that’s the trap. The AISI report found only minimal correlation between capability and robustness — an R² of about 0.097 — and universal jailbreaks across every system it tested (UK AISI). A more capable model makes fewer accidental errors but is not automatically more bounded or more reliably aligned with what you actually wanted. Picking the smartest model reduces undetected mistakes; it does not retire the verification layer. The two are complements, not substitutes.

And the model tier is not the whole system. Gartner forecasts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear value, and inadequate risk controls (Responsible AI Labs carries the same Gartner figure). Defaulting to the most expensive model with no workflow design around catching its errors is its own failure mode. The architecture around the model — checks, evals, human sign-off — matters as much as which model you call.

On the raw hallucination floor: a vendor summary (Claritybot, citing the Vectara leaderboard) puts frontier hallucination rates above 10% on grounded summarization and research tasks, with top models still fabricating citations that pass human review (via Claritybot). Treat the number as directional — it comes through a commercial intermediary, not the primary leaderboard — but the shape isn’t controversial: even the best models fabricate often enough that “it’s the smartest one, so it’s fine” is not a verification strategy.

What to actually do #

Make the smartest model your default, and make yourself argue your way down from it. The burden of proof belongs on cheapness, not on quality. For any decision where you cannot cheaply verify the output yourself — anything novel, consequential, or at the edge of your competence — pay for the best, because that is exactly where a weaker model’s errors go undetected and the leverage on each call is highest. Route down to a smaller or cheaper model only where the task is well-scoped and an automatic downstream check will catch a mistake before it leaves the building.

Then build the layer the model tier can’t give you. The smartest model lowers your rate of undetectable errors; it does not zero it. Evals, test harnesses, retrieval grounding, and a human who actually signs off are what convert “probably fine” into “I can stand behind this.” Intelligence is an unalloyed good. It is just not a complete one.