How to choose an AI model without looking at the leaderboard

Model leaderboards answer a question almost nobody asks: who wins on a fixed set of someone else's tasks. Your problem is different. You have a use case, a budget and a tolerance for error. The decision comes from there, not from a table.

What benchmarks really measure

A benchmark is a standardised exam: the same questions, a correct answer, controlled conditions. It is useful for comparing raw capability. It is not useful for predicting what will happen with your billing form.

Why it does not predict your case:

• Contamination. The cases may be in the training data: the model does not solve them, it remembers them.
• Single task. They reward one correct answer, not an acceptable one within a workflow with tools.
• None of your context. It does not include your repo, your data or your business rules.
• Format. A small change in the answer moves the score more than capability does.
• Variance. The difference between two adjacent places often falls within the noise.
• Cost ignored. The leaderboard does not tell you how much it costs to get it right.

The underlying bias is that benchmarks reward the single answer. Your work almost never has a single answer; it has an acceptable answer, which a human recognises in seconds and which the model cannot define on its own.

A model can rise in the leaderboard and get worse at your task. It is not a contradiction. They simply measured something else.

The five criteria that do matter

• Quality on your task. The real question: does it get it right with your inputs and your criteria? Measure it with 30 real cases, evaluated blind. Warning sign: it gets clean examples right and fails on real ones.
• Cost per operation. The real question: how much does a finished task cost, including retries? Measure it as total cost divided by accepted tasks. Warning sign: the price per token is low and the bill is not.
• Latency. The real question: how long does a person wait in the real workflow? Measure p50 and p95 per step. Warning sign: the average looks fine and the tail feels slow.
• Context window. The real question: does it handle your longest document without losing the middle? Test it with your documents, not with numbers. Warning sign: the nominal window is huge and the result ignores half of it.
• Privacy and lock-in. The real question: where does the data go and how expensive is it to leave? Review the terms and test a switch. Warning sign: you cannot change provider without rewriting the product.

The order of these criteria is not universal. In an interactive tool, latency may be worth more than a point of quality. In a batch process, the opposite. What does not change is that all five are decided with your use case, not with a public table.

The nominal context window is not the useful one. A model that accepts a huge document can lose the middle. Measure with your longest contract and your most tangled support thread.

How to build a home-made evaluation with 30 cases

1. Take 30 real inputs. From the history, from tickets, from emails. Do not invent them.
2. Respect the distribution: 20 typical, 7 difficult and 3 rare. The rare ones are what break the average.
3. Define what is acceptable before you see results. Three levels: usable, usable with editing, not usable.
4. Freeze the prompt and the configuration. Change one variable at a time.
5. Run each model two or three times. If your product uses a temperature above zero, variance is part of the result.
6. Evaluate blind. Remove the model's name from the outputs.
7. Save the failures with the exact input. Failures are worth more than successes.

Thirty well-chosen cases beat three thousand generic ones.

What you need is not statistical coverage: it is knowing where it breaks and whether that failure costs you money.

Two warnings. Evaluate the whole workflow, not the isolated call: pre-processing, tools, retries and final format. And if two people score the same case differently, the problem is not the model; it is that your criterion is not defined. That disagreement is the most useful finding of the evaluation.

Cost per task, not price per token

The price per token is a billing unit, not a decision unit.

Cost per accepted task = inference and tools, multiplied by the attempts needed, plus human review, plus the cost of errors.

A relative example. The cheap model costs a fifth as much per token, but fails one in five cases and needs two turns in half of its successes. The expensive one costs five times more and gets it right first time in four out of five. With retries and review, the difference can disappear. If the failure reaches the user, the expensive one turns out cheaper.

• Inference. Input and output tokens per task. It is the only component that shows up on the bill, so it gets all the attention.
• Retries. The average number of attempts until an acceptable output. It gets forgotten because it is seen as a prompt failure, not as a cost.
• Tools. External calls, searches, executions. Forgotten because they are billed somewhere else.
• Human review. Minutes per output times the hourly cost. Forgotten because it does not appear on any dashboard.
• Errors. Support, refunds, reprocessing, trust. Forgotten because they get put down to bad luck.

The component that usually dominates is human review. A model that is slightly more accurate and saves you thirty seconds per case can justify a price several times higher. One that is just as accurate and takes twice as long cannot.

The two-model strategy

An expensive model for the hard cases, a cheap one for volume. It sounds obvious and is almost always implemented badly.

How to do it well: route by type of task, not by prompt length. Length is a poor proxy for difficulty. A short prompt can ask for a delicate decision; a very long prompt can ask for a mechanical summary.

• Cheap: classification, extraction, formatting, drafts and high-volume summaries.
• Expensive: decisions with consequences, ambiguous cases and generation that a person will read carefully.
• Verification: a cheap step, with rules or a small model, decides whether the cheap model's output is acceptable or should be escalated.

Two rules. The cheap model must cover most of the volume, or you are paying for two systems. And two models mean two prompts, two evaluations and two behaviours: do not duplicate the architecture without measuring the maintenance.

The expensive one does not always win: sometimes the cheap one is better tuned to the format or responds faster. Only your evaluation can tell you.

Signs that you should switch models

• Your acceptance rate drops after an update, even though the provider's ranking goes up.
• The cost per task rises without a measurable improvement in review.
• A new task appears that the model fails systematically, not randomly.
• Tail latency degrades and it shows in the workflow.
• The privacy, retention or data location terms change.
• You can no longer switch provider without rewriting prompts, tools or format.
• The team spends more time correcting outputs than building.

What is not a signal: a headline, a new leaderboard, an impressive demo. Switching models has a real cost in prompts, evaluation and behaviour. You switch when the cost of staying is higher, not when moving looks cheap.

A model is not chosen once. It is adopted with conditions, measured and reviewed. The day you stop measuring, the leaderboard decides for you.

Keep reading

• Open or closed AI models: when it makes sense to run a model yourself.
• Prompt, RAG or fine-tuning: what to try before changing models.
• Tokens and the context window: what the context window really means.
• AI agents: how to design one that does not fail in production: evaluating trajectories, not just answers.