Use casesAI evaluation
Before you trust it, test it. Before you buy it, test the number.
Most organizations are experimenting with AI. Few have much in production, and fewer can say how well it works. The gap is evaluation. We build the exam for an AI system, yours or a vendor’s, the way we grade our own: held-out data, the simplest baseline it has to beat, planted failures it has to catch, and a report that says where it stops.
What gets in the way
A vendor shows a 0.95. A pilot wows the demo. Six months later nobody can say whether the thing is right more often than the spreadsheet it replaced, and the project sits in limbo, neither killed nor shipped.
Most accuracy numbers come from the easy test: shuffled data, the model studying tomorrow to predict yesterday. The number is real and the system is not.
The market is buying AI the way nobody buys anything else: without checking the number. That is the gap between the pilots and the production systems, and it is closable.
What we build
An exam built from your data and your decisions, run on their system or ours.
- A held-out set the system never saw, cut by time, so it is graded the way it will be used.
- The simplest baseline you already have, run beside it: last week’s number, the customer’s track record, the current rule. If the system does not beat it, you learn that first.
- Planted failures the system must catch: a missing field, a shifted decimal, a question the documents cannot answer, a request past a limit. A system that cannot fail the test has not been tested.
- For language systems, a question set your experts wrote with known answers, and a check without any model that each citation actually contains the claim.
- A report in plain words: the score, the baseline, the failure modes, the cap or hand-off rule we recommend, and the things we could not test.
How we know it works
It is how we grade our own work, and it has changed our own results.
- On Winland Foods, the honest exam turned a vendor’s 0.95 into a 0.81, and that 0.81 cleared $174K of deductions in its first month-end close.
- On Acceleration Partners, planting defects into the verification exposed a real flaw in our own checker. We fixed it, and the same checks found 13 errors in numbers the clients already had.
- On Verdict, every prediction is registered before the experiment, and the failures stay in the record. One feature reached 31% against a predicted 60% and ships switched off.
- The exam is yours when we leave: the held-out set, the baselines, the planted failures and the scripts, so you can run it again every quarter.
Built from this work
Every evaluation we have run so far was on our own work. The three below show the method, and they are the reason we trust it.
28% of claims clear automatically at the recommended bar
92% of cleared claims were real shortages, on months the model never saw
$174K in deductions cleared through the model at the first month-end close
Read the story4,756 figures compared across three independent sources
41 of 52 client units that match the client’s own report exactly
43 client report programs specified for one engine
Read the story86% right cause ranked first on a public benchmark, against 14% by chance
8.7% right cause first on a simulated chemical plant, our open problem
10,300 labeled anomaly windows behind the measurements
Read the story
Where it fits
Manufacturing
AI for manufacturers, where the count has to be right.
Insurance
Insurance, from the question a customer asks to the systems that answer it.
Financial services
AI for financial services, built to pass an audit.
Franchising
AI for franchise brands, researched at a scale no team does by hand.
Marketing
AI for marketing, where one wrong number costs more than it saves.
Questions
Can you evaluate a system we are buying?
Yes, and that is the best time. We build the exam from your data and run their system through it, with the vendor in the room if they want to be.
What if it fails?
Then you found out for the price of an exam instead of a rollout. The report says what failed and whether a hand-off rule, a cap or more data would fix it.
How long does it take?
Weeks, not months. The exam reuses your existing history and decisions. The slow part is usually finding out that the history has traps in it, which is itself a finding.
Do we need you after the exam?
Not necessarily. The exam is yours: the held-out set, the baselines, the planted failures and the scripts. Run it again every quarter.
Bring us the project that matters most.
Tell us what you’re building and what’s in the way. You’ll hear back within 24 hours.