AI Accuracy: The Study Where AI Made Experts Worse

AI accuracy is the benefit most often claimed and least often qualified. The honest version is that AI improves accuracy on work it handles well and degrades accuracy on work it does not, and the gap between those two is not obvious from the outside.

What the experiment found about AI accuracy

Dell’Acqua and colleagues ran a preregistered randomized experiment with 758 management consultants at Boston Consulting Group, published in Organization Science. Participants worked without AI, with GPT-4, or with GPT-4 plus prompt-engineering training.

On tasks inside the model’s capability range, the AI groups completed 12.2% more tasks, worked roughly 25% faster, and produced quality rated more than 30% higher, among people who are already expensive and already good.

The researchers also included a task chosen to sit outside what the model handled well. On that task, participants using AI were 19% less likely to reach the correct answer than participants working without it.

The consultants did not get worse. Their judgment did. The authors describe a false-sense-of-confidence effect: the model produced output that was wrong and confidently worded, and experienced professionals accepted it.

Why this matters more for a small company

A large firm has review layers that catch this. A partner reads the deck, a second analyst checks the model.

A ten-person company often has one person owning a task end to end with no second reader. That is how confidently wrong output reaches a client proposal or a published price.

The risk is not that AI is unreliable. It is that AI is unreliable in a way that does not look unreliable.

AI accuracy depends on matching the task to the tool

There is no clean list, because the boundary moves as models change. Reliable patterns hold, though.

Work AI handles well shares three traits: the task has been done many times in a recognizable form, the inputs are supplied rather than discovered, and a knowledgeable person can verify the output quickly. Summarizing a transcript, drafting a standard email, or reformatting data all fit.

Work it handles badly is the reverse: it needs information the model does not have, depends on context specific to your business, or involves a judgment where being confidently wrong is expensive. Pricing, legal positions, technical specifications, and anything about your own customers fit here.

The last trait is the practical test. If checking the output is as much work as doing the task, the tool has not saved anything and has added a failure mode.

Building the review step in

The fix is unglamorous and it works: define where a human looks before output ships. The NIST AI Risk Management Framework organizes this as govern, map, measure and manage, and names unclear human-oversight roles as a risk in itself.

Decide which outputs require review before anyone uses the workflow, and write it down, because a review step held in someone’s head disappears the first busy week. Match the depth of review to the cost of being wrong. When an error type recurs, fix the workflow rather than reminding people to be careful.

Frequently asked questions

Does AI reduce human error or add to it?

Both, depending on the task. The evidence shows fewer errors on suitable work and more on unsuitable work, which is why matching tool to task is the actual skill.

Can better prompting solve this?

Only partly. In the BCG experiment, participants given prompt-engineering training still fell into the same trap.

How do we know if a task is inside the frontier?

Test it. Run the task both ways on cases where you know the right answer. Assumptions about capability are unreliable in both directions.

Does this mean AI is too risky for regulated work?

It means unreviewed AI is. Human review is the control, and it needs to be defined rather than assumed.

Design the workflow before you trust it

Most accuracy problems are workflow problems. The tool gets applied to the wrong task, or the right task with no checkpoint.

Bring a process where being wrong is expensive to a working session. See AI Consulting & Training for how review steps get built and documented, or read the full AI benefits for business overview.