“94% faster” and “8 hours a week”: how I read vendor case study numbers and count my own

In one October week, OpenAI published four case studies with numbers, Close described how one of its sales managers saves 8 hours a week, and every number was counted a different way. These are the figures a business owner gets in a sales proposal: “94% faster,” “88.6% agreement with the expert.” I'll show how I read these numbers and how I count my own, because my site has percentages and hours on it too.

Six case studies and six ways of counting

OpenAI, October 20, put a Schlotzsky's and Cinnabon franchisee in the headline: cash reconciliation 94% faster with ChatGPT Work. The page description already gives a different number: menu repricing 95% faster, and a scenario in which $100,000 in cash flow could be freed up. I don't have the full text, so I'm going by the headline and the description, and even they show two different processes and conditional wording on the main sum.

OpenAI, October 22, reported the rollout of ChatGPT Work at Uber to more than 30,000 employees after a pilot with 1,000 people. There are no results in the post; the companies promise them later. OpenAI, October 21, described the plans of AT&T's legal department: a “front door” for requests in natural language, and precedent lookup. Also no numbers, but with a principle: decisions stay with the lawyers.

OpenAI's Terminal X case study, October 20, gave the most detailed figure: on 500 evaluated judgments, its reviewing agent on GPT-6.1 Sol agreed with the senior reviewer in 88.6% of cases, against 86.4% for the next model. The difference was measured against a rubric of 54 criteria, so the method is described. The $2.4 million in annual savings for one client, though, comes from Terminal X's own words, and the case study doesn't say how it was counted.

Close, October 20, published its sales manager's account of how the Chloe Chat assistant saves him more than 8 hours a week: meeting prep went from 15 minutes to a couple of minutes, the email after a call from 10 to 15 minutes to a couple of minutes, and on a day with five meetings that adds up to more than an hour. The numbers are his own and depend on his schedule. OpenAI, October 16, showed a Benefit Street Partners example in a different genre: a portfolio manager built a loan rating tracker himself, 10 to 12 hours in total including revisions, and a credit agreement analyzer for documents of 300-plus pages produces a first pass in 20 minutes, after which a person reads for an hour or more. Machine time here stands next to human time.

What should stand behind a number

Take the owner of a service company with ten managers. He reads “8 hours a week,” multiplies by ten and expects 80 hours. But the manager in the case study has five meetings a day and calls around the world, while his own people have two meetings and messenger chats. So a number can move to another business only together with the conditions it was obtained under. In any figure I look for four things: what was measured, on what volume, over what period, and who checked the result after the machine.

How I count my own numbers

I put the same four questions to my own numbers: what was measured, on what volume, over what period, and who checked the result.

For a manufacturing company, in 2024 I built from start to finish a program that takes Excel specifications, breaks a product down into parts and calculates where, how much and of what, with materials checked against the 1C catalog. We prepared the technical specification from February, active development ran from April, and I handed over the working version in the summer. The program processes one order in 65 seconds on average, where it used to take 4 to 5 hours. It's used daily by 11 people, each saving about 4 hours a day, which comes to about 900 hours a month. The total breaks down into clear parts: 65 seconds per order, 11 people, about 4 hours a day each. At the car rental in South Africa, where I was responsible for the website and integrations, cars are handed over 3 times faster with the new system, the average check grew by 30%, and maintenance costs fell by 25%. How much the idle time between rentals shrank, which is what I built the calendar for, I can't say precisely, so I don't have that number.

I started building document parsers for a PDR company in the US in 2023. From mid-2023 about half of the documents went through without manual correction, in 2024 about 75%, in 2025 about 95%, and in 2026, when I added AI validators and shipped the release version, about 99.9%. These shares apply to the documents that went through the parsers. The parsers free up about 60 hours of manual work a week: several employees used to retype these documents into a spreadsheet. When a client asks how much a parser will save him, I answer that I won't promise a number until I've looked at sample documents.

What I would do with someone else's case study

Find the baseline. “94% faster” without “it was this much, now it's this much” is a percentage of an unknown. If there's no baseline, the number is good for curiosity, not for a budget.

Separate measured from estimated. At Terminal X the 88.6% was measured on 500 judgments, while the $2.4 million comes from the company's own words. Both numbers may be correct, but they carry different weight.

Look for the human after the machine. At Benefit Street Partners the 20 minutes of machine and the hour of human are written side by side; if there's no human in the case study, ask who checks the result and what that costs.

Count on your own data before you buy. Take 20 of your documents or 20 of your meetings, run the scenario, and do the math yourself.

On where AI helps in data processing and how to check the result: AI in data processing: extraction and verification. By area: AI in a process where the result can be verified. The cases with the numbers discussed here: document and photo parsers, order calculation in 65 seconds and car rental in South Africa. On the same topic: what happened at our company when the parsers freed up 60 hours a week and metrics that teach a team bad decisions; earlier I wrote about the cost of AI per accepted task. If someone brought you a case study with percentages and you need to know what applies to your documents, tell me what documents they are: Discuss your task.