How to check a productivity claim before you believe it

Numbers about how much time a new tool saves are everywhere at the moment, and most of them will not survive four questions. This is how to ask them. It uses today's legal examples, but the method works on any industry and any tool.

One. Who paid for the number to exist?

Not who published it. Who benefits from it being large. A supplier measuring its own product is not disqualifying, but it is the single most useful thing to know, and it is usually one click away. Look for the company name on the study, the case study or the survey.

If the figure comes from a survey, ask who was surveyed. A survey of a company's own customers tells you what its customers believe. That is real information, and it is not the same as a measurement.

Two. What was actually counted?

Time, or something that stands in for time? A vacancy count is not a measure of hours worked. A self-reported estimate of hours saved is not a stopwatch. Both can be useful and neither is what the headline usually claims.

The giveaway phrase is up to. Up to six hours a week is compatible with most people saving nothing.

Three. Where did the trade press get it?

Industry publications often carry a line at the foot of an article recording that it was submitted by the company it is about. That line is doing a lot of work and it is easy to miss. If you find it, you have found the origin, and the origin is a press release.

A useful habit: search the exact figure in quotation marks and sort by date. The earliest result is usually the source, and it is usually a supplier.

Four. Does a second, unrelated source say the same thing?

This is the one that settles it. If the supplier says a tool saves five hours and an independent study of the same task says something similar, the claim is sturdy. If only the supplier says it, hold it loosely.

For AI in UK legal work, we have looked for that second source and not found one. That is a statement about our searching rather than about the world, and it is the only version of it we can stand behind.

What this is useful for

If you run a small firm and you are deciding whether to buy something, these four questions are the difference between a purchase and a bet. If you are deciding whether to hire someone junior, they matter more, because a hiring decision made on an unverified productivity claim is a decision about a person's first job.

The Quernal provides general information, not financial advice.