BuzbyTimeRead the requirements guide

Reading the numbers · 2.5

Industry benchmarks, and where they come from

Industry benchmarks, and where they come from. What decides it, what it costs, and what usually goes wrong. For a practical comparison point, see Monitask's developer time tracking.

Where an industry benchmark comes fromThe provenance question

Almost always one of three places: a vendor's own client base, a survey of self-selected respondents, or a consultancy's engagement history.

None is a sample of the industry. Each is a sample of who buys from that vendor, who answers surveys, or who hires that consultancy, and the difference between those populations and yours is unstated. For another perspective on the surrounding workflow, see Slack.

Definitions do more damage than sampling

Utilisation calculated against a forty-hour week and against available hours after leave are different numbers. Billable including or excluding travel. Overheads counted as non-billable or not counted at all.

A benchmark quoted without its definition is not a figure about anything, and the definition is usually the part omitted from the headline.

Four questions before using one

  • Who was sampled, and how many?
  • How is the number defined, exactly?
  • Who published it and what do they sell?
  • What year is it from?

Most benchmarks fail the first two. That does not make them worthless; it makes them a prompt rather than a standard.

The comparison that means something

Yourself, last year, on the same definition. It controls for everything a cross-industry comparison cannot: your mix of work, your contracts, your overheads, your definition.

It is also available to you and to nobody else, which makes it more valuable rather than less.

When a benchmark earns its place

As a reason to ask a question. If a published figure sits far from yours, the useful response is to establish whether the definitions match before concluding anything about your business.

Roughly half the time the gap is definitional and the investigation is over in an hour. The other half produces something worth knowing.

What a benchmark must never be

A target. Everything in the entry on measurement applies with full force: a figure imported from elsewhere, attached to a consequence here, is optimised here, and the optimisation is against a definition somebody else chose.

Publishing your own

If you do, publish the definition, the sample and the year alongside it. It costs a paragraph, it makes the figure usable, and it distinguishes you from the material this entry is about.

The circular benchmark

A vendor publishes a figure from its client base, prospects compare themselves against it, those who differ buy the product to close the gap, and the client base shifts toward the figure.

Nothing dishonest is required and the number becomes self-confirming. It is worth knowing before treating any vendor-published benchmark as a description of an industry.

Peer comparison that works

A handful of comparable businesses agreeing to share a defined figure with each other, having agreed the definition first. Small, unglamorous, and worth more than any published set because the definitions are known and the businesses are actually comparable.

What we publish

Nothing yet, and if we do it will carry its sample, definition and date. A benchmark from our own client base would describe our clients, and saying so is the minimum this site can do given everything else on it.

Ratios that travel and ones that do not

Some are robust enough to be worth comparing across businesses: the proportion of hours never attributed to anything, for instance, measures the same failure everywhere.

Others are almost pure definition: utilisation, billable percentage, average project size. Treat the first kind as informative and the second as local.

Age

A benchmark from before 2020 describes a different arrangement of work, and a good many still in circulation predate it. Check the year before the definition; it disqualifies more figures than anything else.

Where to start

Take the benchmark somebody in your business quotes most often and find its source. Fifteen minutes, and it will either become usable or stop being quoted.

What to say when somebody quotes one at you

Ask what it is defined as. It is a courteous question, it is not adversarial, and about half the time the person quoting it does not know, which answers the question of how much weight to give it.

Building your own comparison over time

Pick three or four figures, define them once in writing, and record them monthly. Within two years you have a benchmark that describes your business exactly, on definitions you control, with a history no external source can offer.

This is slow and it is the only benchmark that will ever be properly comparable to you.

A short checklist

Ask who was sampled, how the figure is defined, who published it, and from what year. Use it as a prompt. Never as a target. Prefer your own history.

One sentence to carry

A benchmark without its definition, sample and year is not a figure about anything.

The comparison people forget

Against your own targets, set in advance, rather than against anybody's industry figure. If a business decided last year what it wanted utilisation to be and why, that number is more relevant than a published one and it comes with a rationale attached.

Two figures worth tracking regardless

The proportion of hours with no attribution, and the proportion entered on the day. Both measure the health of the record itself rather than of the business, both are comparable over time, and neither is available from any published benchmark.

A last word on comparison

The instinct to know how you compare is reasonable and the available answers are mostly poor. Where a properly defined comparison exists it is worth having; where it does not, your own history is not a consolation prize but the better instrument.

Also in reading the numbers