BuzbyTimeRead the requirements guide

Getting people to record · 1.5

Running a pilot that answers something

Running a pilot that answers something. What decides it, what it costs, and what usually goes wrong. For another implementation reference, see this Monitask guide.

A pilot is not a small rolloutThe distinction

It is an experiment with a stated question, and most pilots fail because nobody wrote the question down. Running one team for a month and then asking whether it went well produces an opinion.

Write down beforehand what would count as working: completion above some level, entries made the same day rather than in batches, the categories used as intended, and nobody asking to stop. For broader background, see Notion.

Which team

Not the most enthusiastic and not the most difficult. A team whose work is representative, whose manager is respected, and who will say so if it is annoying.

The enthusiastic team proves nothing, because their result will not repeat. The difficult team proves nothing either, and burns the goodwill you need for the actual rollout.

How long

Six weeks. Four is not enough to get past the novelty and eight is long enough for the organisation to lose interest in the answer.

Within that, the useful observation is the third and fourth weeks, once the initial attention has passed and before anybody has decided anything.

What to ask at the end

  • What was annoying?
  • What did you do when the correct route was inconvenient?
  • What would make you stop using it?
  • What came back to you that was useful?

The second question is the one that finds the workaround while it is still one person's shortcut, and it only produces an answer if nothing is at stake in giving it.

Rolling out afterwards

Team by team, with the pilot team's people helping, and the categories already settled. Simultaneous organisation-wide launches concentrate every problem into one week and give whoever is supporting it no chance to fix anything before the next group arrives.

What a failed pilot tells you

Frequently that the categories were wrong or the entry point was too slow, both of which are fixable. Occasionally that the organisation does not have a question this data would answer, which is worth discovering in six weeks rather than after a year of licences.

A supplier who cannot hear that answer is a supplier to be careful with.

Baseline first

Before the pilot, record what you currently know: how hours are captured now, how long it takes, and how confident anybody is in the result. Without that, the comparison at the end is against memory.

Who runs it

Somebody with time allocated, not somebody fitting it around a full role. A pilot with no owner produces a set of impressions and a decision taken on whoever spoke last.

The decision at the end

Write down in advance who takes it and against what. Pilots that end without a decision drift into permanent trials, which is the worst outcome: the cost is being paid and nobody has committed to the practices that would make it worth anything.

What to measure during it

  • Proportion of entries made the same day.
  • Time to make an entry, timed with a stopwatch on a few people.
  • Use of the catch-all.
  • Number of support questions, and what they were about.

The fourth is the most useful and the least collected. Every question is a place the design is unclear.

Piloting with a supplier in the room

Useful and distorting. A supplier present fixes problems within hours, which will not be the experience afterwards. Ask what support looks like once you are an ordinary customer, and discount the pilot experience accordingly.

The second pilot

If the first fails for a fixable reason, run another with the fix rather than concluding. Organisations that abandon after one pilot usually abandoned a category list rather than a system.

What to do first

Write the success criteria down and send them to whoever will decide. A pilot with criteria agreed in advance produces a decision; one without produces a discussion about impressions.

Cost of the pilot

Licences for a team, the time of whoever runs it, and the time of the people using it. Small, and worth stating, because a pilot presented as free understates what a rollout will cost by the same proportion.

A closing thought

A pilot is cheap and the thing it is testing is expensive. Six weeks and a written question is a small price for finding out that the categories were wrong before three hundred people were using them.

Running two suppliers in parallel

Occasionally proposed and rarely worth it. Two systems doubles the entry burden on the pilot team, which distorts the thing you are trying to measure, and the comparison ends up being between two half-hearted deployments.

Evaluate on demonstrations and references, pick one, and pilot it properly.

Documenting what the pilot found

A page: what was tested, what the criteria were, what happened, what was changed as a result, and the decision. It takes twenty minutes and it is the thing somebody will look for in two years when asking why the system is configured as it is.

Pilots that produce no document produce no institutional memory, and the same questions get relitigated by whoever arrives next.

Extending rather than concluding

A pilot that ran into a genuine obstacle can be extended once, with the obstacle named and a new end date. What it should not do is drift: an unextended pilot still running after four months has become a deployment nobody decided on, at pilot pricing and without the practices a rollout would have brought.

Also in getting people to record