BuzbyTimeRead the requirements guide

Choosing and running the software · 4.2

The demonstration and the reference call

The demonstration and the reference call. What decides it, what it costs, and what usually goes wrong. For another implementation reference, see this Monitask guide.

A demonstration is a performanceWhat you are watching

Prepared, rehearsed, on clean data, by somebody who uses the product every day. Nothing about it is dishonest and nothing about it predicts your experience.

The whole art of evaluating one is removing the parts that were prepared. For additional context, Dialpad is a useful reference.

Four ways to unscript it

Your data. Send a sample of your real categories, clients and sites beforehand and ask to see it in the product.

Your scenario. Describe an awkward real situation and ask them to do it live.

Your device. On the phone people actually have, not the presenter's laptop.

Your hands. Ask to make an entry yourself, timed. Four seconds is the target from the entry on adoption, and watching somebody else do it tells you nothing about it.

Three things to ask to see

  • The export an auditor would read, for a real control.
  • What a person sees about themselves.
  • What happens when the connection drops mid-entry.

The third is the one that produces an honest answer or an evasion, and either is informative.

The reference call

Worth more than the demonstration and asked for less often. Ask for three: somebody similar in size, somebody who has been through an audit with the product in place, and, if they will, somebody who left.

The third request is unusual and the response to it is informative regardless of whether it is granted.

Questions that get real answers

  • What surprised you after the first month?
  • What would you do differently?
  • What does your team complain about?
  • What did it cost beyond the licence?
  • How long did people take to actually use it?

None of these can be answered with a testimonial, which is the point. A reference who has only good things to say has either had an unusually good experience or is not being candid, and the second question usually distinguishes them.

Red flags

A demonstration that will not use your data. A refusal to let you make an entry. No reference who has been audited. An answer about screen capture that begins with a reassurance rather than a yes or no. And a price that changes when you mention a competitor.

The trial

Where one is offered, define what it must show before starting, as the entry on pilots argues. A trial with no criterion becomes a long demonstration and produces the same evidence as a short one.

Who should attend

Somebody who will enter time daily, somebody who consumes the output, and somebody technical. Three, and the first is the one most often left out of a procurement meeting.

Their reaction to the entry screen predicts adoption more reliably than anything else in the process.

Take notes against the list

Not free-form impressions. Go through the must-have list live and record demonstrated, asserted, or absent for each. A week later the memory of three demonstrations has merged and only the notes distinguish them.

Asking about the roadmap

Useful for understanding direction and worthless as a commitment. Evaluate what exists. A requirement satisfied only by something promised should be treated as absent, and said aloud.

The follow-up demonstration

Ask for a second, unscripted, with the questions that arose from the first. A supplier who accommodates it easily is showing you something about how support will feel; one who resists is also showing you something.

Scoring the demonstrations

Against the must list, as demonstrated or not, with a short note for each. Resist a weighted total: the weights are invented, the arithmetic conceals the judgement, and the decision gets taken on other grounds anyway while appearing quantitative.

A one-page comparison with three columns of demonstrated, asserted or absent is more honest and more persuasive to whoever signs.

The reference who left

Where a supplier will not connect you with one, ask why, and consider looking yourself. Former customers are findable and are generally willing to talk, and what they say is the most informative material available in an evaluation.

Discount it appropriately: somebody who left is describing a relationship that ended. Ask what went wrong and also what worked, and the answer to the second question tells you how much weight to give the first.

After the decision

Tell the suppliers who lost, and why, in a sentence. It costs a minute, it is unusual enough to be remembered, and it makes the next evaluation with them substantially easier if you ever run one.

The single most informative moment

When you ask to make an entry yourself, on your own phone, and somebody times it. Everything else in a demonstration is a claim about the product; that is a measurement of it.

Where to start

Send your categories and a real scenario a week before the first demonstration. Suppliers who use them are showing you the product; suppliers who present the standard deck anyway have answered a question you did not have to ask.

One line to carry

Everything in a demonstration was prepared. The only unprepared moments are the ones you create, so create three of them.

Two more things worth watching for

How the presenter handles a question they cannot answer. Somebody who says they will find out and does is showing you the support relationship; somebody who improvises is showing you a different one.

And who else is on the call. A solutions engineer who knows the product deeply is a good sign; three salespeople and nobody technical is a poor one.

Also in choosing and running the software