Adoption telemetry — what to measure when no one is watching

TL;DR — Accuracy tells you the model works; it says nothing about whether anyone uses it. Adoption telemetry instruments the human decision — seen, acted on, overridden, ignored — and it is the earliest reliable signal of whether an AI programme will compound or revert.

Most AI programmes are measured on the wrong thing. The dashboards track accuracy, latency, and cost per call — model health — and go quiet on the only question that predicts survival: did a human do anything different because of the output?

Model metrics measure the tool, not the change

A model at 94% accuracy that nobody acts on is a 0% programme. Accuracy is necessary and radically insufficient. It measures the tool in isolation, in a world with no operators, no habits, and no accountability — which is not the world the tool ships into.

Adoption telemetry measures the tool in situ: at the moment a human meets the output and decides what to do with it.

The four states worth instrumenting

For every AI-surfaced decision, capture which of four things happened:

• Seen — did the output actually reach the operator, in time, in their flow?
• Acted on — did they take the action the system suggested?
• Overridden — did they consciously choose differently? (This is signal, not failure.)
• Ignored — did the output arrive and change nothing?

The mix tells you the problem. High ignored is a trust or workflow problem, not a model problem — and no amount of extra accuracy will fix it. High overridden with good outcomes may mean the human knows something the model does not, which is a training signal. High acted on is the only state that means the programme is working.

Instrument it before launch, not after

The instinct is to add telemetry once adoption looks shaky. By then you are three months in, arguing about the model, with no data on the decision. Instrument the four states from the first release — even crudely — and within a week you know whether you have a trust problem, a workflow problem, or a genuine quality problem. That single week of clarity is worth more than another quarter of model tuning.

What you do with it

Adoption telemetry is not a scorecard; it is a steering signal. Rising ignored on a specific surface tells you exactly where the workflow or trust design is failing, before the programme is written off. It is also the honest number to put in front of an executive sponsor — far more predictive than accuracy of whether the investment will compound.

This is why we build adoption in from day one across every engagement — see how it fits the wider operating-model work and the capabilities that make it real. Measuring the decision, not the model, is the difference between a programme that learns and one that freezes at launch.

FAQ

What is adoption telemetry? Instrumentation of the human decision around an AI output — whether it was seen, acted on, overridden, or ignored — rather than the model's accuracy or latency.

Why isn't model accuracy enough? Because an accurate model nobody acts on delivers nothing. Accuracy measures the tool in isolation; adoption telemetry measures whether it changes behaviour in the real workflow.

When should you start measuring adoption? From the first release. Instrumenting the four decision states early gives you, within a week, a clear read on whether the blocker is trust, workflow, or quality.

Is a high override rate bad? Not necessarily. Conscious overrides with good outcomes can mean the operator knows something the model does not — a useful training signal, not a failure.