Measure an AI pilot without promising savings

Choose a decision the pilot can answer

Start with one bounded workflow, such as preparing an internal summary from an approved set of documents. Define who needs the summary, what makes it usable and who accepts it. Keep customer sending and other external actions outside this example. The decision might be whether to continue the same small trial, change the workflow or stop it. Avoid beginning with a target percentage of savings. That encourages the team to search for numbers that support a promise. Instead, name the practical problem: staff spend time assembling information, reviewers repeatedly correct missing context, or completed work waits in a queue. The pilot should investigate a problem that people actually recognise.

Describe the current process before changing it

Record how comparable work is completed today. Include preparation, drafting, review, correction and final acceptance. Note the mix of straightforward and difficult cases, the experience of the people doing them and any interruptions. A baseline made only from difficult historical cases will flatter a pilot that receives easy examples. Choose a comparison approach suited to the workflow. Where possible, allocate similar eligible tasks across the existing and proposed processes during the same period. If you use before-and-after observations, record changes in staffing, workload, source quality and task mix. Explain those limitations instead of attributing every difference to AI.

Measure people’s effort and elapsed time separately

Active effort is the time people spend preparing, checking and correcting work. Elapsed time includes waiting between steps. They answer different questions. A summary can require less active writing yet arrive later because its reviewer is unavailable. Both observations matter to the operating decision. For each eligible task, record the route used, start and acceptance times, active effort by stage, correction rounds and final disposition. Use a collection method the team can apply consistently without intrusive monitoring. If timing is approximate or self-reported, say so. Do not present an estimate as a precise instrument reading.

Keep quality beside speed

Agree acceptance criteria before reviewing the pilot results. For an internal summary, these could include factual agreement with the source, coverage of the requested questions and a clear indication of missing information. Have an authorised reviewer apply the same criteria to both routes. Where practical, avoid telling the reviewer which route produced a draft until the quality assessment is complete. Record rejected and abandoned attempts as well as accepted ones. If a person rewrites an unusable draft manually, include that effort in the assisted route. Removing failures from the report creates a misleading picture of how the process operates. Keep consequential errors visible rather than averaging them away in an overall score.

Separate ordinary variation from a serious incident

Not every correction has the same importance. A stylistic edit differs from an invented fact that could mislead a decision maker. Agree categories and escalation ownership before the trial, and record examples with sensitive information removed. A privacy or access-control failure may require pausing even if the average completion time looks good. NIST’s AI Risk Management Framework organises risk work around govern, map, measure and manage. It treats risk management as ongoing and context dependent. The worksheet proposed here is a business evaluation aid; completing it is not NIST certification, a complete risk assessment or permission to deploy.

Record costs without turning estimates into savings

List actual charges attributable to the pilot and identify shared subscription costs separately. Include setup, maintenance, staff training, review and exception handling. Explain any allocation method rather than presenting the whole organisation’s subscription as the cost of one task. Do not publish account details or sensitive billing records in a general report. Time released is not automatically cash saved. The team may use that capacity for other work without reducing expenditure. Report the observed effort difference first. Any conversion to money needs an explicit costing method and assumptions about how capacity would actually be used. Keep projected future benefits distinct from measured pilot outcomes.

Show the cases behind the average

Summarise the number of eligible, attempted, accepted, rejected and unresolved tasks. Compare similar categories and show the spread of completion times, not just one average. A small set of unusually easy cases should not stand in for the whole workflow. Keep a record of missing observations and explain why they are absent. An illustrative conclusion could say: “The assisted route reduced drafting effort in straightforward cases, but corrections and unresolved cases prevent a rollout decision.” That is sample wording, not an Agenticures result. If the observations are too limited or uneven to support a conclusion, the useful finding is that more comparable evidence is needed.

Make the next decision explicit

End with a named decision owner, the evidence considered and a limited next step. Continue only when the trial is useful enough to justify its cost and the relevant risks remain controlled. Change the workflow when a specific bottleneck is identifiable. Stop when the task is unsuitable or the required controls cannot be maintained. Preserve the baseline, configuration and evaluation dates so the next trial can be compared honestly. A changed model, source set or review process creates a new evaluation context. Agenticures’ AI strategy work can help define a bounded use case and a measurement plan before a wider implementation is proposed.