Did your AI pilot create value, or just move the work?

An AI trial can feel impressive on day one. A draft appears in seconds; a meeting summary arrives before everyone has left the room. If you have already made the five decisions before buying an AI tool, the owner still has to ask a harder question: did the whole job improve?

If a person spends the time saved checking mistakes, finding missing context or repairing a customer interaction, the speed of the first draft tells you very little. A useful pilot measures the work from the original request through a checked, usable result.

The Australian Government recommends starting with one or two uses, testing whether a tool works as expected and continuing to monitor it. Its guidance also calls for human oversight of AI output. That makes a narrow, measured workflow a better starting point than rolling a tool across the business on enthusiasm alone. Source: business.gov.au

Pick one job and write down the starting point

Choose a repeatable task with a clear finish line, such as preparing a standard enquiry response for a person to check and send. Avoid a pilot where one mistake could have serious consequences until the right controls and specialist advice are in place. The National AI Centre's screening questions are a useful early check for whether a proposed use needs tighter control.

Before switching the tool on, record a small baseline. For two weeks, note how many requests arrive, how long each complete job takes, how many need rework, and how long customers wait. Note what the owner or team cannot do while this work is being done. Use the same definitions during the pilot, or the before-and-after numbers will be hard to interpret.

Decide in advance what would count as a useful result. For example: the checked response should take less total time without increasing late corrections or customer complaints. Write down the minimum improvement worth keeping and who can stop the trial. That threshold is an owner's choice, not a benchmark supplied by a tool vendor. If request complexity or staffing changes during the test, note it; comparing an easy week with a difficult one can make a weak process look strong.

Count the whole job

For four weeks, keep a simple row per job:

This can be a spreadsheet, not a new analytics system. Record enough to make a decision, without putting customer details into the scorecard. The National AI Centre's workflow guide suggests testing exception cases and recording the decision before building further.

Keep a separate count of tasks that the tool could not handle or that needed escalation. Averages can conceal the expensive exceptions. Review a small sample of completed work against the original requirement, not just the draft's appearance. Someone must own that review and have the authority to pause the workflow.

Imagine a hypothetical enquiry process that takes 20 minutes end to end. An AI draft takes six minutes to prepare, but the person needs nine minutes to verify and finish it. The gain is five minutes per usable response, not fourteen. At 30 responses a week, that is 150 minutes of potential capacity. Whether it creates financial value depends on what happens next: more enquiries handled well, shorter response times, paid work made possible, or a cost that can genuinely be avoided. Idle time is not automatically a cash saving.

Separate capacity from cash

There are at least three different outcomes worth reporting:

1. Capacity: Staff can handle more of the same work without extending hours.
2. Service: Customers get a faster or more consistent answer, with no rise in errors.
3. Cash: The business actually avoids a cost or earns contribution from additional work.

Do not multiply every minute saved by an hourly rate and call the result profit. Include the time spent setting up, teaching and supervising the process, plus software and maintenance costs. business.gov.au's digital tools guide recommends comparing total ownership cost, not just a purchase price.

If the team uses the recovered time for higher-value work, name the work and check whether it happened. If a new subscription replaces an existing tool, count the avoided cost only when the old spend is actually removed. This keeps a capacity estimate, a service improvement and a cash outcome from being reported as the same benefit three times.

Make a decision at the end of the pilot

Set a review date before the trial starts. Then choose:

• Continue if the full workflow is reliably better, the costs and controls are acceptable, and someone owns ongoing review.
• Change if drafts are fast but checking, exceptions or handoffs consume the gain. Narrow the task or redesign the process and retest.
• Stop if errors, privacy exposure, customer harm or total cost outweigh the benefit. Stopping a weak pilot is useful information, not a failed investment.

If personal information may enter an AI product, assess the data flow and the product before use. The OAIC's guidance calls for due diligence, human oversight and continued review, and advises particular caution with information entered into publicly available generative AI tools. The appropriate obligations and controls depend on the business and the use.

The owner does not need a grand AI return-on-investment claim. They need evidence that one real job got better, the business kept control, and the next dollar or hour has a sensible destination.