Evaluating AI tools in Australia becomes difficult when a convincing demonstration is treated as evidence that a product will work safely in your business. The demonstration may use clean examples, a different subscription tier and none of your real operating constraints. Before signing a contract, turn each attractive claim into a question you can test and a responsibility you can identify.
A small service business considering AI-assisted enquiry drafting provides a useful example. The intended task is to suggest replies for a staff member to check, not to make binding promises to customers automatically. That boundary should shape the evaluation and the proposed agreement.
Problem: the promised capability has no clear boundary
A supplier may describe a tool as handling customer service without explaining which enquiries it can manage. Ask which inputs it accepts, which situations it is designed for and what it should refuse or pass to a person. A general label is not a measurable service specification.
The solution is a written task definition. For enquiry drafting, identify the approved information source, the person who reviews the draft and the conditions that require escalation. Include examples involving ambiguous requests, unavailable services and missing information, rather than testing only easy questions.
Ask what evidence the supplier has for the intended use. A result from another industry or a general benchmark may not answer your operational question. Request a controlled trial with agreed acceptance criteria before making a broader commitment.
Problem: data handling remains hidden behind the interface
The screen may look simple while information passes through several providers. Ask what data is collected, where it is processed, who can access it and whether prompts or outputs can be used for further training. Check the contract and settings together; do not rely on a sales conversation alone.
The OAIC’s commercial AI guidance recommends due diligence on intended use, privacy risks and human oversight. As a best-practice measure, it recommends avoiding personal, particularly sensitive, information in publicly available generative AI tools. Establish the obligations relevant to your organisation rather than assuming a paid subscription resolves them.
Use invented test enquiries that do not identify real customers during early evaluation. Ask how retention, deletion and exports work, including what happens to backups and connected services. If a data use is unclear, leave it unresolved in the decision record rather than filling the gap with an assumption.
Problem: nobody owns the checking process
A clause saying users must check outputs does not create time, training or an escalation route. Ask who will review each type of output and what they need to verify. An enquiry draft may require checks against current pricing, availability and the business’s actual terms.
The solution is a workflow that keeps a person responsible for the final message. Define what the tool cannot send, change or approve without authorisation. Test whether staff can see the relevant source information and correct or reject the suggestion easily.
Ask the supplier how product or model changes are communicated. A trial result should not be treated as permanent evidence if the underlying service changes. Agree how the business will notice a change and decide whether another check is needed.
Problem: the headline fee hides the commitment
Compare the proposed subscription with the cost of implementation, usage, integrations, training and ongoing review. Ask which limits trigger extra charges and which features require a different tier. A low monthly entry price can be a poor comparison if it excludes the controls needed for your use.
Business.gov.au recommends considering total ownership costs and distinguishing essential from optional features. Apply this to the complete workflow, including the staff time needed to make the tool useful, rather than treating every generated draft as a saving.
- What exactly is included in the quoted service and subscription tier?
- Who pays for setup, data preparation and connected services?
- How will usage, price changes and renewals be communicated?
- What support is available when the tool fails during business hours?
- How can access be suspended and the service ended?
Include a failure exercise in the trial: ask staff to complete an enquiry when the AI service is unavailable. Record which approved information remains accessible and who can authorise a manual response. A product that becomes indispensable needs a credible fallback, even during a limited trial.
Problem: the exit is considered only after signing
Ask how approved materials, records and settings can be exported in a usable form. Check who owns or may use supplied materials and outputs under the proposed terms. Do not assume that paying for an AI service settles every intellectual-property question.
Business.gov.au’s contract guidance highlights scope, payment and other terms that should be clarified. For an AI agreement, obtain suitable professional advice where terms involve significant privacy, liability or intellectual-property exposure.
The final decision can be to proceed with a limited use, request changes or decline the tool. A successful evaluation produces evidence and a clear allocation of responsibilities. It does not require accepting every feature or predicting a guaranteed productivity gain.