14 Questions to Ask an AI Automation Agency Before You Sign

An AI automation agency in Australia builds software that does repetitive business work for you, such as sorting enquiries, updating your CRM, drafting replies or chasing approvals. These 14 questions help you tell the teams that ship reliable systems from the ones that sell a demo.

Want to try them on us first? Book a free 30-minute scoping call and ask all 14.

How to use these questions

Use these 14 questions by asking every shortlisted AI automation agency the same questions in the same order and writing down exactly what each one says.

The questions sit in six groups, and each one has three parts: why it matters, what a good answer sounds like and what a red flag sounds like.

Group	What it tests	Questions
Scope and fit	Whether they understand your problem and have real work to show	1 to 2
Delivery and team	Who builds it and how you see progress	3 to 4
Reliability and human approval	What happens when the AI is wrong	5 to 7
Data, privacy and security	Where your information goes	8 to 9
Ownership and handover	What is yours at the end	10 to 11
Cost, measurement and support	How it is priced, proven and looked after	12 to 14

A scorecard at the end lets you rate each answer from 0 to 2 and compare agencies side by side. Print it, or copy it into a spreadsheet.

Scope and fit: whether they understand your problem

Scope and fit questions test whether an AI automation agency understands your specific problem and can show real, working systems rather than a polished sales demo.

1. Which of my processes would you automate first, and which would you leave alone?

Why it matters	An agency that automates everything is selling volume. The best first project is narrow, repeated often and easy to check.
A good answer sounds like	”Start with this one process. We would not automate that other step yet, because the rules are unclear and a mistake is costly.” They ask about triggers, owners and exceptions.
A red flag sounds like	A full-company transformation pitched before they have asked how any single task works.

If you are not ready to pick a process, that is a sign to start with advice. We run AI consulting separately from builds for exactly that reason. Our guide to AI agency vs AI consultant explains which one suits your situation.

2. Can I see a system you built running in production today?

Why it matters	A demo shows what is possible. A production system shows what survives real data, real users and real mistakes.
A good answer sounds like	A live walk-through, including a case where it got something wrong. Where client work is confidential, they describe the systems connected, the problem and the outcome in specifics.
A red flag sounds like	Only slides, screenshots or a recorded video, and no example of a failure.

Ask which tools were used and whether any part was custom code. Our AI agents service page shows how we describe our own builds.

Delivery and team: who builds it and how you see progress

Delivery and team questions establish who will actually scope, build and support your AI automation project, and how often you will see working software.

3. Who will scope, build and support my project?

Some agencies sell with senior people and hand delivery to others. That can be fine, but you should know before you sign.

A good answer sounds like	A red flag sounds like
Named roles, not names: who scopes, who writes the code, who you speak to each week and what happens if someone leaves. They are open about subcontractors.	”Our team will handle it” with no detail, or a reluctance to say where the work is done.

The Lightning Ventures team is Melbourne-based and 100% Australian. Location matters less than overlap with your hours and how quickly the team replies. Read more about our AI agency in Melbourne work.

4. How will delivery run, and when will I see working software?

Why it matters	Long builds hide problems until the end. Short cycles surface them while they are cheap to fix.
A good answer sounds like	A cadence with something usable at each checkpoint, a named person who reviews it with you and a clear way to change direction.
A red flag sounds like	A long build phase followed by a big reveal, or progress reports that are only status colours.

We work in two-week sprints, and each sprint is agreed before it starts. Our guide to how we scope projects shows what the first conversation covers.

Reliability and human approval: what happens when the AI is wrong?

When an AI automation is wrong, a good agency can show what was logged, who was alerted, and which actions a person approves before anything happens.

5. What gets logged, and who is told when something fails?

AI makes mistakes. Without a record of every action, you cannot find out what happened, fix it or explain it to a customer.

A good answer sounds like	A red flag sounds like
A log of each step and decision, alerts to a named person when something breaks, and a way to replay or reverse an action.	”It rarely fails,” or logs that only the agency can see.

Listen for detail rather than reassurance. A team that has run systems in production can describe a real failure and how it was caught.

6. Which actions will the AI take alone, and which need a person to approve?

Why it matters	Sending an email draft is low risk. Issuing a refund or changing a record is not. The line should be drawn on purpose.
A good answer sounds like	A written split. On our builds we use three tiers: what the AI does on its own, what it drafts for a person to approve, and what stays human-only.
A red flag sounds like	”Fully autonomous” for anything involving money, advice, health or customer commitments.

Ask to see that split in writing before any build starts. Accountability and human oversight are both named in Australia’s AI Ethics Principles, which are voluntary but a useful yardstick.

7. How will you test it before it touches live data?

Why it matters	A system that works on five tidy examples can fail on the messy real thing.
A good answer sounds like	A set of real, awkward examples agreed with you, a trial run alongside the current process and a defined point at which it goes live.
A red flag sounds like	Testing described as “we try it and see,” or going live on day one with no trial period.

Data, privacy and security: where your information goes

Data and privacy questions ask which AI providers see your information, where it is processed and stored, and how an agency limits access to it.

8. Where is my data processed and stored, and which AI providers see it?

Business and customer data may pass through several providers. You are still responsible for it.

A good answer sounds like	A red flag sounds like
A list of the model providers and hosting they would use, where data is stored, whether it is used to train anyone’s models and what the options are if you need data kept in Australia.	”It is all secure,” with no specifics, or an unwillingness to put the answer in the proposal.

9. How do you handle personal information and limit access?

Why it matters	Many Australian businesses have obligations under the Privacy Act 1988 and the Australian Privacy Principles. Regulated sectors have more.
A good answer sounds like	Minimal access to your systems, separate credentials you can revoke, a plan for removing data that is no longer needed and a willingness to work with your legal or compliance adviser.
A red flag sounds like	A request for an admin login to everything, or shared passwords.

This is not legal advice. Ask your own adviser to review the terms if the data is personal, financial or health information.

On security, ask how the agency’s own practices line up with the ACSC’s Essential Eight baseline, such as access control and backups.

For a real example of connected systems, our work for a global sustainability NGO used MCP integrations with HubSpot and Xero, and the organisation cut reporting time by 50%. That kind of access should always be scoped, named and reversible.

Ownership and handover: what is yours at the end

Ownership questions confirm that your business, not the agency, holds the accounts, code, prompts and data, and that another team could take over if you part ways.

10. Who owns the accounts, code, prompts and data?

If the agency holds the logins, you depend on them for something your team may use every day.

A good answer sounds like	A red flag sounds like
You own all of it, on infrastructure billed to your own accounts, written into the agreement.	Everything runs through the agency’s subscriptions, or the prompts and configuration are described as proprietary.

11. If we stop working together, what do I get and how fast?

Why it matters	You are choosing an exit as well as a start. A clean handover makes the relationship honest.
A good answer sounds like	Documentation another team could follow, a list of every tool and credential, and a defined handover period.
A red flag sounds like	Handover is “available on request” with no detail, or only at an extra charge that is not stated.

Cost, measurement and support: how is it priced, proven and looked after?

Cost, measurement and support questions cover how an AI automation agency prices work, proves a return against a baseline, and looks after the system after launch.

12. How is the work priced, and what will it cost to run?

The build is one cost. Model usage and hosting are separate, and they continue after launch.

A good answer sounds like	A red flag sounds like
A conversation and written scope first, then a price. Running costs are estimated separately and billed to your own accounts.	A price on the first call, or a single figure for “an AI agent” regardless of your systems.

Four things move the cost: how many systems are connected, how clean the data is, how much human review is needed and what compliance requires. At Lightning Ventures the scoping call is free and builds are priced per two-week sprint.

13. How will we prove it worked?

Why it matters	Without a baseline taken before the build, any result is a claim, not a measurement.
A good answer sounds like	Agreed measures such as hours spent now, error rate and how often a person steps in, all captured before the build and checked after.
A red flag sounds like	Return described in general terms, or a number that appears for the first time after launch.

Climate Salad is a good example, with a caveat: it was no-code automation across their member systems, not an AI agent. The result was exactly 10 hours a week saved, a figure that depends on knowing the starting point. See the Climate Salad case study and our guide to measuring AI ROI.

14. What happens after launch, and who fixes it when something changes?

Providers change models, and integrations break. A system nobody looks after slowly stops working.

A good answer sounds like	A red flag sounds like
A named owner, a response expectation, a plan for model and integration changes, and a clear way to ask for improvements.	Support described as “we are always here for you” with nothing in writing.

For a system that stays in use, Carino Care is a useful comparison. It was a custom build rather than an AI automation, and the Carino Care case study records 15+ hours of admin removed every week as a retainer client.

Scorecard: how do you compare agencies side by side?

Score each agency’s answer from 0 to 2 across all 14 questions, then add the totals to compare AI automation agencies side by side out of 28.

Use 0 for a red flag or no answer, 1 for a partial or verbal-only answer and 2 for a specific answer in writing.

#	Question	Agency A	Agency B	Agency C
1	Which process first, and what to leave alone
2	A system running in production
3	Who scopes, builds and supports
4	Delivery cadence and visible progress
5	What is logged and who is alerted
6	AI alone, drafts for approval, human only
7	Testing before live data
8	Where data is processed and which providers
9	Personal information and access limits
10	Ownership of accounts, code, prompts, data
11	Handover if we part ways
12	Pricing method and running costs
13	Baseline and proof of return
14	Support and changes after launch
	Total (out of 28)

Treat any 0 on questions 5, 6, 8 or 10 as a reason to pause, whatever the total. Those cover failure handling, human approval, data and ownership, and they are the hardest to fix later. For the other half of the decision, which type of agency suits your problem, read our guide on how to choose an AI agency.