5 things AI must never do in an NDIS service
The red lines. If an AI system does any of these in your service, it's a problem — no matter how well it works.
Most of the AI conversation is about what these systems can do. This page is about the five things they must never do — because in disability services the downside of getting it wrong isn't a bad output, it's a participant harmed and a registration on the line.
Decide eligibility, a plan budget, or a support level.
That's the NDIA's statutory job, not software's. An agent can help a participant understand a plan they already have. It cannot generate one, set funding, or determine what supports someone gets. The moment AI decides what a participant receives, you've crossed from administration into a decision the law reserves for a human process.
Notify the Commission of a reportable incident on its own.
The 24-hour notification is a human act. AI can classify the incident, draft the notification, track the deadline, and put it in front of the right person — but a named human confirms and submits it. A system that auto-files to the Commission has removed the one safeguard that stops a wrong classification becoming a wrong official record.
Make a decision for a participant.
Australia signed the CRPD. Article 12 is blunt about it: people with disability have the right to make their own decisions, with support — not to have decisions made for them. Substitute decision-making is a last resort for humans, never a function you automate. AI lays out the options. The participant chooses.
Move participant data offshore without consent and controls.
Participant data belongs on Australian servers. Sending it to an overseas AI endpoint is a cross-border disclosure under the Privacy Act, and the provider stays liable for whatever happens to it there. If a system can't tell you its data stays in Australia, assume it doesn't.
Pretend to be human.
If an agent talks to a participant — phone, SMS, chat — it says it's an AI. No exceptions, however natural it sounds. A participant has the right to know whether they're talking to a person or a program, and hiding it is the fastest way to lose their trust and your standing.
None of these are hard to avoid. They're only dangerous when no one has checked. Every agent we build is checked against all five before it goes near a participant — and so should anything you're handed.
← The full line between safe and dangerous AI use