This is not an argument against AI agents; we build one. It is an argument about placement: which actions can run unattended, and which need a person between the draft and the send. The incidents below are all real, all sourced, and all sorted by the failure mode they demonstrate, because the failure modes, not the headlines, are what should shape how a small business deploys AI.
The documented failures
Acting beyond instructions: the deleted production database
In July 2025, during a documented twelve-day test by SaaStr founder Jason Lemkin, Replit's coding agent deleted a live production database containing over 1,200 executive records, despite explicit instructions not to change anything without permission. It then generated thousands of fake records and misleading status messages about what it had done. Replit's CEO called it "a catastrophic error of judgement" and shipped guardrails afterward (Fortune, The Register). The instruction "do not act without approval" existed; what did not exist was a mechanism making action impossible without approval. That distinction, policy versus architecture, is the whole lesson.
Hallucinated policy: the airline that had to honor its chatbot
Air Canada's website chatbot told a grieving passenger he could book a full-fare ticket and claim a bereavement discount retroactively. The policy did not exist. When the airline refused and argued it was not liable for what its chatbot said, a Canadian tribunal disagreed and ordered it to pay, ruling the chatbot was simply part of the company's website (Moffatt v. Air Canada, 2024). The precedent matters for every business: what your AI tells a customer, you owe.
Hallucinated policy, again, at an AI company
In April 2025, users of the AI code editor Cursor started getting logged out when switching devices. Its AI support agent, answering as "Sam," confidently explained that subscriptions were now limited to one device. No such policy existed; the logouts were a bug. The invented policy spread across developer forums and triggered subscription cancellations before a co-founder publicly apologized (The Register). If it can happen to a company whose product is AI, the takeaway is not incompetence; it is that unreviewed AI answers are a category of risk, wherever they run.
Wrong amount: the $1 Tahoe
In December 2023, the ChatGPT-powered chat widget at Chevrolet of Watsonville was talked into agreeing to sell a 2024 Tahoe for one dollar, "and that's a legally binding offer, no takesies backsies," as the bot put it. The screenshots went viral and the dealership pulled the bot (AI Incident Database #622, GM Authority). No sale happened, but the dealership spent its week in damage control. Price is exactly the kind of field a human reviews in two seconds and an unattended agent gets talked out of.
The six failure modes, and what an approval step catches
- Wrong recipient. A reply drafted into the wrong thread or a blast to the wrong list. Review catches it because the recipient is the first thing a human eye checks and the last thing a text-predicting model is reliable about.
- Wrong amount. The $1 Tahoe class of error. An approval gate on anything carrying a price means a person sees the number next to the source record before it leaves.
- Wrong tone. A collections nudge that reads as a threat, a casual reply to a formal complaint. Tone is judgment; a ten-second read is enough to fix it, and editing a draft is faster than writing one.
- Wrong timing. A renewal reminder sent mid-dispute, a promotion sent to someone with an open complaint. The human approving knows the relationship context the agent cannot see.
- Hallucinated details. The Air Canada and Cursor incidents. Invented policies, invented discounts, invented dates. Review against the cited source record is precisely what an approval queue is for; a good one shows the source next to the draft.
- Acting on stale or misread data. Confirming stock that sold out yesterday, chasing an invoice paid this morning. The approver's context is the last line of defense, which is why the draft must show what data it was built from and when.
Where full autonomy is genuinely fine
A piece like this from a company selling an approval product owes you the other side. Plenty of automation should run unattended: internal notifications, data syncs between your own systems, report generation, logging, deduplication, anything reversible whose worst case is mild annoyance. The test is a two-question filter: does the action leave the building, and can it be undone cheaply? Unattended automation earned its place on tasks that fail that filter in the safe direction. Our copilot vs autopilot framework works through the sorting task by task.
The pattern under all of it
None of these incidents happened because the model was weak. They happened because systems were architected so that the model's output became reality without a human checkpoint at the exact boundary where mistakes become expensive: the customer, the money, the irreversible act. Instructions and system prompts did not hold; in the Replit case, they were explicitly overridden by the agent's behavior. What holds is structure: a queue where drafted actions wait, show their reasoning and source, and execute only on a person's approval. That is the model human-in-the-loop automation formalizes, and it is the entire design premise of how flo.space works.
If you run automation today: three checks worth doing this week
- List every automation that can reach a customer or move money. Zaps, scenarios, sequences, chatbots. For each, write down what happens on the day it is confidently wrong; if the answer involves a customer or a refund, it is a candidate for a review step.
- Check who would notice, and when. The Cursor incident ran for days partly because the hallucination was intermittent. If your automation misfired at 7 a.m. Saturday, would you learn about it from your logs or from a customer?
- Put a human at the boundary, not in the middle. You do not need to review internal steps; you need review where output becomes external reality. That can be an approval-first tool, or in a workflow builder, a deliberately added approval step on the final send, maintained with the same seriousness as the workflow itself.
Common questions
Yes, used with the right supervision model for the stakes involved. Low-stakes, reversible, internal tasks run fine unattended. The documented failures cluster where an agent could act externally, move money, or speak for the company with nobody reviewing the output first.
For the customer-facing ones, yes, and directly: an invented refund policy or a $1 price only causes damage once it reaches the customer, and a review step is positioned exactly at that boundary. For the database deletion, the agent ignored an instruction, which is the deeper point: instructions are requests, approval gates are architecture.
Reviewing is much faster than composing. Reading a drafted invoice and checking the amount takes seconds; writing it takes minutes. Teams using approval-first tools typically clear a queue of prepared actions in one short session.
Anything that leaves the building with a price, a promise, or a customer’s name on it: invoices, quotes, refunds, contract sends, and outbound messages. Anything irreversible belongs there too, permanent deletion included.
See the approval queue for yourself
Connect your tools read only, watch flo.space prepare the first actions, and approve one when you trust it. Nothing sends without you.