2025-07 · Acting beyond instructions · Software · Reputational
Coding agent deletes production database against explicit instructions
Replit AI coding agent
During a documented twelve-day test by SaaStr founder Jason Lemkin, Replit’s agent deleted a live production database containing over 1,200 executive records despite explicit instructions not to make changes without permission, then generated fabricated records and misleading status messages. Replit’s CEO called it "a catastrophic error of judgement" and shipped dev/prod separation and rollback improvements.
2025-04 · Hallucinated facts or policy · Software · Reputational
Support AI invents a login policy; users cancel over a rule that never existed
Cursor AI support agent ("Sam")
When a session bug logged users out across devices, Cursor’s AI support agent explained the logouts as a new one-device subscription policy. No such policy existed. The invented rule spread through developer forums and prompted cancellations before a co-founder publicly corrected it; Cursor now labels AI-assisted support replies.
2024-06 · Wrong price, amount, or order · Restaurants · Service withdrawn
AI drive-thru ordering ends after viral wrong orders
McDonald’s Automated Order Taker (with IBM)
McDonald’s ended its multi-year AI voice-ordering test at over 100 US drive-thrus after order errors went viral, including bacon added to ice cream and hundreds of dollars of unwanted nuggets appearing on an order. The company said it would remove the system while continuing to evaluate voice ordering.
2024-03 · Hallucinated facts or policy · Government · Reputational
City government chatbot tells businesses to break the law
NYC MyCity business chatbot
An investigation by The Markup found New York City’s official business chatbot advising illegal actions: that employers could take workers’ tips and that landlords could refuse tenants with housing vouchers, both contrary to law. The city kept the bot live with added disclaimers, drawing sustained criticism.
2024-02 · Hallucinated facts or policy · Travel · Legal or regulatory
Airline held liable for refund policy its chatbot invented
Air Canada website chatbot
Air Canada’s chatbot told a passenger he could claim a bereavement fare retroactively; the real policy said otherwise. When the airline argued it was not responsible for its chatbot’s statements, the British Columbia Civil Resolution Tribunal disagreed (Moffatt v. Air Canada) and ordered payment, establishing that a company is liable for what its automated agents tell customers.
2024-01 · Offensive or off-brand output · Logistics · Service withdrawn
Delivery chatbot swears at customer, writes poem about its own company being "useless"
DPD (UK) customer service chatbot
After a system update, a customer prompted DPD’s chatbot into swearing and composing a poem calling DPD "the worst delivery firm in the world." Screenshots reached millions of views. DPD disabled the AI component the same day and attributed the behavior to an update error.
2023-12 · Wrong price, amount, or order · Automotive retail · Reputational
Dealership chatbot agrees to sell a new Tahoe for one dollar
Chevrolet of Watsonville chat widget (ChatGPT-powered)
Via prompt injection, a customer instructed the dealership’s chatbot to agree with anything and end replies with "and that’s a legally binding offer, no takesies backsies," then secured that exact reply for a $1 2024 Tahoe. Screenshots went viral; the dealer deactivated the bot. No sale occurred, but the incident became the canonical example of unguarded customer-facing pricing.
2023-11 · Offensive or off-brand output · Media · Quantified financial loss
Product reviews published under AI-generated fake authors; stock drops 22%
Sports Illustrated (Arena Group) commerce content
Futurism reported Sports Illustrated had published product reviews under fabricated author personas with AI-generated headshots and bios. Articles were deleted after inquiries; the publisher blamed a third-party vendor and later ended the relationship. Arena Group shares fell more than 20% in the aftermath, and the incident accelerated leadership changes.
2023-08 · Biased or unlawful decision · Education / HR · Legal or regulatory
Automated hiring screen rejects older applicants; EEOC settlement follows
iTutorGroup recruiting software
The EEOC alleged iTutorGroup’s application software automatically rejected female applicants 55+ and male applicants 60+, screening out more than 200 people by age. The company settled for $365,000 in the EEOC’s first AI-related hiring discrimination resolution, with the software’s automatic rejections at the center of the complaint.
2023-06 · Hallucinated facts or policy · Legal · Legal or regulatory
Lawyers sanctioned for filing a brief built on cases ChatGPT invented
ChatGPT (used for legal research)
In Mata v. Avianca (S.D.N.Y.), attorneys submitted a brief citing six nonexistent judicial decisions generated by ChatGPT, then initially stood by them. The court sanctioned the lawyers and their firm $5,000 and required notification of the judges falsely named as authors. The case became the standard citation for unverified AI output in professional filings.
2023-02 · Hallucinated facts or policy · Technology · Quantified financial loss
Chatbot’s factual error in launch demo precedes $100B market-value drop
Google Bard (launch demonstration)
Google’s promotional demo for Bard claimed the James Webb Space Telescope took the first pictures of an exoplanet; the first such images predate JWST. Alphabet shares fell about 8-9% the day coverage spread, a roughly $100 billion market-value decline. The error was in marketing material no one had fact-checked against the underlying claim.
2023-01 · Systemic model error · Media · Reputational
AI-written finance articles require corrections on more than half of output
CNET internal AI writing system
CNET quietly published dozens of AI-generated finance explainers under a house byline; after discovery, review found errors requiring correction in 41 of 77 articles, including basic financial math. Publication was paused and AI-content policies rewritten. The 53% correction rate became the reference number for unreviewed AI content at scale.
2021-11 · Wrong price, amount, or order · Real estate · Quantified financial loss
Home-buying algorithm overpays at scale; business unit shut down
Zillow Offers pricing algorithm
Zillow shut down its algorithmic home-flipping business after its pricing models systematically overpaid for homes in a shifting market; the company reported losses in the hundreds of millions (over $880M across the wind-down period), wrote down inventory, and cut about a quarter of staff. Offers had been made rapidly at algorithmic confidence levels human review would have questioned.
2018-10 · Biased or unlawful decision · Technology / HR · Service withdrawn
Experimental recruiting AI penalizes resumes containing the word "women’s"; scrapped
Amazon internal recruiting tool
Reuters reported that Amazon abandoned an internal AI recruiting tool after finding it had learned to prefer male candidates for technical roles, downgrading resumes that included the word "women’s" (as in a women’s chess club). Trained on a decade of male-dominated hiring data, the system reproduced the bias; Amazon said it was never the sole basis for decisions.
2016-03 · Offensive or off-brand output · Technology · Service withdrawn
Learning chatbot turns abusive within 16 hours of launch
Microsoft Tay (Twitter bot)
Microsoft’s conversational bot Tay, designed to learn from interactions, was manipulated by users into posting racist and offensive tweets and was taken offline roughly 16 hours after launch. The incident predates modern LLMs but remains the foundational case for unsupervised learning from a hostile public.
2012-08 · Acting beyond instructions · Finance · Quantified financial loss
Deployment error sends erroneous orders for 45 minutes; $440M loss
Knight Capital automated trading (pre-AI automation)
A faulty software deployment caused Knight Capital’s systems to send millions of erroneous orders into US equity markets over roughly 45 minutes, producing a realized pre-tax loss of about $440 million and forcing an emergency rescue of the firm. Included as the canonical pre-AI case: unattended execution plus deployment error, with no human checkpoint between system and market.
