7
I finally had a major AI tool fail on me during a live demo yesterday and it was brutal
My automated customer service chatbot (using a popular language model) started giving out fake refund links to users in a 30-minute window before I caught it, so I had to manually shut down the whole system and send apology emails to about 200 people - has anyone else had their AI suddenly hallucinate something that could actually cause real harm?
2 comments
Log in to join the discussion
Log In2 Comments
jake6386d ago
Went through something similar at my last company with a legal document summarizer. It started inserting fake court cases into summaries, just made up references that looked real but never existed. We only caught it because a paralegal had an off feeling about one citation and checked it. Hallucinations are scary enough when it's just bad advice but when the AI starts creating convincing lies that could fool professionals, that's a whole different level of dangerous. What safeguards do you have in place now to catch stuff like that before it goes live?
5
harper6936d ago
That paralegal saved them from a huge lawsuit right there. I saw a similar thing happen with a real estate AI that started making up zoning laws that didn't exist. We ended up having to build a whole verification layer where every single citation gets fed through a separate checker before it reaches a client. The real problem is these things are too confident even when they're dead wrong.
-1