The Therac-25 radiation machine is taught as the literal textbook case study regarding software safety because six patients were injured or killed.
Meanwhile, LLMs Have Led to 33 Deaths
We should also be talking about recommendation algorithms (“social” media especially.) There have been genocides.
1000%
Yeah, I don’t think lumping together all these cases is a good idea. There’s legit cases like AI induced psychosis. In which a chatbot directly contributed to deaths, likely even directly caused them. On the other hand a case when a murderer asks what happens if you mix benzodiazepine with alcohol, and the chatbot says it can lead to death… Or if it’s possible to keep an unregistered firearm at home… Well, clearly the murderer already made up their mind and out of some kind of stupidity, they’re bouncing ideas. All the AI does at that point is gather evidence. So it’s doing a good and legit job? I mean I know something that’s far worse and that is: murder mystery stories and movies. NCIS… Shouldn’t we outlaw those? Those are probably planting new ideas into people’s minds… Whereas ChatGPT only served as some poor-man’s Google here.
On the flipside we’re probably forgetting about the hundreds(?)/thousands(?) of other cases who are on the brink of AI psychosis, or unwell and harmed by (and because of) AI. They’re not dead (yet), but probably not okay either. And pumping out some random number like 33 and setting random standards to qualify is probably doing them a disservice. Or even something fairly simple like being amongst the 30% of staff being fired because of AI, is far worse than it reading back the package insert of some medication.
(Also, please don’t commit suicide on a railway. That’s horrible. And every commuter hates it. And I heard the train drivers hate it as well.)
Interesting, but the very first item on the list is bullshit. Just because someone asked an AI about how to kill some one doesn’t mean the AI is responsible for the murder. This would be way more convincing if it wasn’t grasping at any connection at all. What is the count of direct cause deaths? The Therac-25 directly cooked people’s bodies with radiation. There is no loose connection, yeah that killed people directly.
To argue that ChatGPT et al. can’t cause direct harm (i.e. irradiate people) misses the larger point I think. To me, that’s slicing up the ethical responsibility of a system until no component bears ownership. Extending that logic to Therac-25 might be saying a function or variable can’t irradiate people. Sure, not by itself. But it existed within a system that reliably produced harm.
From the Therac-25 wiki: “These accidents highlighted the dangers of software control of safety-critical systems.”
I’d argue that mental health conversations are safety critical systems. Another safety critical system might be a basic search engine, if a user turns to it with real intent to do harm.
There’s a reason we put safety warnings on products, regulate access to chemicals, etc, etc. I’m not arguing that we should do the same for LLMs, just showing that the Therac-25 example is rookie numbers compared to the reach and risks of modern software.
I feel like that largely ignores the idea of what an actual safety critical system is. It doesn’t cover malicious usage in the browser case. Idk if I would put the amplifying effect it has on the mentally ill in that category either. It’s just a wide net to be casting.
You would have to include stuff like social media, or just simply having conversations with people. Discord would fit in just as much as AI. Most of the internet would tbh.
It should probably be kept away from the mentally ill and children to an extent, but since it’s completely virtual and available on the net, that would mean some intense regulation. No thanks, not interested in the internet being treated as a dangerous chemical. Regulations would help openai a lot more than regular folks in any case.
I think the numbers are way way higher…
