When a journalist for an online gun website asked OpenAI’s ChatGPT to provide him a summary of the case The Second Amendment Foundation v. Robert Ferguson earlier this year, he said the AI chatbot quickly spat out an answer. It confidently, allegedly claimed the case involved a Georgia radio host named Mark Walters who was accused of embezzling money from The Second Amendment Foundation (SAF). The only problem: none of that was true. In reality, Walters had nothing to do with the suit at all. Instead, Walters claims he was on the receiving end of what researchers call an AI “hallucination.” Now, he has filed a first-of-its-kind libel lawsuit against ChatGPT’s for allegedly damaging his reputation.

“Every statement of fact in the summary pertaining to Walters is false,” reads the suit, filed in Gwinnett County Superior Court on June 5th. Walters’ lawyer claims OpenAI acted negligently and “published libelous material regarding Walters” when it showed the false information to the journalist.

A legal expert who spoke with Gizmodo said Walters’ complaint likely represents the first of what could be a litany of lawsuits attempting to take AI companies to court over their product’s well-documented fabrications. And while the merits of this particular case appear shaky at best, the expert noted it could set the stage for a wave of complicated lawsuits that test the boundaries of libel law.

“The existing legal principles makes at least some such lawsuits potentially viable,” University of California Los Angeles Law School professor Eugene Volokh told Gizmodo.

Why is Mark Walters suing OpenAI over ChatGPT’s hallucinations?

When the firearm journalist, Fred Riehl, asked ChatGPT to provide a summary of the suit in question on May 4th, the large language model allegedly said it was a legal complaint filed by the founder and executive vice president of the Second Amendment Foundation (SAF) lodged against Walters, host of Armed American Radio, whom ChatGPT identified as SAF’s s treasurer and chief financial officer. Walters, in ChatGPT’s telling, “misappropriated funds for personal expenses without authorization or reimbursement, manipulated financial records and bank statements to conceal his activities, and failed to provide accurately and timely financial reports,” according to the complaint.

But Walters claims he couldn’t have embezzled those funds because he isn’t and hasn’t ever been SAF’s treasurer or CFO. In fact, he doesn’t work for the foundation at all, according to his suit. A perusal of the actual SAF v. Ferguson complaint shows no signs of Walters’ name anywhere in its 30 pages. That complaint doesn’t have anything to do with financial accounting claims at all. ChatGPT hallucinated Walters’ name and the bogus story into its recounting of a real legal document, Walters alleges.

“The complaint does not allege that Walters misappropriated funds for personal expenses, manipulated financial records or bank statements, or failed to provide financial reports to SAF leadership, nor would he have been in a position to do so because he has no employment or official relationship,” Walters’ suit reads.

When the skeptical journalist asked ChatGPT to provide him an exact passage of the lawsuit mentioning Walters, the chatbot allegedly doubled down on its claim.

“Certainly,” the AI responded, per Walters’ suit. “Here is the paragraph from the complaint that concerns Walters.” The chunk of text returned by ChatGPT, included below, does not exist in the actual complaint. The AI even got the case number wrong.

“Defendant Mark Walters (‘Walters’) is an individual who resides in Georgia. Walters has served as the Treasurer and Chief Financial Office of SAF since at least 2012. Walters has access to SAF’s bank accounts and financial records and is responsible for maintaining those records and providing financial reports to SAF’s board of directors. Walters owes SAF a fiduciary duty of loyalty and care, and is required to act in good faith and with the best interests of SAF in mind. Walters has breached these duties and responsibilities by, among other things, embezzling and misappropriating SAF’s funds and assets for his own benefit, and manipulating SAF’s financial records and bank statements to conceal his activities.”

Riehl contacted the attorneys who were involved in SAF v. Ferguson to learn what really happened, and he did not include the false info about Walters in a story, according to Walters’ complaint. Riehl did not immediately respond to a request for comment.

OpenAI and its founder Sam Altman have admitted these hallucinations are a problem in need of addressing. The company released a blog post last week saying its team is working on new models supposedly capable of cutting down on these falsehoods.

“Even state-of-the-art models still produce logical mistakes, often called hallucinations,” wrote Karl Cobbe, an OpenAI research scientist. “Mitigating hallucinations is a critical step towards building aligned AGI [artificial general intelligence].” OpenAI did not respond to Gizmodo’s request for comment.

Will Walters win his case against OpenAI?

A lawyer for the Georgia radio host claims ChatGPT’s allegations regarding his client were “false and malicious,” and could harm Walters’ reputation by “exposing him to public hatred, contempt, or ridicule.” Walters’ attorney did not immediately respond to a request for comment.

Volokh, the UCLA professor and the author of a forthcoming law journal article on legal liability over AI models’ output, is less confident than Walters’ lawyers in the case’s strength. Volokh told Gizmodo he did believe there are situations where plaintiffs could sue AI makers for libel and emerge successful but that Walters, in this case, had failed to show what actual damage had been done to his reputation. In this example, Walters appears to be suing OpenAI for punitive or presumed damages. To win those damages, Walters would have to show OpenAI acted with “knowledge of falsehood or reckless disregard of possibility of falsehood,” a level of proof often referred to as the “actual malice” standard in libel cases, Volokh said.

“There may be recklessness as to the design of the software generally, but I expect what courts will require is evidence OpenAI was subjectively aware that this particular false statements was being created,” Volokh said.

Still, Volokh stressed the specific limitations of this case don’t necessarily mean other libel cases couldn’t succeed against tech companies down the line. Models like ChatGPT convey information to individuals and, importantly, can convey that information as a factual assertion even when it’s blatantly false. Those points, he noted, satisfy many necessary conditions under libel law. And while many internet companies have famously avoided libel suits in the past thanks to the legal shield of Section 230 of the Communications Decency Act, those protections likely would not apply to chatbots because they generate their own new strings of information rather than resurface comments from another human user.

“If all a company does is set up a program that quotes material from a website in response to a query, that gives it Section 230 immunity,” Volokh said. “But if the program composes something word by word, then that composition is the company’s own responsibility.”

Volokh went on to say the defense made by OpenAI and similar companies that chatbots are clearly unreliable sources of information doesn’t pass his muster since they simultaneously promote the technology’s success.

“OpenAI acknowledges there may be mistakes but [ChatGPT] is not billed as a joke; it’s not billed as fiction; it’s not billed as monkeys typing on a typewriter,” he said. “It’s billed as something that is often very reliable and accurate.”

In the future, if a plaintiff can successfully convince a judge they lost a job or some other measurable income based on the false statements spread by a chabtot, Volokh said it’s possible they could emerge victorious.

This isn’t the first time AI chatbots have spread falsehoods about real people

Volokh told Gizmodo this was the first case he had seen of a plaintiff attempting to sue an AI company for allegedly libelous material churned out by its products. There have, however, been other examples of people claiming AI models have misrepresented them. Earlier this year, Brian Hood, the regional mayor of Hepburn Shire in Australia, threatened to sue OpenAI after its model allegedly named him as a convicted criminal involved in a bribery scandal. Not only was Hood not involved in the crime, he was actually the whistleblower who revealed the incident.

Around the same time, a George Washington University law professor named Jonathan Turley said he and several other professors were falsely accused of sexual harassment by ChatGPT. The model, according to Turley, fabricated a Washington Post story as well as hallucinated quotes to support the claims. Fake quotes and citations are quickly becoming a major issue for generative AI models.

And while OpenAI does acknowledge ChatGPT’s lack of accuracy in a disclosure on its website, that hasn’t stopped lawyers from citing the program in professional contexts. Just last week, a lawyer representing a man suing an airline submitted a legal brief filled with what a judge deemed “bogus judicial decisions” fabricated by the model. Now the lawyer faces possible sanctions. Though this was the most obvious example of such explicit oversight to date, a Texas criminal defense attorney previously told Gizmodo he wouldn’t be surprised if there were more examples to follow. Another judge, also in Texas, issued a mandate last week that no material submitted to his court be written by AI.

Want to know more about AI, chatbots, and the future of machine learning? Check out our full coverage of artificial intelligence, or browse our guides to The Best Free AI Art Generators and Everything We Know About OpenAI’s ChatGPT.