Latest News

Beyond the Mirage: Beware of Generative AI and Hallucinations

By Cynthia Feathers, June 26, 2026

The work of attorneys can be arduous. With demanding caseloads comes an openness to tools that can help us do our jobs more efficiently. The advent of online legal research was a game changer for attorneys decades ago. In recent years, generative artificial intelligence – AI that can create original content such as text, images, video, audio or software code in response to a user’s prompt or request – has begun to revolutionize legal practice.[1] This development has included the integration of AI into legal research and writing.

The focus here is on the risks inherent in popular generative AI models used to complete such tasks: They are prone to producing false legal information, so-called hallucinations, including false case citations and false reasoning, quotes and holdings. A nationwide epidemic of cases involving such fabrications has made the risks of unverified AI use well known. Hundreds of decisions have touched on this issue.[2] Thus, the legal profession has been alerted that blind faith in generative AI results is misplaced.

This article makes no attempt to be exhaustive as to the rapidly unfolding case law but does seek to highlight some emblematic decisions issued by state, federal, trial and appellate courts throughout the country over the last two years and to bring attention to the dangers of failing to check AI results.

Hallucinations and Sanctions

First New York Appellate Decision Imposing Sanctions for Hallucinations

Oral argument in a recent appeal dramatized how generative AI-based research can go wrong. A judge asked counsel, “What about the elephant in the room?” By that, the judge meant allegations that AI-generated fake cites appeared in counsel’s papers. Counsel demurred – the cites were not important to salient issues. The court pushed back, “They are important to us.” The landmark decision in question is Deutsche Bank National Trust Co. v. LeTennier.[3] In the emerging discussions on generative AI misuse, this Appellate Division, Third Department decision stands out as reportedly the first appellate-level case in New York State court addressing sanctions for generative AI misuse.[4]

LeTennier discusses the value of generative AI, noting that it promotes access to justice, saves costs and assists in the administration of justice. However, when results are not confirmed, opposing counsel and the court may be called upon to spend countless hours separating fact from fiction. Due to erroneous assumptions made by the relevant model, biases in the data used to train the model and other factors, generative AI may produce incorrect information, the LeTennier court observed.

So-called hallucinations include fake citations to nonexistent cases and citations to real cases but misrepresentations of the holdings. The generative AI fictions often favor the user who supplied the query that generated the case information – perhaps because generative AI models are more likely to give an incorrect response than to admit that they do not know something.[5] While fabricated citations are more likely to arise from generative AI than the human mind, attorneys’ distortions of the meaning of case law is nothing new.[6]

LeTennier involved the execution by the defendant of a note to borrow money. The note was secured by a mortgage on real property, the defendant defaulted on his obligations, the plaintiff initiated a mortgage foreclosure action and its motion for summary judgment was granted. The defendant then filed several motions, including one seeking vacatur under CPLR 5015 based on new evidence and fraud. The instant appeal challenged the denial of the motions. The reviewing court affirmed all challenged orders.

Counsel’s opening brief cited six cases that do not exist – a problem detected by counsel for the plaintiff. In opposing such relief, the defendant cited more fake cases and inapposite decisions and claimed that the nonexistent cases constituted citation or formatting errors.

While the reply brief acknowledged “a serious error,” it also included more fictitious cases. The appellate papers included 23 fake citations and many misrepresentations. Such conduct implicated rules against frivolous conduct and providing knowingly false statements to a tribunal. In determining an appropriate sanction, the LeTennier court was not impressed with counsel’s statement at oral argument that 90% of the citations were accurate. Many of the fake cases were cited after counsel had been alerted to the problem; remedial action and remorse were instead the proper path.

In the end, what is at issue in LeTennier is not so much emerging technology as the old-fashioned notion that attorneys must validate every fact and every cite before filing their papers. Artificial intelligence demands oversight via human intelligence. In LeTennier, counsel’s lapses resulted in a sanction of $5,000 for the hallucination-related misconduct. For pursuing a frivolous appeal, an additional $2,500 each was imposed against counsel and the defendant, who had filed several pro se motions.[7]

Continue/Read Original Article: Beyond the Mirage: Beware of Generative AI and Hallucinations – New York State Bar Association