Back to Blog
When AI Hallucinations Reach the Courtroom: Two Real Cases
AI Research

When AI Hallucinations Reach the Courtroom: Two Real Cases

Editorial Team5 min readOctober 15, 2026

AI hallucinations usually get filed under "funny" — a chatbot inventing a book, a biography, a historical event. But when the invented detail is a legal citation and the document is a court filing, the same failure lands people in front of a judge.

Two documented cases show what that actually looks like.

Case One: New York, 2023

In 2023, a judge fined two New York lawyers, Steven Schwartz and Peter LoDuca, $5,000 after they filed a court brief containing quotes from cases that did not exist. The research behind those citations had been produced by ChatGPT.

What made the filing convincing was its ordinariness. The cases had plausible names, correct citation formatting, and detailed holdings. Nothing about the brief announced that the underlying authority was invented. Opposing counsel could not locate the cases, and neither could the court.

The detail that makes this case worth remembering: when the lawyers questioned whether the citations were real, the chatbot confirmed they were and pointed to sources supporting them. The confirmation was as fabricated as the citations. A model asked to verify its own output has no way to know the difference between a fact it was trained on and a pattern it generated.

Case Two: Alabama, 2025

In 2025, three private attorneys representing Alabama's prison system were sanctioned and removed from a federal case after a judge found that they had submitted court filings containing fake case law generated by ChatGPT.

Removal from a case is a heavier consequence than a fine. It means the work is taken away, the client has to find new counsel, and the record shows why. In both cases, the failure was not a typo or a bad argument. It was invented authority presented as established law.

Why This Keeps Happening

A language model is trained to produce the most plausible continuation of a text, not the most accurate one. There is no step where it checks a citation against a court database.

Legal citations are especially vulnerable for three reasons:

  • The format is rigid and well represented in training data, so the model can reproduce its shape perfectly
  • A citation is short, so there is little text for a person to read for errors
  • Readers treat citations as checkable, which makes an unchecked one feel verified

The result is a document that is structurally flawless and substantively empty.

What It Costs

The visible costs are the fine, the sanction, and the removal. The larger ones are slower.

Judges now read AI-assisted filings with suspicion, which adds friction for every lawyer who uses these tools carefully. Clients pay for work that has to be redone. And in both cases, someone signed a document attesting to research they had not confirmed existed.

How to Check a Citation Before You File

The fix is not complicated, and it does not require abandoning the tools.

  1. Pull every citation out of the draft into a separate list.
  2. Open each one in a real database or reporter. If you cannot open it, you cannot cite it.
  3. Confirm the quoted language appears in the actual opinion, not just the citation.
  4. Check that the holding supports the sentence it is attached to. A real case cited for the wrong proposition is its own problem.
  5. Keep the verification trail with the draft so the work can be checked again later.

Any citation that survives those steps is safe to file. Any citation that skips them is a liability with a docket number.

Where Detection Fits

Detection cannot tell you whether a case exists. It can tell you whether a passage looks machine-generated, which is a reasonable hint about where unverified specifics are likely to be hiding. Run a check, then verify the flagged sections against primary sources by hand.

The pattern across both cases is the same: a fluent, well-formatted document, a citation nobody opened, and a signature at the bottom. Grammar was never the problem. Verification was.