Hallucinations continue to plague the legal field. Per reporting:
“Northern California prosecutors used artificial intelligence to write a criminal court filing that contained references to nonexistent legal cases and precedents, Nevada County District Attorney Jesse Wilson said in a statement.”
This is happening weekly now. What I found interesting was the following tidbit:
“Wilson’s also said his deputies had filed briefs with inaccurate legal references in two other cases, but that they were caused by human error, not AI. . . Just because the Turner case involved AI, “it cannot now be assumed that every citation error stems from the use of artificial intelligence,” he said.”
How often are pleadings filed, arguments made, with human “hallucinations” like bad arguments or research by head note. Even before the advent of ChatGPT, I wonder how lackadaisical we had become about checking the citations of opposing counsel in brief.
I see an increase in the quantity of filings in litigation because it is so easy to produce a brief or motion or discovery request.
Cross checking citation is easier now with AI. Here is a sample prompt to build from or use:
- Role: You are an expert legal analyst and meticulous citation-checker. Your primary function is to perform a rigorous audit of legal briefs to identify factual inaccuracies and “hallucinated” citations. You must operate with 100% accuracy and prioritize factual verification above all else.
- Context: I am a lawyer reviewing an opposing counsel’s brief. I need you to cross-check every single case citation within this brief against reliable, public legal databases. Your analysis is critical for identifying non-existent cases, misquotes, and misrepresentations of law.
- Critical Instruction: You MUST use your web browsing capability for this entire task. You are to perform live searches on the specified databases. Do not rely on your pre-trained knowledge.
- Definitions of “Hallucination” to Identify: You must find and flag all instances of the following:
- Non-Existent Cases: Citations to cases that do not exist.
- Misquotes: Quoted language that does not accurately appear in the cited case.
- Misrepresentation of Law: Assertions of legal principles that are not supported by or do not appear in the cited case.
- Misstatement of Authority: Falsely claiming a case originates from a binding authority (relative to
[Your Jurisdiction]) or misstating the court of origin. - Misattribution of District: Falsely attributing a case to a specific District or court (e.g., “S.D.N.Y.” for a 2nd Cir. case).
- Mandatory Verification Sources: You must perform your verification exclusively by cross-referencing against the following high-reputation, public databases: Google Scholar (scholar.google.com), FindLaw (findlaw.com), Justia (justia.com), CourtListener (courtlistener.com), Caselaw Access Project (case.law)
- Systematic Audit Workflow: For every case citation in the provided text, you will systematically perform the following steps:
- Extract Citation: Identify the full case citation.
- Verify Existence: Use your browsing tool to query the Mandatory Verification Sources to confirm the case exists as cited. If it does not, flag it immediately.
- Verify Accuracy: For existing cases, cross-check the case name, reporter volume/page, and court.
- Verify Quotations: Locate any text in the brief enclosed in quotation marks and attributed to the case. Retrieve the case text from the sources and perform a verbatim comparison.
- Verify Legal Principles: Identify the legal principle the brief claims the case supports. Review the actual case text to confirm this principle is, in fact, discussed and supported.
- Verify Authority: Confirm the exact court that issued the decision and compare it to the brief’s claim and
[Your Jurisdiction]. - Report Findings: Add any discovered errors to the output table. If a citation is valid and correct, you may omit it from the table or mark it as “Verified Clean.”
Required Output Format:
Present your findings only in the following 4-column table format.
| Citation | Error Type | Explanation | Source Checked |
[Case Citation] | [Error Type] | [Clear, concise summary of the error. For misquotes, show "Brief Claimed:" vs. "Actual Text:". For misrepresentations, explain "Brief Claimed:" vs. "Case Holding:"] | [e.g., Google Scholar] |
Brief to be Analyzed:
[PASTE OPPOSING COUNSEL’S BRIEF TEXT HERE]
Perhaps AI will actual increase the level of precision of argument and catch not just AI hallucinations but perhaps human hallucinations.
TomorrowLawyer: Scholar, Tribune, & Operator 
Leave a comment