We tested 5 AI chatbots on brief citations and quotes. Two confirmed a wrong pin cite.

Short answer: we gave five AI chatbots - ChatGPT, Gemini, Claude, Copilot and Perplexity - the same 10 case citations and 6 quotations, with planted errors. All five spotted the invented cases and the altered quotations. But four of the five made at least one mistake a lawyer could act on: ChatGPT and Copilot both confirmed a pin cite that falls outside the opinion, ChatGPT and Claude flagged a real, correctly cited case as a problem (Claude advised removing it from the brief), and Gemini gave the wrong page for two quotations. Gemini and Perplexity scored 10 of 10 on citations. LegiMatch Brief Check, which compares each citation with the stored court opinion, got every item right on both tests. And there is a second risk the scores don't show: to check a real brief in a chatbot, you have to paste your client's brief into it.

Disclosure: we make LegiMatch. This is our own test; the prompts are below so you can repeat it.

How we tested

Test 1 - citations. Ten federal and state case citations: five correct (each checked against the published opinion) and five with one planted error each - two invented cases, a real citation with the wrong case name, a pin cite outside the opinion, and a wrong year.

Test 2 - quotations. Six sentences attributed to real opinions: three quoted word for word, and three with a few words changed so the meaning flips (for example "entering and exiting a door" changed to "mopping the floor").

Each chatbot got each test in a new chat, with the same prompt and nothing else, on October 17, 2026: ChatGPT (free plan), Gemini (default model), Claude (Opus 5.5, paid plan), Microsoft Copilot (the app built into Windows) and Perplexity (free plan), each with its default settings. We ran the same items through LegiMatch Brief Check.

Test 1: citations

Test 2: quotations

What the results show

1. Chatbots with web search are good at well-known published cases. Every chatbot caught both invented cases and all three altered quotations. If a case and its text are freely online, a web-searching chatbot can often find them. Credit where it's due: Gemini and Perplexity got every citation right.

2. The dangerous error is a confident "yes". On citation 9, the case is real but the pin cite isn't. ChatGPT answered that page 560 "is within the reported opinion's range"; Copilot wrote that "the opinion extends beyond page 560, so the pinpoint is valid." The opinion ends at page 552. A real-looking citation, confidently confirmed, is the pattern behind the AI citation sanctions.

3. False alarms cost good authority. Citation 5 is a real case, correctly cited. ChatGPT couldn't verify it; Claude called it "likely nonexistent" and advised pulling it from the brief, apparently because the free website it searched didn't list it. A checker is only as good as its source.

4. Every chatbot was different. Four of the five made a different mistake, and chatbot answers can change from one chat to the next. You can't know in advance which answer you'll get.

The bigger risk: pasting a client's brief into a chatbot

Our test used published cases. A real check means pasting a real brief - your client's facts, names, strategy and arguments - into a chatbot. That is a confidentiality question before it is an accuracy question.

• Your duty doesn't change with the tool. ABA Model Rule 1.6 requires reasonable efforts to prevent unauthorized disclosure of client information, and ABA Formal Opinion 512 (2024) tells lawyers to understand how a generative AI tool uses what they enter - including whether it can be used to train the model or seen by others - before inputting client information.
• Consumer chatbots are built for consumers. Depending on the provider, your plan and your settings, conversations may be kept, reviewed or used to improve the service. Many lawyers never check those settings, and a free account is the most common way a brief ends up in one.
• You'd have to do it for every citation in every brief. A 40-citation brief means pasting the whole document, every time.

How LegiMatch handles a brief:

See the Security page and our client confidentiality checklist for AI tools.

Where LegiMatch differs

LegiMatch compares each citation with the stored court opinion, so it can answer exactly - and say what the right answer is ("78 N.Y.2d 1 is Reed v. State", "the decision is dated 2002", "pin cite 560 is outside pages 541-552"). It also:

• gives the same answer every time;
• checks every citation in a whole brief at once, with good-law flags and a table of authorities;
• shows the page where each quotation appears;
• keeps client briefs out of consumer chatbots: encrypted, never used for AI training, and the full brief text isn't kept after the check.

One honest limit: LegiMatch reports a citation it doesn't hold as "not found - verify it", not "fake", because every database has gaps (unpublished and very recent decisions, some older cases).

The limits of this test

• It's a small test: 10 citations and 6 quotations. It shows the kinds of errors to expect, not error rates.
• The items are published cases that are easy to find online. Real briefs - 30 to 80 citations, unpublished decisions, good-law questions - are harder.
• Chatbots change often, and the same chatbot can answer differently in another chat.
• No tool replaces reading the cases you rely on.

Part 2: a whole brief

In Part 2 we gave the same five chatbots a 22-citation brief with 8 planted errors. Results spread much wider - from 3 to 8 caught - and some chatbots skipped citations or called real cases fake.

Try it yourself

Paste these into any chatbot, then run the same list through LegiMatch Brief Check - unlimited on every plan and in the 14-day free trial - or the free citation checker, no sign-up.

Citation test prompt:

I'm a lawyer finalizing a brief. Please check each of these case citations for accuracy. For each one, tell me: (1) does a case exist at this citation, (2) is the case name correct for that citation, (3) is the pinpoint page inside the opinion's page range, and (4) are the court and year correct. Answer each with "Correct" or "Problem", and explain any problem. [the 10 citations in the table above]

Quotation test prompt:

I'm a lawyer finalizing a brief. For each quotation below, tell me whether it appears word-for-word in the cited opinion. Answer "Exact", "Altered" or "Can't verify" for each, and if altered, say what the opinion actually says. [the 6 quotations]

Frequently asked questions

Can ChatGPT check legal citations?

Partly. In our test it caught invented cases and altered quotations, but it confirmed a pin cite outside the opinion and flagged a correct citation as a problem. Use chatbots for drafting, and verify every citation against the actual opinion.

Which AI chatbot is best at checking citations?

In this test, Gemini and Perplexity scored 10 of 10 on citations and all five chatbots scored 6 of 6 on quotations - but four of the five made at least one consequential mistake, each a different one, and results vary between chats. A checker that compares each citation with the stored opinion gives consistent answers.

Is it safe to paste a client's brief into ChatGPT or another chatbot?

It depends on the provider, your plan and your settings, and on your duty of confidentiality under ABA Model Rule 1.6. ABA Formal Opinion 512 asks lawyers to understand how an AI tool uses the information they enter before inputting client information. LegiMatch never uses briefs for AI training, encrypts them, and doesn't keep the full brief text after the check.

What is the most dangerous citation error?

A real case with a wrong detail - a pin cite outside the opinion, a misquotation or the wrong case name - because it looks legitimate. Two of the five chatbots confirmed exactly this kind of error.

Did LegiMatch find the invented cases?

Yes. It flagged both as not found in its library and advised verifying them in an official reporter. It reports them as "not found" rather than "fake" because a real case could be missing from any database.