We gave 5 AI chatbots a 22-citation brief with 8 planted errors. They caught between 3 and 8.

Short answer: we uploaded the same 3-page brief - 22 case citations, 5 quotations and 8 planted errors - to five AI chatbots. Claude (Opus 5.5, paid) caught all 8 and added useful notes on whether cases support the argument. The others caught between 3 and 6: ChatGPT 5, Gemini 5, Copilot 4 and Perplexity 3. Copilot checked only 17 of the 22 citations, Perplexity left 16 unconfirmed, and Gemini called two real cases "fabricated". LegiMatch Brief Check caught all 8 with no false alarms in about 6 seconds.

Disclosure: we make LegiMatch. This is our own test. The brief, the prompt and the answer key are below so you can repeat it. Part 1 tested a short list of citations.

The test

The brief: a realistic opposition to a summary-judgment motion in a Texas grocery-store slip-and-fall case, with 22 real-looking case citations, 5 quotations attributed to opinions, a short-form cite and two statute sections. Download the test brief (Word).

The 8 planted errors:

The other 14 citations and 3 quotations are correct.

How we ran it: each chatbot got the Word file in a new chat with the same prompt, on October 17, 2026: ChatGPT (free plan), Gemini (default model), Claude (Opus 5.5, paid plan), Microsoft Copilot (Windows app) and Perplexity (free plan).

I'm a lawyer about to file the attached brief. Please check every case citation and every quotation in it. For each citation, tell me whether the case exists at that citation and whether the case name, pinpoint page, court and year are correct. For each quotation, tell me whether it appears word-for-word in the cited opinion. List every problem you find, and tell me how many citations you checked in total.

Results: the 8 planted errors

✓ caught with the right diagnosis · ~ flagged vaguely or with the wrong diagnosis · ✗ missed or not checked

Results: false alarms and skipped citations

What the results show

1. A whole brief is much harder than a list. In Part 1, with 10 citations, every chatbot scored 8 to 10 of 10. With 22 citations inside a real brief, four of the five caught 5 or fewer of the 8 errors. Some simply stopped checking: Copilot reviewed 17 citations and called them verified; Perplexity confirmed 5 and listed 16 as "not independently confirmed".

2. "Fabricated" is a serious accusation. Gemini called two real cases fabricated - one with only a wrong case name in the brief, one entirely correct. A lawyer acting on that would delete good authority or accuse opposing counsel of inventing a case.

3. Claude did excellent work. Claude (Opus 5.5, paid) caught all eight errors, gave the right fix for each, and went further than any checker in this test, including ours: it noted that Duncan and Bagley cut against the argument they were cited for, and that Tex. Civ. Prac. & Rem. Code § 33.001 does bar recovery above 50% responsibility, contrary to the brief. It ran several research passes and took several minutes.

4. Speed and consistency. LegiMatch checked the brief in about 6 seconds and gives the same answer every time. Chatbot answers change between chats and versions, and a lawyer can't know in advance which answer they'll get.

The bigger risk: uploading a client's brief

This test used a made-up brief. A real check means uploading a real one - your client's facts, names and strategy - to a consumer chatbot. Under ABA Model Rule 1.6 and ABA Formal Opinion 512, lawyers must understand how an AI tool uses what they enter before inputting client information. Depending on the provider, plan and settings, chatbot conversations may be kept, reviewed or used to improve the service.

LegiMatch never uses briefs for AI training (ours or our AI provider's), encrypts them, stores them on US servers, and doesn't keep the full brief text after the check - only an encrypted excerpt of its facts and argument, used for "Cases you may have missed". See the Security page.

What LegiMatch doesn't do

To be fair about our own tool: LegiMatch checks that each citation and quotation matches the opinion, flags later negative treatment, and builds a table of authorities. For statutes, an AI check flags a section cited for something it isn't about. For cases, it does not judge whether a case actually supports the proposition you cite it for - the kind of reading Claude did for Duncan and Bagley. That remains the lawyer's job (or a slower, deeper AI review). And LegiMatch reports a citation it doesn't hold as "not found - verify it", not "fake".

Limits of this test

• One brief, 8 planted errors. It shows the kinds of errors to expect, not error rates.
• Chatbots change often; results vary between chats, plans and versions.
• Paid and free plans differ: Claude was a paid plan; ChatGPT and Perplexity were free.

Evidence

The chatbots' own answers, as shared by each service: ChatGPT · Gemini · Claude · Perplexity · Copilot (its share link requires a Microsoft work sign-in; transcript available on request).

Frequently asked questions

Can ChatGPT check all the citations in a brief?

Not reliably in our test. Given a 22-citation brief with 8 planted errors, the free version of ChatGPT correctly identified 5, left two wrong citations as merely "unverified", and said a correct quotation was not in the opinion.

Which AI chatbot was best at checking a brief?

In this test, Claude (Opus 5.5, paid) caught all 8 planted errors with no false alarms and also flagged authority that cut against the argument. The others caught between 3 and 5.

Do AI chatbots skip citations in long documents?

In this test, yes: Copilot reported checking 17 of 22 citations, and Perplexity confirmed 5 and left 16 "not independently confirmed". Always count the citations a tool says it checked against the brief.

How long does LegiMatch take to check a brief?

About 6 seconds for this 22-citation brief, with a report of every citation, every quotation, later negative treatment and a table of authorities. Brief Check is unlimited on every plan and in the 14-day free trial.