AI hallucinates. The way it is trained and evaluated tends to reward a confident guess over an admission of uncertainty, because a fabricated answer has some chance of being right and "I don't know" has none. The result is an answer that sounds authoritative even where the underlying citation, case or proposition does not exist.
Placed in an Indian courtroom, this is no longer a hypothetical; petitions may already have been dismissed or allowed, and penalties confirmed, on the strength of citations and precedents that never existed, and the damage may already have been done.
In July, the Supreme Court set aside NCLT and NCLAT orders in the Essel Infraprojects insolvency after finding the tribunal had relied on non-existent, hallucinated judgments [Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd. (2026 INSC 668)]. In September, it set aside a ₹425 crore customs penalty, holding that the adjudicating officer's reliance on fake citations was fatal to the order, one the Gujarat High Court had already confirmed [Vijay Ghanshyam Gadiya v. Union of India (2026 INSC 947)].
SCAORA, in its Comments and Recommendations on the Draft Regulations for the Use of Artificial Intelligence (AI) in Courts, 2026, said generative AI produces citations that look genuine but do not exist, and that citation verification must remain a human function.
AI is already becoming part of legal work, and the answer is not to stop using it.
The problem, then, is not whether lawyers will use AI. It is what happens between an AI-generated answer and a document being filed in court. A citation that looks correct is not necessarily a citation that is correct. Verification must become a distinct step in the legal workflow, particularly where AI has been used to generate or assist with research, drafting or case analysis. Responsible use includes verification before filing. Grayscope Verify is built for that step. Upload your petition or draft and get a report that checks each extracted citation and flags citations that could not be verified. Each citation is compared against Grayscope Library, which is free, open to all, designed that way on purpose so anyone can audit it: a judge, a client, or opposing counsel, and keeps growing; today it holds reported Supreme Court judgments from 1950 to date, and High Court and Tribunal judgments are being added as we speak. Verify is not a substitute for legal judgment or independent research. It is a verification layer designed to help catch citation errors before they become part of a legal submission.
The report is a fully transparent audit trail, with custody logs showing exactly how the verification was done: when the file was received, when the citations were pulled out, and when each one was checked. Every citation links to Grayscope Library and to the eCourts record. For every citation it verifies, the report also gives the Court's own neutral citation, the INSC number, along with the SCR reference where one exists, so the judgment can be looked up on eCourts directly even if the draft cited it from SCC or another reporter. Every verified citation also carries its own cryptographic fingerprint, making subsequent alteration of the verified citation detectable. The report itself is fingerprinted too, so it is tamper-proof: change anything and it no longer matches; attach it to your file and it stays checkable. It doesn't matter what you use: ChatGPT, Claude, or another legal tool. Use Grayscope Verify as a sanity check on your own petitions, and on the counterparty's. The Supreme Court has made the message clear: AI can assist legal work, but legal authorities must still be independently verified.
Chirag Jhumkhawala is the co-founder of Grayscope.
Disclaimer: The opinions expressed in this article are those of the author(s). The opinions presented do not necessarily reflect the views of Bar & Bench.