AI companies must be held responsible for the hallucinations that bog down courts

Tom Macintosh Zheng is the co-founder of Courtready.ca, which builds tools for Canadians who represent themselves in court, and which conducts research on access to justice issues.

By now, most Canadians are already familiar with the concept of AI “hallucinations.” These are the factually false responses produced by a generative artificial intelligence chatbot in response to prompts entered by the user.

The federal discussion paper on AI transparency, whose consultation closed this week, provides an example of this in our legal system. In recent years, Canadian courts and tribunals have seen an increase in the number of instances where parties who appeared before them had relied on law that does not exist. In most cases, this fictitious law was created by generative AI.

Our courts and tribunals have been sanctioning parties who have done this primarily through warnings and monetary penalties. But there is a missing defendant: the creators of those tools.

Economists have a name for a cost that a business imposes on third parties uninvolved with that business’s activity: Externality. For example, a factory that pollutes a lake and leaves the cleanup costs to the communities relying on that lake for water has created a negative externality for those communities.

Chatbots that invent the law are doing something similar to our justice system. So, as Ottawa decides what to do with its consultation, it should consider making the makers of those chatbots carry the cost.

Opinion: Is it reasonable to be afraid about Big Tech’s ability to regulate AI? Yes

General-purpose chatbots may not look up the law when asked for it. Instead, they predict what the law should look and sound like, based on their training data. The result is legal advice that is fabricated but believable, and most Canadians without a law degree have no way to detect that this advice is, in fact, wrong. The only warning is a generic line at the bottom of the website saying that AI can make mistakes.

We have been tracking this issue for some time. The federal discussion paper cited our count of 132 decisions as of March in which non-existent laws were cited. As of Oct. 6, 2026, Canadian courts and tribunals have flagged at least 241 instances where a party relied on fictitious law. In 80 per cent of them, the adjudicator explicitly identified generative AI as responsible. There are now more such decisions in 2026 than during the entirety of 2024 and 2025 combined, and these decisions are only the ones the adjudicators caught and reported. The real number is likely to be higher.

Every decision carries a cost to our justice system that no one is currently paying. Judges, tribunal members, clerks and counsel are now spending more time checking the legal precedents submitted by parties to determine whether it is real or fabricated. There was a time in the legal system where, as lawyers, we may have argued over the law’s interpretation, but never on whether the law exists. That is no longer the case.

In addition to the fake law issue, at a webinar earlier in September, Justice Maya Hamou of New Brunswick’s Court of King’s Bench described an “explosion of paperwork” as AI-generated submissions are filed. Justice Peter Browne of Newfoundland and Labrador’s Supreme Court warned that disputes over AI-generated evidence may lead to longer trial time and delays.

A product that doesn’t work, or is otherwise defective, will lose customers over time. But that principle does not apply here. The third parties harmed by generative AI’s defectiveness (e.g., courts, adjudicators and taxpayers who fund our justice system) did not participate in creating this harm. The users of generative AI, in most cases, cannot see the defect. In 82 per cent of the decisions in our database in which fabricated law was cited, the parties are self-represented litigants. These individuals use generative AI in good faith, often because they cannot afford a lawyer. They will continue to turn to it because it is the only legal help they can afford.

Opinion: Writing should look natural. At work, stop using AI for every step of the process

Of course, you are ultimately responsible for what you put before the court. Blaming generative AI has not been, and should not be, an effective strategy when the court catches someone filing fake law. However, how is someone with no legal training supposed to verify the law? They cannot tell that the output produced by generative AI is wrong, especially when the chatbot delivers it with total confidence.

Ottawa’s proposed Bill C-34, the Safe Social Media Act, is a first step in correcting this wrong. It intends to bar generative AI chatbots from deceptively posing as lawyers or other licensed professionals. That is not enough. When a chatbot produces legal precedents or provides legal advice, its developers should be required to warn its users, plainly and at that moment, that the law it cites may not exist. This modest demand is the least the developers can do to absorb the costs of their defects.