As an Engagement Manager at Proactive Worldwide, I’ve spent considerable time analyzing the intersections of law, technology, and strategic business operations.

Over the years, the traditional way of practicing law—particularly at mid-sized, large, and global law firms—has come under increasing pressure from clients. They scrutinize bills more closely, driving firms to find ways to ‘more efficiently’ execute tasks that junior and mid-level associates traditionally managed. In this quest for efficiency, technology, especially in the realm of document review, has played a pivotal role.

Legal technology companies have responded by expanding their offerings to include workflows, forms, clause libraries, and practice guides, among other resources, to help streamline drafting and research on core tenets of law.

AI promises more: deep legal research and document drafting performed entirely through the Large Language Models (LLMs).  However, early attempts by legal practitioners to use AI to conduct research and drafting did not go well.  Attorneys have presented legal motions containing citations to cases that do not exist, or to citations that actually stand for the opposite legal proposition than what it was cited for.  The collective term for these inaccuracies is “hallucinations.”

Recently, the big players in the legal technology space have pounced on the promise of AI, rolling out their own solutions for AI-driven legal research and drafting. And with those rollouts come big promises. Lexis, in announcing its Lexis + AI solution, touted, “Unlike other vendors, however, Lexis+ AI delivers 100% hallucination-free linked legal citations connected to source documents, grounding those responses in authoritative resources that can be relied upon with confidence.” Casetext asserted, “Unlike even the most advanced LLMs, CoCounsel does not make up facts, or ‘hallucinate,’ because we’ve implemented controls to limit CoCounsel to answering from known, reliable data sources—such as our comprehensive, up-to-date database of case law, statutes, regulations, and codes—or not to answer at all.” The other competitors in this space make similar claims.

Unfortunately, for most practicing attorneys, these claims might run a bit short.  A recent study out of Yale and Stanford tested many of the leading Legal AI tools and found that they hallucinate – a lot: between 35%-55% of the time, an AI answer to a legal query was incorrect or ungrounded in law.

According to the study, these legal tools struggle to: accurately describe the outcome (holding) of a case, distinguish between legal actors (the court, attorneys, parties), and understand the hierarchy of the legal system, including jurisdiction.

The law is not a field where “that’s about right” cuts muster – so the bar is high for legal AI solutions.  With such high miss rates, lawyers are faced with three choices for now:

1. Check every single citation and legal proposition that the AI churns out. This option adds significant labor and cost to a solution that is fundamentally positioned as a cost saving tool.
2. Trust that the AI got it right. This is a high risk move most attorneys would be loath to make, as getting caught misrepresenting the law in court will assuredly give rise to discipline and sanctions at a minimum.
3. Wait out the AI for the time being.

At Proactive Worldwide, we continue to monitor these developments closely. Our goal is to equip our clients with strategic insights that not only inform but also anticipate shifts in legal tech. The promise of legal AI is immense, and the potential is undeniable. Established players—and perhaps some surprising new entrants—should persist in their efforts to develop a truly reliable, hallucination-free legal AI solution. It will undoubtedly be a ‘killer app’. However, as of June 2024, we’re not quite there yet. Until then, we recommend a cautious approach, leveraging AI where it adds value without compromising the integrity of legal work.