What is ‘legal checking’ and how can AI accelerate it?

By Celine El Moghraby and Rita Saliba, Youth 4 Governance interns

25 July 2024

Facts and accurate information are the lifeblood of a healthy democracy. With electoral integrity increasingly challenged by disinformation, rigorous fact-checking has become ever more important. 

Politicians and public figures often appeal to the law in statements to bolster their credibility, but their statements do not always hold up to legal scrutiny. They may promote interpretations of the law that are incorrect, misleading or that are considered fringe.  

Responsible journalism of course involves checking the accuracy of such claims, rather than taking them at face value. This can be a difficult process. Not only can it be time consuming to locate the relevant legislation and legal texts, but the law is often open to interpretation.  

French media outlet Les Surligneurs has developed a methodology to ensure the highest standards in ‘legal checking.’ They recently came to Beirut as part of an OIF funded collaboration to share their legal checking methodology with 12 talented jurists and students on the Arab Master in Democracy and Human Rights at Saint Joseph University of Beirut, and to explore how it can be enhanced through AI. So, what does it involve? Below are some takeaways from the workshop. 

Identifying a claim 

The legal checking process obviously starts by identifying a claim to be checked. But given the volume of questionable information in the public realm, legal checkers should be selective.  

For Les Surligneurs, as signatories of the European Fact-Checking Standards Network (EFCSN) , this means focusing one’s efforts on claims that are in the public interest. This is defined by the EFCSN’s Code of Standards as “issues that concern the welfare of society or individuals.”  

Claims made by public figures and people with large platforms tend to have greatest impact, so these should usually be prioritised over claims made by people who are relatively unknown. Alternatively, fact-checkers can prioritise potentially harmful claims that are going viral.    

Locating legal references 

Next, the fact-checker begins their legal analysis. The EFCSN’s Code of Standards states that fact-checkers should rely on primary sources where possible, so this means locating the relevant legal texts that can support or contradict the selected claim.  

In general, relevant primary sources can be found online, but it’s preferable to avoid using sources that are locked behind paywalls.  

The fact-checker can also turn to lawyers or legal experts specialised in the subject matter for further analysis. Here, however, they must exercise caution since lawyers, as subject matter experts, may face conflicts of interest that could bias them towards promoting specific interpretations rather than adhering to the established legal consensus.

For this reason, Les Surligneurs has built a network of ‘vetted’ legal experts who are considered impartial on various topics. Such networks should be continually maintained to ensure that individuals whose interests change in a way that compromises their impartiality are no longer called upon for independent advice.  

Writing the article  

“It is not the mark of a learned man to express himself in language which others cannot understand; it is the sign of a fool who cannot think clearly” - Arthur Hoole

When drafting articles, the goal is to dissect the legality and legitimacy of each assertion.

A top tip is to use accessible and comprehensible terms, and when using legal terms, fact-checkers should define them thoroughly.  

Generally, a fact-check won’t be more than 2000 characters as people’s attention spans are limited. If possible, it also helps to use a title and subtitles that are funny, catchy, or not too serious.  

Augmenting legal checking through AI  

The process outlined above is primarily manual, making it time consuming and prone to human error. Siren and Les Surligneurs have been developing an AI tool that accelerates this process for claims involving International Humanitarian Law (IHL).  

This involves building a database covering every facet of IHL, whether they are Conventions, Protocols, International Case Law, UN Resolutions and Declarations and so on.

A large language model (LLM) is built on top of this database, accessing the information in a comprehensible and efficient way. When a user inputs a statement or a claim to be checked, the tool extracts the relevant terms and documents from the database, evaluates whether the assertion aligns with IHL principles, and generates a summarised analysis on why a claim may or may not be correct, referencing the relevant legislation and explaining the thought process used to achieve the given conclusion.  

During the workshop, the students and the jurists had the opportunity to try the tool, testing various claims. The tool revealed crucial references and sources that had escaped the participants during their initial searches, demonstrating its potential to streamline and enhance the accuracy of legal analyses. 

Importantly, the tool’s output are being evaluated by a team of legal experts from Les Surligneurs, helping to ensure the accuracy of generated legal analyses. Through this process, the tool not only verifies the legal validity of statements but also provides nuanced insights where necessary, contributing to clearer and more informed discussions on global issues.  

For more information on legal checking and AI’s role in it, check out our podcast on the subject with Vincent Couronne , co-founder of Les Surligneurs.  

You can also learn more about the use of LLMs in the legal sector by listening to our podcast with Siren's head of AI, Amer Mouawad.

To discuss how Siren can help accelerate your access to accurate and detailed legal information through cutting edge technologies, please contact info@sirenanalytics.com