Taking Legal Analytics to the Next Level

I originally posted this article at my blog on 13 April 2020.

COVID-19 Prelude

I started writing this blog post about LegalMation just as the COVID crisis was ramping up. By the time I finished it, the world had changed. As I noted in my blog post last week about Fastcase, this crisis will last a while. And what happens after it remains uncertain. Already, many large law firms have cut costs or withheld partner draws to conserve cash. We are now past the end of the beginning of the crisis. Consequently, law firm management should now plan for what comes next. 

Many industries are already struggling and that almost certainly will continue. At least some practices therefore likely will face fee pressure. It may not be happening yet with “the house on fire”. But it almost certainly will come. And when it does, firms need to be ready to offer better value. That will drive the need to understand costs and manage matters more effectively.

LegalMation offers an interesting approach to cost and outcome analytics. As usual, when I blog about a provider, I do so because I find the product or service interesting and new or different. I am not comparing it to other approaches or reviewing it; rather explaining it.

——————-

The legal analytics revolution continues. LegalMation has developed a new approach to litigation data analytics. I learned about this from the Artificial Lawyer post LegalMation Expands Into Cost Analysis + Lawyer Comparison (19 March 2020). Previously, the company earned kudos in 2018 for its AI-driven approach automating draft answers to complaints.

I wanted to learn more about the new offering so I spoke with James Lee, co-founder of LegalMation and report here. 

The Analytic Paradigm of Predicting Outcomes

Models and Analysis Can Improve Outcomes. Clients want to minimize litigation costs: the sum of legal fees, expenses, and the judgment or settlement. Smart law firms want to achieve better results for clients at lower cost. If lawyers understand how case strategy affects outcomes, then they can improve outcomes and/or reduce cost. For example, if investing more time early in a matter leads to better outcomes, then do that. Or if lawyers knew that using more senior lawyer time early in a matter, say during depositions, improved outcomes, then modify the staffing mix.

How to Model + Analyze Case. To analyze the relationship between outcomes and cost and their drivers requires data and analysis. A word about analysis before turning to the data. Analysis can be bivariate: compare outcomes to just one input. Or it can be multivariate, comparing outcomes to multiple drivers. Either way, lawyers need a good theory of why one variable affects the other because correlation doesn’t equal causation. 

The Data Needed for Models and Analysis. A statistically reliable model requires good data and enough of it. While publicly available data provides some insights, 97% currently lies beneath the surface in the document and data repositories at the law firms and corporations. LegalMation’s approach offers a new approach to surface these data, an approach that may uncover insights previously not attainable. To get to those insights, its software: 

1. Characterizes what cases are about and classifies them.
2. Explains the work lawyers did on the case.
3. Measures the case outcome.

In the next section, I discuss how LegalMation gets these. 

Generating the Data and Analyzing It

High Volume Data and Outcomes Data. LegalMation focused on two types of high volume cases: employment (such as age or disability discrimination claims) and personal injury claims (such as slip-and-fall cases). Most organizations with high volumes of litigation use enterprise legal management software (ELM) to track legal costs. Pulling cost data for analysis is easy.

Characterizing Cases. LegalMation’s key insight was that they could use their underlying machine learning software to characterize both cases and the work lawyers do. Its software extracts information (“entities”) from documents. Examples include jurisdiction, parties, injuries alleged, and dollar claims. By extracting enough entities, the company systematically describes what cases are about. LegalMation captures and identifies up to 500 unique entity-relationships in areas of law: employment, personal injury, insurance defense, and financial services litigation.

Characterizing Lawyer Work. The software also uses entity extraction to analyze lawyer time narratives. LegalMation determines both the phase of litigation and the type of work, for example research, writing, deposition preparation, taking a deposition, or appearing in court. (Assigning the phase depends on both machine learning on narratives and external task set definitions.)  

Phase and Work Type Classification. LegalMation’s first product in 2018 analyzed complaints and automatically drafted answers and discovery requests. To do this on par with a good junior associate, the company had to build and train its artificial intelligence to read passages from filings and comprehend their meaning. That, in turn, required developing a taxonomy and classification system to differentiate text at a granular level. The ability to analyze outcomes and work is an extension of these capabilities.

In analyzing matters by phase, existing standards for phase and work type were not helpful. The ABA phase and task codes are not very granular and, in any event, the lawyers who use them do not do so well. Moreover, LegalMation learned from law departments that most just wanted hours by phase; task-level detail proved distracting. The company extended its software to capture events in litigation from beginning to end of cases. Getting this right required deep and iterative work with lawyers in law departments and firms. Beyond automating a tedious task, using software to define phases yields more consistent data than human coding. This allows more reliable comparison of billing data across lawyers and firms.   

Visualize Relationships. To analyze these data, LegalMation provides visualization tools using capabilities from Tableau and IBM Cognos.  

Implications 

Legal analytics blossomed in the 2010s. Entrepreneurs realized that public information contained data to provide insight about litigation, data that incumbent providers had failed to tap. Early and successful legal analytic start-ups included Lex Machina, Casetext, Ravel Law, and Judicata. At the risk of oversimplifying, those companies initially focused on the law, judges, and citations. 

Now, LegalMation is focusing on case facts and what we might think of as case metadata. James showed me several visuals that illustrate intriguing and actionable relationships. For example, plotting the age bracket of a plaintiff against case outcomes in the sample data set shows that the most costly outcomes are for plaintiffs age 50 to 55. That contradicts many labor lawyers’ instinct that older plaintiffs are more costly. It is also actionable: knowing this should affect the type of work lawyers do and their settlement negotiations.

The figures below illustrate this type of analysis for disability harassment claims based on physical (top) vs. mental disability (bottom). The average settlement amounts are quite different. Today, most legal professionals lump all disability harassment claims in one bucket. The data, however, clearly show meaningful differences among sub-types. Knowing this allows lawyers and managers to make better decisions about handling each type. (These charts also illustrate the detailed analytics possible. The panes to left are facets / filters that allow an analyst quickly and easily to select different data slices.)

Another example compares the cost of lawyer time by phase with outcome. In the sample data, cases with more investment in early phases (e.g. fact finding and depositions) achieve better outcomes than investing more later in the case. That should drive more lawyer work to early in the matter. (Spend by phase is an example of what I view as matter metadata.) The figure below illustrates the part of the analysis that looks at the difference in hours between partners and associates by phase. It is for a single firm. One can produce charts like this for multiple firms and compare outcomes.

James also showed me charts that compare outcomes to a set of specific plaintiff law firms. These clearly show some firms cost more to battle than others. The two figures below illustrate summarizing this type of analysis in a single metric – the dial here – to measure risk. The first shows a high risk score of 114.5 for one plaintiff’s attorney and the second a low-average score of 56.81. The first \s an attorney who takes cases to the end, seen in the bar chart to the left, with a high percent of time on trial and appeal. The second one tends to settle in the oral and written discovery phases – at typically lower amounts. James reports that the risk index compiles multiple measurements / statistics that relate to settlement amounts, hours incurred, and strength of counsel He said “the risk score is working really well in forecasting the level of risk.” The blurred orange / blue bars in the lower right of each figure show the sub-scores for each component of the risk index.

A geek note: The real driver of the difference might not be firm but rather the judicial district (especially if plaintiff firms operate mainly in select districts). Where two explanatory variables correlate with one another, reliable modeling may require some more serious statistical chops. 

James showed me multiple interesting visual comparisons of potential drivers and outcomes. In my former work as an econometrician, I’d call this exploratory data analysis, the practice of examining multiple relationships using visual displays. That stands in contrast to testing specific hypotheses. Good modeling requires both and, as important, having a sound hypothesis for any unexpected relationships that exploration yields.

Conclusion

LegalMation’s ability to characterize matters and lawyer work means that legal professionals can now model how decisions about case strategy affect outcomes. In-house counsel, legal operations professionals, and law firm partners who manage portfolios of related litigation should be all over this. Done properly, it will improve case strategy, and for in-house lawyers, selection of outside counsel. We could even develop best practices for legal strategies.

Even with fabulous findings to improve litigation outcomes or reduce costs, we will face a change management challenge. Empirical evidence alone rarely persuades lawyers to change what they do. To change what lawyers do, be prepared to offer carrots and sticks – and for some hard work.