After Goldman Sachs Flip-Flop on Generative AI in Legal: Who is the Customer?

Edward Bukstel

The experts at Goldman Sachs told us in early 2023 that Generative AI would create a significant increase in productivity across multiple industries and markets. The Macro effects on GDP were going to be massive, including a 7% increase in Global GDP.

The boost to global labor productivity could also be economically significant, and we estimate that AI could eventually increase annual global GDP by 7%. Although the impact of AI will ultimately depend on its capability and adoption timeline, this estimate highlights the enormous economic potential of generative AI if it delivers on its promise.

Legal was one of the top industries to be impacted by the GenAI onslaught. The reporting was everywhere. Law.com featured the March 26, 2023 Goldman Sachs report as a cover story on March 29, 2023.

Artificial intelligence has long been seen as a tech supplement — and not a wholesale replacement — for legal professionals. But with advancements in generative AI, including large language models such as GPT-4 and the rise of popular AI-powered applications such as ChatGPT, that perception is beginning to change.

It’s been over a year since the experts at Goldman Sachs made these statements, and now we are beginning to see a little pushback by the experts on the numbers.

I saw the reference to the Bain & Company report from Peter Duffy’s excellent Legal Tech Trend newsletter. The June 20, 2024 Bain & Company report presents a startling amount of disillusionment among the legal community regarding Generative AI.

Across industries, conversations about generative AI are more earnest, moving from excitement and hype to more realistic assessments. Concerns about security and conversations around implementation are more deliberate and informed as companies have a better understanding of the challenges based on learnings from their pilot programs. Concerns about organizational readiness grew while those around quality and risk have declined

The wretched opinion of Generative AI in the legal community may be directly attributed to the major amount of hype around the potential of GenAI that was generated by the Goldman Sachs findings in March 2023.

Goldman Sachs has definitely changed its tune in June 2024. The new expert report by Goldman Sachs was released on June 25, 2024, and it is a doozy.

The wildest part of the Goldman Sachs June 25, 2024 report is the fact that the legal sector is NEVER even mentioned in any graphic or content. It also appears that the writers of the new report are completely different from the folks that did the March 2023 analysis. You literally can’t make this sh@t up.

While not necessarily taking on their massive mistake in their March 2023 analysis of the effect of Generative AI in legal, Goldman Sachs did address their total whiff on the GDP and macro economic analysis of Generative AI effects on significantly larger trends.

Goldman Sachs highlighted the work of MIT Professor Daron Acemoglu who wrote a powerful paper citing many of the problems in the Goldman Sachs early estimates.

The work of Daron Acemoglu is extraordinary and pre-dates the recent negative analysis of Generative AI in legal by a couple of months. The research he published is also pretty dense with a lot of equations that are definitely above my pay grade.

This paper evaluates claims about large macroeconomic implications of new advances in AI. It starts from a task-based model of AI’s effects, working through automation and task complementarities. So long as AI’s microeconomic effects are driven by cost savings/productivity improvements at the task level, its macroeconomic consequences will be given by a version of Hulten’s theorem: GDP and aggregate productivity gains can be estimated by what fraction of tasks are impacted and average task-level cost savings. Using existing estimates on exposure to AI and productivity improvements at the task level, these macroeconomic effects appear nontrivial but modest — no more than a 0.66% increase in total factor productivity (TFP) over 10 years.

What does all of this mean for Legal Technology?

Seeing the Thomson Reuters Quarterly report for the Legal Sector was somewhat jolting and could be the canary in the coal mine for Generative AI in Legal. Their Q1 2024 Financial results showed an embarrassing 1% increase in revenues after they spent $650m on Casetext. Maybe they still need more time to integrate the product suite and hopefully the numbers will be significantly better in the coming months as we close out 2024.

Thomson Reuters CEO, Steven Hasker, is optimistic about the companies generative AI roadmap.

“We have delivered an encouraging start to 2024, underscored by a strong financial performance and raised outlook, building on the momentum of the past year,” said Steve Hasker, President and CEO of Thomson Reuters. “As we chart our course through 2024, we remain committed to investing in content-driven technology that helps professionals make complex decisions with confidence. With an exciting AI product roadmap and strategic acquisitions shaping our core operations, we are confident we will continue to lead the way in transforming professional work.”

Buried in a recent Thomson Reuters Survey at Figure 22, the company admits that respondents aren’t being asked to utilize Generative AI in legal practice.

Still, that does not mean that clients do not want their firms to be using GenAI — on the contrary, in fact. More than half of corporate respondents surveyed — including more than half of respondents from corporate legal and tax departments — indicated that their outside firms should be applying GenAI to their work. And less than one-quarter of corporate respondents in each segment surveyed said their firms should not be applying GenAI to their work.

Goldman Sachs didn’t ‘swing and miss’ on everything in there report. There are some areas that are more client facing that they clearly got right. Content and customer service jobs are getting slashed left and right, and there is clearly an appetite for bots that utilize Generative AI.

Klarna and companies that are in customer service have replaced significant numbers of employees with Generative AI. The companies Q1 2024 financials demonstrate that a whopping $10m was saved due to implementation of GenAI.

Regulations and the importance of getting cases factually correct in front of Judges, put a higher bar in the implementation of Generative AI in Legal. Marketing gets a bit of a pass if someone makes a mistake (unless its to the FDA) things can be quickly remedied.

It seems that there is a new Contract Management system using Gen AI announced every day. Who is buying all of this stuff and why are customer adoption rates in legal so abysmal?

One group of experts that seem to have all of the answers for implementation of Generative AI is the consultant class. Yep, it looks like businesses are too stupid to deploy Generative AI on their own, so Mckinsey, KPMG, and Boston Consulting Group is coming to the rescue.

A June 26, 2024 New York Times article talks about the expert boom for Generative AI “sherpas” to help guide companies on the use of the new technology.

The next big boom in tech is a long-awaited gift for wonky consultants. From Boston Consulting Group and McKinsey & Company to IBM and Accenture, sales are growing and hiring is on the rise because companies are in desperate need of technology Sherpas who can help them figure out what generative A.I. means and how it can help their businesses.

It’s obvious that I have not been a fan of Harvey AI and their strategy and rollout to the legal community, but the company may have been on the right track in partnering with PwC early on in March 2023.

Storm Clouds?

One of the most alarming findings in the Bain survey for any Legal Technology Vendor is the finding regarding Law Firms wanting to do Generative AI on their own.

Companies are buying third-party solutions when available but investing in tailoring them for their needs. As we might expect with a new wave of technology, many applications are built in-house simply because off-the-shelf versions aren’t ready yet or don’t meet expectations

Bain recognized that the quality of some of the technology vendor offerings has begun to increase, so maybe we are seeing an uptick in buying vendor products, but that 58.8% number has to change if the legal technology market is going to grow. This trend may point to a mix and match approach by law firms.

The numbers of legal technology choices continues to grow with Generative AI features. It boggles the mind how all of these systems are going to be implemented into the legal tech ecosystem. I’m reminded about the 350k Health and Wellness application ecosystem that can be found on the App Store and Google Play. Out of all of these applications, only the top 100 make any real money.

We are still in the first inning of the Generative AI in Legal game and a lot of attention has been put on Thomson Reuters and LexisNexis for their product offerings. There are many other GenAi’s that we haven’t yet heard from , regarding their deployment and usage rates. Litify and Evenup has Litty, Case Status has Casey, and there a plethora of others to choose from.

Even if some of the deployment numbers indicate storm clouds, it’s worth remembering that a lot of good things grow after a good rain, maybe some of that growth will be in legal.

Edward Bukstel

CEO

Giupedi