Weekend Reading: The 3LoD — “A Human Interaction System”

By: Lara Warner , Past-CRO/CCO of Credit Suisse

This piece first appeared in Starling Insights' newsletter on October 13, 2024. If you are interested in receiving our thrice-weekly newsletter, among many other benefits, please consider signing up as a Member of Starling Insights.

On Monday last week, I was invited to participate in a class at Harvard Business School (HBS). The class focused on a case study about HSBC , and efforts to trial new AI-powered tools to improve the function of the bank’s Three Lines of Defense (3LoD) risk management model. 

Anyone who has worked in risk/compliance in banking will recognize the central dilemma of the case: despite billions of dollars of investment in governance, risk, and compliance infrastructure, HSBC continued to experience Operational Risk (OpRisk) management failures at an unabated pace.

The Board wanted to know why. The Global Head of the firm’s 2nd Line was thus tasked with explaining what was going on and determining how performance would be made to improve. Should he trial a new and unproven ‘regtech’ tool, produced by a small and thinly-resourced startup, to support his investigations? 

The unnamed regtech company — Starling — had demonstrated an ability to distill signals from within non-sensitive and readily available data sources that identified where HSBC was likely to experience OpRisk management mishap. Should HSBC adopt broader use of the company’s “Predictive Behavioral Analytics” tools, the HBS students were asked. Why, or why not? 

The Case

Following the Great Financial Crisis, HSBC had paid out over USD $5 billion in fines and penalties as a result of employee misconduct, and was the subject of a Deferred Prosecution Agreement (DPA) with the US government following a series of high-profile money laundering scandals. Per the terms of the DPA, the bank was placed under a monitorship and faced intense pressure to demonstrate a more effective OpRisk management framework.

This required a wholesale update of the bank’s 3LoD implementation, including hiring and reallocating thousands of executives, assigning new roles and reporting relationships, and implementing expensive new systems and processes. This is no small matter for a bank of nearly 200,000 employees deployed across some 60 countries.

Despite over $3 billion invested, the firm’s Global Head of Operational Risk at the time, Mark Cooke , still struggled to anticipate where processes were breaking down, he remained unsure whether he was getting accurate risk and control reporting and, perhaps most difficult of all, he was insufficiently able to demonstrate to the bank’s monitor and its regulators that HSBC could be trusted to manage its OpRisks successfully. 

Specifically, the bank’s leadership faced:

• Long periods of uncertainty between self-reported updates on risk conditions;
• Reliance on ‘executive intuition’ to identify management blind spots;
• Risk management failures discovered only during regulatory reviews; and
• Punitive fines in the wake of these lapses.

The Problem

As Mark Cooke terms it, a capable 3LoD is inherently a "human interaction system.” It relies on a complicated web of interactions across business lines, management tiers, geographies, time zones, and languages. As risk professionals will know, these realities make it all the more difficult to know — or to demonstrate — that desired risk management behaviors are in fact taking hold. Starling has written about these challenges here, in an article well worth reading.

The root of the problem is that the standard toolkit for managing risk has focused on policies, processes, controls, and monitoring systems. Implemented well, such tools are effective at indicating what risk managers intend to happen (e.g., ‘Tone from the Top’) and they serve as a good ‘system of record’ to capture data about what actually happens, as eventually seen down the road.

But when it comes to understanding how people are in fact engaging with one another (or not), what expectations they share (or not), how they actually collaborate day-to-day (or not), a focus on these OpRisk management inputs is of little help.

As such, we’re left awaiting the evidence of subsequent outcomes to learn whether our good intentions merely paved the road to Hell. All too often, this proves to be the case. Standard approaches to OpRisk leave us perpetually operating from the back foot.

It is this that makes Starling’s solutions so interesting. Indeed, after trialing Starling’s tools, HSBC’s Mark Cooke described the company’s offerings as providing risk managers with a ‘Holy Grail’ of sorts. But how to persuade the rest of the ExCo of this?

The Solution

Rather than compare policies and processes (‘inputs) to eventual risk management outcomes (‘outputs’), Starling focuses on the bit in between — the operational ‘throughputs.’ 

After mapping the internal networks of people who enliven policies and processes, and testing management presumptions about how work is done against digital evidence of the actual practices that are ongoing, Starling identifies gaps that impact performance outcomes and levers of change that might be engaged when such outcomes prove to be problematic.

By bridging the gap between inputs (policies & processes) and outputs (performance & problems), Starling affords leaders with an end-to-end view of the who, how, and why behind OpRisk as it plays out in real-time. Such capabilities move us off the back-foot.

At HSBC, Starling showed it could distinguish between teams that were providing effective OpRisk oversight and those where problems were brewing. By back-testing against past events, Starling was able to accurately identify groups that would later report a deficiency in their risk and control environments, and those where regulators later discovered deficiencies. Starling was also able to identify specific data signals that differentiated high-performing risk teams from those that were more challenged — signals, in short, that accurately predicted trouble.

The Need

Since this case study was published, banks have faced increasing pressure to demonstrate greater OpRisk management capabilities. Certainly, the “banking sector turmoil” of spring 2023 has added a still greater sense of urgency — an experience I know a fair bit about given my own personal experience at Credit Suisse. But CS is far from alone …

As readers here will have seen reported by Starling Insights, banks worldwide struggle with OpRisk challenges:

• In Canada, TD Bank pleaded guilty to criminal money laundering charges this week, and agreed to pay a staggering $3 billion in fines;
• In the US, Citi has paid over $500m in fines for inadequate systems and is facing calls for even more extreme sanctions for not moving fast enough in its efforts to improve; and
• In Australia, ANZ is reeling after its regulator increased its OpRisk capital add-on to $750 million due to persistent shortcomings in non-financial risk management, governance, and culture. 

It would be painfully easy to add to this list.

While banks continue to invest in a standard OpRisk toolkit that repeatedly offers poor return on investment, advances in AI and analytics offer new tools that will surely become increasingly important features of any successful compliance and risk management framework. 

Some firms have brought AI to bear in more forward-looking risk management applications but, to date, these efforts have largely focused on better management of traditional financial risks. When it comes to non-financial risks, however, we remain wedded to tools that provide “detection,” “monitoring,” and “surveillance.” While such backward-looking tools are necessary, they do not help us to achieve forward-looking insights into people, presumptions, and practices.

The Students

I brought this context to life for the HBS students whom I was so pleased to meet last week. As one might expect, they were deeply engaged in the case study discussion and offered many thoughtful reactions. Many argued for HSBC’s broader adoption of Starling’s tools. And many argued against doing so. Their arguments served as a good ‘microcosm’ reflecting the current state of regtech/suptech adoption.

Students rightly highlighted the many challenges that come when large, complex, globally distributed behemoths hope to work with small companies that are pioneering new technology solutions. However, despite these perhaps unavoidable challenges, it was clear students felt there were many advantages that can accrue to firms willing to try new approaches. 

Ultimately, most were enthusiastic because they recognized the difficulty — or near impossibility — of managing a firm like HSBC without effective OpRisk metrics. They also quickly recognized the goodwill firms might generate with their regulators by demonstrating a readiness to push the envelope on perennial struggles, and how such firms might win competitive advantage.

Below, I’ve captured both sides of the classroom debate, in hopes of prompting reactions from my industry peers and others in the Starling Insights community. What points were missed? Which deserve underscoring? How can we assure that opportunity overcomes obstacles in our efforts to field these new tools? 

My Conclusions

Our current approach to managing existentially important non-financial risks is clearly in need of some improvement. I know from experience how challenging it is to implement an effective risk management framework across a firm with a global footprint, and how best efforts can go awry. Doing more of the same will get us more of the same. 

We have robust, and largely effective, metrics to guide us in managing the whole gamut of financial risks — credit risk, counterparty risk, etc. But when it comes to risks that flow from organizational culture, and the conduct that it permits or promotes, we’re still relying on ‘management intuition’ or blunt instruments better suited to the pre-digital era. 

Experience teaches me that non-financial risks are, inevitably, financial risks. Just as we apply forward-oriented analysis to all categories of financial risk, to address non-financial risk with equal rigor, we need tools that allow us to ‘stress-test’ our people, presumptions, and practices before failures of policy and process are made evident in performance problems.

That’s why I was eager to join Starling. As the HSBC case study illustrates, the Predictive Behavioral Analytics tools that Starling has pioneered put leaders on the front foot, enabling them to become proactive regarding misconduct scandals that have real financial and reputational costs.

The bank failures of 2023 make plain just how important this work is, for the industry and its overseers alike. The HBS students — half of whom had worked in banking before attending Harvard — clearly recognized this. Policymakers and practitioners should demonstrate at least as much awareness.

Lara Warner is a member of Starling’s Industry & Regulatory Advisory Board. Throughout a nearly 20-year career at Credit Suisse, she held several leadership roles, concluding her tenure as Group Chief Risk & Compliance Officer and serving for six years on the firm’s Executive Board.

CASE STUDY CLASS NOTES

Points of critique / reasons not to roll out

Use-Case Concerns

• HSBC was troubled by KYC issues, not culture issues/ mis-match between problem and solution?
• Was the pilot project window-dressing to justify HSBC’s move to an ‘activity based’ 3LoD model’?
• Is regtech used as a stick to enforce the 3LoD model or as a carrot to drive change?

Startup Related Concerns

• Would be tough for a startup to scale to the size of HSBC
• Insufficient training data problem / hard to generalize results when you’re first to trial
• Vendor lock-in risk / HSBC should pre-negotiate put-option to buy or invest in Starling
• Not an off-the-shelf offering / hard to roll-out at scale / senior leadership buy-in challenge

Privacy & Incentive Related Concerns

• ‘Big Brother’ concerns among employees 
• Analytics outputs may affect employee interest / how do we know they’re right?
• ‘Guinea-Pig’ problem / fear that particular employees may be personally singled out
• Concern that tech serves to replace human interaction / what role for human judgment?
• Do Starling’s tools work in concert/conflict with existing incentive structures? 
• Liability risk / tool would produce potentially incriminating data discoverable by lawyers
• Banking is heavily regulated + regtech is unproven = use of regtech adds risk & greater scrutiny

Technology Related Concerns

• The ‘black-box’ & ‘explainability’ problem inherent in AI
• Not clear how use of the tech results in new workflows
• Risk of employees gaming the system
• Cross-cultural challenges / are ‘predictive’ signals consistent across national cultures?
• As presented in the case, tools might not capture all relevant data / accurate enough?
• As presented in the case, tools offered limited, high-level predictive capability / specific enough?
• As presented in the case, tools don’t promise less issues, only predict them / effective enough?

Points in favor / reasons to roll-out broadly 

Current Approaches are Failing

• Without new tools, we’re left hoping more-of-the-same will produce different outcomes
• We can’t afford to throw people at these problems / we need more efficient, scalable tools
• The only way to improve is to experiment with new tools / trials must be run
• For this, we need to overcome a chicken-egg problem / new tools can’t be proven if not tried
• Reliance on ‘tone from the top’ is unwise in context of operations across 70 countries 

We Need New Metrics

• If incentives are tied to risk mitigation, how do you demonstrate that success?
• These new tools = ability to evidence success ex ante (or to proactively identify trouble)
• The 1st line struggles with conflicting incentives + reliance on efforts among 2nd & 3rd Lines 
• And yet we have no good measures of efficacy on any line, let alone across them all
• If we can achieve greater visibility through metrics, we also achieve greater accountability

Working with a Startup = Opportunity 

• Because Starling’s tools are not ‘off the shelf’ adoption shows effort to solve specific problems
• Because trial requires testing management hypotheses, the tool leads to continuous questioning
• Applied well, new tools help to make risk and compliance a competitive differentiator
• Adoption of successful new tools prompts regulatory ‘homogenization’ / new ‘best practices’
• Firms that help establish new ‘best practices’ enjoy reputational boost 

Regulatory Goodwill

• Allowing Starling to do more = opportunity to demonstrate that effort is not ‘window-dressing’
• Use of a new tool = a potential “reputational hedge” against the headwinds of past sins
• Trialing new approaches shows effort to move past the fines = ‘cost of doing business’ mindset
• Over time this becomes an insurance policy against massive costs / small fix = huge benefits