Who Approves Your Model – Why Simulations are Key

Updating a risk-assessment model to match shifts in reality is fundamentally essential. If the model fails to adapt, or in other words, doesn’t ‘breathe’ with current circumstances, the outcomes can be detrimental. The performance of a risk-assessment model is directly influenced by how well it aligns with evolving data inputs. A model built during one period may not be relevant for another. Take the COVID-19 period as an example. Let’s say a risk-assessment model was developed to estimate property damage for vehicles during that time. Is it relevant today? Probably not. Another example would be, as the world is moving toward electric vehicles, which are currently harder to steal than non-electric vehicles, a model built primarily on data from non-electric vehicles becomes increasingly irrelevant in countries where electric vehicles are becoming more common on the roads.

That being said, it’s important to note that identifying a new period requiring a model update is often done in comparison with system performance and interpretation of the results. Tools like the Seenity Platform offer solutions for monitoring these outcomes relative to preset expectations. We’ve mentioned more than once the complex process involved in implementing a risk-assessment model. Yet, today, model validation is not a trivial matter. Multiple positions within a company bear responsibility for how the model calculates risk. It’s not solely the domain of actuaries; they are part of it, but not the whole picture.

When it comes to actuarial science, running simulations against a test group is indispensable. You simply cannot approve a model without comparing its outcomes to a test sample. It’s crucial to ensure that the results are either better or at least equivalent to the model currently evaluating risk in a production environment. Moreover, the outcomes from the test group using the new model should be scrutinized using well-established metrics like the Lift and Gini Index (an interesting article on this topic could be referenced here). But it doesn’t stop there: you need a tool that can simulate, in real-time, what happens with each decision outcome generated by the model, and this should be conducted against the test group for accurate assessment.

On the legal and regulatory front, it’s vital to know which data sets are being utilized by the model. Avoiding bias is crucial for a model’s legality, making this issue, which pertains to a company’s legal position, an essential part of the approval process. Any data feeding into the model must have legal clearance.

Additionally, the financial side of an organization needs to understand and be aware of the monetary implications of running and executing predictions. In other words, deploying a new model that updates according to changing realities requires an established ML (Machine Learning) workflow.

Creating a new model to adapt to ever-changing realities is an important necessity, especially for companies based on risk assessment. Seenity’s platform offers a Workflow (WF) capability for model approval, allowing stakeholders to view data relevant to the approval process according to their role or position within an insurance or credit company.