Best Practices for CCAR -PPNR Model Validation: Ensuring Accuracy and Reliability
Nawal Kashyap FRM®
Vice President & Head of Non-Retail Credit, ICAAP Model Validation
In the dynamic world of finance, the US Comprehensive Capital Analysis and Review (CCAR) exercise stands as a monumental milestone. Financial institutions have invested significantly in quantifying and managing risk to meet the ever-evolving compliance standards set by regulators. With each CCAR submission cycle, the complexity of risk quantification techniques employed by Bank Holding Companies (BHCs) has grown. These techniques have given rise to a vast and independent body of work dedicated to the validation of these models.
The validation of stress testing models presents a unique challenge. The Federal Reserve System (the Fed) has identified shortcomings in how many participants validate their models. Ideally, deficiencies not spotted during model development should be caught and corrected through effective validation. This would shield BHCs from the reputational risk of stress test failure and the costly measures needed for remediation.
The Importance of Model Validation
Why is model validation so crucial in the financial sector?
BHCs face several challenges in the validation of CCAR models, including:
Additionally, gauging and responding to evolving regulatory expectations is a key challenge. Instances have been observed where a BHC's progress in addressing Matters Requiring Attention (MRA) from prior submissions was deemed unsatisfactory by regulators. Common pitfalls include the level of validation not matching the Fed's expectations, thresholds for model performance shifting between submissions, and a lack of empirical evidence supporting the use of overlays.
CCAR Model Validation Framework
Effective validation relies on two critical components: providing an independent view of model validation and adhering to a transparent, repeatable, and conceptually sound framework. Developing such a framework necessitates comprehensive coverage of both qualitative and quantitative aspects of model validation. The framework below illustrates this approach:
Applying the CCAR-PPNR Model Validation Framework
In this section, we will apply the CCAR-PPNR Model Validation Framework to describe the quantitative and qualitative model validation approaches for one of the key inputs to capital projections: Pre-Provision Net Revenue (PPNR).
1. Model Development Data
1.1. Data Appropriateness: The first step in the CCAR-PPNR Model Validation Framework involves assessing the appropriateness of the data used during model building. This includes verifying that all relevant drivers have been collected and that the data meet the model's core purpose. Key considerations include:
a) Examine the definitions of each line item and check for consistency with financial reporting/segmentation standards.
b) Evaluate the segmentation of FRY-14 and how modelled line items are mapped to it.
c) Assess the relevance of characteristics to the line item (aggregation methods, candidate variables).
d) Examine the reasoning and testing behind the use or lack of use of internal variables.
e) Assess the logic for the use of dummy variables, and proxy variables, and whether they align with the model's purpose.
1.2. Completeness and Accuracy: The next aspect of data validation involves checking the accuracy of data by comparing it against a benchmark, reviewing the data collection process, analysing outliers, missing values, and means, and assessing data quality discussions from Line of Business (LOB) stakeholders. This step includes:
a) Examining if the development data maps to the source system.
b) Reviewing the steps taken to collect, pull, and segment the development data.
c) Independently preparing the development data sets and comparing them to the development data done.
d) Evaluating data quality by performing standard tests for missing values, outliers, trending, and seasonality.
e) Assessing the rationale behind the treatment of outliers and missing values using techniques like capping or flooring.
1.3. Design of Data Sample: Assessing the design of the data sample ensures that it aligns with the model's intended business purpose and similar industry models. This step includes:
a) Evaluating if the sample size used for model development and testing meets industry standards.
b) Evaluating if the data period used for model development covers a full business and macroeconomic cycle, and if the testing dataset is sufficient for assessing model performance.
c) Examining segmentation criteria and ensuring they support the model's choices and business justifications.
1.4. Assessment of Data Cleaning Practices and Impact of Data Exclusion: Identifying sources of data weaknesses created by data cleaning and exclusion processes is essential. This step includes:
a) Identifying if there are data cleansing rules applied due to mergers, acquisitions, restatements, or other operational changes.
b) Evaluating data exclusions and their business justifications, especially during events like recessions.
1.5. Use of External Data: Assessing the appropriateness of external data and proxies when internal data cannot be used or is unavailable is crucial. This step includes:
a) Examining definitions and logic behind macroeconomic variables used and their link to modelled line items.
b) Reconciling macroeconomic data with the source and alternative sources, ensuring data consistency.
c) Checking for data restatements, inflation cycles, and other adjustments.
d) Verifying the integration of external data sets with other data sets in a consistent manner.
e) Ensuring the authenticity of the external data source.
2. Conceptual Soundness
2.1. Model Design: Evaluating the appropriateness of the overall model design is vital for consistency with its intended use and industry approaches. This step includes:
a) Assessing whether the modelling approach develops meaningful, robust models that align with business logic.
b) Checking if the modelling approach is supported by published research and industry standards.
c) Comparing and contrasting alternative modelling approaches.
d) Evaluating the interpretation associated with the chosen methodology.
e) Examining the logic for choosing the modelling methodology among alternatives.
f) Ensuring that the model theory/approach justifies its impact on business decisions.
2.2. Variable Selection: Assessing whether the selection and structure of variables align with similar industry models and the modelling objective is essential. This step includes:
a) Evaluating the initial pool of macroeconomic variables and the variable reduction process.
Recommended by LinkedIn
b) Reviewing and replicating variable selection phases to test their robustness and adherence to industry standards.
c) Assessing the appropriateness of variables in each model and the level of involvement of business experts.
d) Evaluating if any key variables are missing and quantifying their impact on stress scenarios.
e) Examining variable transformations and their impact on stress scenarios.
f) Identifying other possible internal variables and benchmarking against industry experience.
2.3. Assessment of Assumptions and Model Limitations: Assessing the validity of assumptions and model limitations is crucial. This step includes:
a) Verifying that the data supports the assumptions of the framework.
b) Conducting an assessment of the validity of assumptions.
c) Evaluating the evidence and support for each assumption.
d) Listing and assessing implicit and explicit assumptions, ensuring alignment with business strategy.
e) Ensuring the coverage of all assumptions.
f) Identifying and documenting restrictions on model use due to limitations or shortcomings.
3. Technical Soundness
3.1. Evaluating the soundness of the mathematical structure of estimated relationships is essential. This step includes:
a) Ensuring calculated fields are correctly computed and appropriate for the model's use.
b) Evaluating whether estimated statistical relationships for each variable are sound and supported by tests and analysis.
c) Re-estimating statistical testing and metrics.
d) Identifying any approximations used.
e) Examining acceptance criteria for each metric.
3.2. Parameters of Estimated Relationships: Assessing whether the chosen relationship produces parameters consistent with expectations is crucial. This step includes:
a) Testing the direction and stability of relationships.
b) Evaluating the magnitude of parameter estimates and their justification.
c) Assessing how the direction and magnitude impact model performance during stress.
3.3. Diagnostic Tests: Reviewing whether standard diagnostic tests confirm model performance as intended is essential. This step includes:
a) Reviewing the model selection criteria and their effectiveness.
b) Providing an independent analysis with interpretations.
3.4. Software Configuration: Ensuring that the computer software implementing the statistical estimation technique is properly configured is vital. This step includes reviewing the model code line by line.
4. Outcome Analysis
4.1. Evaluating the robustness of the tests used to evaluate the model's output is essential. This step includes:
a) Reviewing various tests to assess model performance and stability.
b) Evaluating in-sample model fit.
c) Performing back testing out of time and out of sample.
d) Establishing standard performance measures.
5. Computer-Based Processes
5.1. Model Implementation: Checking the accuracy of the computer processes used and the model's implementation into the production system is crucial. This step includes:
a) Verifying that the model is accurately run for its intended use, such as CCAR.
b) Assessing discrepancies between model estimation during development and production.
6. Ongoing Monitoring
6.1. Data Inputs: Evaluating data inputs into the model for data controls is essential. This step includes:
a) Assessing the process documentation and actions for model input and output data.
6.2. Model Monitoring: Evaluating the robustness, completeness, and accuracy of tests performed to monitor the model is vital. This step includes:
a) Conducting performance monitoring tests to assess the model's predictive power.
6.3. Evaluation of Ongoing Monitoring and Governance
Verify that the controls and governance structure within the first line of defence are appropriately designed and in accordance with Federal Reserve Board (FRB) requirements. Ensure that the documentation encompasses elements such as Change Management, Operational Controls, and the completeness and accuracy of Model Documentation. The testing results should provide insights into the robustness of the model and, where applicable, suggest potential model updates.
Conclusion: In the intricate world of finance, ensuring the accuracy of models used in processes like CCAR is crucial. This article has explored how to validate these models, focusing on the data, model design, technical aspects, and monitoring.
Model validation is not just about ticking boxes; it's about making sure these models work well in real-world scenarios. It helps reduce risks, ensures compliance with regulations, and builds trust among everyone involved, from customers to regulators.
As finance continues to evolve, having strong model validation processes in place is vital. It's a key part of staying prepared for whatever the financial world throws our way.
References
Disclaimer: The opinions expressed in this article are solely my own and do not reflect the official position or opinions of the organization I am affiliated with.
Business Science | Data, Risk Management | Transformation & Strategy Specialist I Program Management l Nordea
1yThanks for sharing Nawal Kashyap FRM® Indeed an excellent read.
Business Analyst: Wolters Kluwer || EY || Cognizant
1yVery informative....☺️