Model risk is a risk in its own right. With the release of the Prudential Regulation Authority’s (PRA) Supervisory Statement (SS1/23) in May 2023, it is clear that UK regulators are serious about enforcing effective and sustainable model risk management (MRM) standards. Critically, UK banks* with approval to calculate regulatory capital requirements through internal models will need to assess whether their model risk management frameworks meet the expectations of the PRA. This is only the beginning of the process as the standard is expected to cover the entire banking sector in due course.
Most large banks will have established frameworks that cover model governance, lifecycle management, validation and model risk mitigants that are in line with the PRA’s principles. However, the statement’s intentionally broad definitions of what constitutes a model as well as how to apply model risk tiering could become a chronic impediment if not adequately and effectively addressed early on in the remediation journey.
As the 17 May 2024 deadline approaches, banks will need to identify and remediate deficiencies across their model risk management systems, processes and governance in a manner that is both rigorous and consistent.
*UK incorporated banks, building societies and PRA-designated firms.
The Supervisory Standard (SS) aims to support firms in strengthening their policies, procedures and practices across the model lifecycle and model risk management through five core principles:
Model Identification and Model Risk Classification
Model Development, Implementation and Use
Independent Model Validation
Model Risk Mitigation
A significant challenge for financial institutions will be interpreting and applying the principles outlined in the standard – especially those that are intentionally broad or non-prescriptive (i.e., model definition and model risk classification). As a result, firms may find themselves stretching the existing capacity of their MRM capabilities as they apply a much broader definition of what constitutes a model.
However, the standard’s proportionality and risk-based considerations will be significant levers that senior management can use to direct their effort and oversight where it is especially required.
While the SS will stand alongside existing supervisory expectations, firms will need to conduct an initial selfassessment of their implemented MRM frameworks against the principles and prepare a remediation plan to address identified deficiencies and shortcomings.
Firms should have an established definition of a model that sets the scope for MRM, a model inventory and a risk-based tiering approach to enable the identification and management of model risk.
The PRA’s broad model definition could expand the scope of models beyond what the existing MRM capabilities can effectively manage.
A poorly defined model definition can have significant long-term consequences, particularly if model scope needs to be adjusted after initial deficiencies have already been addressed.
Firms should have strong governance oversight with a board that promotes an MRM culture from the top through setting clear a model risk appetite. The board should approve the MRM policy and appoint an accountable individual to assume the responsibility of implementing a sound MRM framework that will ensure effective MRM practices.
Assigning roles and responsibilities as well as enforcing MRM policies and procedures may be difficult to coordinate consistently across a firm, particularly if model scope expands substantially.
With responsibility assigned to boards and senior management, ensuring adequate MRM compliance of third-party vendor models may pose a challenge if vendors cannot provide satisfactory model validation compliance.
Firms should have a robust model development process with standards for model design and implementation, model selection, and model performance measurement. Testing of data, model construct, assumptions, and model outcomes should be performed regularly in order to identify, monitor, record, and remediate model limitations and weaknesses.
Model development testing, adjustments and documentation may become onerous as scope expands. This will require model tiering to guide prioritisation and the level of oversight required.
With comprehensive requirements regarding quality and change controls, documentation, performance monitoring, periodic revalidation and model weakness/limitation management, to name a few, existing MRM systems, tools and environments may need to be upgraded or in some cases completely overhauled, which can be a costly and intensive exercise.
Firms may not have adequate capabilities to effectively manage risks related to dynamic models (models able to adapt/recalibrate/ automatically change) such as machine learning models which could require significant enhancements to MRM technology platforms and frameworks.
Firms should have a validation process that provides ongoing, independent, and effective challenge to model development and use. The individual/body within a firm that is responsible for the approval of a model should ensure that validation recommendations for remediation or redevelopment are actioned so that models are suitable for their intended purpose.
An expanded scope will put pressure on model validation functions that have historically had limited spare capacity due to a scarcity of specialised validation skill sets. Firms will need to ensure their chosen model risk tiering approach (principle 1) effectively allocates validation time and effort to models with higher complexity and materiality, while utilising technology and light-touch oversight with lower ranked models.
Firms should have established policies and procedures for the use of model risk mitigants when models are under-performing and should have procedures for the independent review of post-model adjustments.
Economic shocks, including the Covid-19 pandemic and recent inflation peaks, have highlighted the need for more agile and robust processes to address model limitations, particularly regarding recalibration or redevelopment of existing methodologies.
Large banks can have hundreds or even thousands of models across the enterprise. The foundation of model risk management is therefore a comprehensive, reliable and risk-based view of all these computations with specific consideration for:
Model Definition. The PRA has provided a broad model definition that is wide enough to include computations
that may not ordinarily or historically have been classified as a model.
In order to meet the requirements of the PRA while ensuring a manageable and realistic model scope, Monocle recommends firms take a practical and structured approach to defining their model definition.
Therefore firms should consider a variety of factors that are critical for an effective model tiering framework such as model complexity, model use, business impact and a host of additional factors that have been incorporated into Monocle’s tiering approach.
As a starting point in model identification, firms should determine the models that are critical for strategic and operational decision-making. Further engagement should be considered with specific business stakeholders as well as model users/owners/developers to identify models and computations that were not previously included as part of model risk management oversight.
The PRA defines a model as “A quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into output. The definition of a model includes input data that are quantitative and/or qualitative in nature or expert judgement-based, and output that are quantitative or qualitative.“
“Notwithstanding the above definition, where material deterministic quantitative methods such as decision-based rules or algorithms that are not classified as a model have a material bearing on business decisions and are complex in nature, firms should consider whether to apply the relevant aspects of the MRM framework to these methods”
Model Tiering. A risk-based model tiering approach sets out which models will require prioritised validation and risk controls throughout the model lifecycle. The PRA requires financial institutions to assign a risk-based materiality and complexity rating to each model.
To meet the requirement of consistent, firm-wide model tiering, Monocle utilises a proprietary Model Tiering Framework including a combination of scorecard and decision tree structures to ensure both quantitative scoring and qualitative decisions are accounted for.
While model scope may expand to include computations previously considered as calculators and tools, risk tiering is an effective mechanism to allocate limited capacity and expertise to where it is
Financial institutions will need to refine their model risk management frameworks to align with the PRA’s expectations across the five principles. Most firms will already have the necessary documented policies and procedures in place and will therefore refine and enhance their frameworks rather than completely overhaul them.
Once all applicable models have been identified and categorised by risk, a gap assessment should be performed to ensure proportional oversight and controls are in place – particularly for tier 1 and tier 2 models.
Tier 1 – High Risk:
Highly complex with critical impacts
Tier 2 – Moderately High Risk:
Complex with material impacts
Addressing all identified shortcomings and deficiencies will require a structured remediation plan with achievable deadlines, considering the 12-month compliance period. A sprawling project such as this will require active
contribution from, and collaboration between, the three lines of defence – model owners/users/developers, model validators and internal audit – as well as review and approval from senior management.
Additionally, remediation offers the opportunity for enhancements and strategic improvements to be implemented, particularly as financial institutions make increasing use of advanced analytics that have inherent characteristics not accommodated by traditional model risk management frameworks.
Machine learning (ML), a prevalent form of artificial intelligence, uses quantitative models to fit a set of data to make predictions, recommendations or decisions without explicitly being programmed. This falls neatly into the Supervisory Statement’s expanded model definition without being mentioned directly.
Banks will need to consider two significant features of their ML models – their complexity as well as their business impact.
Complexity: ML applications are capable of being retrained on new data as it becomes available in order to adjust and potentially improve the model over time. This dynamism is a key difference compared to traditional statistical models but introduces the risk of model drift, which should be addressed through enhanced and continuous monitoring (using key model performance indicators) and validation.
Additionally, the statement refers to the risk factors of model “explainability” and transparency, as well as the risk of data bias – two significant issues particularly for convoluted neural networks. Financial institutions will need to pay special attention to this when risk tiering their ML models.
Business Impact: The Bank of England’s and Financial Conduct Authority’s latest machine learning survey found the business area of Treasury and Credit – arguably the most important functions in a bank – to hold the highest percentage of ML applications that were considered critical to the business area. With the SS’s focus on business decision impacts, banks will need to be aware of how machine learning outputs are supporting materially important business decisions, and categorise these models accordingly.
Climate risk management is becoming a standard part of the risk management and reporting landscape –particularly climate risk scenario analysis and stress testing. However, climate forecasts contain substantial uncertainty due to a number of factors including non-linear outcomes, significant data gaps, reliance on various assumptions and proxies, long time frames and the novelty of forecasting physical and transition risks for the banking industry. Much like the intricacies of ML, banks will need to consider the idiosyncrasies of their climate risk models including their rapid advancement as well as their business impacts.
Rapid Advancement: Risk modelling capabilities around climate are improving as granular, sector-specific data becomes more readily available, benchmark climate scenarios become accepted by the industry and climate modelling becomes increasingly embedded in banks’ risk management frameworks. However, the pace of change requires banks to adopt a more agile approach to model validation that must find alternatives to back testing and other traditional model validation techniques ill-suited to climate risk. Banks can opt for sensitivity testing, benchmarking and reviewing the conceptual soundness of methodologies, key drivers and assumptions on a regular basis.
Business Impact: For most banks, their climate risk modelling will be focused on the impact on credit risk and its various stages including credit underwriting, impairment, collateral management, and regulatory capital. Climate models with significant impacts like these will need to be identified, categorised and managed accordingly. As is the case with AI and ML, the impact and rapid pace of change in the climate modelling space will need to be adequately assessed and addressed by banks.
With Monocle’s four workstream project approach we aim to assist our clients to interpret and apply the Supervisory Statement while identifying and remediating any shortcomings and deficiencies.
Following an extensive assessment of the statement, Monocle has developed a robust methodology that will aid banks in their interpretation of each principle and guide them in the implementation thereof.
Monocle’s vast experience and comprehensive understanding of the statement ensures that, through a detailed model level gap analysis and the subsequent refinement of existing MRM frameworks, our clients will have a wellstructured workflow embedded in their business. Ultimately, this will optimise model risk operations and improve efficiency across all aspects of their model risk management capabilities.