Over reliance on risk management models

I have posted previously on the dangers of relying too much on the preciseness that risk management numbers purportedly gives you. We seem to be assured that by having risk models that we are getting somewhere in progressing risk management without thinking about the quality of information that comes from our risk models, the application of the results of such models in risk decision-making, the assumptions behind such models and more importantly, the risks that are not captured in risk models.

While I am a strong believer in number crunching by way of statistical analysis, spreadsheets and presenting these numbers in a meaningful manner to facilitate decision-making as well as providing an indication of how much risks we are taking, we should not be lulled into being too "confident" with numbers that we think we have all our risks in our radar.

I propagate the following as a guide for all levels of management in this organization, with particular emphasis to risk managers whom I believe play an important role in communicating this message:

1. Risk models supported with established methodology should not replace sound business judgment. In the context of this organization, sound business judgment comes from business lines and it is our role as risk managers to understand the business, its decision-making process, and the risks that businesses take in that process.
2. Numbers can never capture market preception of the company, loss of confidence, reputational risks and in this crisis, I would question whether even liquidity risk can be captured accurately in risk models. In our context, our risk analysis assumes that the risks can be hedged or positions can be liquidated assuming there is liquidity. Therefore risk managers need to communicate to management what risks are represented in risk models and what risks are not.
3. In relation to no 2, for those risks represented in risk models, we must communicate that these risk models attempts to model the reality of risk out there using assumptions that more often than not does not represent the reality of risk out there. Questioning assumptions, I believe, first will make us understand better what these risk models represent and forces us to think what other risks lurks out there that we have not captured. In the recent article in FT "Maths and Mayhem", Lord Turner, chairman of the Financial Services Authority, blamed "misplaced reliance on sophisticated maths" for lulling banks' top managers into a false sense of security about the risks they were taking. This article asserts that contrary to Lord Turner's assertion, the banks' sums were not sophisticated enough. They over-simplified, and assumed away the limitations and caveats of their models. They did this to convey an illusion of accuracy and precision, and so convince the market that they had everything under control.
4. Making decisions in accepting risks based on the results of these risk models and not considering tail-risk may lead to accepting risks that can be beyond what an organization could sustain in a worst case scenario (and when all assumptions in risk models break down). According to a Wall Street Journal article, AIG said in a 2006 SEC filing that its credit default swaps had never experienced high enough defaults to consider the
   likelihood of making a payout on its credit-default-swap protection products more than “remote, even in severe recessionary market scenarios” (Refer to “Behind AIG’s Fall, Risk Models Failed to Pass Real-World Test,” Wall Street Journal, WSJ.com, October 31, 2008
5. Worst case scenarios have to complement the results of risk models and we should question the company's ability to weather these worst case scenarios and assess its impact on P&L and capital.

I hope this and references to previous postings would serve as a guide to

1. senior management in understanding and questioning the analysis that are presented to them and the models used to support such analysis.
2. risk managers in communicating to management on the level of risks that an organization has and in facilitating discussion about risks in business decision-making.