In reading the published accounts of the U.S. Securities and Exchange Commission ("SEC") settlement with the AXA Rosenberg Group and various related entities, I'm reminded of many articles I've written that warn about the limitations of models and the critical need for ongoing tests and oversight of said models. According to the document published today regarding the AXA Settlement With the SEC, certain senior managers who were apparently made aware of a coding error (that "was introduced in 2007" and discovered in late June 2009) responded by instructing junior staff to "keep quiet about the error."

Models are designed by human beings and require constant care and feeding. For one thing, market conditions change which therefore requires that models be revisited for their accuracy and the extent to which underlying assumptions still hold true. Second, most investment models are complex in that multiple calculations are used to generate final outputs. Third, some models are "nested" in the sense that data inputs for an ultimate algorithm are themselves modeled.If the modeled inputs are flawed, it's no surprise that Garbage In, Garbage Out ensues.

For example, interest rate paths (levels, direction, volatility) and prepayment assumptions are integral inputs for mortgage backed security pricing models yet must first be modeled. If rates have just fallen considerably and a large number of home owners have refinanced, another fall in interest rates is unlikely to encourage yet another large spate of prepayments. The underlying assumptions that drive interest rate paths directly influence the estimates of time-valued projected cash flows associated with the mortgage bond so questions must be asked about how the inputs and the outputs are generated.

Model errors are seldom self-contained and create a Pandora's box for those investors or traders who commit money to the "wrong" numbers. Let me elaborate.

Suppose a trader hedges a portfolio of securities based on incorrect model outputs. The net result will be protection that is excessive (thereby costing that trader in the form of opportunity costs) or insufficient (thereby forcing the trader to realize losses that were meant to be avoided).

Another situation relates to end users that allocate part of their portfolio to a quantitative investment strategy such as pension funds or endowments. If the "quants" get it wrong because of sloppy modeling work (and that includes errors that are not caught right away because of poor oversight, bad math or both), these institutional or individual investors have unnecessarily lost money as well as the opportunity to have allocated elsewhere. According to "AXA Rosenberg to Pay $242 Million Over 'Coding Error'" by Jakema Lewis (Securities Technology Monitor, February 3, 2011), the School Employees' Retirement System of Ohio, Los Angeles Fire and Police Pensions, the City of Fresno Retirement System, Florida State Board of Administration and the Montana Board of Investments voted with their dollars by terminating their respective relationships with AXA Rosenberg. According to "AXA Rosenberg finds coding error in risk program" by Mark Weinbraub with Jennifer Ablan, Reuters, April 24, 2010, the Marin County Employees' Retirement Association pulled $16.5 million from an AXA Rosenberg international small-cap portfolio.

Email contact@fiduciaryleadership.com if you are interested in learning more about investment model due diligence. In the meantime, articles listed below provide information about model risk:

• "The Risks of Ignoring Model Risk" by Susan Mangiero, Chapter 5B, 2005 Cumulative Supplement to the Litigation Service Handbook: The Role of the Expert, John Wiley & Sons, 2005
• "Asset Valuation: Not a Trivial Pursuit" by Susan Mangiero, FSA Times, Institute of Internal Auditor, Q1-2004
• "Financial Model Risk Looms Large" by Susan Mangiero, The Investment Lawyer, November 2002
• "Model Risk is Serious Business" by Susan Mangiero, "Derivatives Diary," AFP Exchange, September/October 2002

Every investor must exercise care and diligence in asking tough questions and being satisfied with the answers. At a bare minimum, this requires that each investor:

• Gain an intuitive understanding of the model and what results should raise red flags because the "smell test" fails
• Ask to meet with the architects of the model and have them explain how they stress the model and whether there is a consistent error rate associated with outputs
• Understand the data issues and whether quality varies across vendors, how outliers are identified and addressed and whether the model is sensitive to frequency and form of data inputs
• Inquire as to who has the authority to modify the model and on what basis
• Query whether internal and external auditors are comfortable with the traders' model(s) and if not, explain their concerns.

Risk management is about so much more than good numbers. Process is everything and that includes comprehensive oversight of the models and when they are likely to fail and for what reasons. Common sense is a critical component of managing risk and will never be replaced by a number or computer. Keep in mind too that with FAS 157 and international accounting equivalents, not to mention U.S. Department of Labor mandates for ERISA plans, an investment fiduciary who invests in complicated strategies and/or securities on the basis of a "black box" approach is just asking for trouble.