SPECIAL REPORT - 2007 ONWARDS
Number-crunchers crunched
Feb 11th 2010
From The Economist print edition
The uses and abuses of mathematical models
IT PUT noses out of joint, but it changed markets for good. In the mid-1970s a few progressive occupants of Chicago’s options pits started trading with the aid of sheets of theoretical prices derived from a model and sold by an economist called Fisher Black. Rivals, used to relying on their wits, were unimpressed. One model-based trader complained of having his papers snatched away and being told to “trade like a man”. But the strings of numbers caught on, and soon derivatives exchanges hailed the Black-Scholes model, which used share and bond prices to calculate the value of derivatives, for helping to legitimise a market that had been derided as a gambling den.
Thanks to Black-Scholes, options pricing no longer had to rely on educated guesses. Derivatives trading got a huge boost and quants poured into the industry. By 2005 they accounted for 5% of all finance jobs, against 1.2% in 1980, says Thomas Philippon of New York University—and probably a much higher proportion of pay. By 2007 finance was attracting a quarter of all graduates from the California Institute of Technology.
These eggheads are now in the dock, along with their probabilistic models. In America a congressional panel is investigating the models’ role in the crash. Wired, a publication that can hardly be accused of technophobia, has described default-probability models as “the formula that killed Wall Street”. Long-standing critics of risk-modelling, such as Nassim Nicholas Taleb, author of “The Black Swan”, and Paul Wilmott, a mathematician turned financial educator, are now hailed as seers. Models “increased risk exposure instead of limiting it”, says Mr Taleb. “They can be worse than nothing, the equivalent of a dangerous operation on a patient who would stand a better chance if left untreated.”
Not all models were useless. Those for interest rates and foreign exchange performed roughly as they were meant to. However, in debt markets they failed abjectly to take account of low-probability but high-impact events such as the gut-wrenching fall in house prices.
The models went particularly awry when clusters of mortgage-backed securities were further packaged into collateralised debt obligations (CDOs). In traditional products such as corporate debt, rating agencies employ basic credit analysis and judgment. CDOs were so complex that they had to be assessed using specially designed models, which had various faults. Each CDO is a unique mix of assets, but the assumptions about future defaults and mortgage rates were not closely tailored to that mix, nor did they factor in the tendency of assets to move together in a crisis.
The problem was exacerbated by the credit raters’ incentive to accommodate the issuers who paid them. Most financial firms happily relied on the models, even though the expected return on AAA-rated tranches was suspiciously high for such apparently safe securities. At some banks, risk managers who questioned the rating agencies’ models were given short shrift. Moody’s and Standard & Poor’s were assumed to know best. For people paid according to that year’s revenue, this was understandable. “A lifetime of wealth was only one model away,” sneers an American regulator.
Moreover, heavy use of models may have changed the markets they were supposed to map, thus undermining the validity of their own predictions, says Donald MacKenzie, an economic sociologist at the University of Edinburgh. This feedback process is known as counter-performativity and had been noted before, for instance with Black-Scholes. With CDOs the models’ popularity boosted demand, which lowered the quality of the asset-backed securities that formed the pools’ raw material and widened the gap between expected and actual defaults (see chart 3).
Number-crunchers crunched
Feb 11th 2010
From The Economist print edition
The uses and abuses of mathematical models
IT PUT noses out of joint, but it changed markets for good. In the mid-1970s a few progressive occupants of Chicago’s options pits started trading with the aid of sheets of theoretical prices derived from a model and sold by an economist called Fisher Black. Rivals, used to relying on their wits, were unimpressed. One model-based trader complained of having his papers snatched away and being told to “trade like a man”. But the strings of numbers caught on, and soon derivatives exchanges hailed the Black-Scholes model, which used share and bond prices to calculate the value of derivatives, for helping to legitimise a market that had been derided as a gambling den.
Thanks to Black-Scholes, options pricing no longer had to rely on educated guesses. Derivatives trading got a huge boost and quants poured into the industry. By 2005 they accounted for 5% of all finance jobs, against 1.2% in 1980, says Thomas Philippon of New York University—and probably a much higher proportion of pay. By 2007 finance was attracting a quarter of all graduates from the California Institute of Technology.
These eggheads are now in the dock, along with their probabilistic models. In America a congressional panel is investigating the models’ role in the crash. Wired, a publication that can hardly be accused of technophobia, has described default-probability models as “the formula that killed Wall Street”. Long-standing critics of risk-modelling, such as Nassim Nicholas Taleb, author of “The Black Swan”, and Paul Wilmott, a mathematician turned financial educator, are now hailed as seers. Models “increased risk exposure instead of limiting it”, says Mr Taleb. “They can be worse than nothing, the equivalent of a dangerous operation on a patient who would stand a better chance if left untreated.”
Not all models were useless. Those for interest rates and foreign exchange performed roughly as they were meant to. However, in debt markets they failed abjectly to take account of low-probability but high-impact events such as the gut-wrenching fall in house prices.
The models went particularly awry when clusters of mortgage-backed securities were further packaged into collateralised debt obligations (CDOs). In traditional products such as corporate debt, rating agencies employ basic credit analysis and judgment. CDOs were so complex that they had to be assessed using specially designed models, which had various faults. Each CDO is a unique mix of assets, but the assumptions about future defaults and mortgage rates were not closely tailored to that mix, nor did they factor in the tendency of assets to move together in a crisis.
The problem was exacerbated by the credit raters’ incentive to accommodate the issuers who paid them. Most financial firms happily relied on the models, even though the expected return on AAA-rated tranches was suspiciously high for such apparently safe securities. At some banks, risk managers who questioned the rating agencies’ models were given short shrift. Moody’s and Standard & Poor’s were assumed to know best. For people paid according to that year’s revenue, this was understandable. “A lifetime of wealth was only one model away,” sneers an American regulator.
Moreover, heavy use of models may have changed the markets they were supposed to map, thus undermining the validity of their own predictions, says Donald MacKenzie, an economic sociologist at the University of Edinburgh. This feedback process is known as counter-performativity and had been noted before, for instance with Black-Scholes. With CDOs the models’ popularity boosted demand, which lowered the quality of the asset-backed securities that formed the pools’ raw material and widened the gap between expected and actual defaults (see chart 3).
A related problem was the similarity of risk models. Banks thought they were diversified, only to find that many others held comparable positions, based on similar models that had been built to comply with the Basel 2 standards, and everyone was trying to unwind the same positions at the same time. The breakdown of the models, which had been the only basis for pricing the more exotic types of security, turned risk into full-blown uncertainty (and thus extreme volatility).For some, the crisis has shattered faith in the precision of models and their inputs. They failed Keynes’s test that it is better to be roughly right than exactly wrong. One number coming under renewed scrutiny is “value-at-risk” (VAR), used by banks to measure the risk of loss in a portfolio of financial assets, and by regulators to calculate banks’ capital buffers. Invented by eggheads at JPMorgan in the late 1980s, VAR has grown steadily in popularity. It is the subject of more than 200 books. What makes it so appealing is that its complex formulae distil the range of potential daily profits or losses into a single dollar figure.
Only so far with VAR
Frustratingly, banks introduce their own quirks into VAR calculations, making comparison difficult. For example, Morgan Stanley’s VAR for the first quarter of 2009 by its own reckoning was $115m, but using Goldman Sachs’s method it would have been $158m. The bigger problem, though, is that VAR works only for liquid securities over short periods in “normal” markets, and it does not cover catastrophic outcomes. If you have $30m of two-week 1% VAR, for instance, that means there is a 99% chance that you will not lose more than that amount over the next fortnight. But there may be a huge and unacknowledged threat lurking in that 1% tail.
So chief executives would be foolish to rely solely, or even primarily, on VAR to manage risk. Yet many managers and boards continue to pay close attention to it without fully understanding the caveats—the equivalent of someone who cannot swim feeling confident of crossing a river having been told that it is, on average, four feet deep, says Jaidev Iyer of the Global Association of Risk Professionals.
Regulators are encouraging banks to look beyond VAR. One way is to use CoVAR (Conditional VAR), a measure that aims to capture spillover effects in troubled markets, such as losses due to the distress of others. This greatly increases some banks’ value at risk. Banks are developing their own enhancements. Morgan Stanley, for instance, uses “stress” VAR, which factors in very tight liquidity constraints.
Like its peers, Morgan Stanley is also reviewing its stress testing, which is used to consider extreme situations. The worst scenario envisaged by the firm turned out to be less than half as bad as what actually happened in the markets. JPMorgan Chase’s debt-market stress tests foresaw a 40% increase in corporate spreads, but high-yield spreads in 2007-09 increased many times over. Others fell similarly short. Most banks’ tests were based on historical crises, but this assumes that the future will be similar to the past. “A repeat of any specific market event, such as 1987 or 1998, is unlikely to be the way that a future crisis will unfold,” says Ken deRegt, Morgan Stanley’s chief risk officer.
Faced with either random (and therefore not very believable) scenarios or simplistic models that neglect fat-tail risks, many find themselves in a “no-man’s-land” between the two, says Andrew Freeman of Deloitte (and formerly a journalist at The Economist). Nevertheless, he views scenario planning as a useful tool. A firm that had thought about, say, the mutation of default risk into liquidity risk would have had a head start over its competitors in 2008, even if it had not predicted precisely how this would happen.
To some, stress testing will always seem maddeningly fuzzy. “It has so far been seen as the acupuncture-and-herbal-remedies corner of risk management, though perceptions are changing,” says Riccardo Rebonato of Royal Bank of Scotland, who is writing a book on the subject. It is not meant to be a predictive tool but a means of considering possible outcomes to allow firms to react more nimbly to unexpected developments, he argues. Hedge funds are better at this than banks. Some had thought about the possibility of a large broker-dealer going bust. At least one, AQR, had asked its lawyers to grill the fund’s prime brokers about the fate of its assets in the event of their demise.
Some of the blame lies with bank regulators, who were just as blind to the dangers ahead as the firms they oversaw. Sometimes even more so: after the rescue of Bear Stearns in March 2008 but before Lehman’s collapse, Morgan Stanley was reportedly told by supervisors at the Federal Reserve that its doomsday scenario was too bearish.
The regulators have since become tougher. In America, for instance, banks have been told to run stress tests with scenarios that include a huge leap in interest rates. A supervisors’ report last October fingered some banks for “window-dressing” their tests. Officials are now asking for “reverse” stress testing, in which a firm imagines it has failed and works backwards to determine which vulnerabilities caused the hypothetical collapse. Britain has made this mandatory. Bankers are divided over its usefulness.
Slicing the Emmental
So chief executives would be foolish to rely solely, or even primarily, on VAR to manage risk. Yet many managers and boards continue to pay close attention to it without fully understanding the caveats—the equivalent of someone who cannot swim feeling confident of crossing a river having been told that it is, on average, four feet deep, says Jaidev Iyer of the Global Association of Risk Professionals.
Regulators are encouraging banks to look beyond VAR. One way is to use CoVAR (Conditional VAR), a measure that aims to capture spillover effects in troubled markets, such as losses due to the distress of others. This greatly increases some banks’ value at risk. Banks are developing their own enhancements. Morgan Stanley, for instance, uses “stress” VAR, which factors in very tight liquidity constraints.
Like its peers, Morgan Stanley is also reviewing its stress testing, which is used to consider extreme situations. The worst scenario envisaged by the firm turned out to be less than half as bad as what actually happened in the markets. JPMorgan Chase’s debt-market stress tests foresaw a 40% increase in corporate spreads, but high-yield spreads in 2007-09 increased many times over. Others fell similarly short. Most banks’ tests were based on historical crises, but this assumes that the future will be similar to the past. “A repeat of any specific market event, such as 1987 or 1998, is unlikely to be the way that a future crisis will unfold,” says Ken deRegt, Morgan Stanley’s chief risk officer.
Faced with either random (and therefore not very believable) scenarios or simplistic models that neglect fat-tail risks, many find themselves in a “no-man’s-land” between the two, says Andrew Freeman of Deloitte (and formerly a journalist at The Economist). Nevertheless, he views scenario planning as a useful tool. A firm that had thought about, say, the mutation of default risk into liquidity risk would have had a head start over its competitors in 2008, even if it had not predicted precisely how this would happen.
To some, stress testing will always seem maddeningly fuzzy. “It has so far been seen as the acupuncture-and-herbal-remedies corner of risk management, though perceptions are changing,” says Riccardo Rebonato of Royal Bank of Scotland, who is writing a book on the subject. It is not meant to be a predictive tool but a means of considering possible outcomes to allow firms to react more nimbly to unexpected developments, he argues. Hedge funds are better at this than banks. Some had thought about the possibility of a large broker-dealer going bust. At least one, AQR, had asked its lawyers to grill the fund’s prime brokers about the fate of its assets in the event of their demise.
Some of the blame lies with bank regulators, who were just as blind to the dangers ahead as the firms they oversaw. Sometimes even more so: after the rescue of Bear Stearns in March 2008 but before Lehman’s collapse, Morgan Stanley was reportedly told by supervisors at the Federal Reserve that its doomsday scenario was too bearish.
The regulators have since become tougher. In America, for instance, banks have been told to run stress tests with scenarios that include a huge leap in interest rates. A supervisors’ report last October fingered some banks for “window-dressing” their tests. Officials are now asking for “reverse” stress testing, in which a firm imagines it has failed and works backwards to determine which vulnerabilities caused the hypothetical collapse. Britain has made this mandatory. Bankers are divided over its usefulness.
Slicing the Emmental
These changes point towards greater use of judgment and less reliance on numbers in future. But it would be unfair to tar all models with the same brush. The CDO fiasco was an egregious and relatively rare case of an instrument getting way ahead of the ability to map it mathematically. Models were “an accessory to the crime, not the perpetrator”, says Michael Mauboussin of Legg Mason, a money manager.
As for VAR, it may be hopeless at signalling rare severe losses, but the process by which it is produced adds enormously to the understanding of everyday risk, which can be just as deadly as tail risk, says Aaron Brown, a risk manager at AQR. Craig Broderick, chief risk officer at Goldman Sachs, sees it as one of several measures which, although of limited use individually, together can provide a helpful picture. Like a slice of Swiss cheese, each number has holes, but put several of them together and you get something solid.
Modelling is not going away; indeed, number-crunchers who are devising new ways to protect investors from outlying fat-tail risks are gaining influence. Pimco, for instance, offers fat-tail hedging programmes for mutual-fund clients, using cocktails of options and other instruments. These are built on specific risk factors rather than on the broader and increasingly fluid division of assets between equities, currencies, commodities and so on. The relationships between asset classes “have become less stable”, says Mohamed El-Erian, Pimco’s chief executive. “Asset-class diversification remains desirable but is not sufficient.”
As for VAR, it may be hopeless at signalling rare severe losses, but the process by which it is produced adds enormously to the understanding of everyday risk, which can be just as deadly as tail risk, says Aaron Brown, a risk manager at AQR. Craig Broderick, chief risk officer at Goldman Sachs, sees it as one of several measures which, although of limited use individually, together can provide a helpful picture. Like a slice of Swiss cheese, each number has holes, but put several of them together and you get something solid.
Modelling is not going away; indeed, number-crunchers who are devising new ways to protect investors from outlying fat-tail risks are gaining influence. Pimco, for instance, offers fat-tail hedging programmes for mutual-fund clients, using cocktails of options and other instruments. These are built on specific risk factors rather than on the broader and increasingly fluid division of assets between equities, currencies, commodities and so on. The relationships between asset classes “have become less stable”, says Mohamed El-Erian, Pimco’s chief executive. “Asset-class diversification remains desirable but is not sufficient.”
Illustration by Tim Marrs
Not surprisingly, more investors are now willing to give up some upside for the promise of protection against catastrophic losses. Pimco’s clients are paying up to 1% of the value of managed assets for the hedging—even though, as the recent crisis showed, there is a risk that insurers will not be able to pay out. Lisa Goldberg of MSCI Barra reports keen interest in the analytics firm’s extreme-risk model from hedge funds, investment banks and pension plans.
In some areas the need may be for more computing power, not less. Financial firms already spend more than any other industry on information technology (IT): some $500 billion in 2009, according to Gartner, a consultancy. Yet the quality of information filtering through to senior managers is often inadequate.
A report by bank supervisors last October pointed to poor risk “aggregation”: many large banks simply do not have the systems to present an up-to-date picture of their firm-wide links to borrowers and trading partners. Two-thirds of the banks surveyed said they were only “partially” able (in other words, unable) to aggregate their credit risks. The Federal Reserve, leading stress tests on American banks last spring, was shocked to find that some of them needed days to calculate their exposure to derivatives counterparties.
To be fair, totting up counterparty risk is not easy. For each trading partner the calculations can involve many different types of contract and hundreds of legal entities. But banks will have to learn fast: under new international proposals, they will for the first time face capital charges on the creditworthiness of swap counterparties.
The banks with the most dysfunctional systems are generally those, such as Citigroup, that have been through multiple marriages and ended up with dozens of “legacy” systems that cannot easily communicate with each other. That may explain why some Citi units continued to pile into subprime mortgages even as others pulled back.
In the depths of the crisis some banks were unaware that different business units were marking the same assets at different prices. The industry is working to sort this out. Banks are coming under pressure to appoint chief data officers who can police the integrity of the numbers, separate from chief information officers who concentrate on system design and output.
Some worry that the good work will be cast aside. As markets recover, the biggest temptation will be to abandon or scale back IT projects, allowing product development to get ahead of the supporting technology infrastructure, just as it did in the last boom.
The way forward is not to reject high-tech finance but to be honest about its limitations, says Emanuel Derman, a professor at New York’s Columbia University and a former quant at Goldman Sachs. Models should be seen as metaphors that can enlighten but do not describe the world perfectly. Messrs Derman and Wilmott have drawn up a modeller’s Hippocratic oath which pledges, among other things: “I will remember that I didn’t make the world, and it doesn’t satisfy my equations,” and “I will never sacrifice reality for elegance without explaining why I have done so.” Often the problem is not complex finance but the people who practise it, says Mr Wilmott. Because of their love of puzzles, quants lean towards technically brilliant rather than sensible solutions and tend to over-engineer: “You may need a plumber but you get a professor of fluid dynamics.”
One way to deal with that problem is to self-insure. JPMorgan Chase holds $3 billion of “model-uncertainty reserves” to cover mishaps caused by quants who have been too clever by half. If you can make provisions for bad loans, why not bad maths too?
Copyright © 2010 The Economist Newspaper and The Economist Group. All rights reserved.
In some areas the need may be for more computing power, not less. Financial firms already spend more than any other industry on information technology (IT): some $500 billion in 2009, according to Gartner, a consultancy. Yet the quality of information filtering through to senior managers is often inadequate.
A report by bank supervisors last October pointed to poor risk “aggregation”: many large banks simply do not have the systems to present an up-to-date picture of their firm-wide links to borrowers and trading partners. Two-thirds of the banks surveyed said they were only “partially” able (in other words, unable) to aggregate their credit risks. The Federal Reserve, leading stress tests on American banks last spring, was shocked to find that some of them needed days to calculate their exposure to derivatives counterparties.
To be fair, totting up counterparty risk is not easy. For each trading partner the calculations can involve many different types of contract and hundreds of legal entities. But banks will have to learn fast: under new international proposals, they will for the first time face capital charges on the creditworthiness of swap counterparties.
The banks with the most dysfunctional systems are generally those, such as Citigroup, that have been through multiple marriages and ended up with dozens of “legacy” systems that cannot easily communicate with each other. That may explain why some Citi units continued to pile into subprime mortgages even as others pulled back.
In the depths of the crisis some banks were unaware that different business units were marking the same assets at different prices. The industry is working to sort this out. Banks are coming under pressure to appoint chief data officers who can police the integrity of the numbers, separate from chief information officers who concentrate on system design and output.
Some worry that the good work will be cast aside. As markets recover, the biggest temptation will be to abandon or scale back IT projects, allowing product development to get ahead of the supporting technology infrastructure, just as it did in the last boom.
The way forward is not to reject high-tech finance but to be honest about its limitations, says Emanuel Derman, a professor at New York’s Columbia University and a former quant at Goldman Sachs. Models should be seen as metaphors that can enlighten but do not describe the world perfectly. Messrs Derman and Wilmott have drawn up a modeller’s Hippocratic oath which pledges, among other things: “I will remember that I didn’t make the world, and it doesn’t satisfy my equations,” and “I will never sacrifice reality for elegance without explaining why I have done so.” Often the problem is not complex finance but the people who practise it, says Mr Wilmott. Because of their love of puzzles, quants lean towards technically brilliant rather than sensible solutions and tend to over-engineer: “You may need a plumber but you get a professor of fluid dynamics.”
One way to deal with that problem is to self-insure. JPMorgan Chase holds $3 billion of “model-uncertainty reserves” to cover mishaps caused by quants who have been too clever by half. If you can make provisions for bad loans, why not bad maths too?
Copyright © 2010 The Economist Newspaper and The Economist Group. All rights reserved.
0 comments:
Publicar un comentario