The trouble with predicting the markets

The state of the stock market displayed on a trader's screen is history. Big changes can occur in the fraction of a second that it takes for information to reach the screen. Computers have replaced people in executing deals. Humans are so slow that they are irrelevant for many market functions. In today's electronic trading, the markets are driven by computers rather than people.

The software to operate in the new conditions is designed by quantitative analysts, or "quants", who often have a background in mathematics or physics rather than finance or economics. They specialise in applying complex mathematical models to analyse market trends and execute trades. The computer-based algorithms, or "algos", use all available relevant data – far beyond the analytical capacity of humans – to predict prices.

Financial organisations now employ staff who have no background in finance but who have excellent problem-solving skills. Many have PhDs in mathematics, physics or engineering. The methods used include stochastic differential equations, numerical analysis and complexity theory.

An equation published in 1973 by Fischer Black and Myron Scholes led to an explosive growth in the trading of financial products called derivatives. Basically, these are options to buy or sell on a fixed date at an agreed price. Black and Scholes assumed that changes in stock prices have two components: an upward or downward trend, and random fluctuations or jiggles. The jiggles determine market volatility, and their size affects a derivative’s value.

The Black-Scholes equation is a partial differential equation, similar to the heat equation in physics. This is no surprise: the molecules of a gas move in random fashion, with innumerable jiggles, like market prices, so their mean behaviour, or temperature, is governed by a similar equation.

Bell curve

Black and Scholes assumed that volatility followed the so-called normal distribution, or bell-shaped curve. The bell curve is found throughout science and economics. For example, if you toss a coin many times, heads and tails should show up roughly an equal number of times. The likelihood of a particular number of heads follows the bell curve. The crucial factor is independence: each coin toss is unaffected by the previous ones.

The Black-Scholes equation enabled analysts to reduce investment risk and increase profits using mathematics. But on several occasions following its emergence, major market upheavals showed its limitations. The most dramatic of these was in 2007, when the housing bubble burst and markets collapsed. Global shockwaves left the world economy reeling and the consequences are still being felt. Government intervention was needed to bail out banks that were “too big to fail”.

A common factor of the turbulent episodes was the inability of mathematical models to simulate highly volatile markets. The bell curve is fine for modelling numerous uncorrelated events. But if investors stop acting independently and follow their herd instincts, greed induces reckless behaviour or fear leads to panic. Extreme conditions ensue and they are beyond the range of validity of simple models such as Black-Scholes.

Refining the equations

Mathematicians are working to develop refined models and equations that give more realistic descriptions of market behaviour. They include intermittent surges that fatten the tails of the bell curve, making extreme conditions much more likely. With huge profits to be made, these efforts will continue. In markets that are ever more complex, mathematical models are essential, but they must be used responsibly and with due recognition of their limitations.

Given human frailty and irrationality, there will always be surprise crashes and booms. Even the great Isaac Newton was caught offside in 1720 by the infamous South Sea Bubble, one of the early market crashes. He lost more than £20,000, north of a million pounds today, and is said to have remarked: “I can calculate the movements of the stars, but not the madness of men.”

• Peter Lynch is emeritus professor at the school of mathematical sciences, University College Dublin. He blogs at thatsmaths.com