The great mathematical minds used to become physicists or maths professors; now they develop complex financial algorithms for Wall Street hedge funds. But have their innovations destabilised the very markets they were supposed to rationalise and are we wasting a generation of innovative talent?

‘Beware of geeks bearing formulas,” warned Warren Buffett in a letter to shareholders in his Berkshire Hathaway company in 2008, following record losses there.

Buffett was referring to the mathematical tools that did things like calculate and mitigate risk, price derivatives and buy and sell stock in the most efficient and lucrative way possible.

To his mind these equations and algorithms, created by some of the greatest mathematical minds alive, were not quite smart enough to function in what is an unpredictable world.

It is a view that Ian Stewart, emeritus professor of mathematics at the University of Warwick and author of the book Seventeen Equations that Changed the World, also holds.

“There are some very successful and very good models in the financial world but they come with limitations,” he says. “If for a while they fit the market well then a lot of people who don’t entirely understand the maths think it works and think they can use it too.”

Buffett’s warning ultimately proved to be a prescient one.

In May 2010 the Dow Jones suffered, and then largely recovered, from a 9 per cent loss in a matter of minutes. For a brief period of time $1 trillion disappeared from the value of major markets and all for reasons that no one could explain.

The so-called Flash Crash was ultimately found to have been caused by a number of things, however at its core was a $4.1 billion sale by an algorithm that did not take certain market factors into account.

High-frequency trading (HFT) algorithms – which can automatically undertake trades in microseconds – then exacerbated the problem, reacting rapidly to the initial sale with thousands more. Ultimately this created a negative feedback loop where various algorithms sold in response to other algorithms selling, pushing prices down further and further.

The Flash Crash only ended when trading in the initial contract was stopped for five seconds, allowing the market to readjust. The majority of the losses were returned and normality had returned a half hour after the initial sale created the problem.

Algorithms were also the underlying force in collateralised debt obligations (CDOs), which saw subprime mortgages fragmented and wrapped up with other debt until investors ultimately did not know what they owned.

“That kind of use of algorithms in the financial world has been blamed partially for the financial crash and I would be of that view too,” says Prof James Gleeson, chair in industrial and applied mathematics at the University of Limerick. “All models are models and they’re crude; if those models are used naively they can lead to false answers.”

Indeed very many of the contributing factors to the global economic collapse have the fingerprints of algorithms all over them, though that is not to draw a correlation between their use and problems in the world economy.

“Algorithms have always been important in the industry; the industry is growing and so their use is growing,” says Prof Gleeson.

Put another way, algorithm- and formula-based economics is now a significant, perhaps even dominant, force in many parts of the financial world. As a result they are bound to have had a role where things go bad; however, that is not to say they are dangerous by default.

Indeed there are many sectors that would simply cease to function without them, purely because of their ability to rapidly interpret and translate huge swathes of data to help inform decisions.

Far from being burned by algorithms, financial companies are increasingly seeking the help of academics to solve their problems.

Prof Gleeson’s work at UL includes involvement in the Mathematics Applications Consortium for Science and Industry (Macsi), a leading European example of this new relationship.

Every year Macsi seeks requests from companies who need mathematical help with a particular problem; it chooses six of them and spends a week brainstorming for a solution.

According to Prof Gleeson, submissions from finance-related companies have been on the rise.

“We get a variety of different problems from fluid mechanics to optimisation but we typically always have one or two from finance or finance-related companies,” he says. “When we started in 2008 it was quite difficult to sell the idea to Irish industries, I think because maths had a quite a bad reputation here . . . this year we were over-run with problems.”

In UCD Dr Conall O’Sullivan, director of the master’s programme in quantitative finance in UCD, is developing a new formula for derivatives pricing. Rather than trying to predict prices, however, he instead aims to create a system that recognises that markets do not behave like a perfect bell curve and respond accordingly.

“We’re working on numerical methods that try to make the stock or market price more realistic,” says Dr O’Sullivan. “We’re looking at capturing things like tail risk, and factoring in the fact that volatility itself changes over time.”

Dr O'Sullivan acknowledges the presence of algorithms in the market has, by definition, had an impact on them. However, he argues that there are potential benefits to their existence beyond the high-profile downsides that have been attributed to them in the recent past.

"There is no doubt that algorithms affect markets, though in theory they should make them more rational," he says. "Humans often react with emotion, sometimes they get too worried and some times they get too optimistic. Some argue they have improved liquidity too, which makes it a bit cheaper for the average trader to sell shares."

He accepts, however, that when things go wrong the way they respond can often prove inadequate, for example leading to a withdrawal of liquidity just when the market needs it most.

Put simply, the problem is that numbers and equations are very rigid, predictable things. Markets – as a result of their dependence on human behaviour – are not.

"At some point the nature of the market changes," says Prof Stewart. "A kind of ideology has grown around these models that says this is how the markets behave; that it is the rational, free market. There's no such thing."

Prof Stewart points out that one major failing of applying such scientific work to finance is that it is impossible to test properly, like you might in any other field of research.

Ultimately the actions and limitations of an algorithm cannot be truly known until it is put into action in the real world.

Coupled with that is the fact that, as algorithms become more central, they are largely responding to the decisions of other complex equations; much like what happened in the Flash Crash.

Indeed, a whole industry has now grown around the concept of identifying and undermining competing algorithms operating on the market.

For example, many algorithms are designed to spread sales of a large quantity of shares over a period of time, so as to make the sale go unnoticed and so not affect the price. In response, other algorithms have been designed to identify these sales patterns and take a market position that will generate profit from the result of that knowledge.

Speed of response is also crucial, with a five-millisecond advantage being enough to scalp other traders and win big. This is evidenced by the fact that companies are now buying office space purely so as to be close to network exchanges, while others are even investing millions in high-speed data cabling to gain a competitive edge.

All of this has the potential to disrupt markets significantly, albeit in a different way than human actions might.

It has also created entirely new methods of fraudulent and illegal market behaviour, putting regulators in a tailspin as they seek to get a hold of an area that is literally moving at an unnatural pace.

Since the Flash Crash "circuit breakers" have been installed in US markets to try to shut down trades once unusually high-volume activity is detected. Trade limits, fees and taxes are also being debated as a way of controlling the seemingly uncontrollable.

However, the desire for the algorithm remains. With demand for equations to create and counter market opportunities higher than ever, it is no surprise that the finest maths minds are now highly sought after in the financial world.

After all, anyone who can create a piece of software that gives a company a new, quicker or smarter angle to the rest would be a very lucrative asset for any company. They are also sure to be rewarded significantly for their endeavours.

"There's not many academic jobs out there so finance is a nice alternative," says Dr O'Sullivan. "The rewards are substantial so there's quite a lot of competition and these places can be difficult to get into too."

This has raised fears that a disproportionate amount of the best maths minds will be drawn to finance, risking a drying up of innovative thinking in other, traditionally attractive, areas.

Prof Stewart says there is definite potential for this to happen, but he points out that other factors could also be to blame for the shift in focus to finance. Perhaps wider economic trends and not just the algorithm are to blame for the move to finance.

"There's a lot of talk about the lack of engineers in Britain, but sometimes it's not so much that there's not being enough trained but that people who could work in the area are going somewhere else," he says. "The opportunities are not as great here anyway because of the loss of manufacturing."

Prof Gleeson also sees the trend leaning towards finance but does not see it as a worrying development.

"Fields always come along where talent gets sucked into them for a while," he says. "When the dotcom boom started most people went into software, for example."

In fact both Prof Gleeson and Prof Stewart both argue that finance needs to employ even more capable maths minds, not fewer.

Algorithms in finance are a reality now and what institutions need is a workforce that understands them, from their abilities right through to their limitations.

"I worry that if you give an equation that can be used without the person understanding what's going on you're always in danger that it will be used incorrectly," says Prof Gleeson. "There's no substitute at this stage for human intelligence used in the right way, along with experience and training."

This is a position that Warren Buffett himself would likely agree.

His warning against "geeks bearing formulas" was not one of a Luddite dreaming of simpler days gone by, it was merely an attempt to encourage caution.

As he put it himself, people had become too trusting of "a nerdy-sounding priesthood, using esoteric terms such as beta, gamma, sigma and the like."

He was right.

"Often times chief executives asked their staff to come up with models to do something and then they were too quick to trust what they were given," says Prof Stewart. "There needs to be more awareness from chief executives and boards that math recipes are not magical talismans that will turn everything into gold. Managers need to stop asking very narrow, specific questions of their maths modellers and the people doing the maths should have a stronger role in explaining what its limitations are too."