Book Review: John Authersenjoys an enthusiastic study of the academics whose theories have revolutionised global markets, but whose forays into the world of making money have been rocky

When Fischer Black, a Massachusetts Institute of Technology academic, arrived in Manhattan to work for Goldman Sachs Asset Management, he complained that "the market appears a lot more efficient on the banks of the Charles River than it does on the banks of the Hudson".

Black, an inventor of the Black-Scholes options pricing theorem, which opened the way to a wave of new capital markets, was at the beginning of an attempt to apply revolutionary insights from financial academics to the real world task of making money. Like many others, he found an often yawning gap between theory and practice.

That gap has now been bridged. The quotation comes from Peter Bernstein's Capital Ideas Evolving, and it tells the story of how a group of brilliant academics, several of them Nobel laureates, have transformed the worlds of capital markets and fund management.

This may sound like unbearably dry fare, but it is not. Bernstein, now 88, has spent a lifetime in the investment industry. His enthusiasm is undimmed, and he knows and has interviewed all the main academics involved and the fund managers who tried to implement their ideas. His book, which runs to less than 300 pages, often functions as an improbably gripping narrative. It is also a great primer in the ideas that currently govern the way the world's money is invested.

"Capital ideas" is Bernstein's shorthand for four academic breakthroughs. These are the concept of balancing risk and return through tracking the mean and variance of market prices; the Capital Asset Pricing Model, which divides the return on a security into "beta" of sensitivity to the market and "alpha", an uncorrelated return; the efficient market or "random walk" theory that holds that the market always attempts to incorporate all available information; and the Black-Scholes theory, which makes it possible to put a notional "fair value" on an option.

All these ideas emerged between 1952 and 1973, but have revolutionised investment only in the past decade. It is now understood that "beta" is a commodity that can be bought cheaply through passive index funds. It can thus be separated from "alpha" - returns not correlated to the market.

The efficient market theory predicts, correctly, that alpha will be very elusive. But the use of options and other derivatives have opened new avenues for discovering it. Private markets will be less efficient than public ones. So "capital ideas" have underpinned the vogue for private equity and for big institutions to explore alternative assets, from hedge funds to real estate.

Many of the academics who made theoretical breakthroughs have sought to apply them. This has had hitches. Most famously, Myron Scholes and Robert Merton, progenitors with Black of the Black-Scholes theorem, were partners in Long-Term Capital Management, the hedge fund whose 1998 near-meltdown briefly endangered the world financial system. For many, this was a story of hubris, as academics saw their theories disproved before their eyes.

This was always overdone. The meltdown did not disprove the theorem any more than the meltdown at Chernobyl disproved basic tenets of nuclear physics. But it did reveal that understanding of how to manage risk in the world unleashed by the "capital ideas" was badly limited. In combination with the bursting of the technology bubble two years later, it also gave ammunition to the new school of behavioural finance. Behaviouralists substitute insights from experimental psychology for the assumption of traditional economists that people act rationally. Diagnosing systematic flaws in human reasoning allowed them successfully to predict many cases of market inefficiency.

How does Bernstein resolve the tension between efficient markets theory and behavioural finance? He believes the behaviouralists are doing God's work, ensuring that the market evolves to be ever more efficient: greater understanding of these inefficiencies has led people aware of those insights to take advantage of them, earning "alpha" and making the market more efficient in the process.

"Every form of alpha pursued by sharks with insatiable appetites has a short half-life," he says. "Why should the markets not reach a point where the sharks have consumed everything in sight? Then we would end up with a fully efficient market where the trade-off between risk and return is precisely aligned. We would have attained . . . nirvana."

Once in nirvana, the paradox is that nobody would have an incentive to trade. That means that "the sharks, sensing a meal of unlimited abundance, would swarm back into the markets as fast as they could travel".

All good reasons for confidence that Bernstein's "capital ideas" will continue to evolve, and that the markets will look as efficient from the skyscrapers along the Hudson as from the ivory towers on the Charles.

• Capital Ideas Evolving By Peter L Bernstein Wiley. €24