When the Berlin Wall fell, it marked the end of perhaps the most extensive controlled experiment ever undertaken in economics. One country, divided into two zones: one organised as the archetype of the social market economy, the other on the basis of central planning within a controlled trading bloc. After forty years, the difference in living standards were not just appreciable, but very large – perhaps a factor of two or three. This experience is repeated even more starkly if one compares the post war economy development of Estonia and Finland, or – in the most bizarre comparison of all – North and South Korea.

That outcome did not seem inevitable before it was observed. In the advanced economies of the West, increased government intervention in the economy was more or less unchecked through the twentieth century. In the 1960s most European governments adopted national economic plans and it was generally assumed that the route to successful development in decolonised states lay through state direction. These trends were first halted and then reversed after 1980 by the ideologically conservative governments of Reagan and Thatcher. Their policy innovations were widely if often reluctantly imitated elsewhere. Eastern Europe moved, slowly at first, towards freer markets, and then changed rapidly as Russian influence collapsed. In Asia, China and India followed some of their smaller neighbours into the market economy and the global trading system.

These developments provoked the hubris famously, or perhaps notoriously, framed as The End of History (by Francis Fukuyama), who argued that a combination of liberal democracy and lightly regulated capitalism was now the inevitable form of political and economic organisation. If one country was the standard bearer for that vision of the twenty-first century, it was the United States: if one industry was the standard bearer for that vision, it was the financial services industry.

The term ‘market efficiency’ is used in both a broad and a narrow sense. The broad sense concerns the efficiency of markets as an economic system. A debate about this concept poses the largest economic question of the twentieth century – why did market economies so dramatically outperform planned societies?

But ‘market efficiency’ is also used in a narrower, technical sense. In financial markets, the efficient market hypothesis, which has strong, semi-strong and weak forms – the differences will not be important here – asserts that all information is incorporated in market prices.

There is a relationship between these notions of market efficiency, but they are distinct. It is possible to believe that markets are efficient in a broad sense – that the market economy is in practice superior to feasible alternative economic arrangements – without believing that financial markets are efficient in a narrow sense. And it is also possible to believe in at least weak forms of financial market efficiency – that it is in the nature of securities markets to reflect public information – without feeling committed to belief in the necessity of the capitalist organisation of production.

But recent events have shaken faith in both concepts of market efficiency. And the doubts about the validity of both broad and narrow concepts of market efficiency are related. The evident failures of financial markets have, for the first time in a generation, called into question the apparent triumph of the market economy.

The end of history no longer looks plausible, even to Fukuyama himself.

* * *

There are three elements in the success of the market economy. The first I will describe under the heading of ‘prices as signals’: the price mechanism is generally a better guide to resource allocation than central planning. The second element is ‘markets as a process of discovery’: a chaotic process of experimentation is the means through which a market economy adapts to change. The third heading is ‘diffusion of political and economic power’. The economic point here is that prosperity and growth require that entrepreneurial energy should be focussed on the creation of new wealth, rather than the appropriation of the existing wealth of other people.

In what we teach, in what we say, in our economic research and most importantly in the policies we adopt – we put too much emphasis on the first of these elements over the, possibly more important, second and third elements. We stress prices as signals to guide resource allocation at the expect of markets as process of discovery, and markets as mechanism for the diffusion of political and economic power.

The model of ‘prices as signals’ describes how self-interested agents – individuals or firms – might, through independent decisions, make consistent and efficient choices about how to organise production and distribution and the allocation of capital, labour and other resources. In a loose formulation, this idea has been around since the beginnings of economics. Many people attribute it to Adam Smith, and it is a possible interpretation of his (recently) famous remark about ‘the invisible hand’, and his observation that it was not the benevolence of the baker, but his self-love, that furnishes our table.

In an astonishing demonstration of the power of spontaneous order, decentralised markets manage the process of coordinating complex production systems better than centralised direction. Although it appears to be an empirical fact that markets achieve this, economists did not offer a comprehensive explanation of how such coordination was possible until the 1950s. Such an account had to await the modern application of mathematics to economics.

Powerful mathematical tools were needed to provide a theoretical demonstration, both that a competitive equilibrium might exist – that the price mechanism could coordinate supplies are demands in ways that would eliminate surpluses or deficits – and that, if such an equilibrium did exist, that equilibrium might be efficient. That general equilibrium model (concisely ‘the model’) proved influential in shaping the research agenda of the economic profession. The assertive self confidence in the economics profession that followed would provide an intellectual basis for economic policy among people who knew nothing of the abstract underlying arguments.

An implication of ‘the model’ is that profitable transactions are normally socially beneficial: indeed that their social benefit is demonstrated by their profitability. A corollary is the ‘market failure doctrine’, which is today central to the formulation of economic policy in treasuries and international agencies. Intervention in markets is justifiable by reference to a narrowly defined list of market failures based on potential deviations between assumptions of the model and the world. The terms of market failure is revealing. Deviations between empirical fact and model assumptions are considered deficiencies in the world, not deficiencies of the model.

The model provides a rationale for a certain kind of market fundamentalism. Not only is interference with market forces usually inappropriate, but market outcomes are efficient, even morally justifiable, simply by virtue of being market outcomes. Not only are markets good, but more markets are better than fewer markets. The emergence of new markets is presumptively beneficial and, in general, the more trading there is in markets, the better.

Among economists, the popularity of this approach is often attributed to physics envy: the model provides a universal explanation of economic affairs which resembles in many ways the equilibrium models that have proved so powerful in the natural sciences. Rigour has become the measure of the quality of a theoretical economic argument, where rigour means the logical consistency which readily finds mathematical expression. Most people who are not economists would be startled by the degree to which such rigour trumps empirical relevance in peer review of economic research. But this is the explanation of why modern macroeconomics has so little to say about the current economic conjuncture, and why arguments over economic policy reprise so substantially the similar debates of the 1930s.

But these are professional disputes. Among practical people, largely indifferent to the content though not necessarily to the existence of these theoretical arguments, the simple message that government should go away and leave business alone has wide appeal to business men and women. The simple message that greed can serve a constructive social role has equally wide appeal to greedy people. The claim that profitability demonstrates, is even the measure of, public benefit relieves those who benefit from such profits of any worries they might have harboured about the utility of their activities. These worries are not, it should be acknowledged, common.

These simple messages are, however, easily resistible to intellectuals who are not economists. These simply messages are also resistible among the population at large, which does not run business, which benefits only indirectly from the activities of business, and which is not enamoured of greed. Most voters, and European political parties of the left, today acknowledge the empirical success of the market, but dislike almost every aspect of that market. ‘The market’ and ‘market forces’ are the source of our prosperity, but these phrases are also terms of abuse. The reason is that we have succeeded in providing a description of how markets work that is at once repulsive and substantially false.

* * *

‘The model’ probably contributes something to our understanding of how markets work. But that contribution is largely misunderstood and grossly over-emphasised. One fundamental problem is that there is no real acknowledgement of uncertainty in the model. Or, to be more precise, uncertainty is acknowledged only in essentially formal ways. This omission is of fundamental importance when the model is used to describe financial markets, in which trading in risk is the essence of the transaction.

In such markets, the means of incorporating uncertainty into the model requires, in effect, that there is some true underlying value of an asset; that this value is independent of beliefs about the value of the asset: and that market transactions involve a process of convergence towards the true value. This is the mechanism that establishes the connection within the model between market efficiency in the broad sense, and market efficiency in the narrow sense.

But experience has demonstrated clearly that this process is a hopelessly inadequate account of market behaviour. The world is uncertain: not just risky, but uncertain, in the sense used by Keynes and Knight. Not only do we not know which future outcomes will happen: we are unable to specify at all fully what these possible outcomes will be. If we could predict or anticipate the invention of the wheel, we would have already invented it. Market economies do not predict the future, they explore it.

This feature is the second, and perhaps most important, element in the success of the market economy. Hayek continues to be the most eloquent exposition of the concept of market as process of discovery. Hayek’s argument was a priori , but vindicated by the failures of the eastern bloc in the post-war era. These planned economies failed in the development, not just in the development of consumer products, but in pioneering new business methods. Their technological achievements were disappointing in almost all industrial activities not related to military hardware.

Anyone who has spent time in any large bureaucracy appreciates why. Centralised systems experiment too little. Committees find reasons why new proposals are likely to fail – and mostly they are right in finding reasons why they will fail, because most experiments do fail. Market economies thrive on a continued supply of unreasonable optimism which is occasionally vindicated. When the experiments of entrepreneurs do succeed, the processes of the market economy ensure that they are quickly imitated. It is a striking feature of the market economy that even among those innovations that are commercially successful, few are commercially successful for the innovator.

If market economies are better than planned societies at the origination and diffusion of new ideas, they are also better at disposing of failed ideas. Honest feedback is rarely welcomed in large organisations. In authoritarian regimes, such feedback can be fatal to the person who delivers it. In less draconian contexts, unwanted messages are simply fatal to careers. And when I talk about large bureaucracies here, I am talking just as much about large private bureaucracies as large public ones. Disruptive innovations most often come to market through new entrants – from Google, EasyJet, Amazon. Incumbents have good reasons to be suspicious of novelty and protective of their established markets and activities.

The health of the market economy depends, therefore, on constant replenishment of the business sector by new entry. If you had been planning the development of the computer industry in the 1970s, would you have asked Bill Gates and Paul Allen for their advice? If you had been planning the future of aviation in the 1980s, would you have sat on a committee with Stelios Haji-Ioannou? If, as planner or sponsoring department, you had been planning the future of retailing in the 1990s would you have thought of consulting Jeff Bezos? Of course not: whether you were a member of the politburo or the permanent secretary of a government department, you would have talked to men in suits like yourself.

The men in suits watch their children and grandchildren and acknowledge that the future lies in their hands. Committees of the middle-aged Twitter about technology like embarrassing adults trying to have fun at the teenagers’ disco. But, like those adults at the party, they are not really serious or competent in their observation. Yesterday’s teenage fad is tomorrow’s failed acquisition, as established businesses trade half the historic legacy of Time Warner for AOL or lose millions in My Space.

These occasionally costly frivolities are distractions from the reality of regulatory capture and the power of conventional wisdom. Whether planners or governments of a market economy, we see industries through the eyes of established firms in the industry. And in doing so. miss the pluralism that is the market economy’s central dynamic.

* * *

That leads directly to the third group of reasons for the superior performance of market economies. If I were to offer a one sentence description of why some countries are poor and others rich, it would be that the politics and economics of poor countries are dominated by rent-seeking and the politics and economics of rich countries are not.

Rent seeking is the process by which ambitious people find it more rewarding to batten on the wealth created by others than to create it themselves. Rent seeking takes, and has taken, many forms – castles on the Rhine, the Wars of the Roses; ten per cent on arms sales, or seven per cent on new issues: awarding yourself control over former state assets, stealing the revenues from the sale of your country’s resources: seeking protection from foreign competition, blocking market access by new entrants; winning sinecures or overpaid positions by ingratiating oneself with public servants or corporate employees. The mechanisms of rent-seeking range from the application of armed force to victory in democratic election; the methods pursued involve lobbying on Capitol Hill and in the restaurants of Brussels. The route to rents once took you to the royal court but now more often leads through the C-Suite.

There is a strong tendency for private concentration of economic power to be self-reinforcing. In America’s ‘gilded age’ the well-founded fear was that the new mega-rich – the Rockefellers, the Carnegies, the Vanderbilts – would use their wealth to enhance their political influence and hence enhance their economic power still further, subverting both the market economy and the democratic process. These concerns were the origin of anti-trust legislation, a point today often forgotten. The process that concerned Americans then is the problem we see in Russia – and elsewhere in the world – today.

But while rent seeking is ineradicable, we can have more or less of it. Politics everywhere used to be dominated by rent seeking; factions would battle for control of the state and when they won such control would use it to steal as much as they could get their hands on. In much of the world, it is like that still. It’s Our Turn to Eat is the stomach churning title of one recent book about Kenya’s corrupt though not entirely democratic politics.

The resource curse – wealth from national resources does more harm than good in many countries because of the rent-seeking it attracts – is widely recognised. Foreign aid may have some of the same characteristics. The ability of a political and economic system to resist rent seeking depends on the degree of economic decentralisation. Individuals will try to get their hands on the rents concentrations of power attract wherever they are found; in the public sector, in private businesses, or in groups of private business.

The ability of a market economy to restrict rent-seeking, and its capacity to channel the desire for acquisition into channels that create wealth rather than extract it depend on measures both to prevent the concentration of economic power and to limit the terms of access to such concentration. These are constraints on the economic power of the state: constraint on the concentration of economic power in large businesses: constant vigilance at the boundaries between the state and business: and a mixture of external supervision and internal restraint which prevents individuals who pull levers of economic power from using these levers to direct rents to themselves.

* * *

In the last two decades, we have emphasised the first of these concerns – reducing the economic role of the state. Under each of the other headings – limiting the concentration of economic power, policing the boundaries between state and market, and policing a culture which emphasises public service over personal aggrandisement in both public sector and private business – we have been moving more or less rapidly in the opposite direction.

The reasons are many, but I will focus on two elements in that explanation – the intellectual climate created by economic thinking centred on ‘the model’, and the associated celebration of the growth of the financial services industry. The economic analysis that influences policy centres round ‘prices as signals’.

Trade in markets provides valuable price information. From this feature of the model it follows that the more markets the better, and the more trade that occurs in them the better. In policy towards financial markets, it is often sufficient argument for the benefit of an activity that this activity increases liquidity: more trade is desirable simply because it stimulates more trade.

The view that not only is trade good but more trade is always better is not complete nonsense, but it is facile and misleading. As Joel Mokyr explains in his paper to this conference, one of the great insights of eighteen century economics was that trade is not necessarily a zero sum game in which one side gains only what the other must lose. Exchange can benefit both parties to a transaction. The mercantilist view – in which trade is a mechanism by which one party tricks the other into giving up something valuable for inadequate recompense – has disappeared from economic theory if not popular discourse.

But the possibility that trade might be mutually beneficial does not mean that all trade is actually beneficial. There are many different explanations of why exchange takes place, and correspondingly many different types of trade. Gains from trade are typically the product of differences in capabilities, or preferences. Rembrandt is a skilled painter while I am a knowledgeable economist. Rembrandt gives me a picture in return for economic expertise. (If you think this is not a good deal for Rembrandt, you probably do not realise how inept Rembrandt was in managing his financial affairs). Or I admire Rembrandt, but you prefer Picasso: if my collection includes Picassos, and yours Rembrandts, then exchanges between our collections may make us both better off.

Any accountant will immediately point out a complication. Although the exchange enhances both collections, the list of paintings we jointly hold remains identical, and an appraisal will not register any increase in value. The change is in our perception of value, rather than the market value. The perceived value of a painting, or any other items, will normally not be less than its market value – if it were less, we would sell it. But it might be more. This happens often. We say ‘I got a bargain’, or these shares are selling for less than I think they’re worth.

Where objects are idiosyncratic – like paintings and homes – or where there is uncertainty – as over the value of shares – there may be wide differences in individual perceptions of value. Exchange may be the result of different perceptions of the merits of an idiosyncratic object – we both agree that a painting is a fine example of Rembrandt’s work but I like it and you don’t. But the presence of uncertainty establishes a quite different opportunity for apparently beneficial exchange. There is considerable dispute about which of the many works of the school of Rembrandt were actually painted by the master himself. I think the painting is probably a genuine Rembrandt, and you are not so sure.

In both cases, the trade in which I buy the Rembrandt from you seems a good deal for both parties. But the second case, the result of uncertainty, is fundamentally different from the first, the result of idiosyncrasy. Uncertainty about the future, although pervasive, is capable of resolution, because the future arrives. The painting is either a genuine Rembrandt, or it is not. One day, we might find out. And when that happens, we will realise that the deal was not in fact good for both parties: one gained, the other lost.

There is a paradox here. Ignorance may create value, by permitting us to sustain our different beliefs: better information may actually destroy value. If the issue is the authenticity of a Rembrandt, the truth may never emerge. But if the issue is the earnings potential of a company, or the value of a complex debt security, the discrepancy created by divergent valuations based on divergent perceptions will eventually be eliminated. Either the business is profitable, or it is not: borrowers ultimately pay what they owe, or they don’t.

The example focuses on extreme cases. Uncertainty and idiosyncrasy are part of life, and often associated. We are more likely to believe that a painting is a genuine Rembrandt if it is a painting we like. Most economic transactions in a complex modern society have elements of both uncertainty and idiosyncrasy about them. The textbook trades of economics 101, in which homogeneous pears are exchanged for homogeneous apples, are infrequent. We buy, not applies, but Class 1 Granny Smith apples from Tesco. Or not.

You might expect that mistakes of assessment would average out. Sometimes we overestimate, sometimes we underestimate. But in markets mistakes of assessment do not average out. Not just because of the university human tendency to overconfidence in our judgments; but also because it is a lot easier to sell fake Rembrandts to someone who believes they are genuine than to sell genuine Rembrandts to someone who suspects they are fake.

Securities are therefore much more likely to be held by people who overestimate their value than by people who underestimate that value. Wherever there is uncertainty, market prices reflect the optimism of those who are more than averagely sanguine about the future. John Kenneth Galbraith famously elaborated the concept of the bezzle – the amount by which the world is better off when a fraudster has stolen money but the people from whom he has stolen it have not yet discovered the theft. Bernie Madoff’s achievement was to create the largest bezzle in history: the $50 billion that he claimed to be safeguarding for his investors appeared as real as any other financial asset until the moment at which he turned himself in to the FBI.

The concept of the bezzle – the illusory value created by divergent and at least partially mistaken perceptions of value – is a good deal wider the Galbraith’s conception implies. Overvaluation does not require fraud: it is in the nature of markets. At any time, there will be a reservoir of overvaluation, the bezzle generated by fraud and mistake. The reservoir is constantly depleted by events – the discovery that Rembrandt’s are indeed fakes, and constantly replenished by new uncertainties, some of them intrinsic to the world, some of them created by agents who seek to bamboozle us with complex products. We do not know how much of the wealth apparently created in the last two decades had this illusory character. It is evident from the crash of 2007-8 that a significant part of it did.

The distinction between trade based on differences in preferences and trade based on differences in beliefs is key to understanding the developments in financial markets that led most directly to that recent crisis. There were always two views of the nature of the profits generated through complex securitisation. In one – differences in preferences – the process enabled instruments to be tailored to the precise risk appetites of each individual holder. In the other view, which emphasised differences in beliefs – the main driver of trade was that some traders were willing to pay more for instruments who value they did not properly understand than those instruments were actually worth.

The denouement leaves little doubt that it was the latter view based on divergent belief, not the former, based on divergent preference, that was generally correct. Trade was the result, not of differences in risk appetite, but on differences in perception. The process of developing complex instruments created a reserve of illusory value. A large part of the return from that illusory value was distributed to the employees of financial services businesses. The illusory value was eliminated when the low quality of the basic lending that underpinned the securities was revealed. The outcome was the result of market inefficiency, not market efficiency.

* * *

So where does this leave the efficient market hypothesis, in either a broad or narrow sense? We have come a long way from the finance theory of the 1970s, a time when Michael Jensen could write ‘there is no other proposition in economics which has more solid empirical evidence supporting it than the Efficient Markets Hypothesis’. More relevant today is Warren Buffett: ‘‘Observing correctly that the market was frequently efficient, they (the academics – and many investment professionals and corporate managers) went on to conclude incorrectly that it was always efficient. The difference between these propositions is night and day.’

Buffett’s observation raises important issues, not just about the operation of financial markets, but about the nature of explanation in economics. The efficient market hypothesis – in either the broad or the narrow sense – is not refuted. It remains true that market economies outperformed planned societies. It is also true that securities prices reflect publicly available information.

What is not true is that either of these claims have universal validity – that market outcomes are necessarily efficient by virtue of the simple fact of being market outcomes, that securities prices are always the best possible estimate of the fundamental value of traded assets. The mistake – an intellectual mistake on the part of economists interested in policy, a convenient elision on the part of political ideologues – is to translate general tendencies into propositions accorded the status of established scientific laws.

The test of an economic theory is not whether it is true, but whether it is useful. It is a mistake to believe that the efficient market hypothesis is true, but equally a mistake to believe that it is false. Market efficiency is an instructive concept so long as we do not make the mistake of according it exaggerated deference. We should heed the words of Keynes:

‘Professor [Max] Planck, of Berlin, the famous originator of the Quantum Theory, once remarked to me that in early life he had thought of studying economics, but had found it too difficult! Professor Planck could easily master the whole corpus of mathematical economics in a few days. He did not mean that! But the amalgam of logic and intuition and the wide knowledge of facts, most of which are not precise, which is required for economic interpretation in its highest form is, quite truly, overwhelmingly difficult for those whose gift mainly consists in the power to imagine and pursue to their furthest points the implications and prior conditions of comparatively simple facts which are known with a high degree of precision.

— John Maynard Keynes ’Alfred Marshall: 1842-1924’ (1924). In Geoffrey Keynes (ed.), Essays in Biography (1933)