Although Britain has an established church, few of us today pay it much mind. We follow an even more powerful religion, around which we have oriented our lives: economics. Think about it. Economics offers a comprehensive doctrine with a moral code promising adherents salvation in this world; an ideology so compelling that the faithful remake whole societies to conform to its demands. It has its gnostics, mystics and magicians who conjure money out of thin air, using spells such as “derivative” or “structured investment vehicle”. And, like the old religions it has displaced, it has its prophets, reformists, moralists and above all, its high priests who uphold orthodoxy in the face of heresy.

Over time, successive economists slid into the role we had removed from the churchmen: giving us guidance on how to reach a promised land of material abundance and endless contentment. For a long time, they seemed to deliver on that promise, succeeding in a way few other religions had ever done, our incomes rising thousands of times over and delivering a cornucopia bursting with new inventions, cures and delights.

This was our heaven, and richly did we reward the economic priesthood, with status, wealth and power to shape our societies according to their vision. At the end of the 20th century, amid an economic boom that saw the western economies become richer than humanity had ever known, economics seemed to have conquered the globe. With nearly every country on the planet adhering to the same free-market playbook, and with university students flocking to do degrees in the subject, economics seemed to be attaining the goal that had eluded every other religious doctrine in history: converting the entire planet to its creed.

Yet if history teaches anything, it’s that whenever economists feel certain that they have found the holy grail of endless peace and prosperity, the end of the present regime is nigh. On the eve of the 1929 Wall Street crash, the American economist Irving Fisher advised people to go out and buy shares; in the 1960s, Keynesian economists said there would never be another recession because they had perfected the tools of demand management.

The 2008 crash was no different. Five years earlier, on 4 January 2003, the Nobel laureate Robert Lucas had delivered a triumphal presidential address to the American Economics Association. Reminding his colleagues that macroeconomics had been born in the depression precisely to try to prevent another such disaster ever recurring, he declared that he and his colleagues had reached their own end of history: “Macroeconomics in this original sense has succeeded,” he instructed the conclave. “Its central problem of depression prevention has been solved.”

No sooner do we persuade ourselves that the economic priesthood has finally broken the old curse than it comes back to haunt us all: pride always goes before a fall. Since the crash of 2008, most of us have watched our living standards decline. Meanwhile, the priesthood seemed to withdraw to the cloisters, bickering over who got it wrong. Not surprisingly, our faith in the “experts” has dissipated.

Hubris, never a particularly good thing, can be especially dangerous in economics, because its scholars don’t just observe the laws of nature; they help make them. If the government, guided by its priesthood, changes the incentive-structure of society to align with the assumption that people behave selfishly, for instance, then lo and behold, people will start to do just that. They are rewarded for doing so and penalised for doing otherwise. If you are educated to believe greed is good, then you will be more likely to live accordingly.

The hubris in economics came not from a moral failing among economists, but from a false conviction: the belief that theirs was a science. It neither is nor can be one, and has always operated more like a church. You just have to look at its history to realise that.

The American Economic Association, to which Robert Lucas gave his address, was created in 1885, just when economics was starting to define itself as a distinct discipline. At its first meeting, the association’s founders proposed a platform that declared: “The conflict of labour and capital has brought to the front a vast number of social problems whose solution is impossible without the united efforts of church, state and science.” It would be a long path from that beginning to the market evangelism of recent decades.

Yet even at that time, such social activism provoked controversy. One of the AEA’s founders, Henry Carter Adams, subsequently delivered an address at Cornell University in which he defended free speech for radicals and accused industrialists of stoking xenophobia to distract workers from their mistreatment. Unknown to him, the New York lumber king and Cornell benefactor Henry Sage was in the audience. As soon as the lecture was done, Sage stormed into the university president’s office and insisted: “This man must go; he is sapping the foundations of our society.” When Adams’s tenure was subsequently blocked, he agreed to moderate his views. Accordingly, the final draft of the AEA platform expunged the reference to laissez-faire economics as being “unsafe in politics and unsound in morals”.

So was set a pattern that has persisted to this day. Powerful political interests – which historically have included not only rich industrialists, but electorates as well – helped to shape the canon of economics, which was then enforced by its scholarly community.

Once a principle is established as orthodox, its observance is enforced in much the same way that a religious doctrine maintains its integrity: by repressing or simply eschewing heresies. In Purity and Danger, the anthropologist Mary Douglas observed the way taboos functioned to help humans impose order on a seemingly disordered, chaotic world. The premises of conventional economics haven’t functioned all that differently. Robert Lucas once noted approvingly that by the late 20th century, economics had so effectively purged itself of Keynesianism that “the audience start(ed) to whisper and giggle to one another” when anyone expressed a Keynesian idea at a seminar. Such responses served to remind practitioners of the taboos of economics: a gentle nudge to a young academic that such shibboleths might not sound so good before a tenure committee. This preoccupation with order and coherence may be less a function of the method than of its practitioners. Studies of personality traits common to various disciplines have discovered that economics, like engineering, tends to attract people with an unusually strong preference for order, and a distaste for ambiguity.

The irony is that, in its determination to make itself a science that can reach hard and fast conclusions, economics has had to dispense with scientific method at times. For starters, it rests on a set of premises about the world not as it is, but as economists would like it to be. Just as any religious service includes a profession of faith, membership in the priesthood of economics entails certain core convictions about human nature. Among other things, most economists believe that we humans are self-interested, rational, essentially individualistic, and prefer more money to less. These articles of faith are taken as self-evident. Back in the 1930s, the great economist Lionel Robbins described his profession in a way that has stood ever since as a cardinal rule for millions of economists. The field’s basic premises came from “deduction from simple assumptions reflecting very elementary facts of general experience” and as such were “as universal as the laws of mathematics or mechanics, and as little capable of ‘suspension’”.

Deducing laws from premises deemed eternal and beyond question is a time-honoured method. For thousands of years, monks in medieval monasteries built a vast corpus of scholarship doing just that, using a method perfected by Thomas Aquinas known as scholasticism. However, this is not the method used by scientists, who tend to require assumptions to be tested empirically before a theory can be built out of them.

But, economists will maintain, this is precisely what they themselves do – what sets them apart from the monks is that they must still test their hypotheses against the evidence. Well, yes, but this statement is actually more problematic than many mainstream economists may realise. Physicists resolve their debates by looking at the data, upon which they by and large agree. The data used by economists, however, is much more disputed. When, for example, Robert Lucas insisted that Eugene Fama’s efficient-markets hypothesis – which maintains that since a free market collates all available information to traders, the prices it yields can never be wrong – held true despite “a flood of criticism”, he did so with as much conviction and supporting evidence as his fellow economist Robert Shiller had mustered in rejecting the hypothesis. When the Swedish central bank had to decide who would win the 2013 Nobel prize in economics, it was torn between Shiller’s claim that markets frequently got the price wrong and Fama’s insistence that markets always got the price right. Thus it opted to split the difference and gave both men the medal – a bit of Solomonic wisdom that would have elicited howls of laughter had it been a science prize. In economic theory, very often, you believe what you want to believe – and as with any act of faith, your choice of heads or tails will as likely reflect sentimental predisposition as scientific assessment.

It’s no mystery why the data used by economists and other social scientists so rarely throws up incontestable answers: it is human data. Unlike people, subatomic particles don’t lie on opinion surveys or change their minds about things. Mindful of that difference, at his own presidential address to the American Economic Association nearly a half-century ago, another Nobel laureate, Wassily Leontief, struck a modest tone. He reminded his audience that the data used by economists differed greatly from that used by physicists or biologists. For the latter, he cautioned, “the magnitude of most parameters is practically constant”, whereas the observations in economics were constantly changing. Data sets had to be regularly updated to remain useful. Some data was just simply bad. Collecting and analysing the data requires civil servants with a high degree of skill and a good deal of time, which less economically developed countries may not have in abundance. So, for example, in 2010 alone, Ghana’s government – which probably has one of the better data-gathering capacities in Africa – recalculated its economic output by 60%. Testing your hypothesis before and after that kind of revision would lead to entirely different results.

Leontief wanted economists to spend more time getting to know their data, and less time in mathematical modelling. However, as he ruefully admitted, the trend was already going in the opposite direction. Today, the economist who wanders into a village to get a deeper sense of what the data reveals is a rare creature. Once an economic model is ready to be tested, number-crunching ends up being done largely at computers plugged into large databases. It’s not a method that fully satisfies a sceptic. For, just as you can find a quotation in the Bible that will justify almost any behaviour, you can find human data to support almost any statement you want to make about the way the world works.

That’s why ideas in economics can go in and out of fashion. The progress of science is generally linear. As new research confirms or replaces existing theories, one generation builds upon the next. Economics, however, moves in cycles. A given doctrine can rise, fall and then later rise again. That’s because economists don’t confirm their theories in quite the same way physicists do, by just looking at the evidence. Instead, much as happens with preachers who gather a congregation, a school rises by building a following – among both politicians and the wider public.

For example, Milton Friedman was one of the most influential economists of the late 20th century. But he had been around for decades before he got much of a hearing. He might well have remained a marginal figure had it not been that politicians such as Margaret Thatcher and Ronald Reagan were sold on his belief in the virtue of a free market. They sold that idea to the public, got elected, then remade society according to those designs. An economist who gets a following gets a pulpit. Although scientists, in contrast, might appeal to public opinion to boost their careers or attract research funds, outside of pseudo-sciences, they don’t win support for their theories in this way.

However, if you think describing economics as a religion debunks it, you’re wrong. We need economics. It can be – it has been – a force for tremendous good. But only if we keep its purpose in mind, and always remember what it can and can’t do.

The Irish have been known to describe their notionally Catholic land as one where a thin Christian veneer was painted over an ancient paganism. The same might be said of our own adherence to today’s neoliberal orthodoxy, which stresses individual liberty, limited government and the free market. Despite outward observance of a well-entrenched doctrine, we haven’t fully transformed into the economic animals we are meant to be. Like the Christian who attends church but doesn’t always keep the commandments, we behave as economic theory predicts only when it suits us. Contrary to the tenets of orthodox economists, contemporary research suggests that, rather than seeking always to maximise our personal gain, humans still remain reasonably altruistic and selfless. Nor is it clear that the endless accumulation of wealth always makes us happier. And when we do make decisions, especially those to do with matters of principle, we seem not to engage in the sort of rational “utility-maximizing” calculus that orthodox economic models take as a given. The truth is, in much of our daily life we don’t fit the model all that well.

For decades, neoliberal evangelists replied to such objections by saying it was incumbent on us all to adapt to the model, which was held to be immutable – one recalls Bill Clinton’s depiction of neoliberal globalisation, for instance, as a “force of nature”. And yet, in the wake of the 2008 financial crisis and the consequent recession, there has been a turn against globalisation across much of the west. More broadly, there has been a wide repudiation of the “experts”, most notably in the 2016 US election and Brexit referendum.

It would be tempting for anyone who belongs to the “expert” class, and to the priesthood of economics, to dismiss such behaviour as a clash between faith and facts, in which the facts are bound to win in the end. In truth, the clash was between two rival faiths – in effect, two distinct moral tales. So enamoured had the so-called experts become with their scientific authority that they blinded themselves to the fact that their own narrative of scientific progress was embedded in a moral tale. It happened to be a narrative that had a happy ending for those who told it, for it perpetuated the story of their own relatively comfortable position as the reward of life in a meritocratic society that blessed people for their skills and flexibility. That narrative made no room for the losers of this order, whose resentments were derided as being a reflection of their boorish and retrograde character – which is to say, their fundamental vice. The best this moral tale could offer everyone else was incremental adaptation to an order whose caste system had become calcified. For an audience yearning for a happy ending, this was bound to be a tale of woe.

The failure of this grand narrative is not, however, a reason for students of economics to dispense with narratives altogether. Narratives will remain an inescapable part of the human sciences for the simple reason that they are inescapable for humans. It’s funny that so few economists get this, because businesses do. As the Nobel laureates George Akerlof and Robert Shiller write in their recent book, Phishing for Phools, marketers use them all the time, weaving stories in the hopes that we will place ourselves in them and be persuaded to buy what they are selling. Akerlof and Shiller contend that the idea that free markets work perfectly, and the idea that big government is the cause of so many of our problems, are part of a story that is actually misleading people into adjusting their behaviour in order to fit the plot. They thus believe storytelling is a “new variable” for economics, since “the mental frames that underlie people’s decisions” are shaped by the stories they tell themselves.

Economists arguably do their best work when they take the stories we have given them, and advise us on how we can help them to come true. Such agnosticism demands a humility that was lacking in economic orthodoxy in recent years. Nevertheless, economists don’t have to abandon their traditions if they are to overcome the failings of a narrative that has been rejected. Rather they can look within their own history to find a method that avoids the evangelical certainty of orthodoxy.

In his 1971 presidential address to the American Economic Association, Wassily Leontief counselled against the dangers of self-satisfaction. He noted that although economics was starting to ride “the crest of intellectual respectability … an uneasy feeling about the present state of our discipline has been growing in some of us who have watched its unprecedented development over the last three decades”.

Noting that pure theory was making economics more remote from day-to-day reality, he said the problem lay in “the palpable inadequacy of the scientific means” of using mathematical approaches to address mundane concerns. So much time went into model-construction that the assumptions on which the models were based became an afterthought. “But,” he warned – a warning that the sub-prime boom’s fascination with mathematical models, and the bust’s subsequent revelation of their flaws, now reveals to have been prophetic – “it is precisely the empirical validity of these assumptions on which the usefulness of the entire exercise depends.”

Leontief thought that economics departments were increasingly hiring and promoting young economists who wanted to build pure models with little empirical relevance. Even when they did empirical analysis, Leontief said economists seldom took any interest in the meaning or value of their data. He thus called for economists to explore their assumptions and data by conducting social, demographic and anthropological work, and said economics needed to work more closely with other disciplines.

Leontief’s call for humility some 40 years ago stands as a reminder that the same religions that can speak up for human freedom and dignity when in opposition, can become obsessed with their rightness and the need to purge others of their wickedness once they attain power. When the church retains its distance from power, and a modest expectation about what it can achieve, it can stir our minds to envision new possibilities and even new worlds. Once economists apply this kind of sceptical scientific method to a human realm in which ultimate reality may never be fully discernible, they will probably find themselves retreating from dogmatism in their claims.

Paradoxically, therefore, as economics becomes more truly scientific, it will become less of a science. Acknowledging these limitations will free it to serve us once more.

Main image: Maxian/Getty/iStockphoto/Guardian Design

This is an edited extract from Twilight of the Money Gods: Economics as a Religion and How it all Went Wrong by John Rapley, published by Simon & Schuster on 13 July at £20. To order a copy for £17, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.