Our financial institutions rather desperately need some sort of help. The core of many of our problems is that our institutions are increasingly unable to keep up with the accelerating rate of technological and social change. Moreover, the rate of change is likely to continue to accelerate as people have become more and more connected hrough digital communications, travel and trade. This dynamic is especially acute in the context of climate change, which forces us to consider costly actions to mitigate future risks that are both massive but also fluid and uncertain. This reality is one that the global financial system is confronting only slowly, and in fits and starts.
One reason our institutions are failing is because they were built using 18th- and 19th-century views about human behavior and social structure that imagined people as strictly rational, and strictly individual, actors. Inspired by inventions such as steam engines and spring-powered clocks, many believed at the time that human society could be understood as a huge mechanical system, ticking along like some eternal, all-encompassing clockwork. In addition, the popular view derived from theories by thinkers such as Adam Smith and Charles Darwin was that competition was the clock spring that drove both economic and political progress.
However, as Adam Smith explained in his often-neglected book, “Theory of Moral Sentiments,” society is not just about competition, but also about collaboration within communities. “People also cooperate and learn from each other in order to innovate, and it this trading of favors and ideas that makes society responsive to individual needs and capabilities,” Smith wrote. Our current social institutions, unfortunately, largely ignore the patterns of how people connect with other people and deploy systems and technology that treat people as if they make decisions completely independently of each other.
Today’s science tells us that it is in fact the person-to-person trading of favors, stories and collaborative effort that creates innovation and growth. Importantly, the recombinant idea-sharing process that develops new additions to our common knowledge is a far more powerful search process than checking the possibilities suggested by deduction from current knowledge. Just like biological evolution, it can cover a much larger space of possibilities. The ability to explore “edge cases” or counter-intuitive possibilities is likely key to long-range success.
Optimizing decision-making
As I explore in my forthcoming book “Shared Wisdom” (MIT Press), there is a deeper truth that suggests that these “irrational” patterns of behavior are not accidents or mistakes. Instead of thinking of people as individuals who are just trying to reason things out, it is more accurate to think of human behavior as a collective search process where groups of agents continually search for new opportunities and new, more effective behaviors — quite often without fully understanding why those behaviors are effective.
In unknown or uncertain environments, there is always the question of how much effort to spend on familiar daily activities, and how much effort to spend on finding new, better ways of living. Done correctly, a balance between those pursuits allows both the individual and the community to achieve optimal “minimum-regret” decision-making. The criterion of minimum regret means that an agent or group of agents makes the best possible decision at each instant, given the information and previous experience available at the time. In other words, it is doing the best you can do.
Classic minimum-regret decision-making is often referred to as a “bandit problem,” because of the formal equivalence to the question of which slot machine (sometimes known colloquially as “one-armed bandits”) a gambler ought to try in a gambling casino. In the last decade, the mathematical solution to such problems has been extended to distributed agents: for example, a gambler observes the payouts to other casino patrons and combines those observations with personal knowledge to decide which slot machine to try next.
The importance of these mathematical results is that they provide a strategy that a group of agents may use to form an optimal minimum-regret policy for action. The core of this strategy is for each agent to use a store of previously observed strategies to estimate the prior probability of success for a potential action. Next, the agents use this prior probability to moderate the probability of success as learned from personal experience, which then determines the posterior probability of success for the potential action. The agents then typically spread their resources among the top few actions most likely to work.
Because of the need to minimize uncertainty and maximize reward, they must also learn more about their environment. This means that they must experiment with actions that have uncertain reward and even try actions that are unfamiliar. This trade-off is known as the exploit-explore dilemma — known strategies should be used for surviving, but the search for better strategies is needed for continued thriving.
This strategy for optimal minimum-regret decision making has demonstrated excellent performance in many domains, enough so that it is a standard approach in domains such as signal processing, medical decision making and finance. It shows good generalization to new and changing situations, and the ability to work with noisy, long-tailed and ill-conditioned data inputs. If the agent is smart enough to be capable of a little mental reflection, it can also handle situations where some agents have different — and even adversarial — goals. Agents decide who they should trust to be part of their learning (or “story sharing”) network by comparing the choices they have made to the choices of the other agents, and from that comparison select the best subset of agents with whom to trade stories.
But what about the “rational agents” on which almost all current economics and finance theory is based? Are experts who are trained in the mathematics of finance and have decades of experience true rational individuals, rather than the social explorers? To answer this question of how experts who participate in a sharing network perform versus experts who make decisions alone, my colleagues and I conducted an experiment with 1,700 mid-career financial advisors. These were people who handle millions of dollars every day and work in financial centers around the globe.
To test the performance and reliability of expert networks, we asked each of these financial experts to forecast the future price of various assets and gave them the opportunity to be part of a community consisting of all the experts, so that they could see summaries of how similar other experts were investing. In general, all the experts performed well (as one might expect for such highly trained and highly paid people). However, experts who were not swayed by what others were doing and made their decisions by relying only on data frequently did very slightly better than those who changed their investments based on what other experts were doing.
But those more independent experts, while more frequently making a bit more money, also more frequently made serious mistakes. Because of their mistakes, in the long run, the experts who were completely quantitative in their decision-making and uninfluenced by the community of other experts actually did worse than those who took the opinions of others into account.
The importance of being part of the expert community was nicely illustrated by one phase of our experiment, which spanned the Brexit period when the United Kingdom left the European Union. During that period, we found that the more isolated experts suffered big losses, while the sharing network experts did fairly well. From the point of view of long-term financial returns, the difference was huge. The otherwise slightly better returns of the isolated experts were completely drowned by losses from their mistaken Brexit investment strategy. Experts need some sort of shared wisdom — sometimes called “the street” in finance communities — to make reliably good decisions.
Synergistic human-AI decision-making
Problems that come from neglecting person-to-person interactions happen so frequently that I call them the “four horsemen of failure.”
Cascades of coordinated behaviors, my first horseman of failure, can be illustrated in the in the spectacular collapse of Silicon Valley Bank in 2023. Here, rumors on social media caused a digital run on the bank. The spread of digital media, together with digital funds transfer, meant that rumors of problems at the bank could cause coordinated movement of large amounts of money by big depositors.
A second horseman of failure is the inability of people to account for inevitable but rare events (“grey swans”), such as what caused the 2008 financial crash, where supposedly safe financial instruments failed to consider the possibility that there would be a spike in mortgage defaults in almost every major city at the same time.
My third horseman is non-stationarity, meaning that we overlook change in our environment. Too often we assume our environment to be constant, and as a consequence fail to notice critical changes as they’re happening. The closely related fourth horseman is the problem of myopia: important factors we have neglected because they are difficult to measure or thought to be inconsequential. A good example of this problem of myopia can be found in today’s high-speed stock market trading, where these trading platforms suffer market crashes almost daily because algorithms too often fail to take into account the possibility of liquidity constraints.
Understanding this model of human decision making as a collaborative search can tell us what sort of artificial intelligence tools will likely work best. We don’t want to remove the core processes of human decision-making — the use of social learning, methods for finding consensus and our peculiar way of choosing actions — because the quirks that we often think of as bias or ignorance are often mechanisms for more fully exploring risks and opportunities. In this sense, AI can be quite helpful in supporting human social learning, consensus decision making and action selection with useful information. AI can provide the support we need to act successfully in our increasingly complex world. For instance, even simple machine learning AI programs can provide feedback about what everyone else in the community is doing, helping recreate the kind of collaborative search process that is more effective.
‘The person-to-person trading of favors, stories and collaborative effort is what creates innovation and growth’
Opportunities to apply AI in the domain of sustainable finance range from identifying green investments and processing environmental data to forecasting changing weather patterns and the risks they pose. More will surely emerge in the future. Whatever applications do prevail will be most effective by integrating our uniquely human capacities.
Historically, the best defense against the “four horsemen” errors has been many minds and many opinions working to detect otherwise overlooked changes, to detect exceptions from the general rule, to remind people about the possibility of rare events and try to notice new factors at play. Fortunately, these are all things that generative artificial intelligence appears to be particularly good at doing, and so there’s good reason to believe that human decision making supported by AI would generally work quite well. Sustainable finance needs AI, but it should be the kind that supports rather than replaces human decision-making.
