The Art Of Superforecasting

Is it possible to forecast major events accurately in an increasingly dynamic and evolving world? How are the best of the best anticipating opportunities & risks ahead of the competition?

The last 5 years have seen a wave of significant global events happening in rapid succession. From the pandemic, rising tensions in the Russia, China, and US triangle, potentially changing global world orders, historically high inflation levels, and the likelihood of a prolonged recession. The boom/bust cycles are likely to accelerate as technology positions us to become increasingly global (and in some ways more isolated due to increased polarization of views). However, depending on your perspective, we are also in a world with wonderful opportunities. The key is with so much uncertainty, how do we predict more reliably so we can capture the right opportunities and mitigate risks?

One of my mentors once said the best CFOs are masters at anticipating trends before others. If you can do this, you can bypass the competition capturing more of the profit pools in an industry before others realise what is happening.

I recently finished reading ‘Superforecasting’ by Phil Tetlock. Sharing the ideas below that resonated with me and a few more based on experience.

What is Superforecasting and why is it important?

Phil Tetlock and his team are famous for a study that ran over 20 years from 1984 to 2004 looking at thousands of predictions on the economy, stocks, elections, wars, and other global issues. On assessing the results, he found that the average ‘expert’ did as well as random guessing, or as some articles coined, as well as ‘a dart throwing chimp’. In the years that followed, his team then ran a series of competitions on behalf of the US Intelligence Agency (IARPA) who were trying to find better ways to anticipate complex geopolitical events. The idea was to find the best forecasters in the world and then learn from them to improve the agency’s success.

In short, the best forecasting team (the good judgment project) beat more sophisticated teams (including universities) by 30-70% and even outperformed professional intelligence analysts who had access to classified data. This revealed two key themes: some people have real foresight and an ability to predict events better than others, and second, it is not who they are that made the difference it is what they do.

If you can get good at forecasting accurately, the benefits are apparent. You will be able to react faster than others to both opportunities and risks. This will give you massive advantages compared to the competition.

What are some of the biggest flaws in thinking about forecasting today?

There are many but some of the more common ones:

• Deterministic thinking - seeing a single view of the world rather than considering multiple scenarios
• Skimming the surface - in the excitement to act, a top-level analysis of issues rather than getting to the real underlying drivers
• Biases or anchoring – inward-looking views that miss the outside perspective; biases that distort the ability of an organisation to see what is happening; lack of a data-first approach
• Singular views – inability to leverage diverse perspectives to build a more complete picture
• Going for total certainty – spending too much time going for 100% certainty rather than knowing what is sufficient to act and move.
• Slow or ineffective processes to update beliefs with changing events
• Knowing the trigger points or signposts that tell you if your forecasts are off
• Lack of assessing or learning from past forecasts – repeating the same mistakes all over again

What are the biggest myths about how to forecast well?

There are various myths around forecasting which are worth challenging:

• Only the best companies with the best technology and resources can do it: this is not true. Adopting some of the principles I share below will directly improve your ability to forecast. Some of the best forecasters in Phil’s studies were retirees or hobbyists armed with only their curiosity and the internet.
• Innate talent: there is a view that the best forecasters are born with innate talent or ability. Phil’s study showed that it was less about innate talent and more about skills the best forecasters had acquired and how they approached forecasting – patterns and way of thinking, gathering information, and updating beliefs
• AI and Machine Learning will fix it: the best algorithms in the world will not improve forecasting if the assumptions and data being fed into them are flawed. The ideal path is a blend of human and technology-led forecasting

So how can we radically improve our ability to forecast?

1. Probabilistic Thinking

Ask 10 people what defines a high, medium, or low outlook when forecasting, and you will get multiple answers. High will often range from 60-100%, low will range from 0-30%, and medium anywhere in between. Would a leader react differently if a decision was 60% likely vs 90% likely to succeed? Of course, they would. And yet, we often do not even give leaders a clear view of what high, medium or low really means.

The best forecasters give a confidence or probability around their forecasts. This gives leaders the ability to decide based on their risk appetite if a decision is worth taking or whether to find ways to increase the probability or take an alternate course of action.

An easy way to start this journey in finance teams is to ask ‘What is your confidence level on your forecast?’ and start making this more visible when presenting outlooks to leadership. As discrepancies between divisions or areas become easier to see, you can start aligning before making a decision. More importantly, leaders know the likelihood of a forecast happening to help inform and improve the quality of their decisions.

2. Break the decision into parts then predict up from there

Teams will often try to predict an event in totality without breaking things into their component parts. In his book, Phil gives a great example of Enrico Fermi, the Italian American Physicist who built the world’s first nuclear reactor. As a test, he would ask students ‘How many piano tuners are there in Chicago?’. Average students would rattle off an answer from the top of their heads. However, better students would ask ‘what information do I need to answer this question?’. This would lead them to 3-4 questions such as: what is the population of Chicago? how many pianos are there in this city? how many times does someone tune their piano and how long does it take? how many hours does a piano tuner work? You can see that by breaking the problem into these 3-4 questions, you can start to predict each part and then consolidate the response. Students that applied this thinking were remarkably close to the actual number even when they had to guess certain parts due to limited information.

For finance teams, this means not just simply forecasting sales based on a single line trend from the past. It is understanding the underlying value drivers (ie. what info would help me predict volume, what would help me predict price? etc) and then building up the forecast from there. The fun part is that as you get good at this, you can also sense check forecasts you receive as breaking things into the 3-4 big questions or drivers and consolidating from there becomes natural.

3. Focus forecasting where there is the maximum benefit

Linked to the above idea, the best teams applied good judgment in knowing where there was a benefit to spending time and effort to forecast accurately and where certain parts of the problem were so complex or intractable that it was better to take an educated guess as the effort to be more accurate would not yield a better view.

Examples here are trying to predict accurately the oil price or when the pandemic would come to an end. You can spend a lot of time predicting exactly but it will be difficult to do accurately. For such situations, my view is to work with scenarios instead that consider ranges of outcomes in these areas and how we would act differently at the edges of each scenario. Then, as signals appear, we know how to react. For the pandemic, many companies for instance looked at what it would mean to have a V-shaped fast recovery vs an L-shaped much longer one and understood what signals would tell them which scenario was emerging.

4. Use System 2, and then System 1. Not the other way around.

Daniel Kahneman is a Nobel prize-winning economist and author of Thinking, Fast and Slow. Daniel has published some classics around decision-making, behavioral economics, and how to improve judgment. In Thinking, Fast and Slow, he discusses the concept of System 1 (fast and intuition-based) and System 2 (slower and logic/data-based) thinking.

A classic pattern is to default to System 1 thinking as it is intuitive, somewhat automatic, and allows for faster decision-making. However, it is also where many of our blind spots exist. Just because something ‘feels right’ does not make it so. It needs to be informed by taking a step back and working through a complex problem to de-risk it where possible using data and then using intuition to come to a decision.

The best forecasters in the competition were the ones who were able to suspend intuition for a while to use data and logic to test their assumptions before relying on intuition to decide. They were open to changing their views if data told them something important. Check out Adam Grant’s Work Life podcast on TED with Daniel where he dips into this in more detail.

For finance teams, a simple way to check we are not rushing to a decision is to ask questions like: “What do I need to believe for this to be true?”, “How could I be wrong about this?”…

5. Leverage diverse perspectives

Phil Tetlock showed in his book that truly diverse teams were on average 23% better at forecasting outcomes compared to their peers.

Four themes I want to highlight here:

Diversity is not about quantity but quality. It is not adding 10 more people to review a forecast but more about finding a group of people that have sufficiently diverse perspectives and pieces of information that may be relevant and could inform a much better picture. An example of finance teams is how we involve our sales, operational, strategy, and finance teams in the build-up of forecasts. Too often, it is the finance team with 1-2 others in the business but this is not true diversity.

• Psychological safety is key. For diversity to work, each person has to feel safe to speak up about what they are thinking and where assumptions or ideas are flawed.
• Check for bias. Superteams were able to check each other for potential biases and there was an openness to adjust if this was happening.
• Collective goal is clear. Superteams are good at challenging each other but it is always to advance the common goal of understanding and getting to a more accurate forecast.

The ways in which we can bring wider perspectives are to reflect up front who are the people that are likely to have important information that could influence the outlook (eg. Sales teams will know their customer orders and S&OP outlook, Operations and Supply Chain teams will know what is happening internally to fulfill customer orders, etc).

6. Take the Outside and then Inside View

So often, the process in forecasting is to look at last month’s forecast, adjust for this month’s results and then come up with a new forecast. The problem with this approach, apart from missing various sources of insight, is that it anchors us in an internal view of the world. It misses what is happening externally and more importantly the potential options or ideas that might be there if we open our minds to different perspectives.

The best forecasters start with an Outside View that helps reflect what is happening externally and the full range of options that could impact a forecast. They then use this information and blend it with an internal view that now considers unique factors of the company. This results in much higher quality forecasts and more importantly ideas to help maximize them towards full potential.

Within Finance, we can ask our business teams what external reports or benchmarks exist for the most important drivers and how that compares to what we are assuming. Competitor accounts and investor presentations can also be a great source to see whether their belief in particular assumptions is radically different from ours.

7. Frequently updating beliefs

The highest performing forecasters were frequently tested and then updated their beliefs around a forecast. This runs counter to the idea in many companies that want to ‘simplify’ forecasting and keep to quarterly or half-year updates. With a world moving so fast, this makes no sense.

For finance teams, my view is that we need to have a constant and evergreen forecasting process that is grounded in what is happening externally. Indeed, many top quartile companies such as Equinor, Bayer Pharmaceuticals, Volvo, and others have adopted a ‘beyond budgeting’ evergreen approach that has significantly reduced their budget and forecasting process efforts overall, led to more dynamic resource allocation based on emerging events, and increased top-line growth as a result.

Note this does not mean updating our beliefs for every change happening. It is having the judgment to know the drivers that matter, monitoring them and updating when these move materially, and doing so quickly so the business can reflect and react to the new information.

8. Understanding the true underlying drivers of a forecast

You will see a common theme across several of the ideas above. The best forecasters have an intimate understanding of the true underlying drivers that impact business outcomes and bottom lines. Without working to understand the key levers that generate the ending profit or loss, it will be difficult to increase the quality of forecasts.

Finance teams can start to uncover true drivers by resisting the urge (System 1) to stop at the first explanation they receive for a variance by asking ‘why’ again and again until they get to the real root cause.

9. Being a dragonfly

I love this saying in Phil Tetlock’s book. A dragonfly has several thousand lenses on its eye, the combination of which produce a picture of what is around it. Similarly, the best forecasters are good at taking information from multiple sources, insights from the principles above, considering different perspectives, and then synthesising this into an overall forecast. Yet very few will do this consistently. There are usually one or other parts missing impacting the quality of the forecast.

10. Continuous learning

Perhaps the most important of all of the above is the conscious effort to regularly spend time reflecting on past forecasts to get to the root causes of why it was off. This requires real effort as often forecasts are impacted by multiple events. However, the effort put here can improve our understanding of what is driving business outcomes, and how to react to them, and in turn, improve our forecasting capabilities for our leadership.