Data-driven: what AI experts got wrong

I recently had the opportunity to speak at the curious 2022 conference in Darmstadt, Germany. It took me some courage to go on stage and share a rather unconventional view on artificial intelligence (AI): AI cannot predict the future. That AI, or more specifically time-series forecasting machine learning models, cannot predict the future is something that should be rather obvious. However, the current hype blurred our understanding of what AI can do, and how to best leverage it.

Planning a trip

I have recently had to plan a holiday trip from Zürich, where I am currently living, to Florence. On the road with my family, we explored two options: travelling by train and by car.

The first step in our decision making was to check travel times on maps. However, travel time estimates (forecasts) alone are not sufficient. As part of the decision making processes, we sat down together and evaluated a number of scenario. For example:

• Travel by train: what is the risk of train delays? The risk of strikes? What happens if we miss our connection in Milan? How will we entertain the kids?
• Travel by car: what is a worst-case scenario? How likely is it that we will get stuck in a jam at the Gotthard tunnel? What is the risk that our daughter will feel car sick? As young children can sit too long in a car, how many breaks should be plan?

Most decision makers agree with me: this was a reasonable approach. They would have done the same. However, in their professional life, their decision making process looks completely different.

Predict-then-act

We have just discussed a very simple example from everyday life. However, this everyday life example is already complex enough to challenge the decision making process that many organizations have put in place: a process where a model takes some historical data (travel times, in the example) and produces a forecast. The most likely outcome (expected value) computed by the forecasting engine is then going to be the basis for the decision (e.g. minimize travel time). This approach is commonly referred to as predict-then-act.

There is lots of value in forecasts. Especially if forecasting is done right: alongside the most likely outcome, the model also provides an estimate for the prediction interval, an estimate of an interval in which a future observation will fall, with a certain probability (wiki).

However, the predict-then-act approach does not take advantage of the creativity and knowledge of domain experts, and their ability to challenge assumptions and explore alternative scenarios. Relying exclusively on historical data bakes in a very strong assumption: that historical data reflects all the drivers of uncertainty. This is never the case.

Agree-on-assumptions

Agree-on-assumptions is a decision-making process alternative to predict-then-act. Agree-on-assumptions leverages tools and methodologies to

• identify new sources of uncertainty and collaboratively challenge assumptions
• generate new forecasts based on an alternative sets of assumptions
• build alternative scenarios and run simulations

Predict-then-act is a straight-through process that can be automated without manual intervention. Agree-on-assumptions is an iterative process that takes advantage of the knowledge acquired by the business but not yet available as data for analytical purposes. This includes, for example, all the information collected through informal conversations with suppliers, clients and peers as well as attending trade fairs or reading newspapers and magazines.

Why agree-on-assumptions?

Agree-on-assumptions stresses the importance of working together as a team in refining assumptions and scenarios. The focus is shifted from forecasting accuracy to the ability of an organization to anticipate scenarios.

Generating a richer set of scenarios brings an incredible value to the decision making process of a company. Scenario analysis allows to stress-test a business model which, in turn, leads to resiliency and agility - the ability to quickly adapt if things do not turn out as expected.

In today's fast-moving word, robustness, resiliency and agility have an increasing strategic value. Successful companies are increasingly willing to trade-off performance (optimality) under the most likely scenario for anti-fragility.