Why experimentation is critical for long-term success

Image: Shutterstock

Failure and invention are inseparable twins.

Crises offer a unique opportunity to keep trying new things. It also offer a unique opportunity to learn from natural experiments.

In a letter to shareholders in 2015, Jeff Bezos, the world’s richest entrepreneur, called out Amazon for being distinctive in its failure. “To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organisations embrace the idea of invention, but are not willing to suffer the string of failed experiments to get there.” This letter has some very good insights on the need for adaptive experimentation.

To come up with a winning strategy, we must understand causality i.e, the link between what we do and its outcome and only experimentation will allow us to do this. Crises, purely by the sheer magnitude of change it sometimes brings out, like in the case of Covid-19, give us leaders no choice but to keep trying new things, either to ensure business continuity or simply because it creates an environment suitable for experimentation.

For instance, conferencing services have been around for years but were never used so much by families to get together. Today, families spread around the world do not feel the physical distance they used to pre-Covid-19. Now, you really didn’t need a health crisis to get people to use existing services. But the inertia to adopt the service was overcome by the sheer need to connect.

How important is experimentation Most often a company’s risk appetite is dictated by the mental model of its business leader. If the management of the organisation is averse to risk owing to fear of failure it is very unlikely that the company will be a leader in its sector. In a democratic business environment, where competition can come from any player in any industry, the need for increased risk appetite is crucial for long-term success.

If we take advertising spends as an example, in the traditional route, if a single strategy is in play, you have no way of knowing the exact impact of your spends—was the increase in revenue purely because of ad spends or was it due to other contributing factors like an improved distribution system or change in the packaging, pricing and more. Essentially when you engage in a single strategy, regardless of results, the information is inconclusive. Only experimentation tells you what strategy worked best and needs to be done next.

A few years ago, when advertisers asked Microsoft to provide larger ad spaces on its search engine Bing, the company was particular that its revenue stream didn’t impact the quality of user experience. So it ran a series of A/B tests which proved that the company could in fact accomplish this, as experimentation showed that users were fine with fewer but larger ads. Authors Ron Kovahi, GM of the analysis and experimentation team at Microsoft and Stefan Thomke, Professor of business administration at Harvard Business School, in an HBR review article pointed out that an “experiment-with-everything” approach has helped Bing identify dozens of revenue-related changes that helped MS collectively increase revenue per search by 10 to 25 percent each year.  Similarly, the authors also pointed out that Amazon through a series of experiments arrived at the placement of credit card offers on the shopping cart page instead of the home page, adding tens of millions of dollars to its bottom line.

The 80/20 rule
Growth in the new normal cannot be dictated by historical data. Clearly, the pandemic has accelerated certain changes in how customers experience and make buying decisions. Not investing in studying or evaluating the impact of these changes on a brand would be a gross error. Experimentation at a small scale can help assess and create future strategies. Leaders should ideally account for at least 20 percent of their overall budgets towards experimenting with new ideas such as a change in pricing, packaging, distribution, new market access etc. and spend the remaining 80 percent on proven strategies of revenue growth.

Legacy companies have been accumulating lots of unnecessary functions, processes and systems which often bring down their efficiencies. Several dollars can be saved by optimising certain systems which can then be channeled into funding newer experiments. While publicly traded companies may have the challenge of balancing the investments required for experimentation with myopic QoQ expectations of shareholders, they could consider highlighting either the cost benefit or create a portfolio of experiments that could be of immediate requirement or even long-term.

Incentivise success and celebrate failures
Involve the organisation's architectural processes and structures, including reward systems and incentives to create a culture that encourages experimental learning. Adaptive experimentation is a continuous and iterative process rather than a linear process. Ensure that failures are tolerated more and successes rewarded. In fact, compensate the experimental team if the results of the control are better than that of the test. This will help reduce the resistance to change. In fact, it would be ideal to develop a whole set of processes and incentives for lessons learned from failures.

The need of the hour is to be agile and display adequate nimbleness and humility to make a course correction. In fact, one of the best investments a company can make in its journey of adaptive experimentation is to hire an in-house person with knowledge of statistics, experimental psychology, or even experimental design to help implement the same.

The data that will be generated through experiments can help the company make empirical generalisations that could form the core of the knowledge base, the management information systems, and even the knowledge sharing systems in the company.

The writer is Co-author of Transformation in Times of Crisis and CEO of Mphasis

The thoughts and opinions shared here are of the author.