What happens when business owners turn to chatbots for advice

Can generative AI help entrepreneurs solve their business problems? Yes, but mostly when companies are already thriving, suggests research by Rembrand Koning that reveals the potential for AI-powered mentors to reach owners around the world

By Ben Rand

Published: Oct 23, 2024 11:05:38 AM IST

Updated: Oct 23, 2024 11:14:51 AM IST

The AI helped some of the entrepreneurs in the study highlights the potential of AI mentors to enable entrepreneurship business growth across the globe. Image: Shutterstock

AI can help you become a better writer, coder, and researcher. But can the powerful technology boost business acumen and economic performance?

Sort of, says a recent study. An experimental artificial intelligence (AI) mentor designed to answer business questions from Kenyan entrepreneurs helped some become more successful—but led others down the wrong path, says Rembrand Koning, the Mary V. and Mark A. Stevens Associate Professor of Business Administration at Harvard Business School. How much strategic insight and profit boost AI can provide depends on the condition of a business at the onset of the study.

Businesses that were already financially stable used the AI-generated advice to produce measurable improvements, while struggling businesses ran into further difficulties, say Rembrand and collaborators Rowan Clarke, a doctoral student at HBS; Solène Delecourt and David Holtz, assistant professors at the University of California, Berkeley; and Berkeley doctoral student Nicholas G. Otis.

While Koning describes the paradoxical outcomes as a “real puzzle,” he believes the study offers encouraging findings about AI’s capabilities to help solve business problems, particularly in parts of the world with fewer resources and less access to technology.

“The fact that generative AI was a little over a year old, and we could use it to advise several hundred Kenyan entrepreneurs how to improve their businesses blows my mind,” Koning says. “Already we’re seeing how generative AI can change people’s lives.”

Chatbot versus an old-fashioned business guide

The study enrolled 640 entrepreneurs in Kenya across various sectors, including transportation, food, clothing, jewelry, cosmetics, construction, carpentry, and agriculture.

Researchers randomly divided the entrepreneurs into two groups. One group used the AI mentor to seek advice, while the second group received a placebo training guide.

Surprisingly, low-performing entrepreneurs who had below-median performance before experienced a 10 percent decline in revenues and profits. Yet, high-performing entrepreneurs who had above-median performance got even better, with the AI mentor leading to a 15 percent increase in performance, according to the study.

Also read: How and when will AI make money, investors are asking. But is that the real question?

Difficult questions stump AI

All told, the entrepreneurs asked the specialized AI chatbot thousands of questions. The mixed results likely stem from struggling performers asking more challenging questions based on their more dire circumstances.

“Our best guess is that entrepreneurs in really harrowing straits ask a difficult question, and the AI mentor produces what seems like good advice, but it’s not enough to solve their problem,” Koning says.

“Maybe it’s that you have a food stall, but your food is terrible, your servers are mean, or you’re simply on the bad side of the street,” he says. “The mentor might suggest, ‘Why don’t you do a better marketing campaign?’ The problem is, I’m not sure money spent on ads would improve help, and the cost of marketing might even lead to lower profits and performance.”

The mixed outcomes may also be due to factors such as “overconfident” or “flawed recommendations” from the AI mentor that can lead to negative results.

However, that the AI helped even some of the entrepreneurs in the study highlights the potential of AI mentors to enable entrepreneurship business growth across the globe.

“Unfortunately, getting advice is hard for many entrepreneurs,” Koning says. “It’s especially hard if you run a car wash outside Nairobi. And now, here’s a technology that allows us to put a phenomenal business adviser in the pockets of billions of people.”

This article was provided with permission from Harvard Business School Working Knowledge.