The AIcarus Trap

How Your Shiny New AI' Wins' Can Send Your Bottom Line Crashing Back to Earth

Almost exactly 10 years ago, on December 15th 2014, Man Haron Monis took hostages in the Lindt cafe in Sydney's Martin Place. The standoff lasted 17 hours and claimed the lives of three people: hostages Tori Johnson and Katrina Dawson, as well as the perpetrator himself.

As passers-by tried to flee the area, many pulled out their phones and requested Uber rides. Uber's algorithms detected increased traffic and automatically introduced surge pricing, with a minimum fare of $100.

The untimely price gouging, understandably, led to a massive backlash. Uber was forced to apologise and offered free rides to rescue its reputation. What started as an algorithmic revenue grab by Uber (and a feature that made Uber a lot of money overall) turned into a costly mistake - both for its bottom line and, more significantly, its reputation.

The AIcarus Trap

The mythical Icarus flew too high with his wax wings, got too close to the sun, and crashed when the wax melted. Uber's algorithm similarly soared with its revenue-maximising surge pricing, only to crash when it faced public backlash. (I'll be milking this flying metaphor - bear with me.) This pattern - the euphoria of algorithmic efficiency followed by a harsh reality check - is something we're seeing more and more. I call it the AIcarus Trap.

The AIcarus Trap is when organizations, dazzled by AI's immediate gains, implement radical changes without considering long-term consequences - soaring high on early wins only to crash when hidden costs and complexities emerge.

Consider Amazon's recent AI wins. CEO Andy Jassy reported that AI helped upgrade 50% of their Java systems in six months, with 79% of auto-generated code reviews shipping without changes - saving an estimated 4,500 developer-years of work(!). Impressive, certainly. But experienced developers know the real challenge isn't writing code - it's maintaining it. While AI-generated code might be faster to produce, it risks multiplying technical debt.

The AIcarus Trap might show up as productivity spikes in profits or productivity, but—if not managed well—lead to long-term losses, customer churn and hidden costs down the line.

The rise and fall of UnitedHealth

In 2022, the UnitedHealth Group (yes, the company whose Insurance Division CEO was recently killed in New York) deployed an AI system called nH Predict to automate decisions about elderly patients' post-acute care needs, automating an estimated 50-75% of human labour. The algorithm determined whether patients needed skilled nursing facility care and for how long. It was initially celebrated for reducing costs and speeding up decisions.

You already know where this is going: a lawsuit revealed that the AI system was systematically denying care to eligible patients, particularly affecting those with complex chronic conditions. The investigation found the algorithm had been trained primarily on cost data rather than patient outcomes. While it produced impressive financial savings for UnitedHealth Group, it led to worse health outcomes (with an alleged astounding 90% error rate!). It also increased hospital readmissions and, ultimately, higher long-term costs.

UnitedHealth Group had to face both regulatory scrutiny and multiple lawsuits. The company's initial efficiency gains were dramatically overshadowed by the downstream consequences of over-automating complex healthcare decisions.

How to Tell You're Flying Into an AIcarus Trap

Your organisation might be heading for a fall if you're implementing AI to fully automate critical decisions, rushing deployment without redundancy, or dismissing human expertise as "legacy overhead." Like in UnitedHealth's disaster, full automation of critical decisions rarely ends well.

The good news? Such a fall is not inevitable. Organisations that avoid the AIcarus Trap treat AI as an augmentation tool, not a replacement. They maintain human oversight, set clear boundaries for AI systems, roll out gradually, and regularly audit both immediate gains and long-term impacts.

Getting Back to Earth: How to Avoid the AIcarus Trap

Implementing AI doesn't have to end in a crash landing. Here's what you need to think about:

• Think Beyond the Sugar Rush. Sure, your new AI tool might save thousands of work hours, like at Amazon. But will these savings compound or crumble? Ask the hard questions about sustainability before slashing teams (or redeploying them to help grow your business in other areas).

• Count ALL the Costs. That AI system looks cheap until you factor in infrastructure, training, and maintenance. Like a cheap printer with expensive ink cartridges, your cost-saving initiative might hide costly surprises.
• Keep the Humans in the Loop. Those productivity gains look amazing until everyone who knows how to fix things has left. Build your playbooks, maintain expertise, and run your fire drills. When algorithms go rogue, like Uber's did in Sydney, you need more than an apologetic press release.

The goal isn't to avoid AI - it's to implement it without melting your wings. Smart organisations aren't just asking, "How high can we fly?" but "How do we stay airborne?"

Flying Smart in the AI Age

I am not trying to scare you from using AI. The potential gains are real: improved efficiency, faster decisions, and better resource allocation. Potentially a complete transformation of your business. But I advocate for balanced ambition. Organisations that treat AI as an augmentation tool rather than a replacement, that plan for sustainability rather than just immediate gains, are the ones that will thrive.

Success with AI isn't about how high you can fly but how long you can stay airborne.

Ok, enough Icarus metaphors for today.

See me live:

• Thursday, 16 January 2025 - Saturday, 18 January 2025, Munich, Germany: The Economy of Algorithms, book signing (free book copies for participants). Register here.
• Monday 20 January - Thursday 23 January 2025, Davos, Switzerland: World Economic Forum
• Thursday, 6 March 2025, Brisbane, Australia: Curriculum connect, opening keynote. Register here.
• Thursday, 13 March 2025, Austin, Texas: SXSW, keynote. Register here.

Recent podcasts I spoke at:

• What's the Buzz?
• AI in 5
• Trusted Authority Podcast
• Doughnut Jar: Conversations in AI

Prof. Marek Kowalkiewicz is a Professor and Chair in Digital Economy at QUT Business School. Listed among the Top 100 Global Thought Leaders in Artificial Intelligence by Thinkers360, Marek has led global innovation teams in Silicon Valley, was a Research Manager of SAP's Machine Learning lab in Singapore, a Global Research Program Lead at SAP Research, as well as a Research Fellow at Microsoft Research Asia. His newest book is called "The Economy of Algorithms: AI and the Rise of the Digital Minions".