Misadventures in AI: Moving from Play to production

Theres an old joke that goes "How do you know if someone does crossfit? Dont worry, they'll tell you."

Its starting to feel like that with AI - anyone whos doing it will tell you they're doing it. Its getting to the point where even products which dont need to have their own AI have managed to build it in. Logitech now have a mouse with an AI button, and even Adobe Acrobats AI tool feels like more like a marketing gimmick than need. The more I see AI being integrated into products, the more it reminds me of a Peanuts comic strip from nearly 70 years ago.

Once a company finishes telling you they have AI, they’ll invariably say that getting started is easy and recommend running a Proof of Concept (PoC).

If you're familiar with the idea of a POC, skip this paragraph. If you're not, a PoC is a demonstration designed to verify that certain concepts or theories have the potential for real-world application. It is typically used to determine the feasibility of an idea or to verify that a particular approach will work as intended. In the context of technology and business, a PoC is often a small-scale project that is used to test the viability of a larger project or system.

At first glance, PoCs and AI go together like cheese and crackers - before you fork out for subscription fees and lock yourself in to the tool, you really want to make sure it does what it says on the tin.

But there's a couple of problems with this relationship:

1. You need to ensure that the Proof of Concept (PoC) measures the right aspects.

One of my clients is currently running a PoC using a combination of Optical Character Recognition (OCR) and Generative AI to standardise customer inputs. The issue is that their client base uses bespoke forms to submit information. While these forms contain the same information, the format is inconsistent. This inconsistency impacts productivity because staff spend more time scanning the forms for information than actually inputting it into their system to create a quote (1).

We know that AI can solve this issue, so the PoC isn’t about proving the technology works. Instead, the PoC aims to measure the accuracy of the AI, and how much time will be saved on average. Essentially, we’re using the PoC to assess the return on investment. Sometimes, the only way to measure the benefit is to run the process, and this PoC is a perfect example of that.

2. Theres a big difference between Play and Production.

Asa Cox and I were talking the other day about how vendors often fall short when it comes to transitioning from a PoC to full production. The things you dont get told about include:

• Scalability: During a PoC, the solution is tested on a small scale. Moving to production means handling larger volumes of data and more users, which can reveal performance bottlenecks and require additional resources.
• Integration: Integrating the new solution with existing systems and workflows can be complex. It often involves ensuring compatibility with legacy systems, data migration, and seamless communication between different software components.
• Performance: Ensuring the solution maintains high performance and reliability under production conditions is crucial. This includes handling peak loads, minimising downtime, and ensuring fast response times.
• Security: Addressing security concerns involves implementing robust security measures to protect data and ensure compliance with relevant regulations and standards, such as GDPR or HIPAA.
• Cost: The financial investment required for full-scale implementation can be significant. This includes costs for hardware, software, training, and ongoing maintenance, which may exceed initial estimates.
• Change Management: Training staff and managing the organizational changes required to adopt the new solution can be challenging. It involves overcoming resistance to change, ensuring user adoption, and providing ongoing support.
• Technical Debt: Addressing any shortcuts or temporary fixes used during the PoC phase is essential for long-term sustainability. This may involve refactoring code, improving documentation, and ensuring the solution is maintainable.
• User Acceptance: Gaining buy-in from end-users is critical. The solution must meet their needs and expectations, which may require additional features, user interface improvements, and thorough testing to ensure usability.

This isn’t a criticism of vendors; the real challenge lies in the variability of AI use cases, the customer frameworks being developed, and the specific market conditions each business operates in. We cant expect vendors like Microsoft, AWS, and others to step into this space because its not their core business model.

Well, what do we need?

While the excitement around AI is justifiabe, it’s essential to approach its implementation with a clear strategy. Moving from a Proof of Concept to full production is not just about proving the technology works; it’s about ensuring it delivers real value at scale. If you're fortunate enough to have the skills in house to manage this, you're well placed.

If you dont have the skills, then you should consider partnering with experts who understand the intricacies of this transition who can support you to navigate the complexities and maximise their return on investment.

Two companies that I know of operating in this space are Asas own Arcanum AI , or Unisphere Solutions Limited - Cyber, Transformation and Strategy Specialists with people like James Dickinson , Mike Merry (and more smart people who I wont tag in. Oh, and me!) (2).

These are people who have done this often enough that they know how to make the considerations for moving from PoC to production seamless and efficient. They understand the challenges and have the expertise to navigate the complexities, ensuring a smooth transition and maximising the return on investment.

Transitioning from PoC to production requires a solid strategy; without it, you’re just playing around.

The purpose of Misadventures in AI is that Im trying to cut through the hype on AI and focus on the reality of what it truly means (3). I do use copilot to curate ideas, but the bulk of this post is written by me!

Feedback is always a gift, so please leave yours.

(1) Now before you get on me about straight through processing, well get there - were just focusing on solving one problem at a time.

(2) This is my article, so Im being selective about what horns I toot. If you know of others, feel free to add them in here too.

(3) Youll also find Misadventures in AI on my substack - its curently free to sign up! Misadventures in AI | Ant McMahon | Substack