How can you effectively evaluate the success of an AI development project in a large organization?

Many organizations are at the nascent stages of AI maturity. They may overestimate the value of their data, ease of building models, and its impact on business. You may have to analyze each of these aspects and educate them about the real state. It will not make you popular but serve as a foundation for an iterative plan for building AI-assisted offerings. The success criteria should include measuring the data quality, data collection coverage (the data present vs data needed), business metrics, and model performance. You may need to show the path and a positive reinforcement cycle: how each stage/model leads to collecting more detailed data, which then leads to better/different models, and defang the notion that AI is a magic wand.

A fintech startup, PayStream, developed an AI for fraud detection. They set success criteria based on false positive rates, user transaction security, and customer satisfaction scores. Their goals mirrored those of industry leaders like Stripe and Square, focusing on precision and user trust.

A crucial aspect of an AI project in a large organization lies in bridging the gap between AI metrics and business objectives. It is essential for the AI and business teams to collaboratively establish a common language when defining achievable and measurable metrics. Recognizing that AI metrics may differ from traditional business metrics underscores the necessity for alignment between the technical aspects of the project and the overarching goals of the organization. This synergy ensures that the success criteria resonate with both the intricacies of AI development and the broader strategic vision, fostering a comprehensive evaluation framework that captures the true impact of the project on business outcomes.

I feel the best way to eat an elephant is one bite at a time. Different functions across the organization would require different applied models from both functional and economic perspectives. Making your Sales & Marketing or Operations AI powered will have a more tangible ROI for instance.

In my sector - Healthcare - First Contact Resolution (FCR) Rate: FCR measures the percentage of customer inquiries or issues resolved during the first interaction. A high FCR rate suggests effective communication and problem-solving skills.

PayStream adopted a mix of A/B testing and real-time monitoring, similar to approaches used by Netflix to gauge user experience. They also integrated customer feedback mechanisms akin to Salesforce’s IdeaExchange for direct user input.

- Conduct thorough testing with real users to assess usability, functionality, and satisfaction. - Have domain and AI experts review the solution for quality, accuracy, and relevance. - Compare the AI solution against industry standards or competitors. - Analyse the broader effects on the organisation, users, stakeholders, and society. - Track key performance indicators (KPIs) defined in the success criteria. This includes metrics like accuracy, efficiency, and return on investment. - Establish mechanisms for continuous feedback from users and stakeholders, ensuring the AI system evolves in response to real-world use and challenges.

The method we recommend is Data Thinking. It ensures close collaboration from ideation to implementation and also functions as a great leadership tool to make investment decisions. Data Thinking is a methodology combining design thinking principles but is optimised for data & AI solutions.

Use rigorous techniques, like effect analysis, user input, key performance indicators (KPIs), and deadline adherence, to assess the success of an organization's AI development project

Select appropriate tools and methods to measure progress against set criteria. Utilize benchmarks, performance metrics, and user feedback for comprehensive evaluation.

To enhance decision-making, steer clear of superficial vanity metrics that lack significant value and may lead to self-deception or misleading others. Instead, focus on meaningful metrics, which are often expressed as ratios. For example, rather than solely considering the size of a fine-tuning dataset (a superficial metric), evaluate the ratio of the fine-tuning dataset size to the model's accuracy (an effective metric). This approach offers a more insightful and practical measure of performance. For more on this, refer to the excellent book Lean Analytics by Alistair Croll and Ben Yoskovitz

1.Diverse Data Sources: Emphasizing the need to gather data from a range of sources like the AI system itself, users, stakeholders, and external databases ensures a comprehensive understanding of the system's performance. 2.Data Types and Formats: Recognizing the variability in data types, whether structured or unstructured, numerical or textual, is crucial. This reflects the different ways AI systems process information and the need for tailored evaluation methods. 3.Data Quality and Relevance: Stressing the importance of data being relevant, reliable, valid, and representative is key. 4.Ethical and Legal Considerations: Highlighting the necessity for secure, ethical, and legal data handling practices is vital.

Like Palantir, PayStream employed advanced data aggregation tools to collect data from transaction logs, user feedback, and security breach attempts. They then used statistical analysis and pattern recognition to understand the efficacy and reliability of their AI system.

A major organization's AI project's success can be determined by gathering and evaluating pertinent data, such as performance metrics, user reviews, and project schedules, and making sure they are in accordance with corporate goals

This is maybe the most important stage of AI solution development. With the increased amount of data, there is a need to understand the data before building on top of it. Data security and solution scalability are two most important factors for enterprise.

Clear Reporting: Prepare comprehensive reports presenting the findings, including successes, challenges faced, and areas for improvement. Use visualizations and narratives to make the results easily understandable. Stakeholder Engagement: Share the findings with stakeholders at various levels within the organization, highlighting the impact of the AI project on key metrics and business objectives. Iterative Improvement: Use the results to refine the AI model or processes. Implement changes based on identified weaknesses or areas for enhancement. Documentation and Learning: Document the evaluation process, results, and actions. Use this as a learning tool for future AI projects within the organization.

Inspired by Slack’s methodology, PayStream communicated their findings internally using dashboards and externally through white papers and case studies. They took a page from Google’s AI principles by not only sharing successes but also being transparent about areas needing improvement

Assess the success of an AI project implemented by a large organization by encouraging efficient communication and taking timely action on findings. Transparently communicate findings, deal with difficulties, and modify plans of action to maximize project results

1. Tailor the communication to different audiences - organisational leadership, technical teams, stakeholders, or the public. 2. Utilize various formats like detailed reports for internal stakeholders, dashboards for ongoing monitoring, infographics for broader audience communication, and narrative stories to illustrate impacts and insights. 3. Emphasize the most critical findings, implications, and actionable recommendations. 4. Ensure the communication is clear, avoids jargon, and accurately represents the data findings and their significance. 5. Present findings in a manner that clearly outlines potential actions, improvements, or strategic decisions that can be taken based on the evaluation results.

Share findings transparently with stakeholders. Act upon insights derived from evaluation to refine strategies or make necessary adjustments for improvement.

The journey didn’t end with initial success. The team knew that the AI landscape was as dynamic as the cryptocurrency markets. They committed to a culture of continuous learning, ensuring their AI algorithms were adaptive and ethical, reflecting the evolving tech ethos exemplified by leaders like Satya Nadella.

I believe in failing fast when it comes to AI projects. This means that when things are going wrong, don't drag it on. It's okay to go back to the drawing board with some good lessons learned from your failed attempt. Don't be afraid to fail, you an always try again but don't drag failure on as you'll end up with some serious technical debt.

From my perspectives, the success can be measured by having 1) solutions that are fit for purpose (e.g., bias risk is mitigated) and 2) well adopted by the users (e.g., solution readily available in production and addressing users requirements)

Ok, let’s face it. AI IS BLACK BOX. AI is new, and so the evaluation methods are also evolving everyday. When we build a product, what do we look for to conclude it as “success”. It is user satisfaction. The success of AI lies in rapid prototyping and constant evolution. For eg, when you build a chatbot with GPT-4, see if it answers the query the way you intend to it. If not, create a better prompt or fine-tune your model and then deploy again, and let users try again. AI can’t get better, unless it makes mistakes, so rapid prototyping is the new normal in AI 💡

Factor in organizational context (change management, infrastructure) and long-term impact. Consider ethical implications and potential biases in your AI system. Regularly revisit and adapt success criteria to align with evolving goals and business landscape. Remember, a successful AI project evaluation is iterative and involves collaboration across various teams within the large organization.