How can you define the scope of an ML project?

When defining the scope and problem for a ML project it is good to think of thag project as a research driven project not a purely objective driven project. ML life cycles are very different from software lifecycle and laying down all things upfront is good but remember that as you draw more insights the scope can evolve.

Step 1 in the machine learning process involves identifying the problem to be solved. This includes understanding the business or social need driving the project, defining the current situation and desired outcome, identifying stakeholders and users, and assessing how the ML solution will bring value and improvement. Background research and reviewing existing literature related to the problem are essential components of this initial step.

Often, I find an outside framework like DMAIC (Lean Six Sigma) can help in identifying/objectifying a business problem. But at the end of the day, it's about finding a framework for problem definition that works for you and leveraging it. All of the big cloud companies have data science lifecycle frameworks documented which address problem definition; it's just a matter of researching and adopting a robust framework.

Defining an ML project's scope is like drawing the boundaries of a masterpiece: Clear Objectives: Set specific goals. Data Parameters: Define data sources and limits. Stakeholder Alignment: Ensure everyone's on the same page. Timeline: Establish project duration. Flexibility: Allow for adjustments when needed. With this artistic precision, your ML project will shine!

The cornerstone of any successful machine learning project lies in defining clear and precise objectives. It is like a compass, guiding the team through the complexity that AI projects inherently possess. It sets the tone for the algorithm selection, data collection, model training, and evaluation metrics. Without it, the project risks becoming a directionless endeavor, leading to wasted resources and dissatisfied stakeholders. To ensure shared understanding and commitment, I use the SMART (Specific, Measurable, Achievable, Relevant, Time-bound) framework. This systematic approach will sharpen the project's focus, align stakeholders, and set the stage for quantifiable success.

In Step 2 of the machine learning process, you define the project's objectives. This involves specifying what you aim to accomplish with the ML solution, how success will be measured, and any constraints or limitations. Additionally, you outline the scope of the ML solution, including its functionality, features, user interface, and integration with other systems. Prioritizing objectives and identifying critical goals are also crucial aspects of this step.

In Step 3 of the machine learning process, you analyze the data for your project. This involves collecting, exploring, and preprocessing the data in alignment with your project's objectives and the problem you aim to solve. You assess the quality, quantity, availability, and relevance of the data, identify data sources, formats, types, and attributes. Additionally, you conduct descriptive and exploratory analysis to gain an understanding of the data and its characteristics.

In this step, we must be careful and ensure that everything we build while developing the solution can be replicated in production. Sometimes we come up with ideas or features that, in fact, we won't have access to them in production. Here, we also have to be very organized to replicate it correctly in production. Analyzing features and getting feedback from business experts about their perspective on data, or if it's possible to apply feature engineering, creating new features can make a great difference to the success of the solution. Always keep in mind that we can optimize any model to perform better, but the greatest impact relies on the inputted data.

Step 4 involves selecting the methods for your ML project. This includes choosing the suitable ML techniques, algorithms, models, and tools that align with your objectives and data. You also consider the trade-offs, assumptions, advantages, and disadvantages of each method. Planning for implementation, testing, and validation of these methods is crucial, and documenting the methods and their rationale is essential for transparency and reproducibility.

This a crucial step, because you should have a clear understanding about what are the priorities of the project. Not always your solution will met all the client’s expectations, so you have to know which kind of outcome you have to prioritize over others and translate it to the ideal optimization function and metric score to focus on. The best example it’s on classification problems where we have to understand the upside and downside from a business perspective of having Recall better than Precision and the opposite

In Step 5 of the machine learning process, you define the evaluation criteria for assessing the performance and quality of the ML solution. This involves specifying metrics, benchmarks, baselines, and thresholds that indicate how effectively the ML solution meets its objectives and addresses the problem. Additionally, you define procedures for collecting, analyzing, and reporting results. It's essential to consider how errors, uncertainties, biases, and ethical issues will be handled in the evaluation process.

I would like to add here a Code Review and tasks to double-check the final solution to ensure that we don't have data leakage or any other kind of problem. Of course, this revision must be done by Data Scientists who didn't come up with the final solution.

In the final step, Step 6, of the machine learning process, you review and refine the project's scope. This involves ensuring that the scope is realistic, feasible, and in line with the problem and objectives. You communicate the scope to stakeholders and users, seeking their feedback and approval. If there are any changes or new requirements, you revise the scope accordingly. It's essential to document the scope and any modifications made to it for clarity and transparency throughout the project.

In-practice observation: Often ignored in the scope of traditional project planning is "model adoption". Even if the success criteria (such as accuracy on historical data) are met, the end user needs to build "trust" in the ML models before adoption. This is because of the following reasons: 1) The end users are typically not data scientists. The ML model output is typically probabilistic. The end users may not be comfortable with probabilistic outcomes. 2) Building "trust" in the AI is a journey rather than a destination. When new users come on board, they need to start their journey. Thus the scope of an ML project should include the adoption, putting best practices in place, and training exercises to build trust.

At the beginning of the project it must to be clear what kind of solution the end users want to and understand how they are going to use it. We, most of the time, build solutions for others, so we have to make sure that everyone involved has its expectations aligned about how that solution has to be delivered. Always keep in touch with the end user of your solution and the business expert. Besides that, from my experience, I’ve noticed that usually the end users enjoy being a contributor of the solution, this way they don’t have that feeling of they’re going to use a black box tool which they don’t have any idea how it’s working, then you increase the chances of user’s engagement on your solution after deployment