Can you solve complex Machine Learning problems?

I always start by asking what business problem is the model trying to solve. I then ask about how to find and leverage the relevant data. Often the data itself serves as a model constraint because the ideal data might not even be available.

Ultimately it’s all about solving business problems. And using ML has to be justified against other possible solutions. This goes deep into the project, if you are doing it right. You can read tutorials on classification threshold optimization. But what are you optimizing for? You need to know the costs of each type of error in every class. This gets complicated quickly as the number of classes grows. The only way to create a “best” solution for the optimization is in terms of a business cost function. In my work I often apply weights to the objective (e.g. accuracy, MSE, etc.) for each class. The weights reflect the business cost for errors in each class. You need to discover these up front, by talking to stakeholders.

That first sentence really hits home at the development and failures of large language models. There is no sign of any domain knowledge of linguistics, semantics, semiotics, symbolic logic or knowledge engineering in any LLM or transformer technology in the teams who are hacking their way to ever more complex and kludged systems of code. If they had even the very basics in some of these, they would know that they have no chance of success. Domain knowledge is a critical factor.

I think solving the problem starts way before the "machine learning project" is identified. I'm a big proponent of data literacy without which you cannot have data governance. In particular, making sure the entire organization understands the difference between business intelligence and data science. The latter departs from analytics in that data scientists leverage machine learning algorithms to design statistical methods to generate information from training data. Ideally, the data scientist would be able to report/communicate the data in a way that informs improvements to business (or in medicine, clinical) processes. There is a hierarchy of team competencies relative to machine learning explained further below.

Two of the more important goals for most businesses is what can ML do for the company in terms of cutting costs and/or increasing sales. If other simpler approaches can accomplish either of these goals, there is no reason to use complicated methods.

We often try to use algorithms to solve really tough problems, but there's a lot of low-hanging fruit for applying algorithms as well. Things like fixing spelling through fuzzy matching algorithms are one example of this! Data exploration also helps us better understand what the data looks like before we even run a model on it. Visualizing the data also helps us better understand the patterns in the data points, which include the trends, groups, and outliers/anomalies. Once we determine the anomalies or outliers for example, we can then determine how to handle them.

A quick way to kickstart your data exploration is to use open-source Python libraries like 'sweetviz' or 'ydata-profiling' (formerly 'pandas-profiling'). These are designed to generate high-density visualizations and statistics to help understand and compare different features of your data. Apart from EDA, 'sweetviz' also allows creating comparative reports to compare two datasets, typically a training dataset and a testing dataset or a dataset before and after preprocessing.

Instead of using mean or median, leverage covariances for gap filling in datasets. Covariances capture relationships between variables, offering a more informed approach to fill missing values.

- Finally put the exploration into the reproducible feature engineering pipeline as a script - add essential unit testing for the script - in case of text data always try to have a strategy for masking personal and classified information before training

It is extremely important to make sure that the data for the training set is the same distribution as the validation set. Otherwise, your model will perform poorly.

As my work has shifted towards using more foundation models and LLMs, model selection has taken on additional meaning. With Generative AI it is important to first identify the task you want to solve for and tune a model to support that task. The shift in thinking is around how we select the right model for the task. There are several variables to consider (FURM): Is the model fair? Does it reflect the diversity of those who will use/benefit from system. Is the model useful to complete the desired task? Is the model reliable? Can it deliver consistent results? Is it monetarily viable? What is the cost to get a model to deliver the outcome you are looking for. Model selection is very closely tied to Model evaluation and assessment.

If you have the resources, integrate model selection and architecture into hyperparameter search and optimize everything. If you’re short of resources, try a few baselines and go from there. Ask more senior Data scientists what they would choose. Consider deployment and what compute resources will be available.

When having multiple team members trying out several models always set up a baseline model along with a fixed set of evaluation KPI against which all the models will be compared.

Beware that a complicated model might look good in terms of performance metrics. But it could be the wrong model to use because it doesn't provide sufficient insights about the solutions proposed by the model.

Also do not miss out on the explainability to see when a model is taking certain decisions which are the features the model is focused on. That might reveal gender bias, social bias or racial biasness in the model if exist. It is important for ehhical AI.

Use a real optimizer like Optuna vs grid search. Once you master real optimization you’ll never do things the same again. Optimization using smart optimizers can address non-convex loss surfaces and allow you incredible flexibility compared to grid or random search.

Adding more data, when available, is a beneficial approach for model optimization. It expands the training set, improving generalization and addressing overfitting. New data uncovers hidden patterns and enhances the model's understanding. However, ensure the quality and relevance of the additional data to avoid biases and improve performance

If you are deploying language models always try to implement some OQ/PQ test to see if it does not leak personal data or any classified information.

One simple way to deploy AI applications is as a service of APIs that other applications can call. Each caller has an Application ID that identifies their application and allows the AI service to load configuration information for each caller. This configuration can include where the input data is coming from, parameters, features to run, output data, storing of log files, etc. The actual process of deploying the AI model is using standard Devops that run all code through security checks and can deploy to develop, testing, and production servers.

This article has some nice basics. Though it's important to understand that without the right stack, nothing magical or useful comes out of ML. Below is a hierarchy of data skills starting at item 1 foundational to making it practically useful at scale and over time, #7 7) Ongoing Testing & Tuning for model integrity 6) ML 5) Data science (data scientists leverage ML models to impact the business ideally) 4) Data engineering 3) Data management 2) Data quality 1) Infrastructure

It would be better to start with understanding what complexity is . Is it the Time and Space complexity as perceived in Computer Science ? Is it Model Explainability one is seeking to demystify ? Depending upon this one can tackle solving the complexity issue