¿Cuáles son las prácticas esenciales de preparación de datos para el aprendizaje automático?

From my experience the Data Sourcing and Management Strategy is the key critical function of a successful AI-ML Project. At the start of project it is highly recommended to develop the Data Inventory across Source Data Systems, ETL, Data Pipelines, Enterprise Data Warehouse, Data Lake and Databases. The next critical thing is to develop the ML Objectives, MLOps Hyper-parameters, Algorithms and put in Place Data Observability and Continuous Training of ML Model

Distill your ML objectives as cleanly as possible by asking "why would we do that?" until the answer is so fundamental that it does not need further explanation. For example, someone tells you "we want to use ML on our CRM data". Why? - to improve win rates Why? - because they're too low Why? - because salespeople are doing the wrong things Why? - because they're misunderstanding the buyer signals Just getting to this point would illuminate that you need to get data about "buyer signals". This helps define which data is and is not valuable, which dimensions are relevant or not, and even what type of algorithmic reasoning might produce helpful answers.

Building machine learning model requires well prepared data. Considering techniques like oversampling, undersampling, or using specialized algorithms to balance the data and also usinf robust algorithms to remove outliers.

💡 In my view, Data preparation is vital in machine learning. These practices guarantee high-quality training data and improved model generalization. 🧪 Data cleaning Eliminating noise, inconsistencies, and errors from your data. This ensures accurate and reliable model training. 📊 Data feature engineering Transforming data into more informative features. This may involve creating variables, binning continuous data, and converting categorical data to numerical values. ⚖️ Data balancing Maintaining an equal number of examples for each class or target variable to prevent model bias. 📌 Splitting data into training, validation, and test sets: Assessing model performance on unseen data and preventing overfitting.

Engage with SMEs early at this stage to have a clear understanding of the problem being solved with ML. This will allow you to set shared, clear goals and metrics to attribute the model output to desired business outcomes. The purpose of using ML as a solution must be well defined from the onset, and there must be a rationale that there are no alternative or simpler approaches to solving the problem. The objectives must not only focus on the model output, but the entire ML lifecycle: how are we going to research? for how long? What are we going to test and why? How frequently are we going to demo the solution to stakeholders? The definition of an acceptable model and so on, What is considered to be correct and accurate data?

Data is one of the key ingredient of any ML project. I will say it’s the single most important factor in determining the quality of output that your model will produce. All new ML developers should be ready to spend 60-70% of their time on data related activities. It may not be glorious work but it is something that can make your project successful.

This really comes down to understanding the domain and then mapping your data. I.e. dependencies that would describe the correlation of data entities etc. Contextual dependencies can only be explored by examining data hands on. Ensure that all dependencies, structural and contextual are documented in your metadata repository In the end, we need to answer the question: what is the problem we are solving? Is the data we have “good enough” to give us the answers? I agree with some of the commentaries earlier that repeatability is key. If many manual touches are required to procure and cleanse the data, then your process is broken or inefficient. You need to establish a repeatable quality assurance framework to succeed.

Exploring and understanding data, often through exploratory data analysis (EDA), involves visualizing and summarizing datasets to grasp their characteristics and relationships. In a production company, if the goal is to optimize the manufacturing process, EDA might entail examining variables like material quality, production speed, and defect rates. Visual tools like histograms could reveal the distribution of production times, while scatter plots might highlight correlations between machine settings and product defects, guiding subsequent data cleaning and feature engineering efforts to improve predictive models.

Utilize the Data Catalog to familiarize yourself with the dataset. A foundational understanding of domain knowledge will enhance your comprehension of the dataset's nuances. Consider the nature of the System of Records as it relates to the work at hand. These elements will contribute to a deeper understanding. Additionally, strive to gain a better grasp of the data types and key transformations involved.

In particular, examine how many "manual touches" of the data were involved between the real life event that you are measuring and the data being consumed by the ML algorithm. Examine whether those touches could insert human interpretations (ie bias) into the data. For example, raw data from Amazon about who clicked on what buttons on what page while shopping, could be fed into an ML model. That would data would require zero human touches. But CRM data about deals might have been typed by a salesperson. That is many human touches. More human touches -> larger probability of interpretation errors, bias, incompleteness.

We use the saying "garbage in, garbage out" in terms of the management of your data. Your model and output will only be as good as the input you have prepared. If you are going in without the foundation, you will skew, bias, or inaccurately reflect in the result. For example, missing data, lack of data availability could occur intentionally or at random causing a bias output.

Some people interpret dimensionality reduction as part of data transformation so I will comment about that here. Consider that when you linearly interpolate dimensions in a high-dimensional model, you are creating a new, abstract dimension that is less inherently understandable to most humans. Is this a smart tradeoff for faster computation, for your particular problem? Will an actual human ever have to look at the dimension labels and understand them?

Cleaning and transforming data is a critical process that sets the foundation for successful model development. Data preprocessing is where domain knowledge meets data understanding, where raw information is refined into a format that can be harnessed for predictive analytics. It's a blend of science and art, where thoughtful decisions on how to handle each data quirk impact model performance. It's essential to keep a record of these transformations for reproducibility and auditing.

When imputing missing values be sure to avoid "data leakage" and specifically "future leakage." Data leakage means that information from outside the training dataset is used to train the model. Future leakage is a specific type of data leakage where the model accidentally gains access to future data points, which is especially problematic in time series analysis or any predictive modeling scenario where temporal ordering matters. A typical example with time-series stock data is when there's a missing value, and one mistakenly uses the subsequent day's value to fill it in, rather than the preceding day's value. You have introduced new data!

Most AI/ML algorithms are designed to work on clean/high-quality data, whereas most real-world data is not in that way. So, data cleaning and data quality are very important. There is no one standard approach to this, it is dependent on the actual data set and the problem we are trying to solve.

You need sampling for two key purposes: creating representative subsets and avoiding modeling errors. Sampling comes into play here to guarantee that the subsets, that you feed into your ML models, are representative of the entire dataset. Properly sampled and class-balanced data helps build more robust and accurate ML models. In this sense, sampling is not about Data Science only – it is also about business understanding, what the costs of different mistakes are, and how to sample the data to minimize these costs.

In general, we should randomly split our data in a 30-30-40 ratio . -For example, if the company have 1000 customer records in the customer profile, we should randomly take 300 records for training the algorithm, another 300 for testing the model and the final 400 for validating the results.

When we talk about splitting data, we mean dividing the data into different subsets. These subsets are mainly: Training Set: This is the largest subset, and it's like your study material. It's what you use to teach or 'train' your model. Validation Set: Think of this as your practice tests. Once you've studied (or trained your model), you want to test its understanding. You use the validation set to fine-tune your model, ensuring it's learning correctly. Test Set: This is your final exam. It's data your model has never seen before. You use it to see how well your model has learned from the training and validation sets. It gives you an unbiased evaluation of the model's performance on new, unseen data.

✍🏽Documenting the data used in every iteration of an ML experiment is a good practice. Using a system like a feature store to maintain the data is highly recommended. The advantage is tracking the lineage of the data and integration with data cataloged in your organization. ✒When relaying a feature, store the data, and the pipelines become repeatable and easy to document. Incremental changes are traceable and easy to integrate with production workflows in the MLOps phase. 📃As a best practice, following the CRISP-DM standards in documenting the data and data processing. 📀One of the pitfalls in ML experiments in the enterprise is creating data silos. Every step to keep the data used for ML experiments close to data systems is best.