How can you prepare data for machine learning algorithms?
Machine Learning
Perspectives from experts on the questions that matter for Machine Learning
This article was an early beta test. See all-new collaborative articles about Machine Learning to get expert insights and join the conversation.
Prior to applying machine learning algorithms, it is essential to ensure the data is formatted in such a way that the algorithms can be used effectively. Taking the time to do this work up front may ultimately lead to a more accurate and robust machine learning model. The exact steps required for data pre-processing will vary depending on the specific dataset, but here are a few techniques to clean, format and prepare data before it is used.
1. Explore and analyze the data: When working with machine learning algorithms, it can be extremely helpful to understand what information the dataset contains, the data types and the relationships between different features and target variables. This information will ultimately inform the next steps needed to clean or organize the data moving forward.
"[...] The data must be carefully selected based on any understanding of the problem at hand. [...] It is a common misunderstanding that one just needs to feed the available data to ML and the algorithm will 'find' the information in the data. This is why many data analytics projects end up disappointing."
— Andrea Malagoli is a structured products specialist at insurance company Swiss Re Corporate Solutions. He holds over 12 years of experience in data science and earned his PhD in physics and computational science from the Sapienza University of Rome, as well as his MBA from the University of Chicago.
2. Clean the data: This step involves correcting erroneous data values, filling in missing values, and removing duplicate entries. There are various options for dealing with invalid or missing data, such as replacing invalid entries with the mean, median or mode of the column, or removing rows with missing or invalid data altogether.
"Check and double check for completeness of data. Don’t make any assumptions. It is very easy to introduce bias with incomplete or poorly sampled data."
Recommended by LinkedIn
— Evan Follis is a data scientist at investment management firm AllianceBernstein. He holds over 3 years of experience in data science and earned his master's degree in analytics from Georgia Institute of Technology.
3. Format the data: Data often needs to be formatted in a way that is compatible with the algorithms being used. For example, categorical data often needs to be encoded into numbers so that machine learning algorithms can use them effectively. Additionally in some cases, redundant or unimportant features need to be removed or combined.
4. Normalize or standardize the data: Normalization rescales the data so that all values fall between 0 and 1, while standardization applies a z-score transformation to rescale the data to have a mean of 0 and a standard deviation of 1. These steps help to ensure that different features contribute equally to the machine learning model, and that the model is not dominated by a few high-magnitude features.
5. Split the dataset: Splitting the data into training, validation and testing sets allows for model evaluation and comparison by ensuring that the machine learning model is tested on unseen data. This allows for less bias and greater accuracy when the model makes predictions.
Explore more
This article was edited by LinkedIn News Editor Felicia Hou and was curated leveraging the help of AI technology.
Senior Information System Security Officer @IBM | Lifelong Learner| U.S. Army Veteran| Problem Solver| Information Systems Engineer with Empathy
2yInitially we can not be too data centric. We must first work to understand what problem needs to be solved. Understand our customers need and where it provides business value. We must also work to ensure what type of data needs to be collected and where it will be stored whether it’s confidential or public and what laws my impact our collection and experimentation efforts. Next before we can automate we must validate the data for accuracy this will likely take the longest time before any algorithms can be developed.
CIO Driving Digital Transformation | Leadership Force Multiplier
2ySome of the most important preparations and considerations for machine learning are the following: 1. Imperfection in the Algorithm when data grows 2. Irrelevant data being collected for training 3. Lack of training data to provide the desired outcome 4. Data overfitting
Statistician - Data Scientist and Instructor | Helping your organization get value from data through machine learning and data analytics | Shiny Developer | Mobile Data collection | R, STATA, PYTHON, SPSS, SQL, XLSForm
2yIt's also important to handle missing values. When missing values are randomly distributed, median imputation can help. In case they aren't randomly distributed, Knn imputation can do the job. For other algorithms, scaling can also be thought.
Market Research
2yI think most of the ideas behind data wrangling have been covered. One must always remember to go through the data and data type thoroughly. Only then will data wrangling be effective. A very common idea is to fill nulls with a mean value, sometimes it may make sense to see the distribution of data and take a judgement call on whether to replace it with the mean, median, mode, or maybe just remove it
Global Head of Research | Chief Data Scientist | VC Advisor | Faculty
2yHire a good Data Engineer...