How do you conduct your own data science experiments?

I agree, breaking down a problem into the most simpler form can give us more insight in finding the right solution. Sometimes the business problem may seem complicated at first but when we narrow it down the logic for solution will be simple. Problem solving is the key as the first step.

As an economist, econometrician, and data science/AI/ML leader, conducting data science experiments involves a rigorous methodology: defining clear hypotheses, collecting and cleaning data, choosing appropriate models, and systematically testing while controlling for bias and variance. From a think-tank leader's perspective, it's essential to approach these experiments with an interdisciplinary mindset, incorporating economic theory, statistical rigor, and computational efficiency. This holistic approach ensures that experiments not only yield technically sound results but also offer actionable insights that address real-world challenges, driving societal and economic progress.

First things first, it is important to scope the problem and lay down the steps we can take to achieve the result. In the process of step-wise breakdown of the problem, it makes us understand the complexity of problem and in fact the downstream usage of the result. For example, to answer a question of how much of revenue is generated last month by a subscription based company? We have to first lay down, what are the sources of revenue. In the process, we have to decrement the revenue loss due to churn, increment the revenue coming due to new customers, how much renewal has happened etc. All these sub problems can be combined to answer a larger question of what is the total revenue.

Conducting data science experiments is a systematic process that begins with defining a precise question to address through data analysis. This initial step is crucial for guiding the entire experiment and ensuring meaningful insights are derived. By establishing a clear and specific question, researchers can effectively narrow down their focus, identify relevant data sources, and formulate hypotheses.

This is the core to not only data science, but applied mathematics, computer science and I believe science and engineering in general. Find the core, root, simplest representation of the problem to solve. Understand the fundamental problem before looking for a solution. Often not easy. I think there is a suitable quote from Einstein "make it as simple as possible but no simpler".

Without visualizing the data early on you run the problem of getting stuck with the wrong identifiers a lot further down the road, when the only way for fixing the issue becomes erasing the whole process and starting from scratch. Even if you are missing vital data, through proper data visualization you may arrive at similar anomalies using different univariate or multivariate processes and proactively replace missing data with equivalent quality replacement and equal accuracy rates. Through data visualization, 40-60% you are likely to be proven wrong on your original individual hypothesis, even if you are a long-term practitioner of the given trade.

Collecting and exploring data is a pivotal phase in the data science experiment, following the formulation of a clear research question. This step involves gathering relevant data and delving into its intricacies using descriptive statistics and visualizations. Various data collection methods, such as web scraping, utilizing APIs, accessing databases, or conducting surveys, can be employed based on the nature of the question and data availability. For instance, if investigating the influence of weather on ice cream sales, data collection might involve obtaining weather data from meteorological sources and sales figures from relevant businesses.

Explore like a detective by sifting through messy data to find anomalies and clues for analysis! This part has the thrill of the unknown; you don't know what insights will emerge once you dive in. Yet, it's crucial to avoid making quick assumptions. This is also where you become familiar and comfortable enough with the data to make executive calls on what to drop, modify, omit, or add. This initial groundwork not only helps focus the problem but also paves the way for actionable insights.

When approaching data collection and exploration, it's crucial to align your methods with your specific objectives. While tools like pandas and matplotlib are effective, their usage should be guided by the insights you seek. Consider the relevance of missing values and outliers in the context of your question. If they don't provide meaningful insights or you're not developing an AI model, these aspects might not be your primary focus. Tailor your data exploration strategy to extract the most relevant information for your business question, rather than following a generic process just because someone told you to do.

Once you have your question, the next adventure is data collection and exploration. It's like a treasure hunt; you may use various tools—web scraping, APIs, databases, surveys—to gather your data gems. 1.Diversify Data Sources: More sources, richer insights. 2.Master Data Wrangling Tools: pandas is your best friend. 3.Visualize to Understand: Let matplotlib and seaborn guide you. 4.Look for Missing Values: They hide secrets and pitfalls. 5.Spot and Handle Outliers: They can skew your story. 6.Understand Data Distributions: Know what's normal and what's not. 7.Seek Correlations Carefully: They whisper of relationships. 8.Document Your Findings: Keep track of your data journey. Remember, this step is about laying a solid foundation for you.

One of the most important part of this step is Feature Engineering. Sometimes the data that we get might not be usable, even for exploration. In that case, we need to generate extra variables from the original data. For example, if your data is audio clips and your data science problem is "Is the tone of the audio sad or happy?" Just converting the audio to text is not enough, because audio has a lot more data than the words itself. Then in such scenarios other variables are extracted from audio using different techniques or libraries like librosa. The point being, as a data scientist we need to think not just how to make the data more uniform but also we need to ask "Is there any additional data that I can extract from the current data?"

This is a very important step that can heavily influence your data analysis. Ensuring you have the right variables at the right format and that you have imputed missing values to the best of your ability can define the results of your analysis.

The term "feature engineering" encompasses a range of activities aimed at transforming raw data into features that better represent the underlying problem to predictive models, thus improving their performance. This process can include cleaning, filtering, imputing, encoding, scaling, or aggregating data. The key is to align these transformations with your business objectives, ensuring that each feature contributes meaningfully to the model's ability to make accurate predictions. Utilizing feature importance metrics can be a valuable strategy to assess and refine the relevance and impact of each feature in your model. Remember, the goal is to enhance the data in the context of your specific problem.

Preprocessing and transforming data represent crucial steps in the data science workflow, aiming to prepare the dataset for effective analysis. Prior to applying any data science techniques, it is essential to undertake processes such as cleaning, filtering, imputing, encoding, scaling, or aggregating data, tailored to the specific type and quality of the dataset at hand. For instance, in the context of investigating the impact of weather on ice cream sales, preprocessing steps might involve cleaning the dataset by removing irrelevant columns that do not contribute to the analysis. Additionally, missing values in the weather or sales data may need to be imputed to ensure a complete and representative dataset.

Data preparation is a very important and time-consuming task. First, you need to understand the dataset by performing exploratory data analysis. Once you're familiar with your dataset, there are several areas to consider, and for each area, there are various techniques that can be applied. These include data cleansing, for example removing duplicates, white spaces. Identifying outliers using a proper methods.Handle null values using suitable methods such as mean imputation. Integrate the dataset with another data sets. Derive variables or identify a class variable. Balanced imbalanced datasets. Handle categorical values by converting to numerical. Identifying features by using suitable method and scale the data set.

Fundamentally, the choice of method should be determined by the problem statement. If the available methods at hand are not complimentary to this problem statement, it is best to revise the problem statement. As an example: Company C is trying to determine a model for the expected sales of Product P. Company C puts forth a problem statement: Based on our current revenue contributions MoM of P, RC(P), what is the next month's predicted sales? The model attempts to provide approximations to this RC(P) of the following month given the previous months. However, if unsuccessful after a suite of tests, consider changing the problem statement to see if the results will change: Instead of previous MoM sales, one might consider adding other factors.

Selecting and applying an appropriate data science method is a pivotal phase following data preprocessing, aiming to address the research question effectively. The choice of method depends on the nature of the question and the characteristics of the dataset, encompassing options such as supervised or unsupervised learning algorithms, statistical tests, simulations, or natural language processing techniques.

I would like to know Poornima's insight on how to select an appropriate data analytics techniques for an analysis. What would be the thought process?

As much as the choice of the method is pivotal, there are cases where simple beats the complex. Therefore, in my opinion, use the simple method first, build a simple model, then work further on modifying/engineering data that can enhance the model, then improve the method itself. For example: regression (can be complex) or binary classification (simpler option) to begin with. It helps often to understand the overall potential.

Effectively evaluating a data science training program involves defining clear objectives, reviewing the curriculum, assessing the instructors, and testing the knowledge and skills of the participants after completion. It's important to anticipate and address competing explanations, determine the best way to evaluate the results, explore the data appropriately, build or apply relevant algorithms, and clearly document findings. A framework for evaluation should be explicitly planned and include measures of individual skills such as drafting policy and procedures. Focus should be placed on continual improvement, with the ability to adapt and respond to changing trends and demands in the field of data science

It's not just about making a model. We need to make sure it works in real life. Ask yourself: Is the model fast enough for daily use? Is it better than what we already have? How much does it cost to use or set up? Does it really help our company's specific needs? It's key to check these things to know if our model can be used in a real-world business setting!

The concluding phase of a data science experiment involves the critical tasks of evaluating and interpreting the results obtained through the chosen methodology. This step is fundamental in determining the effectiveness of the analysis in addressing the initial research question. To assess the performance, accuracy, and validity of the results, researchers should employ relevant metrics and methods, including but not limited to error rates, confidence intervals, or p-values. For example, if a linear regression model was applied to predict ice cream sales based on weather variables, assessing the model's performance might involve metrics such as Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE).

In machine learning, it's not enough to build a smart model. After we create it, we need to figure out which pieces of information are the most important for making predictions. We call this "feature importance." It's like knowing which ingredients make a recipe taste the best. Once we know which ingredients (or features) matter the most, we can use them to make better predictions and understand what's happening in our data. This helps us keep things simple and make our predictions useful for everyone. So, it's not just about building the model; it's about knowing which ingredients make it work its magic!

Metrics like accuracy, precision, recall, or more complex ones like ROC-AUC are standard. Interpreting these results in the context of your question is key. My work in computational cognitive science taught me the importance of not just looking at numbers, but understanding the story they tell about human behavior or system performance.

At this final stage, we measure our journey’s success. Use metrics like error rates or confidence intervals as your compass to assess accuracy and validity. But remember, these numbers are more than just digits; they narrate a story, answering your initial question. Look beyond the numbers; what do they imply in the real world? This isn't just about data; it's about insights shaping our understanding and actions. Stay critical, curious, and always contextualize your findings within the big picture.

In the world of data science, communication is often the bridge between complex analysis and actionable insights. Throughout my career, I've found that the ability to translate intricate data patterns into clear and concise recommendations can make or break a project. Whether it's presenting to stakeholders or collaborating with team members, effective communication ensures that everyone is on the same page. It's not just about sharing numbers or charts, it's about telling a story with data, a story that resonates with the audience and drives decisions. This has become a cornerstone of my approach, allowing me to turn raw data into strategies that make a real impact.

Communicating the results to stakeholders can easily be the most over-looked part of an analysis. It does not matter how fancy the Maths of the model is, if the stakeholder does not see a clear story - the battle is lost ! 'Make the complex simple' for your stakeholders, focus on story-boarding, know your audience and be prepared to tailor your message for effective communication of Data Science projects' outcomes.

An important part of doing data science work is sharing your findings. I have found that the quality of your work is in fact capped by how well you can share what you have found. In the movie 'Margin Call', Jeremy Irons plays a big-wig executive at the start of the 2008 financial crisis. He is talking with an analyst, and he says 'speak as you might to a small child, or a golden retriever'. This reminds us of how complex data science topics and mechanics may be casual to a scientist, but for the customer the broad conclusions are much more important.

To conduct effective data science experiments based on my experience, I'd follow these steps: - identify the problem and metrics, - form a hypothesis, - design and collect data, - clean and prepare it, - analyze using statistical methods and ML, - interpret results, draw conclusions, and make recommendations. Iterate or make improvements as needed based on feedback. For example, in predicting buyers churn, collect data on demographics, purchase history, and support interactions, then train an ML model. Use appropriate tools, document process for reuse, and maintain a critical perspective to ensure reliable results and impactful insights. Feedback and metrics to decide reliability of the experiment.

Lots of good takes on what to do. Let's talk about what NOT to do. -Do not over-filter data. Understand the real life scenario where the data will be used, and use just enough filtrations to make sense of data. Over-filtering in the name of data cleaning often produces useless models - Do not torture data. It is said, if you torture your data enough, it will say whatever you want it to say. Be vigilant when using imputations. Consider, do you even need imputations? Or having a missing information is an information by itself. Are you over-simplifying the problem? Are your data insights aligned with SME expectations? Can you confirm your insights?

As per my experience, the real world data is way messier than the publicly available data that we use for learning. So selecting/rejecting columns and rows so much so it doesn't add error after filling null values and also it doesn't add any kind of bias, is where the art in data science lies. You can give this messy data to 10 different data scientists and most of them will come up with different conclusions. So, to draw correct conclusions it's necessary to get better at devising your own strategies to tackle abnormalities of real-world data.

In my experience one of the most significant aspects of data science experiments is ensuring you have an unbiased, representative dataset that allows you to draw valid conclusions. Unlike controlled lab experiments, data scientists typically start with observational data that reflects existing biases and limitations. Failing to recognize and account for hidden biases and limitations in your training data can lead you to build models that inadvertently perpetuate unfairness or fail to generalize.

n the realm of data science, always remember the human element. Beyond the numbers and algorithms, each data point can represent a real-world story, a human behavior. Embrace the unexpected – those outliers and anomalies might just lead you to groundbreaking insights. And never stop learning; the field is always evolving, just like our understanding of the world around us. In essence, stay curious, empathetic, and adaptive.