Shubham Dubey

The Sparks Foundation Hello, Connections! I am thrilled to share my #Task3 for Data Science And Business Analytics at The Sparks Foundation. ↔ Objective:-Understand purchasing behaviors and preferences. ↔ Methodology:-Used Python (NumPy, Pandas, Matplotlib, Seaborn) for data analysis and machine learning for pattern recognition. ↔ Key Insights: Identified key customer segments based on trends. As a business manager, try to find out the weak areas where you can work to make more profit. What all business problems you can derive by exploring the data? You can choose any of the tool of your choice (Python/R/Tableau/PowerBI/Excel/SAP/SAS) 🔗 Explore the Project: Access the code on GitHub:- https://lnkd.in/gUuXmJ3X #DataAnalysis #MachineLearning #Python #GRIPJULY2024 #TSF #DataScience #TheSparksFoundation The Sparks Foundation

13

9 Comments

I have seen 2 interesting bifurcation within industry profiles of data scientists : 1. Dude Data scientists - These profiles are tech heavy profiles. - These data scientists breathe in & breathe out docker, cloud , kubernetes , SQL , Deployment , production , ML , Analysis, APIs etc - These are not ML engineers either. They fight for this identity but when they are about to get it someone assigns them to an automation project spoiling everything. - These people have panic attacks if model/data sees drift in production. - They are often diagnosed with insomnia. 2. Data science Consultants who call themselves as data scientist just to look cool ! - These profile are blend of data scientist + product consultants - These data scientist spend 50% of their time working on ML , stats and 150% of time explaining to business why these framework will work. They always create 1 .ipynb file & 5 PPT decks. - They have immense confidence while explaining anything to anyone since they have been explaining meaning of p-value to non-tech stake holders often - They are often diagnosed with imposter syndrome - They dont know who they are, some see them as core data scientists , others see them as a consultants. - They face rejections in data science interviews since they are heavily working on consulting & face rejections in consulting roles since in tech rounds they dont know how the hell XGBoost works - They suffocate if chatgpt stops working some day. What kind of data scientist are you ? #datascientist #sarcasm #fun #humour #datascience #python #machinelearning #ai #learning #growth

194

22 Comments

[playground] Quantile Regression (with Keras & Scikit-Learn) Informed decision-making is the cornerstone of effective business. Quantile regression is a technique for estimating the conditional quantiles of the response variable across values of the predictor variables, which is a powerful tool for making actionable insights. Application of Quantile Regression: - Based on distance (and other predictor variables), estimates delivery time (quantile) that covers (percentile)% on-time deliveries. Case study: (check link in my blog) In this tutorial, we will fit a quantile regression model with Keras & Scikit-Learn. https://lnkd.in/ejPSmNC3

26

HOW NOT TO DATA ANALYTICS (or maybe how to?) #IndianElections2024 #TheQuint #BJP #IndianNationalCongress #DMK I saw this post yesterday on Instagram from The Quint’s handle titled - Party wise MPs with declared criminal cases. The post was adorned with colorful visualizations. The Data Scientist in me got very interested in the post, and Curious as I was, I started looking intently at this post. Firstly, Quint has been doing a wonderful job at pointing out misleading factual comments made by the ruling BJP folks. They’re spot on when it comes to making people aware of these fallacies. But they fail miserably when it comes to false facts from anyone opposing the BJP. Now, I was surprised when I first glanced at the visualization after reading the image title. Was surprised that of all the political parties in India, the ruling BJP had the most MPs with criminal cases. But when you look closely, you’ll realize how beautifully the reader is misguided. Though BJP has the most number of MPs in the Indian Parliament, only 39.2% or their MPs have criminal convictions. Whereas, Indian National Congress has 49.5% and DMK has 59% of their MPs with criminal cases. The viz needs to be changed to reflect the facts and not mislead the common citizen. Is this a case of genuine mistake by #TheQuint or was this an intentional anti-BJP propaganda to mislead the common man? One can only wonder.

4

🚀 Day 15/20 of Revisiting Apache Spark: GraphX and Advanced Graph Algorithms 🚀 Today's session delved into graph processing with GraphX in Apache Spark. Here are the key takeaways: GraphX Overview: - Graph Representation : Graphs in GraphX are represented using vertices (nodes) and edges. - Power of GraphX : Combines the advantages of both graph processing and data parallelism. Graph Algorithms Explored: - PageRank : Measures the importance of each node in a graph. Used extensively in search engine algorithms. - Connected Components : Identifies clusters of connected nodes, useful for social network analysis. - Triangle Counting : Counts the number of triangles passing through each vertex. Important for understanding the local clustering coefficient. GraphX empowers us to perform complex graph analytics at scale, harnessing the full power of distributed computing with Apache Spark. #ApacheSpark #GraphX #GraphProcessing #PageRank #ConnectedComponents #TriangleCounting #DataEngineering #LearningJourney

4

This week Recommendation : Satyajit Pattnaik's End-to-End Statistics Series launched last Saturday and everybody liked it, and good great feedback. In case you haven't seen it, here you go. 𝑬𝒏𝒅 𝒕𝒐 𝑬𝒏𝒅 𝑺𝒕𝒂𝒕𝒊𝒔𝒕𝒊𝒄𝒔 - 𝑷𝒂𝒓𝒕 1 👇 https://lnkd.in/gvSyrFY3 𝑬𝒏𝒅 𝒕𝒐 𝑬𝒏𝒅 𝑺𝒕𝒂𝒕𝒊𝒔𝒕𝒊𝒄𝒔 - 𝑷𝒂𝒓𝒕 2 👇 Goes live today at 6 PM IST, It's covering Probability and Probability distribution!! 💪 Link to LIVE Session below 👇 https://lnkd.in/gUYdZkuE #Statistics #Hackweekly #TheHackWeekly #Probability

42

1 Comment

🚀 Week 15 Update: Ultimate Big Data Master's Program by Sumit Mittal at TrendyTech 🚀 I’m thrilled to share that I’ve successfully completed Week 15 of the Ultimate Big Data Masters' program! 📚 This week, we delved into the intricacies of Apache Hive 🐝 and its vital role in big data processing. Here’s a snapshot of what we covered: 1. Transactional System vs. Analytical System 🔄📊 2. Formal Definition of Apache Hive 📖 3. Why is Metadata stored in a Database and not in a Data Lake? 💾🏞️ 4. High Throughput, Low Latency ⚡ 5. Schema on Read and Schema on Write 📝🔄 6. Hive Services (HMS, HiveServer2, Beeline) 🛠️ 7. Two Ways of Connecting to Hive 🔗 8. Creating Databases, Tables, and Inserting Values 🛠️📊 9. Types of Tables in Hive (Managed Table, External Table) 📂📁 10. When to Use External Tables? ❓ 11. Creating and Loading Data into a Hive Managed Table 🗃️📥 12. Setting hive.execution.engine to Spark 🔥 13. Hive Optimizations (Table Structure Level, Query Level) ⚙️🚀 14. Partitioning vs. Bucketing, Join Optimization 📊🔄 15. Types of Partitioning: Static and Dynamic ⚙️🔧 16. Query Level Optimization - Join Optimization (Map-side Join, Bucket Map 17. Join, Sort-Merge-Bucket Join) 🔄🛠️ 18. Internal Working of Map-side Join 🧩 19. ACID Properties in Hive 🛡️ 20. Slowly Changing Dimensions (Fact Table, Dimension Table) ⏳ 21. Delta Files 📑 22. Hive-Spark Integration 🔗🔥 23. Hive MSCK Repair 🛠️ This week’s focus was on mastering the functionalities and optimizations of Hive, understanding the importance of metadata management, and enhancing query performance through effective partitioning and join strategies. Huge thanks to Sumit Mittal and the TrendyTech team for this comprehensive learning experience. #BigData #ApacheHive #HiveOptimization #BigDataLearning #HiveIntegration #Spark #MetadataManagement

19

2 Comments

Excited to share the completion of the 'Apache Spark Caching In-Depth' of week 7th, a significant milestone in my Big Data Journey with TrendyTech and Sumit Mittal Sir. Important topics are included below: ·       Steps for accessing Spark UI ·       Understanding Cache and Persist ·       Parsed | Analyzed | Optimized Logical Plan ·       Cache – In Memory table Cache | Node and Process Local ·       Caching Spark table ·       Spark Catalog, Managed, External tables ·       Cache Performance ·       Understanding Persist Thank you TrendyTech and Sumit Mittal Sir for curating such an amazing program. Looking forward to the upcoming weeks! #Bigdata #Spark #Python #DAG #Join #dataengineer #AWS #Azure #ADLS #HDFS #Linux #SQL #athena #redshift

38

3 Comments

Great to have been a part of Growth Square (By Devoic) Squarelight Session as a guest speaker! It was an amazing experience sharing insights on real-world data applications, building portfolios, and career growth in data science. Big thanks to Shivam S. and his team for this opportunity and to everyone who joined the session! Let's keep pushing the boundaries of learning and development together. 🌟 #DataScience #CareerGrowth #KnowledgeSharing

24

5 Comments

📌 2016 → Completed BTech in ECE 📌 2017 → Data Analyst 📌 Today → Senior Data Scientist This Transition was not easy for me, but it’s been a rewarding journey of continuous learning and growth. 🙌 Though my background was in ECE, I quickly developed a passion for analytics. Working as a Data Analyst at various companies, I became fascinated by it’s impact This curiosity led me to dive deeper into Data Science. I explored the field through a combination of online courses, workshops, and hands-on projects. Here's how I approached it: 👉 Started with mastering Python and strengthening my Math and Stats foundation. 👉 Delved into Machine Learning algorithms and concepts. 👉 The game changer? Gaining practical experience by working on real-world datasets. After over 5 years of persistence and dedication, I made a complete and successful switch to Data Science! 🎉 It wasn’t a short journey, but it was absolutely worth it. 💪 So, if Data Science excites you as it did for me, don’t wait to start today and get the right guidance early on, Bosscoder Academy can help you with the same Check here: https://bit.ly/3U3Qduu They can help you with: ✅ Structured curriculum to learn Data Visualization, Deep Learning, ML etc. ✅ Getting you job-ready for top companies. ✅ Personal mentorship from experienced Data professionals. ✅ Live classes, project-based learning, and access to a talented community of people with similar goals of upskilling.

382

30 Comments

🚀 Week 3 of the Ultimate Big Data Masters Program by Sumita Mittal sir focused on distributed processing with MapReduce and Apache Spark. We explored how MapReduce enables parallel processing by using Mappers and Reducers, and the role of combiners in optimizing performance. A real-world example was processing sensor data to find maximum temperatures. In addition, we dived into Apache Spark, a high-performance in-memory compute engine. Spark offers significant performance advantages over MapReduce and supports PySpark for scalable data engineering. I’m excited to continue honing my skills in distributed computing! 💻🌍 #BigData #MapReduce #ApacheSpark #CloudComputing #TrendyTech

41

1 Comment

New scientist Express batch start alert : Starting a fresh batch for Data Science & Machine learning for career transitions from next week, TUESDAY 9PM IST, THURSDAY 9PM IST, SUNDAY 1PM IST , 8 months industry ML AI course FOR CAREER TRANSITION - Covering * 10+ actual interview screening projects for high ctc offers, covering Telecom, Banking, Credit risk, Ecommerce, Retail interview projects * 3 actual interview case studies for healthcare, banking, retail * Python 8 pillar approach, All ML algorithms with Explainability, Calibration, Monitoring * 10+ high ctc Industry projects A score, B score, C score for Credit Risk, Fraud prediction for Insurance, Forecasting for retail, Lead Prioritization for insurance , ECL model for Banking, Diabetic Retinopathy detection Using Computer Vision & many more with 250+ successful high CTC career transitions into Data Science ML AI across India, US, UK, Canada, Australia, Europe so far!!

2

Hey friends, Continuing with the series for Machine Learning concepts and interview problems for everyone. Q5: What is validation data and why is it important? Validation data is a small part of the training data (10-20% of the training data) which is not used in training but is used to check the performance of the model every x steps to check if the loss on the validation data is decreasing a.k.a. the model is improving or not. The validation data is also important to prevent overfitting by finding out the exact point where both the train loss and validation loss decrease since if train loss continues to decrease beyond this point and validation loss stagnates or starts increasing then the model has started to overfit and we need to stop training at this point. Feel free to add your ideas too! #machinelearning #ML #interview #CareerDevelopment

139

Documenting my 30 day learning journey with Learning Utsav Day 1 : Hypothesis Testing with Python! Today was all about understanding about hypothesis testing, a crucial aspect of data analysis. Here’s what I learned: Purpose of Hypothesis Testing: I explored why hypothesis testing is essential in making informed decisions based on data. To solidify my understanding, I conducted a minor project analyzing user data. Check out my code on GitHub here : https://lnkd.in/d9w4cdYm Statistical Tests: I delved into various hypothesis tests, including: Z-Test: Used for comparing sample and population means when the standard deviation is known. T-Test: Applied for comparing means between two groups. Chi-Square Test: Great for testing relationships between categorical variables. ANOVA: For comparing means across multiple groups. Parametric vs. Non-Parametric Tests: I learned the key differences between these types of tests. Parametric tests assume underlying statistical distributions, while non-parametric tests do not. Wilcoxon Signed-Rank Test: For non-parametric analysis, I focused on the Wilcoxon Signed-Rank Test, which serves as the non-parametric equivalent to the t-test, ideal for paired samples. #LearningUtsav #Hypothesistesting

24

Visualizing Virat Kohli's IPL 2024 Performance with Linear Regression Analysis As part of my recent data analysis project, I created a comprehensive visualization of Virat Kohli's performance in the IPL 2024 season. Leveraging the power of linear regression, I analyzed his scores to identify trends and patterns throughout the season. Colab Link: https://lnkd.in/ggeZJieb Dataset link:(Kaggle) https://lnkd.in/gPCWmtkq Data Overview: The dataset includes detailed statistics from each match, capturing key metrics such as the number of runs scored, balls faced, boundaries hit (fours and sixes), and strike rates. This rich dataset provided a robust foundation for my analysis. Methodology: I employed linear regression to model and predict Virat Kohli's performance over the course of the season. Linear regression, a staple in predictive analytics, helps in understanding the relationship between dependent and independent variables. In this case, it enabled me to predict future scores based on his performance in previous matches. Findings: The linear regression model highlighted significant insights into Kohli's performance dynamics. Notably, it revealed trends that could help in anticipating his scoring patterns in upcoming matches, providing valuable strategic information. Visualization: The visualization showcases the linear regression line plotted against Kohli's actual scores, offering a clear and intuitive representation of the analysis. This visual aid not only makes the data more accessible but also underscores the predictive power of the regression model. Conclusion: By applying linear regression to sports analytics, we can gain deeper insights into player performance and make more informed predictions. This project exemplifies the intersection of data science and sports, demonstrating how analytical techniques can be used to enhance our understanding of the game.

18

Finding an element in a semi-sorted matrix Suppose that we are given an m x n matrix M with integer elements, where m and n are positive integers. All the m rows of the matrix are sorted in increasing order. All the n columns of the matrix are sorted in decreasing order. Given an integer k, design an algorithm to find k in the matrix M in O(m + n) time.

2

A better question to ask might be: Are we addressing the root causes of class imbalance, or are we simply applying surface-level fixes that risk introducing new issues? Rather than finding a "fix," it’s more effective to approach the problem retrospectively by examining its underlying causes and impact: 1. Why does class imbalance exist in the first place?. Is it due to natural rarity, data collection biases, or labeling inconsistencies?. Tackling these root causes can often reduce the severity of the problem and prevent reliance on overly technical fixes. 2. It’s 2024, and we’re still SMOTing up datasets like it’s the ultimate cure for all data imbalances?. Techniques like SMOTE and ADASYN rely on linear interpolation, which often falls apart in high-dimensional spaces, leaving us with synthetic samples that feel more like guesswork than data. Meanwhile, oversampling risks overfitting, and undersampling throws away valuable majority-class information. These quick fixes may work temporarily but rarely address the deeper issues behind the imbalance. 3. Are we distorting the data distribution?. Generating synthetic samples assumes the true minority class distribution is well-captured, but if the original data is sparse or noisy, these methods risk amplifying errors instead of adding value. 4. Are these strategies helping models generalize to real-world imbalanced distributions?. While many techniques improve metrics on artificially balanced datasets, they often fail to address performance on naturally skewed test sets. Finally, instead of treating imbalance as a problem to “fix,” we should ask how models can leverage imbalance to make better predictions—such as prioritizing high-confidence decisions for minority classes or using imbalance to uncover deeper patterns.

2