sarah moussaoui’s Post

Just finished the course “Learning Apache Airflow” by Janani Ravi! Check it out: https://lnkd.in/eMMqwNdY #apacheairflow #itautomation. It's a great course for getting started with Apache Airflow and learning some basics.

🚀 Excited to share a solution I've implemented in Apache Airflow that's streamlined my workflow and reduced redundant coding by leaps and bounds! 🌟 As data engineers, we often find ourselves wrestling with repetitive tasks, especially when it comes to setting up similar DAGs in Apache Airflow. In my journey, I encountered a pattern: I had around 100 DAGs with striking similarities, which led to hours of monotonous coding and maintenance. But hey, necessity is the mother of invention, right? 💡 Introducing my newfound gem: DAG templates coupled with the power of multiple file techniques. 🌐 By harnessing these techniques, I've managed to slash my coding time significantly while ensuring scalability and maintainability across the board. Here's the lowdown on how it works: 1️⃣ DAG Templates: Instead of reinventing the wheel for each DAG, I crafted a versatile template that encapsulates the common structure and logic shared among them. This not only saves time but also promotes consistency and reduces the chances of errors creeping in. 2️⃣ Multiple File Technique: Leveraging this technique, I modularized my DAGs into separate files, each focusing on a specific aspect or functionality. This modular approach not only enhances readability but also simplifies debugging and encourages code reuse. The result? A cleaner, more efficient Airflow environment where managing and scaling DAGs is a breeze! 🌬️

Just finished the course “Learning Apache Airflow”! Check it out:

🚀 Master Apache Airflow with Ease! Learn How to Install Apache Airflow Using Docker and create your very first DAG (Directed Acyclic Graph) in this step-by-step guide. 👉 What you'll learn: ✅ Set up Airflow with Docker ✅ Write your first DAG from scratch ✅ Simplify workflow orchestration Whether you're a beginner or looking to sharpen your data engineering skills, this guide has you covered. Start building powerful data pipelines today! Read the full blog here: https://lnkd.in/dm_xmdjF ✨ #ApacheAirflow #DataEngineering #WorkflowAutomation #Docker #LearnWithMe #TechBlog #AirflowDAGs #CloudComputing #DataPipelines #TechTutorials #DeepSeek link: https://lnkd.in/dfFFWH4D

Have you ever wanted to try out Apache Spark without tons of stitching things here and there? What about Minio or Apache Iceberg? I've found it already done by Tabular folks (https://lnkd.in/gGUw_8ty), so it saved me tons of time battling docker-compose and other time-consuming things. The repository itself is quite good, with examples of how Iceberg works, but you'd have to be more familiar with Apache Spark. So I copied (Yeah, maybe forking would have been better, but it is what it is) and added some introductory things about Spark that I thought were missing. I used this in a workshop during PyCon Lithuania and explained things in greater detail there. Instead of interacting with listing things (that's done in the repo) in Jupyter Notebooks, I went to the Minio console and checked stuff through there. If you want an easy intro to Spark and how Iceberg connects with it - feel free to check and try it out here: https://lnkd.in/gz9iHxJn ---- Stay in the loop for the latest insights from Uncle Data! * Click the notification bell icon 🔔 to receive updates on Linkedin * Subscribe to my Newsletter on Substack ✏ * Listen to Uncle Data Podcast ⌨️ Listen. Learn. Connect ---- #spark #iceberg #learning #workshop

Just finished Learning Apache Airflow! Check it out: Thanks to Janani Ravi for this amazing short and comprehensive course. https://lnkd.in/gn7ej-Vs #apacheairflow #itautomation

🎓 I am thrilled to announce that I have successfully completed the "Learning Apache Airflow" course! This comprehensive program has equipped me with in-depth knowledge and practical skills in Apache Airflow and IT Automation. During this course, I have learned: 📚 Apache Airflow concepts and architecture 🛠️ Modeling workflows with tasks and dependencies 🔧 Creating tasks using Operators 🔄 Passing data using XCom 🗂️ Visualizing workflows using TaskGroups 🕒 Implementing Cron scheduling, catch up, and backfill This learning journey has significantly enhanced my ability to design, implement, and manage complex workflows. #ApacheAirflow #ITAutomation #WorkflowAutomation #LearningJourney #ProfessionalDevelopment

Just finished "Learning Apache Airflow" by Janani Ravi! During this course, I've gained deep insights into: • Apache Airflow's core concepts and architecture • Designing complex workflows with advanced task dependencies • Creating tasks using powerful Operators • Mastering data exchange techniques with XCom • Visualizing workflow structures using TaskGroups • Understanding and using Cron scheduling, catch-up and backfill mechanisms for workflow management Check it out: https://lnkd.in/dj9DVCgc #apacheairflow, #itautomation.

🚀 Excited to announce the second installment of my five-part series on Apache Airflow is now live on Medium! 🎉 In this series, I'm diving deep into Apache Airflow, a powerful tool for orchestrating workflows and data pipelines. Whether you're new to Airflow or looking to enhance your skills, this series has something for everyone. 🔗 Apache Airflow Part 2: Understanding DAGs: https://lnkd.in/eYfsWkeS Next in the Series: Example Data Pipeline with Airflow - Part 3 Advanced DAG Concepts and Best Practices - Part 4 Integrating Airflow with External Systems - Part 5 This second post focuses on Directed Acyclic Graphs (DAGs), the core concept in Airflow. Learn about tasks, dependencies, schedules, and how to create and deploy your own DAGs. This article provides practical examples and step-by-step guidance to help you master DAGs in Airflow. Stay tuned for more updates, and feel free to share your thoughts and questions in the comments. Let's continue this Airflow journey together! 🚀 #ApacheAirflow #DAG #DataEngineering #DataPipelines #MediumBlog #WorkflowAutomation #TechBlog

I'm creating two repo for Kernel Memory extension First one is ElasticSearch support https://lnkd.in/dA2pj3vW, we already have an ElasticSearch support for KM, but I use a different approach (dynamic template mapping) and also support keyword search. In extension package I'm starting to port some extension I wrote for KM, in current version (https://lnkd.in/dyXDP9v5) you have a search pipeline that allows you to pipe multiple search results with re-ranking. (cohere support) Also I'm porting the code to use a local python fast api server for embedding and reranking that you can use to run models directly from huggingface with very little effort. Elasticsearch is already on nuget, extension will be published once the code is ready for at least some minimal demos.