Dagster Labs

Your December dose of data & AI is here! In this month's roundup, you'll get: - Our Data Council 2025 talk tracks (save $ and get Early Bird tickets now) - Feature on LlamaIndex an innovative project for building RAG apps - A new podcast featuring Spencer Kimball, exploring how Cockroach Labs used open source licensing to grow their business - Curated technical reads and select job listings from the data & AI community Happy reading! https://lnkd.in/g82mntTM --- Reads written by: Mosbeh Barhoumi, Animesh Kumar, Eric Xiao, Adel Zaalouk, Simon Späti, Mehdi Ouazza, Julien Hurault, Kevin Liu, LangChain, Siddhant Sahu, Nikos Kafritsas, Adam Bellemare Jobs at: MotherDuck Clusterfudge Hex Dagster Labs Soda Vectara

Community Spotlight! Check out this step-by-step guide from Gianfranco Demarco on setting up interactive debugging with Dagster in Docker! Learn how to: - Debug your assets directly in containerized environments - Bridge the gap between development and deployment - Step through both your code AND Dagster's internals This guide is perfect for data engineers who want their local debugging to match production environments. Check out the full guide in the comments! #DataEngineering #Dagster #Docker #Debugging #TechTutorial

🚀 Ready to transform your data team from firefighters to innovators? Join us on December 19th with Edson Nogueira, who will reveal how Artefact built a modern, proactive data platform using Dagster, dbt, and dlt. You'll discover: ✨ How to seamlessly integrate best-in-class open source tools 🔍 End-to-end observability strategies that prevent fires before they start ✅ Practical approaches to implementing robust data quality checks Don't miss this chance to learn how to spend less time putting out fires and more time driving value. 📅 December 19th, 9AM PT (link in comments)

At Dagster, our mission is to empower every organization to build a productive, scalable data platform. Check out the keynote from Data Platform Week from our Founder and Chief Technology Officer Nick Schrock where he explains the future of data platforms and how Dagster built the right abstractions to fully leverage the value of your data.

dlt or Fivetran? or RudderStack? As far as we are concerned, all tools have their pros and cons. Ours is a devtool, the others are services - which makes them vastly different in practice even if they partly overlap on use case. So check out this video where Alex Chisholm shows us how he uses dlt, Fivetran, RudderStack, Dagster Labs and dbt Labs https://lnkd.in/eegzr8mm

Stuck on Airflow? Is it because of the time & risk involved in migrating? I often hear "It is the devil we know" when discussing Airflow and sticking with the status quo because of the work involved. Dagster Labs recently released a tool kit, codename 'Airlift' for Airflow migration to Dagster. Airlift focuses on incremental migration with low-risk steps to deliver intermediate milestones that deliver value and credibility of the move. Nick Schrock, Dagster's Founder & CTO, discusses Airlift further in our Data Platform Week announcements video, check out the first 5 minutes (link in comments). Link to the Github discussion in the comments as well! #Dagster #dataorchestration #dataplatform #datapipelines #airflow #datamigrations #dataengineering #datascience #machinelearning #softwareengineering #ELT #dataquality #datacatalog #datalineage

Having a composable data platform means easily using the best tool for the job. One of our most popular use cases is our integration with dltHub through embedded ELT to manage ingestion cost-effectively. When we implemented the embedded ELT package internally on our data platform, we saved $40K/annually on software spending for a managed ingestion provider. (Blog post in the comments) Check out the great tutorial from Alex Chisholm!

Wow, truly a powerful year. Let's go for 2025!!

Data Engineers working on data pipeline testing at the moment? See below our Soda's webinars and get started with data quality! 🔹 How to Test Data Quality in Microsoft Azure Data Factory Pipelines 🗓️ December 5, 2024 | 11:00 AM ET | 5:00 PM CET 🖇️ https://lnkd.in/eHV3g2Hk 🔹 How to Build a Self-serve Data Quality Platform on AWS 🗓️ December 11, 2024 | 9:00 AM ET | 3:00 PM CET 🖇️ https://lnkd.in/ep58vMcX 🔹 How to Test Data Quality in Dagster Labs Pipelines 🗓️ December 12, 2024 | 11:00 AM ET | 5:00 PM CET 🖇️ https://lnkd.in/eyYQc7G5 🔹 Tutorials: Adding Data Quality Checks to Data Pipelines Explore step-by-step guidance on integrating Soda into Databricks, Apache Airflow, and more to manage reliable pipelines at scale before migration and during development for error-free operations. 🖇️ https://lnkd.in/dr3XSzU9