dltHub

How do you handle bad data quality? Here are some insights from practitioners: ➊ "Data that gets used gets improved." ➋ "The point of data engineering is to transform data in various ways." ➌ "Store a copy of the raw data for compliance and traceability." ➍ "Bad data is a mix of missing, incorrect, or messy records—but the fix depends on the problem and the context." ➎ "Quarantine bad rows, label issues, and notify responsible teams." These are just a few of the insightful ideas shared. Dive into the full Reddit thread here: https://lnkd.in/eujQQV_k What’s your take on managing bad data?

𝗜𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝘁𝗼 𝗱𝗮𝘁𝗮 𝗹𝗼𝗮𝗱 𝘁𝗼𝗼𝗹 (𝗱𝗹𝘁) In this first post of the dlt series: - Discover the origins of dlt and how it works. - Learn how dlt incorporates data engineering best practices. - Explore real-world dlt case studies and its performance in production. - Get step-by-step guidance on creating your first data pipeline with dlt. - Access resources and join communities to kickstart your journey with dlt. and overall, understand why dlt is a must-try tool for data engineers. Stay tuned for more insights in the upcoming Pipeline2Insights Substack series! created by ( Erfan and Hasan Geren) Adrian Brudaru Matthaus Krzykowski dltHub

This Wednesday I was at Elia Group/50Hertz Transmission GmbH Devcon '24 What I learned is that a cross org data mesh is required for Europe's energy transition. Read more: https://lnkd.in/e25GNUDu

It's been a while since I wrote a connector but after seeing the great work Adrian Brudaru and his team at dltHub I decided to give it a try I figured if it was worth my time to do it for a very cheap and low-volume connector, it would be worth doing it for higher volume, more expensive connectors in the future. It was both an excruciating and enjoyable experience! Forgot how annoying building connectors can be. Best part? Deployment in Orchestra was a breeze. Can check out the article in comments below #dataingestion #dataengineering #analytics #orchestra

Yesterday we had a pleasure to join dltHub meetup. It was quite a long evening with a number of interesting talks. Our favourite: "Accelerating privacy-enhancing data processing" by Florian Stefan from Flatiron Health. The highlight was, of course, the announcement of dlt+ product launch and roadmap presented by dltHub's CTO Marcin Rudolf. We are looking forward to testing it! Many thanks to the organisers & hosts: dltHub, DataTalksClub and the speakers: Matthaus Krzykowski, Douglas Zickuhr, Ajit Gupta, Florian Stefan, Abhishek Choudhary, Alexey Grigorev, Simon Rosenberger (Bumm), Serhii Sokolenko, Adrian Brudaru.

Are you using dlt? show of hands! well if you like dlt, you will love dlt+ for those of you building data platforms at scale, with requirements like portability, decentralisation, and developer efficiency, dlt+ is a platform-building tool much like dlt is a pipeline building tool We are currently accepting pilot users, sign up to our waiting list here: https://lnkd.in/e-M7YwCw

Exciting news! Our Staff Data Engineer Ajit Gupta  recently presented at the dltHub conference Berlin, delivering a fantastic talk on Forto's Data Journey and how DLT helped with ingesting unstructured MongoDB data. His presentation captured the evolution and impact of our data initiatives, showcasing the progress we’ve made as a team and a company. Great job Ajit Gupta 🎉 Matthaus Krzykowski and Marcin Rudolf congratulation on the Product Launch! It's great to see you listening to and supporting open source community 💪 And personally integrating with DLT was as seamless as ever

Our partners at Untitled Data Company helped us build the REST API source. So it was only natural that they talk about how to use it. Check out this Data Talks Club OSS Spotlight on dlt's REST API source. https://lnkd.in/ebuK8vMp

Interested in how to use Dagster Labs with dlt? Alex Chisholm demonstrates in this youtube video: https://lnkd.in/eh5PiWWf

dlt on serverless strikes again Cut your Fivetran cost 10.000x with dlt and github actions.