dltHub reposted this
Cut Data Stack Cost | dbt + Snowflake + Tableau Expert | DM me for data consultation!
I spent a lot of time in 2024 learning dlt (dltHub). I struggled quite a bit to get to an advanced level in dlt, a declarative way to write data pipelines in python, when I could already write bespoke pipelines in Python using imperative code. So why did I stick with it? I just had this feeling it was worth learning because it standardizes so many steps of the data pipeline process: how data is fetched incrementally, normalized, target tables created in the destination, data loaded (using bulk loading best practices), merged, logging, tracking state, etc. These are all things I’ve done a million times (by hand, imperatively), but every time I do it from scratch I do it slightly differently. 👎 Dlt gives repeatability and scalability. So instead of “Jeff’s custom ELT code”, a client can get something that is very standardized and follows industry best practices. 👍 But for me, it was a bit hard to learn! (I could be alone here, lots of people say how easy it is...) It is a tool that seems very easy on the surface, but there’s quite a bit you need to know about how it works and how to deploy it; I struggled to get to that advanced level that I’m at now, after months of practice. But it was worth it! I do feel like I have a unique super power now after gaining this experience! 🚀 Great news for anyone that is looking to learn dlt right now 🚀: You don’t have to struggle as much as I did! They just released a new training program. It starts out extremely basic and works up to more advanced concepts of how dlt works under the hood. It is great for people like me who need to be able to handle all of the edge cases. This new training program is fantastic and I highly recommend it! https://lnkd.in/e89vC4UW
Jeff Skoldberg I frequently have to problem of wanting to just figure things out vs go through the training. Haven’t gone through the dlt training but have spent sometime struggling with it myself - doing the training seems like the right answer. 😁
With snowflake now able to make api calls, Dlt was on my list for 2025 so great timing.
Jeff Skoldberg, This sounds like a fantastic tool! Do you think DLT is a good fit for smaller teams, or is it more tailored to enterprise-level projects?
Where does it store state for your snowflake workflow ?
I'll admit this is over my head technically, but it's super relatable on a few of your keys points - the struggle is worth it, it's how you deeply learn - especially in data tasks driving towards consistency has real value and even more so in consulting/project work It reminds me of the meme for "too busy to improve" where someone is offering to convert from square to round wheels: https://images.app.goo.gl/Q7U3VkFU6ixvCFVPA
> But for me, it was a bit hard to learn! (I could be alone here, lots of people say how easy it is...) Jeff my experience and ideology was very similar to yours. I think it's very easy to get started with out-of-the-box sources and simple data ingestion tasks; however, if you're already an experienced data engineer, and you're not just trying to write "my first EL pipeline", there's a depth of complexity (and power!) hidden beneath the surface. Have appreciated the responsiveness of everybody on the dltHub team in working through these challenges—not every CTO will help you debug a stupid Python bug. 🤣 That said, I'm very bought into the composability aspect and, in general, the benefits of standardized tooling. I know that I can write a Python script myself to parse and populate some dataset, but my code will get more and more unwieldy the second I try to write to a cloud destination or introduce additional logging—all off which has been done in the open already in dlt. Definitely the tool I'm betting on in the EL space!
Totally agree! My first impression of the tool at the beginning of this year (0.4.x) was that the idea behind the tool was right but I the implementation needed to improve. But as of now, I would say it is my go-to choice for every use case it covers - which fortunately is the majority that I find in practice 🚀
Exciting
Great post! Learning a new pipeline or ETL tools can be frustrating. But the journey to standardize can add considerable value. You've got me intrigued so definitely going to check it out!
Open Source Advocate | Top 100% TryHackMe | Ex AWS Security Specialist
1wWow! Alena Astrakhantseva (and others) did a great job with these courses and workshops! 👏👏👏