Data Engineer Things’ Post

💪 No, Data Engineers Don’t NEED dbt. 🖋️ Author: Leo Godin 🔗 Read the article here: https://lnkd.in/eFb9qafH ------------------------------------------- ✅ Follow Data Engineer Things for more insights and updates. 💬 Hit the 'Like' button if you enjoyed the article. ------------------------------------------- #dataengineering #datascience #sql #data

Benn Stancil latest post on dbt hits close to heart. 🥲 In parts because I remember ‘winging it’ myself whilst building an entire warehouse amount of dbt models. I had some medallions to follow, ideas for encapsulating logic, but other than it was wild west sql slinging. But also through our work with reconfigured v1 where we tried to enforce an auto-generated-yet-exposed-to-the-eyes framework for building core data models. It solved the spaghetti issue for sure, but it really didn’t address the underlying problems. Recommend the read. :-) https://lnkd.in/dmBcPhyA

Great post again from Benn! What I've seen is that many times Data Engineers need to clean up the mess that Application/Services create -- which is obviously wrong. You get crappy data in and you need to somehow clean it up and organise to tables and eventually through some magic graph of thousands of lines of SQL into clear and understandable and simple charts and reports. Nobody dares to say that "hey, pls fix this JSON where it is produced". "Programming" on th SQL layer is way harder and a lot more unclear and cumbersome than on the real programming language layer. It's the same thing again, companies need to have e2e approach for data. Otherwise they sink into the complexity.

💡 Enhance Data Quality Tests with the dataform-assertions Package 🖋️ Author: Fumiaki Kobayashi 🔗 Read the article here: https://lnkd.in/e2BrwXan ------------------------------------------- ✅ Follow Data Engineer Things for more insights and updates. 💬 Hit the 'Like' button if you enjoyed the article. ------------------------------------------- #dataengineering #dataform #bigquery #data

😵💫 Ever wrestled with SQL that needs to adapt across data warehouses? At Numbers Station, SQLGlot is our secret weapon, handling the heavy lifting of parsing, transforming, and unifying SQL across dialects. Dive in as Maureen Daum breaks down how this powerhouse library keeps our SQL analysis sharp and our codebase streamlined! https://lnkd.in/gkeuJpd8

Dive into my latest article on "Understanding dbt: Models" where I unravel the essentials of data transformation using dbt. In this article, we explore: 1. What models are and how they work in dbt 2. The importance of modularity in data transformations 3. Different types of materializations and their use cases (Views, Tables, Incremental, Ephemeral) 4. Creating dependencies and building a Directed Acyclic Graph (DAG) with ref() Curious to learn more? Check out the full article: 📖 On Medium: https://lnkd.in/evHFTaaJ 📖 On Substack: https://lnkd.in/ewhW-xTQ #DataEngineering #dbt #DataTransformation #BigData #ETL #DataAnalytics #TechBlog

Easy to generate test data. Databricks https://lnkd.in/gMgAggXU