Recce

Data preparation is an essential step for any data project. How do you know which is the best tool for the job? Recce's own Even Wei shares how he prepped the data for the open-source TodoFEC-dbt project and made the data available to all. TodoFEC-dbt uses dbt to model US campaign finance data. Even explains the challenges he overcame and the tools he used during data prep: - Parquet - Polars - DuckDB Read on to find out how: https://lnkd.in/gp9Dmx4W Next up, data modeling in dbt Labs! #opensource #data #analytics #datascience #dataprojects #dbt

Live preview dbt data model changes without needing to rebuild the model, sound interesting? You can do it right now in Recce, here’s how… We’re working on a feature that will enable you to preview the data from your model changes with a before/after diff - without the need to rebuild your dbt project. 📺 In the video below, Dave Flynn shows how you can do this right now in Recce. We’re currently working on making this workflow smoother, but the value of the feature is clear: https://lnkd.in/gNgQwHxx 👀 Please take a look and let us know if this would fit into your dbt data modeling workflow and, if not, let us know why. 🏗️ The foundation of Recce is built on improving and streamlining the workflow for analytics engineers and PR reviewers. Let's make data productive together! Get started with Recce in 5 minutes: https://lnkd.in/gGgi-R5F #dbt #dataengineering #dataworkflow #analytics #analyticsengineering

Get fast insight into data impact in dbt projects with Recce multi-node impact analysis. Select groups of nodes in the Lineage Diff either by dbt selector or select them yourself, and the perform: - Row Count diffs - Schema diffs - Value diffs (percentage match per column) on ALL selected nodes at once! ✅ Add the data checks to your Recce Checklist for peer review. ✅ Re-run the checks later. ✅ Automate the same checks in CI to cover your critical models. It’s just one of the ways that Recce helps improve your dbt development workflow, and makes the life of your PR reviewer easier! Try it right now in the online demo (no login required): https://lnkd.in/gaWMUb8R Like what you see? Give us a star and support open-source data tools: https://lnkd.in/gsZDPUVP #data #dbt #dataengineering #analyticsengineering #bestpractices #impactanalysis #impactassessment #sql #analytics #dataworkflow

Did you know Recce isn’t just for validating your work in isolation? You can actually share your complete data validation environment with your team - Validate your work, create your checklist, and then send it to your PR reviewer. They can recreate your Recce environment with one command: 💲 recce server --review pr1_recce.json Take it to the next level with Recce Cloud integration and there’s no need to even share the file! ☁️ Checklists are automatically synced 🌥️ PR merging can be blocked unless Recce checks are approved 🌤️ Recce can be opened online, no need to install locally You have a reproducible data pipeline, now you can get a reproducible data validation environment! ♻️ Read about how to do it here: https://lnkd.in/ghtqSxEK #dataengineering #bestpractices #analyticsengineering #dataops #data #analytics

Data pros, being asked to lead GenAI projects? 🚀 Explore why your existing data engineering skills might outshine purpose-built tools for evaluation. https://lnkd.in/gx9WbqG6 This article dives into treating GenAI as a data problem—offering flexibility, control, and seamless integration. Get ahead in the GenAI game! Shout out to Kent Huang for his work and research on this article. #DataEngineering #GenAI #evaluation #data

Checklists are important for ensuring things are done correctly, and that includes validating your dbt data modeling work. That’s why the Checklist is a core feature of Recce. ✅ Curate the data checks that help you verify your intention for this PR was realized. ✅ Share the Checklist with your PR reviewer as proof-of-correctness that your work is complete. It’s a simple but powerful way to perform data impact assessment for your dbt PRs. In a new series of blog posts, we’ll look at how you can make the most of the Checklist in Recce: - During development making modeling changes - For stakeholder sign-off - As part of PR review Check out the first part here: https://lnkd.in/gAaukfhq #data #dataengineering #dataops #dbt #pullrequest #checklist #bestpractices

We previously shared about how the data team at the Rio de Janeiro Department of Health uses Recce for data validation and impact assessment. If you’d like to know more, we now have a Case Study on the website that goes into some more detail of how the team was able to: ✅ Reduce merge times to an hour (from a day) ✅ Bring visibility into data impact and improve stakeholder communication ✅ Ultimately ensure the correctness of the health records of 7 million people following data model changes 📚 Read the case study: https://lnkd.in/grU5i34v Thanks again for Thiago Trabach and the team for sharing their experiences with us! ❤️ If you’d like to know more about how Recce can help improve your PR review process, book a chat with our team: 🗓️ https://lnkd.in/gXSV5t_m

Thanks to all those who expressed interest in contributing to the campaign finance project, TodoFEC. We’ve cleaned up and clarified the project, and also have some cool updates for you on the dbt fork! As a refresher, TodoFEC is a project to demonstrate how to perform data modeling and related BI tasks using different tools and frameworks, all using US campaign finance data. 💰💰💰 The idea is you can see the same tasks completed using different tools to help you evaluate the suitability of those tools for your needs. 🛠️ (The name is borrowed from TodoMVC, a project that compares various MVC frameworks building a todo app) 🗄️ TodoFEC: https://lnkd.in/gRZkXzpZ The TodoFEC repo will be the central repo where the example projects are linked from, and also provide some links to resources such as the data you'll need to get started. (We've provided an s3 bucket with the parquet files). If you need ideas for data and analytical tasks this is also the place to look! 🗄️TodoFEC-dbt: https://lnkd.in/ge2eXG6A The dbt implementation of the project shows ingesting the data, with modeling in dbt, and then an Evidence dashboard. All the steps required to run the project are included in the Readme, thanks to Recce's own Even Wei for that! 🫶 How to contribute You could create your own project using the source data, for instance you might choose to use dlt for ingestion, or SQLMesh for data modeling. Then open a PR on TodoFEC to submit your project to the Implementations section. Or, you could help out in the TodoFEC-dbt project, and help implement some more data analysis tasks, or wherever you feel your skills would help the most! 💾 Where to get the data? Check out the TodoFEC-parser for how to get the data: https://lnkd.in/gYHrwpmV Oh yeah, and don't forget to join the chat on #tools-recce in the dbt Slack. We've got folks available to help and answer questions

Thanks to all those who expressed interest in contributing to the campaign finance project, TodoFEC. We’ve cleaned up and clarified the project, and also have some cool updates for you on the dbt fork! As a refresher, TodoFEC is a project to demonstrate how to perform data modeling and related BI tasks using different tools and frameworks, all using US campaign finance data. 💰💰💰 The idea is you can see the same tasks completed using different tools to help you evaluate the suitability of those tools for your needs. 🛠️ (The name is borrowed from TodoMVC, a project that compares various MVC frameworks building a todo app) 🗄️ TodoFEC: https://lnkd.in/gRZkXzpZ The TodoFEC repo will be the central repo where the example projects are linked from, and also provide some links to resources such as the data you'll need to get started. (We've provided an s3 bucket with the parquet files). If you need ideas for data and analytical tasks this is also the place to look! 🗄️TodoFEC-dbt: https://lnkd.in/ge2eXG6A The dbt implementation of the project shows ingesting the data, with modeling in dbt, and then an Evidence dashboard. All the steps required to run the project are included in the Readme, thanks to Recce's own Even Wei for that! 🫶 How to contribute You could create your own project using the source data, for instance you might choose to use dlt for ingestion, or SQLMesh for data modeling. Then open a PR on TodoFEC to submit your project to the Implementations section. Or, you could help out in the TodoFEC-dbt project, and help implement some more data analysis tasks, or wherever you feel your skills would help the most! 💾 Where to get the data? Check out the TodoFEC-parser for how to get the data: https://lnkd.in/gYHrwpmV Oh yeah, and don't forget to join the chat on #tools-recce in the dbt Slack. We've got folks available to help and answer questions

Tomorrow 11/14 is the Peninsula Data Happy Hour! It's a super fun casual event where you can meet leaders of the post-modern data stack! Join me, Tobiko, typedef, Recce, and Wilson Sonsini Goodrich & Rosati. We’ll start at 5pm with food and drinks and lively conversation around data! https://lu.ma/omfb6f4o