Buf’s Post

SafetyCulture's journey to streamlining their data integration pipelines, powered by #Protobuf, by the numbers 🤯 The results that Thiago Baldim and Richard Gutierrez Montenegro delivered are really impressive: The development time for new data integrations dropped by 90% (‼️), alerts were significantly reduced, and they achieved 100% adoption of the new pipeline across the business in just 12 months. 🚀 Initially using Kafka Connect for two RDS instances, SafetyCulture's architecture evolved to one RDS per service. This change led to scalability issues, maintenance nightmares, and deployment bottlenecks. The lack of standardization across tables and databases created a ripple effect of problems for their data engineering team. 😵💫 To address these challenges, SafetyCulture implemented a phased solution: 1️⃣ Centralization: They created a single Kafka topic to manage all messages, enabling standardization and governance. 2️⃣ Standardization: Utilizing Protobuf, they implemented an envelope structure for messages, enforcing metadata requirements and allowing for better control and validation. 3️⃣ Automation: They built systems to detect new messages and critical data set issues automatically, reducing manual monitoring and improving response times. 4️⃣ Culture shift: SafetyCulture shifted its approach to data integration, bringing software engineers, analysts, and data engineers together to design effective data solutions from the start. This transformation not only dramatically streamlined SafetyCulture's data processes but also empowered teams to measure the success of their features more effectively. By combining Protobuf and #ApacheKafka, SafetyCulture has created a robust, scalable solution for their data integration needs, setting a foundation for future growth and innovation. 🦾 Check out their talk here: https://lnkd.in/gpyX9jp8 #DataEngineering #DataStreaming #FutureOfEngineering #TechTrends #Engineering #BigData #AmazonRDS #AmazonMSK