Buf

#Dart and #Flutter folks! I have an early Christmas present for you. We just shipped Connect RPC 0.1! Check it out here: https://lnkd.in/gyri-bZT

SafetyCulture's journey to streamlining their data integration pipelines, powered by #Protobuf, by the numbers 🤯 The results that Thiago Baldim and Richard Gutierrez Montenegro delivered are really impressive: The development time for new data integrations dropped by 90% (‼️), alerts were significantly reduced, and they achieved 100% adoption of the new pipeline across the business in just 12 months. 🚀 Initially using Kafka Connect for two RDS instances, SafetyCulture's architecture evolved to one RDS per service. This change led to scalability issues, maintenance nightmares, and deployment bottlenecks. The lack of standardization across tables and databases created a ripple effect of problems for their data engineering team. 😵💫 To address these challenges, SafetyCulture implemented a phased solution: 1️⃣ Centralization: They created a single Kafka topic to manage all messages, enabling standardization and governance. 2️⃣ Standardization: Utilizing Protobuf, they implemented an envelope structure for messages, enforcing metadata requirements and allowing for better control and validation. 3️⃣ Automation: They built systems to detect new messages and critical data set issues automatically, reducing manual monitoring and improving response times. 4️⃣ Culture shift: SafetyCulture shifted its approach to data integration, bringing software engineers, analysts, and data engineers together to design effective data solutions from the start. This transformation not only dramatically streamlined SafetyCulture's data processes but also empowered teams to measure the success of their features more effectively. By combining Protobuf and #ApacheKafka, SafetyCulture has created a robust, scalable solution for their data integration needs, setting a foundation for future growth and innovation. 🦾 Check out their talk here: https://lnkd.in/gpyX9jp8 #DataEngineering #DataStreaming #FutureOfEngineering #TechTrends #Engineering #BigData #AmazonRDS #AmazonMSK

Buf's CTO, Akshay Shah, reflects on the complexities of distributed systems and the value of generative testing in ensuring their reliability. He notes that while many models focus on crashes, gray failures—such as reduced CPU capacity due to noisy neighbors or asymmetric network partitions—are more common in cloud environments and significantly harder to anticipate. Akshay advocates for leveraging computers to uncover potential issues, allowing developers to focus on clearly defining and documenting system behavior through assertions and guarantees. He acknowledges the longstanding history of generative testing, tracing its roots to Erlang and Haskell while emphasizing its relatively recent application to distributed systems. Discussing his experience with Kafka, Shah highlights the challenges in defining guarantees for such systems. He credits Kyle's expertise in designing workloads and helping to crystallize the concept of correctness. Shah also points out the varying complexity in testing different properties, from moment-in-time assertions to global, timeless requirements like serializability. Despite the power of automated testing, Shah suggests there may still be a role for testing experts, particularly in crafting sophisticated predicates for complex, global properties. This blend of automated tools and human expertise, he believes, is key to ensuring the robustness of modern distributed systems. #DataEngineering #DataStreaming #FutureOfEngineering #TechTrends #Engineering #Kafka #DevOps #CloudNative #DistributedSystems #BigData #CloudIntegration #Testing #Debugging

Testing distributed systems like #ApacheKafka is hard. Really hard. But we've made progress. 🔬 With Antithesis, we're taking a unique approach to testing Bufstream that's exciting for both Buf and the industry as a whole. Imagine this: A well-defined Postgres workload that anyone could run on their platform to test their new distributed database. Right out of the gate? That would be game-changing! It's like conformance testing that the whole industry could agree on. 🤝 Why is this so valuable? Conventional testing approaches we're used to are sequential. Distributed systems are fundamentally concurrent. But that's not how it works in the real world. We need to do better! We need tests that make concurrent requests and consider ALL possible outcomes. The multiverse of distributed systems! 🌌 Managing this explosion of state spaces is challenging, in no small part because It's non-deterministic. That's why Antithesis is so exciting to us at Buf! #DataEngineering #DataStreaming #FutureOfEngineering #TechTrends #Engineering #Kafka #DevOps #CloudNative #DistributedSystems #BigData #CloudIntegration #Testing #Debugging

The #ApacheKafka protocol's unordered delivery assumption: A double-edged sword for transactions! ⚔️ This assumption creates real-world challenges for operators. Read on 👇 Kafka's protocol assumes unordered delivery, allowing messages to be sent to any node over any TCP connection. This flexibility works well for individual sends and polls to different topic partitions in Kafka or Kafka-compatible systems, like Bufstream. Consider this example: Imagine a timeline where a client interacts with three database nodes (N3, N4, N5). The client sends message 1018 to topic partition 15 and then attempts to commit. The commit is delayed in the network. After receiving no confirmation, the client retries and sends a second commit to N4, which succeeds. From the client's perspective, this transaction is successful. Later, the client starts a new transaction, sending messages 424, 926, and 927 to different partitions. As it completes, the delayed commit from the previous transaction arrives at N3, inadvertently committing the current transaction prematurely. The result? A torn transaction that's half committed and half aborted. The client intended to abort this transaction, but only part of it was actually aborted. We're left with a lost write, an aborted read, and a fragmented transaction state. 😵 This scenario highlights a critical flaw in Kafka's transaction protocol, stemming from its fundamental assumption of unordered delivery. It's a reminder that even well-designed systems can have unexpected vulnerabilities when core assumptions are challenged in complex, real-world scenarios. #DataEngineering #DataStreaming #dataops #ai #FutureOfEngineering #TechTrends #Engineering #ApacheKafka #DevOps #CloudNative #DistributedSystems #BigData #CloudIntegration

Bufstream uses next-gen testing tools like Antithesis and Jepsen to ensure we deliver the most correct implementation of #Kafka's entire protocol (including transactions and exactly-once semantics). We do this by disaggregating storage from compute, which means you can infinitely absorb topic writes using S3-compatible object storage and elastically scaling compute up or down without having to fret about topic partition rebalancing, etc. Oh, it's also fully self-hosted (BYOC). 😜 Distributed systems like Bufstream are fundamentally about handling "rare" events that happen constantly at scale. Here's what most people miss about distributed systems: 1️⃣ Network delays and message reordering are constant challenges, not exceptions. 2️⃣ Clients retrying operations introduces a new layer of complexity and potential bugs. 3️⃣ Fault tolerance mechanisms to improve user experience can introduce more issues. 4️⃣ Testing the resilience of these systems is incredibly difficult due to the near-infinite number of possible failure modes. 5️⃣ Higher expectations for reliability paradoxically create more opportunities for things to go wrong. The key to building robust distributed systems isn't trying to predict every possible failure. It's building systems (like Bufstream) that can gracefully handle the unexpected and recover from failures we didn't anticipate. #DataEngineering #DataStreaming #dataops #ai #FutureOfEngineering #TechTrends #Engineering #ApacheKafka #DevOps #CloudNative #DistributedSystems #BigData #CloudIntegration

We're about to go live at 12:30PM PST/3:00PM EST! Join us for our live Q&A with the team.

Antithesis, Buf, and Jepsen are running a joint webinar on December 5th. We'll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause, rewind, and explore a running cluster in an interactive debugging shell! https://lnkd.in/eRBiQKAk

Antithesis, Buf, and Jepsen are running a joint webinar on December 5th. We'll discuss a Kafka protocol safety issue, talk about the challenges of distributed systems testing, and show how Jepsen and Antithesis helped identify critical safety errors in Bufstream. Come watch Antithesis pause, rewind, and explore a running cluster in an interactive debugging shell! https://lnkd.in/eRBiQKAk

We know organizations rely on Kafka for mission-critical systems, and we couldn't bring Bufstream to market without rigorous correctness testing. Together with Jepsen + Antithesis, we've done it, and we're excited to talk about it. We launched Bufstream earlier this year as a drop-in replacement for Kafka. It supports the full #ApacheKafka API, including transactions and exactly-once semantics (EOS). It works with existing clients, stream processing frameworks, Kafka Connect—the whole nine yards. Pre-launch, we partnered with Antithesis to chaos test Bufstream with their deterministic fault injection platform. Combined with Will Wilson's FoundationDB experience, Antithesis was invaluable. Post-launch, we brought in Kyle Kingsbury for his expertise with generative workloads and sophisticated consistency checkers. And that's when things got interesting. During our exploration of Bufstream's behavior, Kyle's tests unearthed a significant flaw in the Apache Kafka protocol itself. Our ability to reproduce this behavior in a deterministic environment is huge. This isn't just about Bufstream anymore—it's about the foundational protocol powering data streaming for countless organizations. The implications could be far-reaching for the entire Kafka ecosystem. Sometimes, you get a chance to uncover issues that impact the entire industry! #DataEngineering #datastreaming #dataops #ai #FutureOfEngineering #TechTrends #Engineering #DevOps #Kafka #distributedsystems #databases