RisingWave

🔥 Happens TOMORROW! 2025 Crystal Ball: Real-Time Data and AI 🔮 Featuring: 🔹 Rayees Pasha - Chief Product Officer, RisingWave 🔹 Sijie Guo - Founder & CEO, StreamNative 🔹 Chang She - Co-founder & CEO, LanceDB Why attend? ✨ Explore how #AI and #realtime data are shaping the future 🔍 Discover the latest technologies driving real-time processing innovation 🌐 Gain insights into the challenges and breakthroughs we can expect in 2025 Save your spot here: https://lnkd.in/gYh462U4 #risingwave #dataprocessing #streamprocessing

✈️ 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐢𝐧𝐠 𝐭𝐡𝐞 𝐀𝐢𝐫𝐥𝐢𝐧𝐞 𝐈𝐧𝐝𝐮𝐬𝐭𝐫𝐲 𝐰𝐢𝐭𝐡 𝐄𝐯𝐞𝐧𝐭-𝐃𝐫𝐢𝐯𝐞𝐧 𝐒𝐲𝐬𝐭𝐞𝐦𝐬 Building event-driven systems in the airline industry is a complex challenge, often hampered by traditional data silos and outdated legacy systems. These barriers obstruct real-time data processing and seamless information sharing across departments, making it difficult for airlines to react swiftly to critical events like flight status updates, boarding pass scans, or loyalty program milestones. So, how can airlines modernize their operations, deliver exceptional customer experiences, and ensure flights run on schedule? In a recent Solace Scholar blog, Fahad Shah, Developer Advocate at RisingWave, explores how Solace and RisingWave empower airlines to streamline operations by seamlessly integrating real-time data streams to optimize their operations and provide great customer experience. Together, Solace and RisingWave provide the platforms organizations need to build robust and scalable event-driven systems—not just for aviation but also for industries like manufacturing, financial services, telecommunications, and beyond. A big thank you to Giri Venkatesan for reviewing the blog and sharing invaluable insights on how to leverage Solace’s unique features. Special thanks to Rey Riel also for his collaboration throughout the process. 👉 Dive into the blog to learn how real-time data streams can revolutionize aviation: https://lnkd.in/dB9aYFue #EventDrivenArchitecture #RealTimeData #Solace #RisingWave #Aviation

𝐊𝐚𝐟𝐤𝐚 𝐂𝐡𝐞𝐚𝐭𝐬𝐡𝐞𝐞𝐭: 𝐄𝐯𝐞𝐫𝐲𝐭𝐡𝐢𝐧𝐠 𝐘𝐨𝐮 𝐍𝐞𝐞𝐝 𝐭𝐨 𝐊𝐧𝐨𝐰! Apache Kafka has revolutionized data streaming, becoming the backbone of high-throughput, fault-tolerant event-driven architectures. While Kafka is the gold standard for event streaming, platforms like Redpanda, Pulsar, and WarpStream offer Kafka-compatible alternatives, catering to diverse use cases. Whether you're a developer, architect, or data engineer, here’s your comprehensive Kafka cheatsheet: 𝐂𝐨𝐫𝐞 𝐂𝐨𝐧𝐜𝐞𝐩𝐭𝐬 🔹 Broker: Manages message routing between producers and consumers. 🔹 Producer: Sends messages to Kafka topics. 🔹 Consumer: Reads messages from Kafka topics (often part of a consumer group). 🔹 Topic: Categorizes messages into logical channels. 🔹 Partition: Enables parallelism by splitting data across brokers. 🔹 Offset: Tracks the position of a message within a partition. 🔹 Replication: Ensures fault tolerance by maintaining leader and follower partitions. 𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬 ✅ High Throughput: Processes massive data streams efficiently. ✅ Durability: Persists and replicates messages for reliability. ✅ Scalability: Handles thousands of partitions and nodes seamlessly. ✅ Security: Supports SSL/TLS encryption, SASL authentication, and ACLs for fine-grained control. 𝐊𝐚𝐟𝐤𝐚 𝐄𝐜𝐨𝐬𝐲𝐬𝐭𝐞𝐦 🔹 Kafka Connect: Integrates with databases, file systems, and data lakes. 🔹 Kafka Streams: Enables real-time stream processing within Kafka. 🔹 ksqlDB: A SQL-based engine for transformations and aggregations. 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐞𝐬 ⚠️ Complexity: Requires meticulous setup and tuning for large-scale deployments. ⚠️ Latency: Higher compared to in-memory systems like NATS. ⚠️ Storage Costs: Persistent storage can increase expenses over time. 𝐀 𝐁𝐫𝐢𝐞𝐟 𝐇𝐢𝐬𝐭𝐨𝐫𝐲 𝐨𝐟 𝐊𝐚𝐟𝐤𝐚 🔹 2011: Created at LinkedIn 🔹 2012: Open-sourced under Apache 2.0. 🔹 2016: Introduced Kafka Streams for native processing. 🔹 2017: Became an Apache Top-Level Project. 🔹 2022: Introduced KRaft (Kafka Raft) in version 3.3. 🔹 2024: Reached 1000th KIP, reflecting continued innovation and community growth! 🚀 𝐊𝐚𝐟𝐤𝐚 3.9 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬 🔥 Tiered Storage: Lowers costs by offloading cold data to cheaper storage tiers. 🔥 Dynamic KRaft Quorums: Boosts fault tolerance and flexibility. 🔥 ZooKeeper Migration: Streamlines the shift to KRaft protocol for a simpler architecture. As Kafka and other streaming platforms evolve, they’re enabling more scalable, resilient, and cost-effective event-driven architectures. ✅ RisingWave provides native support for Kafka as both a source and sink, making real-time data analytics even more seamless! If you're passionate about Kafka and data streaming, follow these incredible people for more insights: 👉 Stanislav Kozlovski, Sion Smith, Dunith Danushka #Kafk #DataStreaming #DataEngineering #RisingWave #EventDrivenArchitecture

𝐏𝐫𝐞𝐯𝐢𝐞𝐰 𝐨𝐟 𝐑𝐢𝐬𝐢𝐧𝐠𝐖𝐚𝐯𝐞 𝐖𝐚𝐯𝐞𝐊𝐢𝐭 𝐢𝐬 𝐇𝐞𝐫𝐞! 🚀 We're excited to announce the preview of 𝐖𝐚𝐯𝐞𝐊𝐢𝐭, our on-prem tool for managing RisingWave clusters. WaveKit offers a powerful and intuitive web UI that makes managing your on-prem RisingWave clusters seamless and efficient. ➡️ 𝐇𝐞𝐫𝐞’𝐬 𝐰𝐡𝐚𝐭 𝐲𝐨𝐮 𝐜𝐚𝐧 𝐝𝐨 𝐰𝐢𝐭𝐡 𝐖𝐚𝐯𝐞𝐊𝐢𝐭 ✅ 𝐌𝐨𝐧𝐢𝐭𝐨𝐫 𝐂𝐥𝐮𝐬𝐭𝐞𝐫 𝐃𝐞𝐭𝐚𝐢𝐥𝐬: Run control commands directly and view real-time cluster information. ✅ 𝐌𝐚𝐧𝐚𝐠𝐞 𝐌𝐞𝐭𝐚𝐝𝐚𝐭𝐚 𝐒𝐧𝐚𝐩𝐬𝐡𝐨𝐭𝐬: Create, view, and utilize snapshots for cluster migration or upgrades. ✅ 𝐒𝐐𝐋 𝐄𝐝𝐢𝐭𝐨𝐫: Explore table schemas, including internal tables, effortlessly. ✅ 𝐒𝐭𝐫𝐞𝐚𝐦𝐢𝐧𝐠 𝐆𝐫𝐚𝐩𝐡𝐬: Visualize streaming graphs and manage schemas for materialized views, tables, sources, and sinks. ✅ 𝐃𝐢𝐚𝐠𝐧𝐨𝐬𝐭𝐢𝐜𝐬: Collect and store diagnostic information in your database with just a click. 𝐐𝐮𝐢𝐜𝐤 𝐒𝐭𝐚𝐫𝐭 Getting started with WaveKit is very simple! 1️⃣ 𝐂𝐥𝐨𝐧𝐞 𝐭𝐡𝐞 𝐫𝐞𝐩𝐨𝐬𝐢𝐭𝐨𝐫𝐲 𝘨𝘪𝘵 𝘤𝘭𝘰𝘯𝘦 𝘨𝘪𝘵@𝘨𝘪𝘵𝘩𝘶𝘣.𝘤𝘰𝘮:𝘳𝘪𝘴𝘪𝘯𝘨𝘸𝘢𝘷𝘦𝘭𝘢𝘣𝘴/𝘸𝘢𝘷𝘦𝘬𝘪𝘵.𝘨𝘪𝘵 2️⃣ 𝐍𝐚𝐯𝐢𝐠𝐚𝐭𝐞 𝐭𝐨 𝐭𝐡𝐞 𝐞𝐱𝐚𝐦𝐩𝐥𝐞𝐬 𝐟𝐨𝐥𝐝𝐞𝐫 𝘤𝘥 𝘸𝘢𝘷𝘦𝘬𝘪𝘵/𝘦𝘹𝘢𝘮𝘱𝘭𝘦𝘴/𝘥𝘰𝘤𝘬𝘦𝘳-𝘤𝘰𝘮𝘱𝘰𝘴𝘦 3️⃣ 𝐒𝐭𝐚𝐫𝐭 𝐖𝐚𝐯𝐞𝐊𝐢𝐭 𝐰𝐢𝐭𝐡 𝐃𝐨𝐜𝐤𝐞𝐫 𝐂𝐨𝐦𝐩𝐨𝐬𝐞 𝘥𝘰𝘤𝘬𝘦𝘳 𝘤𝘰𝘮𝘱𝘰𝘴𝘦 𝘶𝘱 4️⃣ 𝐀𝐜𝐜𝐞𝐬𝐬 𝐭𝐡𝐞 𝐰𝐞𝐛 𝐔𝐈 𝐢𝐧 𝐲𝐨𝐮𝐫 𝐛𝐫𝐨𝐰𝐬𝐞𝐫 👉 𝐎𝐩𝐞𝐧: http://localhost:8020 👤 𝐃𝐞𝐟𝐚𝐮𝐥𝐭 𝐔𝐬𝐞𝐫: root 🔑 𝐃𝐞𝐟𝐚𝐮𝐥𝐭 𝐏𝐚𝐬𝐬𝐰𝐨𝐫𝐝: 123456 𝐊𝐞𝐲 𝐅𝐞𝐚𝐭𝐮𝐫𝐞𝐬 ➡️ Manage on-prem RisingWave clusters with ease. ➡️ Create snapshots for seamless upgrades and migrations. ➡️ View real-time diagnostic information. ➡️ Explore streaming graphs and manage schemas effortlessly. Try it out and share your feedback! 👉 𝐆𝐢𝐭𝐇𝐮𝐛 𝐑𝐞𝐩𝐨: https://lnkd.in/dGGqPj6N 🎥 𝐖𝐚𝐭𝐜𝐡 𝐭𝐡𝐞 𝐝𝐞𝐦𝐨 to see WaveKit in action! 🔥 🔥 🔥 #RisingWave #RealTimeAnalytics #DataStreaming #WebUI

𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐖𝐨𝐧 𝐭𝐡𝐞 𝐅𝐮𝐭𝐮𝐫𝐞 — 𝐖𝐡𝐚𝐭'𝐬 𝐍𝐞𝐱𝐭 𝐟𝐨𝐫 𝟐𝟎𝟐𝟓? ➡️ As we start 2025, Apache Iceberg has emerged as the dominant open table format in the data ecosystem. ➡️ Let’s explore why Iceberg won and what lies ahead: 𝐊𝐞𝐲 𝐅𝐚𝐜𝐭𝐨𝐫𝐬 𝐁𝐞𝐡𝐢𝐧𝐝 𝐈𝐜𝐞𝐛𝐞𝐫𝐠'𝐬 𝐒𝐮𝐜𝐜𝐞𝐬𝐬 1️⃣ Strategic Acquisitions & Integrations ✅ Databricks Acquired Tabular: Bridged Iceberg and Delta Lake, pushing for a unified standard. ✅ Snowflake Polaris Catalog: Fostered Iceberg interoperability via partnerships and open-source contributions. ✅ Dremio Hybrid Iceberg Catalog: Enabled table management across cloud and on-premise. 2️⃣ Amazon Innovations ✅ S3 Tables for Iceberg: Managed Iceberg tables with 3x query throughput and automated maintenance. ✅ S3 Specialized Buckets: Enhanced performance and streamlined table management. ✅ SageMaker Lakehouse: Zero-ETL analytics with AI/ML support for Iceberg tables. 3️⃣ Ecosystem Growth ✅ Data Ingestion: Kafka, PostgreSQL protocol (via RisingWave). ✅ Querying Engines: Trino, Snowflake, Databricks, RisingWave, and others. ✅ Starburst Icehouse: Integrated Iceberg into managed lakehouses for AI/ML and analytics. ✅ Iceberg Integration with Snowflake and Microsoft Fabric: Improved compatibility and reduced data duplication. ✅ BigQuery Native Iceberg Support: High-throughput streaming ingestion and advanced optimizations. ✅ Upsolver's Iceberg Support: Automated table maintenance for real-time analytics workflows. ✅ RisingWave now supports Iceberg both as a source and sink. 𝐖𝐡𝐚𝐭’𝐬 𝐍𝐞𝐱𝐭 𝐟𝐨𝐫 𝟐𝟎𝟐𝟓? ➡️ Streaming Evolution ✅ CDC with Spec V3 introduces row lineage, boosting real-time workflows and materialized view maintenance. ➡️ RBAC Catalogs ✅ Fine-grained permissions independent of storage or query engines. ➡️ Materialized Views ✅ Faster query performance with automated updates for derived datasets. ➡️ Advanced Data Types & Compliance: ✅ Nanosecond-precision timestamps for finance and telecom. ✅ Binary deletion vectors for GDPR compliance. 𝐓𝐡𝐞 𝐑𝐨𝐚𝐝 𝐀𝐡𝐞𝐚𝐝 ✅ Lightweight compaction frameworks for smaller teams. ✅Greater accessibility for real-time analytics and lakehouses. 🎯 Apache Iceberg as the universal table format is redefining the future of modern data engineering. 👉 Want to learn more about Apache Iceberg? Follow these amazing people: ➡️ Alex Merced ➡️ Roy Hasson ➡️ Yingjun Wu ➡️ Danica Fine #ApacheIceberg #DataEngineering #RealTimeAnalytics #Lakehouse #BigData

𝐀𝐩𝐚𝐜𝐡𝐞 𝐈𝐜𝐞𝐛𝐞𝐫𝐠 𝐖𝐨𝐧 𝐭𝐡𝐞 𝐅𝐮𝐭𝐮𝐫𝐞 — 𝐖𝐡𝐚𝐭'𝐬 𝐍𝐞𝐱𝐭 𝐟𝐨𝐫 𝟐𝟎𝟐𝟓? ➡️ As we start 2025, Apache Iceberg has emerged as the dominant open table format in the data ecosystem. ➡️ Let’s explore why Iceberg won and what lies ahead: 𝐊𝐞𝐲 𝐅𝐚𝐜𝐭𝐨𝐫𝐬 𝐁𝐞𝐡𝐢𝐧𝐝 𝐈𝐜𝐞𝐛𝐞𝐫𝐠'𝐬 𝐒𝐮𝐜𝐜𝐞𝐬𝐬 1️⃣ Strategic Acquisitions & Integrations ✅ Databricks Acquired Tabular: Bridged Iceberg and Delta Lake, pushing for a unified standard. ✅ Snowflake Polaris Catalog: Fostered Iceberg interoperability via partnerships and open-source contributions. ✅ Dremio Hybrid Iceberg Catalog: Enabled table management across cloud and on-premise. 2️⃣ Amazon Innovations ✅ S3 Tables for Iceberg: Managed Iceberg tables with 3x query throughput and automated maintenance. ✅ S3 Specialized Buckets: Enhanced performance and streamlined table management. ✅ SageMaker Lakehouse: Zero-ETL analytics with AI/ML support for Iceberg tables. 3️⃣ Ecosystem Growth ✅ Data Ingestion: Kafka, PostgreSQL protocol (via RisingWave). ✅ Querying Engines: Trino, Snowflake, Databricks, RisingWave, and others. ✅ Starburst Icehouse: Integrated Iceberg into managed lakehouses for AI/ML and analytics. ✅ Iceberg Integration with Snowflake and Microsoft Fabric: Improved compatibility and reduced data duplication. ✅ BigQuery Native Iceberg Support: High-throughput streaming ingestion and advanced optimizations. ✅ Upsolver's Iceberg Support: Automated table maintenance for real-time analytics workflows. ✅ RisingWave now supports Iceberg both as a source and sink. 𝐖𝐡𝐚𝐭’𝐬 𝐍𝐞𝐱𝐭 𝐟𝐨𝐫 𝟐𝟎𝟐𝟓? ➡️ Streaming Evolution ✅ CDC with Spec V3 introduces row lineage, boosting real-time workflows and materialized view maintenance. ➡️ RBAC Catalogs ✅ Fine-grained permissions independent of storage or query engines. ➡️ Materialized Views ✅ Faster query performance with automated updates for derived datasets. ➡️ Advanced Data Types & Compliance: ✅ Nanosecond-precision timestamps for finance and telecom. ✅ Binary deletion vectors for GDPR compliance. 𝐓𝐡𝐞 𝐑𝐨𝐚𝐝 𝐀𝐡𝐞𝐚𝐝 ✅ Lightweight compaction frameworks for smaller teams. ✅Greater accessibility for real-time analytics and lakehouses. 🎯 Apache Iceberg as the universal table format is redefining the future of modern data engineering. 👉 Want to learn more about Apache Iceberg? Follow these amazing people: ➡️ Alex Merced ➡️ Roy Hasson ➡️ Yingjun Wu ➡️ Danica Fine #ApacheIceberg #DataEngineering #RealTimeAnalytics #Lakehouse #BigData

🔥 Join us on January 28 for an expert panel: 2025 Crystal Ball: Real-Time Data and AI 🔮 No doubt, AI is to shape our future in the years to come. But without real-time data processing—enabling anything from low-latency predictions to real-time feedback loops and beyond—AI’s true potential will remain untapped. Featuring three industry leaders: ▶️ Rayees Pasha, CPO at RisingWave Labs ▶️ Sijie Guo, Founder and CEO at StreamNative ▶️ Chang She, Co-founder and CEO at LanceDB What to expect: 🔍 insights on the evolving relationship between AI and real-time data processing ⚙️ deep dive into the key real-time processing technologies transforming AI efficiency and accuracy 🌟 predictions on challenges, obstacles, and groundbreaking advancements for 2025 🎫 Save your seat today: https://lnkd.in/gYh462U4 #dataprocessing #ai #realtimedata #streamprocessing

🔥 Excited to invite you to the next Singapore Apache Iceberg™ Community Meetup happening on January 21! We're teaming up with our partners at Snowflake to bring you an event you won’t want to miss! Speakers Lineup: 🌟 BOHAN ZHANG (SWE at RisingWave) Real-Time Data Pipeline with RisingWave for a Streaming Lakehouse 🌟 Manish Maheshwari (Director of Product Management at Cloudera) Apache Iceberg - Upcoming Challenges 🌟 Liang Chen (Data Foundation Partner Solution Architect at Amazon Web Services (AWS)) Iceberg on AWS 🎫 Save your seat: https://lu.ma/25sfndt4 #risingwave #iceberg #apacheiceberg #lakehouse #streaminglakehouse

✨How to build an event-driven agent with streaming databases? ✨ Mike Wang has the answer! 🤖 To implement a proactive agent that can know what to do without instructions from humans, he needs to know what’s happening in #realtime.  How will a proactive agent know that? Well, the streaming database is where the magic happens.🧙♂️ It has everything we need: a stream processing engine, data storage, data serving, and the SQL interface. There are a lot of stream processing engines, but one with #SQL interface can help us save a lot of time to build an event-driven agent. 🌊 And RisingWave is one of the best choices! 🌊 📑 Read the full blog for more details: https://lnkd.in/gnuPHkz7 #risingwave #dataprocessing #streamprocessing #aiagent

Waiting days to see results from your data pipelines? For critical operations like aircraft engine overhauls, delays and inconsistencies aren’t just inconvenient—they're unacceptable. Now, imagine this: insights updated in milliseconds. Metrics defined with simple SQL. Fast, consistent results—every time. 🎥  Watch Christos Anagnostakis's talk at Data Streaming Summit 2024 (hosted by StreamNative) to learn how advanced materialized views transform real-time monitoring for even the most complex processes: https://lnkd.in/gKmT-W6P #DataPipelines #MaterializedViews #RealTimeData #DataAnalytics #apacheiceberg #postgres #streamnative #kafka #apacheflink