Navi’s Post

Just 1 more day to go for the Apache Hudi Meetup happening on May 11th in Bangalore 🇮🇳 📅 Date: May 11, 2024 | 🕥 Time: 10:30 AM - 2:00 PM IST 📍 Navi Technologies, Bangalore Register: https://lnkd.in/dbKB4fVm This is an excellent opportunity to connect with fellow data professionals & gain invaluable insights into lakehouse architecture & Apache Hudi. #dataengineering #softwareengineering

Great article from Toplyne on implementing a data lakehouse to: 🔋 Power AI/ML use cases… 🌺 …with fresh data… 🤑 …while eliminating vendor lock-in and reducing overall costs. 🕵️ Read on to learn more about Toplyne's architecture, including Apache Kafka ingestion to the lakehouse, with Dagster and Apache Spark to run feature engineering pipelines. https://lnkd.in/ggkt_i36 #onehouse #dataengineering #nolockin #ai #datalakehouse #apachehudi #opensource 

Powering the Future of Financial Services with EDB Postgres AI 5 Must-Read Success Stories See how today’s top BFSI firms are navigating increasing regulatory demands, customer expectations and costs and doing more with their data by building their future on EDB Postgres AI. #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource 

Three of the EDB Postgres experts attending PGConf NYC, sharing insights on new Postgres features and navigating common challenges for DBAs. Get a preview of EDB's talks at PGConf NYC this year. Read the blog here: 🔗 https://lnkd.in/gzcycypN My personal favourite is the in-depth exploration of the new incremental backup feature by Robert Haas himself. #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource

🧐 If you’re a data engineer, the data warehouse and the data lake may seem pretty different. 🫶 But the data lakehouse is bringing them together. And @Apache Hudi is at the forefront of this evolution. 🤝 This month’s Hudi newsletter is a good introduction to, and update for the Hudi world.  ⚡️ With articles showing how Kafka and Hudi work together for AI/ML (by introducing the data lakehouse (by Prabodh Agarwal of Toplyne), showing how Hudi works with Apache Spark (by Karl C. of Careem), and describing how to use OpenAI vector embeddings with Hudi (by Soumil S. of JobTarget), among others. 👍 Recommended. https://lnkd.in/gNaDqWdi #apachehudi #opensource #dataengineering #datalakehouse 

🚀 𝐃𝐫𝐢𝐯𝐞 𝐘𝐨𝐮𝐫 𝐃𝐚𝐭𝐚 𝐏𝐢𝐩𝐞𝐥𝐢𝐧𝐞𝐬 𝐰𝐢𝐭𝐡 𝐏𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧: 𝐔𝐧𝐥𝐞𝐚𝐬𝐡 𝐭𝐡𝐞 𝐅𝐮𝐥𝐥 𝐏𝐨𝐭𝐞𝐧𝐭𝐢𝐚𝐥 𝐨𝐟 𝐀𝐃𝐅 𝐓𝐫𝐢𝐠𝐠𝐞𝐫𝐬! 🚀 In Azure Data Factory (ADF), triggers are mechanisms that initiate the execution of pipelines based on specific criteria. They enable automation by allowing you to schedule, repeat, or event-driven execution of data workflows. Triggers are fundamental in orchestrating and managing data workflows, helping to handle tasks like data ingestion, transformation, and movement seamlessly. ADF offers three main types of triggers: Here’s how ADF Triggers can take your data workflows to the next level: 1️⃣ 𝐒𝐜𝐡𝐞𝐝𝐮𝐥𝐞 𝐓𝐫𝐢𝐠𝐠𝐞𝐫: Purpose: Run pipelines on a regular schedule—hourly, daily, weekly, or monthly. Usage: Define the recurrence pattern and start time, with the flexibility to set time zones! Example: Schedule a daily midnight pipeline to keep your data up-to-date automatically. 2️⃣ 𝐓𝐮𝐦𝐛𝐥𝐢𝐧𝐠 𝐖𝐢𝐧𝐝𝐨𝐰 𝐓𝐫𝐢𝐠𝐠𝐞𝐫 Purpose: Execute pipelines repeatedly within specified time intervals. Usage: Set your window size and frequency, and let ADF handle the rest. Example: Process data hourly within a 24-hour window, keeping data flows smooth and consistent. 3️⃣ 𝐄𝐯𝐞𝐧𝐭 𝐓𝐫𝐢𝐠𝐠𝐞𝐫: Purpose: Start pipelines based on external events, like a new file in Azure Storage or a message from Azure Event Hubs. Usage: Define criteria, and ADF triggers your pipeline as soon as the event occurs. Example: Automatically launch a pipeline when a file uploads to Azure Blob Storage. 📈 Embrace ADF triggers to make your data workflows work smarter, not harder! Tagging Anurag Sharma Abhinav Singh Vishnu Reddy Sagar Prajapati Ajay Kadiyala Harshit Bhadiyadra Vaishnavi MURALIDHAR Devikrishna R 🇮🇳 💎 Thodupunuri Bharath Abhay Singh Subham Khandelwal for better reach. #SQL #azuredataengineer #bigdata #pyspark #python #databricks #ApacheSpark #pysparkcodingquestions #Azuredatafactory #deltalake #spark #DataAutomation #DataEngineering #CloudComputing #MicrosoftAzure #DataIntegration

At EDB, we believe the AI generation is a world in which Postgres is meant to thrive. Postgres has a flexible and extensible architecture that is ideal for transactional workloads. But in order for Postgres to support large-scale analytics, its architecture needs to be extended in a few key ways. Read the blog from EDB Senior Director Franck Sidi to discover the latest insights into using Postgres for analytics, search, and AI ➡️ https://bit.ly/3ZsbYYL #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource #analytics

Kirana Club is India's largest Kirana Network, and their goal is to provide a platform where Kiranas get neutral and trusted information that empowers them to scale their business. See how #ScyllaDB is helping them tackle the challenges of storing impression data so they can continually deliver personalized content to each user. https://ow.ly/RrRE50Qcv29 #NoSQL #database #scaling

At EDB, we believe the AI generation is a world in which Postgres is meant to thrive. Postgres has a flexible and extensible architecture that is ideal for transactional workloads. But in order for Postgres to support large-scale analytics, its architecture needs to be extended in a few key ways. Read the blog from EDB Senior Director Franck Sidi to discover the latest insights into using Postgres for analytics, search, and AI ➡️ https://bit.ly/3ZsbYYL #Postgres #PostgreSQL #justsolveitwithPostgres #EDBPostgresAI #data #AI #database #DBA #opensource

Impressive and insightful material in this Uber Engineering blog about their utilization of Apache Hudi! #data #datalake #dataengineering #apache #hudi