lakeFS

Daniel Beach you caught our attention⚠️ Great post on why #AWS #S3Tables (could) disrupt external data warehouses like Snowflake and Databricks. Will this development shake up the data landscape? 🫨 Here are 3 reasons why it just might: 1. 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹𝗶𝘁𝘆 𝗮𝘁 𝗹𝗼𝘄𝗲𝗿 𝗰𝗼𝘀𝘁𝘀 2. 𝗥𝗲𝗱𝘂𝗰𝗲𝗱 𝗿𝗲𝗹𝗶𝗮𝗻𝗰𝗲 𝗼𝗻 𝗲𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺𝘀 3. 𝗦𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗲𝗱 𝗶𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 What's your take? #iceberg #opentableformats #datalakes

Can you fit a square peg into a round hole? 🟩🔴 If you answered yes, this article is for you! 🚀 Sam Austin dives deep into the art of achieving CI/CD for Machine Learning — tackling the unique challenges of managing code, data, and models in harmony. 💡 From handling data drift to automating training pipelines, this piece is packed with insights to inspire your ML CI/CD strategy. 👉 Read the full article on Medium https://lnkd.in/d5Vzc8Vi #machinelearning #cicd #dataengineering #mlops

𝗪𝗵𝗮𝘁 𝗶𝘀 𝗮 𝗱𝗮𝘁𝗮 𝗴𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸? A #datagovernance framework establishes a standardized set of rules and practices for data collection, storage, and utilization. It ensures that your policies, regulations, and definitions are applied to all data in your organization. A good framework lets you offer trusted data to people in a variety of roles, including business leaders, data stewards, and developers. This type of framework also makes sure that data can be managed, transformed, and delivered across all application and analytics installations, both in the cloud and on-premises. This opens the door to implementing self-service solutions available to non-technical teams, helping them identify and access the data they want for data governance and analytics. Read on to learn more 📖 https://lnkd.in/dPnpanfp #datacollection #data #dataversioncontrol

Be sure to register for this virtual talk presented by Amit Kesarwani at the MLOPs World and Generative AI World Summit. 🎙️ Here's what Amit will demonstrate in this talk: • Training a TensorFlow predictive model on data mounted using lakeFS Mount • Integration with Git to version code and data together • Reproducibility of code as well as data #mlops #datareproducibility #machinelearning #genai

🔍 What if isolating data could 𝘪𝘯𝘤𝘳𝘦𝘢𝘴𝘦 collaboration, speed up development, and even make troubleshooting easier? It's time to rethink the norm and embrace #dataisolation 💡 Here’s why data isolation (with version control) matters: 𝗙𝗲𝘄𝗲𝗿 𝗖𝗼𝗻𝗳𝗹𝗶𝗰𝘁𝘀, 𝗠𝗼𝗿𝗲 𝗣𝗿𝗼𝗴𝗿𝗲𝘀𝘀 🔄 Isolation means each team can work without disrupting others. No more accidental overwrites—just smooth, uninterrupted progress. 𝗥𝗮𝗽𝗶𝗱, 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 ⚙️ With isolated datasets and version control, you can test at full speed, roll back if needed, and never have to wait on others’ changes. 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁 𝗙𝗿𝗲𝗲𝗹𝘆, 𝗦𝗮𝗳𝗲𝗹𝘆 🧪 Data version control ensures every experiment stays contained, letting teams take creative risks without risking production or other teams' data. 𝗦𝘁𝗿𝗲𝗮𝗺𝗹𝗶𝗻𝗲𝗱 𝗗𝗲𝗯𝘂𝗴𝗴𝗶𝗻𝗴 🛠️ Isolation, paired with version control, helps track data changes, making it easier to pinpoint issues and debug faster. Data isolation is a game-changer for teams who need speed, safety, and seamless collaboration. #datacollaboration #dataexperimentation #datatesting #datadebugging https://lnkd.in/djDArDrk

In the world of software development, a 𝗣𝘂𝗹𝗹 𝗥𝗲𝗾𝘂𝗲𝘀𝘁 is a mechanism for proposing changes to a codebase. When a developer makes changes in a separate branch, they can create a Pull Request to ask their peers (or other project maintainers) to review and merge those changes into the main branch. During the process, reviewers can leave comments, suggest improvements, and even run tests to ensure everything works as expected. This is invaluable because it creates a structured process for change: discussions happen in the open, reviews are documented, and changes are only merged when they’re fully approved. This ensures that every change is scrutinized, improving quality and fostering collaboration. By adding Pull Requests to data workflows, lakeFS offers the same benefits seen in software development: 🤝 𝗖𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝗼𝗻 - Multiple team members can collaborate on data changes, review each other’s work, and leave feedback 🔎 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝗰𝘆 - Every change is visible and documented, providing a clear audit trail 🎯 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗖𝗼𝗻𝘁𝗿𝗼𝗹 - Changes are reviewed and tested before they are merged, reducing the risk of introducing errors into the data Watch how to create a pull request in lakeFS and read more (link to post in comments) #pullrequests #dataversioning #dataquality

2 weeks to go until Open Data Science Conference (ODSC) WEST kicks off. Will we see you there? 👀 Be sure to check out Einat Orr's track:  Don’t Go Over the Deep End: Building an Effective OSS Management Layer for Your Data Lake! In this talk, Einat will explore fundamental challenges, focusing on different needs of structured vs unstructured data where each requires its own distinct approach. She’ll dispel some of the chaos and cover key components of a robust data lake management architecture, including open table formats, catalogs and data version control systems. You’ll learn how they contribute to an organized data lake environment, helping you avoid feeling like you’re constantly treading water.   Make sure to attend the talk, meet the team at Booth #26 and grab some spooky swag! 🎃👻🍬🦇 https://lnkd.in/dPv-Mhcu

🚨 𝙒𝙝𝙖𝙩 𝙞𝙛 𝙮𝙤𝙪𝙧 𝙛𝙞𝙡𝙚 𝙨𝙮𝙨𝙩𝙚𝙢 𝙬𝙖𝙨𝙣’𝙩 𝙖𝙨 𝙧𝙚𝙡𝙞𝙖𝙗𝙡𝙚 𝙖𝙨 𝙮𝙤𝙪 𝙩𝙝𝙞𝙣𝙠? 🚨 Many #filesystems make bold promises about handling complex datasets—but the reality? They can struggle with versioning, tracking, and scalability, especially in cloud environments. 🔍 𝗧𝗵𝗲 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝘄𝗶𝘁𝗵 𝗟𝗲𝗴𝗮𝗰𝘆 𝗙𝗶𝗹𝗲 𝗦𝘆𝘀𝘁𝗲𝗺𝘀: 1. Traditional file systems weren’t designed for the complexities of modern #datalakes. 2. They rely on manual workarounds for critical tasks like version control. 3. Gaps emerge when scaling data across distributed cloud architectures. 💡 𝗘𝗻𝘁𝗲𝗿 lakeFS: 1. Provides Git-like capabilities for your file system, making #dataversioning seamless. 2. Automatically tracks every change and allows you to roll back without breaking a sweat. 3. Scales effortlessly with your cloud setup, ensuring performance even with massive datasets. 🌟 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁? You get full control over your data, simplified versioning, and peace of mind that your system can scale without bottlenecks. Read more about how lakeFS redefines file representation: https://lnkd.in/eFKM_avW 

🔄 Ever wonder how #datapipelines keep your data flowing seamlessly? Here’s a breakdown of the 𝟱 𝗸𝗲𝘆 𝘀𝘁𝗮𝗴𝗲𝘀 that keep your data ready for analysis: 1️⃣ 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻: Data is gathered from diverse sources like databases, devices, and applications. 2️⃣ 𝗜𝗻𝗴𝗲𝘀𝘁𝗶𝗼𝗻: The collected data is loaded and organized within systems, making it ready for storage. 3️⃣ 𝗦𝘁𝗼𝗿𝗮𝗴𝗲: The organized data is securely housed in data warehouses, lakes, or other systems. 4️⃣ 𝗖𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻: Data is cleaned, formatted, and transformed to meet company standards. 5️⃣ 𝗖𝗼𝗻𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻: The processed data is available for analysis, visualizations, and business applications. lakeFS introduces a Write-Audit-Publish model, ensuring #datavalidation throughout the pipeline. Its powerful "hooks" functionality automates checks and validation at critical points—such as during data writes and before publishing to production—guaranteeing the integrity and reliability of data versions. With hooks, #dataquality is continuously monitored, flagging any issues before they affect downstream workflows.

👉🥴👈Tired of ETL testing headaches? These 4 essential steps will set you on the path to hassle-free testing! 1️⃣ 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗡𝗲𝗲𝗱: Manual testing can be slow and error-prone; automating #ETLtesting is key to faster, more reliable data workflows. 2️⃣ 𝗦𝗲𝘁 𝗨𝗽 𝘁𝗵𝗲 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸: Discover the right tools and methods to create robust automated tests for your #datapipelines. 3️⃣ 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗲 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: Integrate testing into your #ETLworkflows for continuous validation and peace of mind. 4️⃣ 𝗚𝗲𝘁 𝗦𝘁𝗮𝗿𝘁𝗲𝗱 𝗧𝗼𝗱𝗮𝘆: Dive into the step-by-step guide on setting up automated ETL tests and watch your #productivity soar 💫 https://lnkd.in/dzcNK7qY