lakeFS’ Post

The Databricks Unity Catalog helps organizations seamlessly govern both structured and unstructured data in any format, as well as machine learning models, notebooks, dashboards and files across any cloud or platform. Our own Sun Jang writes about our experience governing data lakes and SQL servers using the Unity Catalog. https://hubs.la/Q02P2Dyl0

How to Expose Delta Tables via REST APIs is a common question. There are multiple solutions to achieve this. I describe three architectures with its pros and cons in this blog. #azure #msftadvocate

Kate hit it out of the park with this blog post on Aurora DSQL. She does a beautiful job of explaining what makes DSQL unique and outlining the use cases that are great fits for this new database from AWS!

The Unified Apache Iceberg Lakehouse. Click below to learn more. #DataLake #DataLakehouse #S3 #ADLS #Azure #Iceberg #ApacheIceberg #SQLQueryEngine #GCP #Snowflake #PolarisCatalog https://lnkd.in/e_4S3Y89

Check out this informative blog post on how to replicate CDC events to CrateDB using AWS DMS! If you're looking to efficiently manage and analyze your data, this is a must-read. Click the link to learn more! #AWS #datareplication https://hubs.ly/Q02L49_W0

Who needs post Re:Invent sleep when there's Aurora DSQL to talk about?!? In my latest blog, I discuss its disaggregated architecture, data patterns that thrive with Aurora DSQL (like high-concurrency and read-heavy workloads), and tips to determine if it’s a fit for your organization. At DoiT, we specialize in cloud consulting and can help you maximize Aurora DSQL or other advanced databases. Read the full post here: https://lnkd.in/eeEvwY-W #aurora #dsql #doit #reinvent2024

Delta Lake UniForm is an open table format that provides a universal data representation for efficient reading by different processing engines. It bridges gaps between data formats and systems, offering standardized data storage/retrieval so Delta Lake tables can be read as Apache Iceberg tables. #aws #awscloud #cloud #amazonemr #analytics #awsglue

Could you stream your database changes in real-time into an event-driven system? Enabling seamless updates to search indexes 🔍, other systems 🔄, and analysis platforms 📊. That's exactly what I've been working on in my current multipart article series, and I’m ready to share it with you! In this part, we go into setting up and configuring the CDC pipeline, powered by Azure, allowing you to capture changes from your database and feed them into your broader architecture. It’s pretty exciting, if you ask me! 🚀 #Azure #Debezium #CDC #SqlServer #ChangeDataCapture #Stream #EventBasedSystem

Keeping resource labels consistent and up-to-date in cloud environments is critical—but also one of the hardest challenges to solve… To tackle this, we at 1KOMMA5° created a #Terraform module that automatically fetches metadata from Backstage components and transforms it into a set of standardized labels for your infrastructure. But there’s a catch: infrastructure provisioning can become dependent on Backstage being online. If Backstage goes down, deployments grind to a halt. To solve this, we worked with Tomas Dabašinskas, the founder of Datolabs and creator of the Terraform Backstage Provider, to extend the provider with resiliency features. These include a fallback to existing metadata, ensuring that even when the Backstage API is unavailable, your infrastructure deployments can continue without interruption. All of this is baked into the module, which I’m now open-sourcing and sharing with the community. If you’re managing infrastructure metadata with Backstage, this module offers a resilient way to fetch and apply labels across your resources. Check it out here: https://lnkd.in/dJRCaHsG Feedback and contributions are always welcome! #terraform #backstage #terramate #oss Vibhor Kainth Martin Lefringhausen Klaus L. Alexander Zenger Ankit Raj Teresa Ingram William Campos

For large enterprise customers, it is often a necessity to use multiple purpose built cloud services and tools to support analytics business use cases. For customers using AWS Glue for Data Lakes and Snowflake for data consumption, they can facilitate data interoperability between platforms using Apache Iceberg file format. This pattern provides flexibility to choose fit-for-purpose tools ensuring no data duplication. #aws #awsglue #snowflake #datainteroperability