Wang Yi’s Post

The new Dedicated access mode (previously Single user) allows you to assign a dedicated all-purpose compute to a group or single user. It's very interesting for people using the ML runtime since it's not supported yet in Shared Clusters. Martin Grund Stefania Leone #Databricks

I keep harping, tirelessly, for all those who.choose Databricks, to consider seriously using Photon Engine and spend a good amount of time designing their Unity Catalog & tagging methodology. here is a very good detailed explanation of why Databricks built the Photon Engine , written by Vu Trinh. #databricks #photon Vu Trinh https://lnkd.in/g4UPFmXS

Kubeflow, Ray, KubeRay,mlflow,Flyte,Argo,JupyterHub,Kserve and many more…. As I am exploring the landscape for OpenSource MLops and ML Lifecycle management, distributed computing and s more… the list just gets longer and the data is showing that different people build different platforms and combination of tools based on very specific needs. If you’re running on OSS, What are your tools ? What is your sauce for running your ML Data Process ? I am curious to know .. #mlops #oss #genai #aiml

In case you missed Jeff Chou's talk at Databricks Data + AI conference, check it out below. This talk covers best practices to managing jobs at scale while lowering compute costs . It highlights common pitfalls and effective strategies for managing Databricks Jobs clusters, SQL warehouses, and ML workloads. 💸 Compare the cost and performance of serverless vs. optimized classic compute clusters. 💡 Get tried and true optimization tactics based on years of research into high-frequency computing  💪 See real-world examples of companies saving up to 65% on their Databricks compute spend Watch it now and let us know what you think in the comments 👇 https://lnkd.in/g8YMZYFQ

Pipeline Optimization: Lessons You Only Learn by Doing When I started, I thought a working pipeline was all that mattered. But after nearly two years in the field, I learned that performance is just as important. I had a pipeline pulling from multiple sources, and it took forever to run. That’s when I dug into optimization. One key technique? Parallelism and partitioning. Tuning parallel copies and splitting large datasets effectively cut the runtime in half. It took some time to get right, but the payoff was huge. If your pipeline’s dragging, ask yourself: can I optimize the data partitions? #DataEngineering #Azure #Optimization

👩💻 From our engineering team's keyboard 👨💻 Our latest Tech Blog is out now, featuring the brilliant minds of Patrick Conway and Philip Catterall. They dive into a new view caching strategy that revolutionized our ELT pipelines at ActionIQ, reducing compute usage by 50%. Check out the full blog here: https://hubs.li/Q02tY90Q0. #Tech #ELTpipelines #TeamWork

A helpful article about integrating MLFlow with Airflow based on the docker-compose file. These 2 tools support us in automating a machine learning/deep learning pipeline while tracking the results as well as providing the UI for easy compared observation https://lnkd.in/eTNxCXEB

I'll be at AWS re:Invent 2024! Would love to connect with developers, ML practitioners or anyone else interested in: - Vector Quantization - Hybrid search architectures - Embedding model capabilities - Search evaluation metrics - Configuring HNSW indexes 🔬 DM me if you're attending and want to dive into the technical nitty-gritty of vector search systems! #AWSreinvent #MachineLearning #MLOps #VectorSearch #AIEngineering"

Under the hood DataChain combines power of warehouses with distributed clusters with proper data access patterns to process millions of video, images audio files: ☁️ Never copy data. Store references to files instead. (while still preserving versioning, data loading, efficient processing) ⚡ Use warehouses under the hood (e.g. ClickHouse) to store metadata and perform as many operations inside it (e.g. filters). ⚙️ Distributed compute that runs close to the data to compute Python-based UDFs 🤏 Data access. Pre-fetch, batching, caching, streaming - different workloads require different ways of using data. #unstructured #datachain #dvc #machinelearning #opensource

🚀 New Blog Post: Mastering Spark - Parallelizing Non-Distributed Tasks 🚀 Spark excels at distributed computing, but what about tasks that aren't inherently distributed? While reading from a Delta table is easily parallelized by Spark’s APIs, things get trickier with user-defined tasks like API calls or custom computations. In my latest post, I show you how to leverage Spark’s distributed nature for tasks that Spark doesn’t natively handle—ensuring that even non-distributed tasks benefit from the scalability of Spark clusters. 👉 Check out the link in the comments #ApacheSpark #MicrosoftFabric #FabricSpark #dataengineering #distributedcomputing