Youssef Mrini’s Post

Kubeflow, Ray, KubeRay,mlflow,Flyte,Argo,JupyterHub,Kserve and many more…. As I am exploring the landscape for OpenSource MLops and ML Lifecycle management, distributed computing and s more… the list just gets longer and the data is showing that different people build different platforms and combination of tools based on very specific needs. If you’re running on OSS, What are your tools ? What is your sauce for running your ML Data Process ? I am curious to know .. #mlops #oss #genai #aiml

Have you heard about lazy evaluation in Spark? It's an interesting strategy that allows Spark to perform tasks in the most optimized way possible. Essentially, Spark doesn't execute transformations on datasets immediately, but instead creates a plan of action called a Direct Acyclic Graph (DAG). This plan is executed only when an ACTION function is called, such as collect, count, saveAsTextFile, and more. The advantages of this approach are significant. Lazy evaluation leads to optimization, reduction in disk memory usage, and more. It's a powerful tool for anyone working with Spark and looking to streamline their processes. Share your experiences in the comments! #Spark #Azure #AzureDatabricks #Dataengineering #Pyspark

I keep harping, tirelessly, for all those who.choose Databricks, to consider seriously using Photon Engine and spend a good amount of time designing their Unity Catalog & tagging methodology. here is a very good detailed explanation of why Databricks built the Photon Engine , written by Vu Trinh. #databricks #photon Vu Trinh https://lnkd.in/g4UPFmXS

In case you missed Jeff Chou's talk at Databricks Data + AI conference, check it out below. This talk covers best practices to managing jobs at scale while lowering compute costs . It highlights common pitfalls and effective strategies for managing Databricks Jobs clusters, SQL warehouses, and ML workloads. 💸 Compare the cost and performance of serverless vs. optimized classic compute clusters. 💡 Get tried and true optimization tactics based on years of research into high-frequency computing  💪 See real-world examples of companies saving up to 65% on their Databricks compute spend Watch it now and let us know what you think in the comments 👇 https://lnkd.in/g8YMZYFQ

Pipeline Optimization: Lessons You Only Learn by Doing When I started, I thought a working pipeline was all that mattered. But after nearly two years in the field, I learned that performance is just as important. I had a pipeline pulling from multiple sources, and it took forever to run. That’s when I dug into optimization. One key technique? Parallelism and partitioning. Tuning parallel copies and splitting large datasets effectively cut the runtime in half. It took some time to get right, but the payoff was huge. If your pipeline’s dragging, ask yourself: can I optimize the data partitions? #DataEngineering #Azure #Optimization

Prototype to Production: Crafting Applications with #AzureCosmosDB - Part 2 This is part two of the three-episode series that dives deep into the challenges faced when moving applications from prototype to production. https://lnkd.in/ef-dXKvC

Prototype to Production: Crafting Applications with #AzureCosmosDB - Part 1 This is part one of the three-episode series that dives deep into the challenges faced when moving applications from prototype to production. https://lnkd.in/ePkypP6N

🚀 New Blog Post: Mastering Spark - Parallelizing Non-Distributed Tasks 🚀 Spark excels at distributed computing, but what about tasks that aren't inherently distributed? While reading from a Delta table is easily parallelized by Spark’s APIs, things get trickier with user-defined tasks like API calls or custom computations. In my latest post, I show you how to leverage Spark’s distributed nature for tasks that Spark doesn’t natively handle—ensuring that even non-distributed tasks benefit from the scalability of Spark clusters. 👉 Check out the link in the comments #ApacheSpark #MicrosoftFabric #FabricSpark #dataengineering #distributedcomputing

Under the hood DataChain combines power of warehouses with distributed clusters with proper data access patterns to process millions of video, images audio files: ☁️ Never copy data. Store references to files instead. (while still preserving versioning, data loading, efficient processing) ⚡ Use warehouses under the hood (e.g. ClickHouse) to store metadata and perform as many operations inside it (e.g. filters). ⚙️ Distributed compute that runs close to the data to compute Python-based UDFs 🤏 Data access. Pre-fetch, batching, caching, streaming - different workloads require different ways of using data. #unstructured #datachain #dvc #machinelearning #opensource

8. Post: SparkContext: The Engine Behind PySpark’s Distributed Power   "SparkContext is the entry point into the PySpark ecosystem, managing the distributed computing resources and job scheduling. Without SparkContext, PySpark wouldn’t be able to coordinate tasks across clusters. #Amazon and #Airbnb rely on SparkContext to handle the orchestration of their massive data processing workloads. Learn about SparkContext’s role: [SparkContext Guide](https://lnkd.in/geQPVMNw). #BigData #DistributedComputing #ApacheSpark"