Compute Grid’s Post

Have you always wanted to look into renting your own GPU, whether it be for ML or just to utilize better graphics for art or video processing, but all of it looked to complex? Well have no fear! Compute Grid is here! Compute Grid aims to demystify the GPU Marketplace by making it simple & easy for any one to get started! Don't believe me? Well just watch the video below to see just how easy it is to get started.

Use this simple estimator to compare the costs and energy consumption of a workload running on an x86 CPU-based server versus an NVIDIA GPU server. https://lnkd.in/eUmvCnAm

🚀 Large Scale Batch Processing with Ollama ⚡ Building on my previous article, LLM Zero-to-Hero, I’ve written a new article that covers creating a scalable LLM-based parallel batch processing system capable of fully utilizing a cluster of multiple GPU servers. I had access to a new pre-production GPU cluster and was struggling to achieve full utilization of multiple GPUs, but after some experimentation, I developed a batch processing client that can fully utilize all GPUs on a single host, or across multiple hosts, to process prompts in parallel. In my tests, I successfully utilized 28 GPUs across 7 hosts, achieving a throughput of nearly 100K prompts per hour (rate varies significantly based on model size and prompt complexity). The second half of this article explores how this method can be used to extract structured data from unstructured clinical notes. If you’re interested, you can check out the article here: https://lnkd.in/gHyXfrR4 #LLM #Ollama #GPU #HPC #Data_Processing

Kubernetes doesn't provide a support for the GPU sharing You must allocate the entire GPU to a workload, even if the actual GPU usage is less than 100% This project helps to improve GPU utilization by allowing GPU sharing between multiple workloads #k8s #gpu

Microsoft's BitNet.cpp: Unlocking the Power of 1-Bit LLMs on CPUs 🤯 Exciting news from Microsoft Research! They've just released bitnet.cpp, the official inference framework for 1-bit LLMs like BitNet b1.58. This game-changer offers: 🚀 Significant Speedups: Achieving 1.37x to 6.17x speedups on ARM and x86 CPUs respectively, with larger models seeing even greater gains! ⚡️ Impressive Energy Efficiency: Reducing energy consumption by 55.4% to 82.2% across different CPU architectures. 💪 Local Device Deployment: Enabling the execution of a 100B BitNet b1.58 model on a single CPU, reaching speeds comparable to human reading (5-7 tokens per second). This opens up incredible possibilities for running powerful LLMs directly on local devices, paving the way for more accessible and efficient AI applications. paper : https://lnkd.in/d-4pDW9x github repo : https://lnkd.in/dE5cT8Nx

vGPU, MIG, & Time-Slicing! 3 ways to optimize the #GPU resources to meet the dynamic demands of computational tasks💻 In this article, Sameer Kulkarni explains & compares them so you can make the best choice for better GPU optimization👇 https://lnkd.in/gFQPX4b4

Is it possible to run an (arguably) moderate sized LLM with 34 billion parameters without any GPUs? Well, surprisingly it’s possible to do that using GGUF (More details about GGUF here: https://lnkd.in/g_vZ83kz) I did a few experiments to compare inferences between gguf and non-gguf in my PC, and here are the results and code: https://lnkd.in/g4_Vp8BR Ok fine, we can run inferences with CPU.  But why should we run it?  Well, BECAUSE WE CAN.  But seriously, Use of gguf models at scale for inferences could help us operate at lower costs.  Of course serious analysis is required over a period of time to get reliable and satisfactory answers, but by-and-large it’s obvious that less resource-intensive machines in AWS for example are tiny bit cheaper at least. In addition to fast inferences, we can also perform POCs faster with APIs.  We can make a lot of automated inferences this way, for example to help with Prompt Engineering. Lastly, it is a tiny bit simpler to work with.  Model downloads and model loading times are significantly faster.

🎉 Thrilled to share that Container Runtime for Snowflake Notebooks is now in public preview! GPUs, no problem! Snowflake is making it easier than ever before to build and deploy models while using distributed GPUs, all from a single platform. Check out the blog to learn more:

NVIDIA’s CUDA platform has emerged as the dominant framework for GPU computing, enabling developers to harness the power of GPUs for a wide range of applications. By combining the capabilities of Kubernetes with the extreme parallel computing power of modern GPUs like the NVIDIA H100, organizations are pushing the boundaries of what is possible with computers, from realistic video generation to analyzing entire novels worth of text and accurately answering questions about the contents. However, orchestrating GPU-accelerated workloads in Kubernetes environments presents its own set of challenges. This is where the NVIDIA Device Plugin comes into play. It seamlessly integrates with Kubernetes, allowing you to expose GPUs on each node, monitor their health, and enable containers to leverage these powerful accelerators. By combining these two best of breed solutions, organizations are building robust, performant computing platforms to power the next generation of intelligent software. https://lnkd.in/gZQcU8Ju