Peter Seddon’s Post

I have thought for a some time about options in designings for hosting infrastructure for either AI training and inference tasks and high performance computing (HPC) tasks. Options have being ranging from traditional batch scheduling solutions with suitable VM’s to various flavors of containerized work loads using Kubernetes or its derivatives, of course with underlying VM and networking infrastructure. This article is on that perspective interesting reading. https://lnkd.in/dFb-FVFS

A big day for Chainguard today with our $140 million Series C round of funding 🐙. Another reason why today is exciting is that we are also announcing General Availability for Chainguard AI Images, a growing suite of CPU and GPU-enabled container images, including Pytorch, Conda, and Kafka, that are hardened, minimal, and optimized for efficient software development. AI Applications heavily rely on open-source software for all their components, and we are on a mission to make sure they are free of vulnerabilities, including GPU workloads. To learn more check out our most recent blog post! https://lnkd.in/gZ5UcT4e #ai #machinelearning #aisecurity #supplychainsecurity

💰Let's examine the pricing of Llama 3.1 405B. ⚡️405B is revolutionizing AI accessibility and affordability so far. As industrial leaders deploy the Llama 3.1 405B model on servers, they offer remarkably competitive pricing. 👇 Check it out: 🟢 Llama 3.1 405B: $3 / $3 per million tokens for both input and output (https://lnkd.in/d9efkw_6) 🔵 Claude 3.5 Sonnet: $3 input / $15 output per million tokens 🟣 GPT-4: $5 input / $15 output per million tokens

Local LLM Model in Private AI server in WSL - learn how to setup a local AI server with WSL Ollama and llama3 #ai #localllm #localai #privateaiserver #wsl #linuxai #nvidiagpu #homelab #homeserver #privateserver #selfhosting #selfhosted

OpenAI announces new o3 models “…. instead of retrieving memorized information, it searches through possible solutions and reasons about them step by step, though this process takes more time and computing power. This addresses a limitation of previous LLMs because it can recombine existing knowledge in new ways to solve novel problems rather than just applying memorized patterns…..”

Unlock 75% cost savings with Gemini Context Caching! 🚀 Imagine this: You’ve got a considerable context size, and every time you make a request, you’re thinking, “There goes my lunch money!” 🍱 Well, worry no more! Context caching has saved the day. 🤖 RAG vs. Caching: Which is the better choice? ⚠️ Limitations: Are there any? Let's see. 💰 Pricing: How does the pricing compare to not using a cache? 🔧 Usage: Step-by-step guide to implementing caching with Gemini 📊 Usage Metrics Confusion: Clearing up the confusion once and for all! 💡 In-context Learning: Unlock huge savings while using extensive examples without fine-tuning! #contextcaching #Gemini #VertexAI #GoogleCloud #GenAI

#Fine #tuning a foundational model has the potential to enhance its operational efficacy. Foundation models are typically trained to serve broad purposes; however, they may not consistently meet desired performance levels, particularly in specialized tasks. This limitation often arises due to the inherent difficulty in imparting specialized task knowledge to the model solely through prompt design methodologies. The process of model tuning entails furnishing the model with a comprehensive training dataset rich in examples pertinent to a specific task. In scenarios involving unique or niche tasks, substantial enhancements in model performance can be achieved through meticulous tuning, even with a modest number of examples. Following the tuning process, the model's reliance on examples within its prompts diminishes significantly. #GCP #VertexAI supports the following methods to tune foundation models: a. Supervised tuning b. Reinforcement learning from human feedback (RLHF) tuning c. Model distillation To tune the models the following sequence happens: i. Pipeline Validation ii. Dataset Export iii. Prompt Validation iv. json to tfrecord conversion v. Parameter Composition for Adapter tuning vi. LLM Tuning vii. Model uploading viii. Endpoint deployment #KnowledgeGraphs #GenerativeAI #LLMs #GraphDB #Neo4j #Cypher #FineTuneLLMs #Langchain #GraphML #NodeEmbeddings #Chatbots #Gradio #GCP #vertexai #PaLM2 #mlops

🚀 vLLM for LLM deployment on AWS At Rootflo, we have a use case of deploying LocalLLM on-premise. I spent my day setting this up and integrating the same with flo-ai (https://lnkd.in/gYSqRVNv) Here are some learning from the exercise: 💭 Deploying fast interference models using vLLM is simple and easy. We used a docker for deployment and did it on an AWS G5 instance with L4 GPU, which has 48GB GPU memory 🏗️ Deploying models with bigger context sizes or bigger parameter counts needs larger GPUs. However, context size directly depends on the KV cache, so it can be reduced to deploy into a smaller system. We deployed a 128k size llama model as a 64k context size deployment. vLLM takes care of lowering the context size using `--max-model-len` 💡 vLLM provides an openai-compactible docker image, making the model available as APIs that can be used with flo-ai (langchain or llamaIndex). Overall we were able to test flo-ai against vLLM deployment pretty easily. The next steps for us are to benchmark vLLM and understand how much load can it handle. Will keep you posted about the progress #ai #generativeai #localllm #llama #phi3

TAI 131: OpenAI’s o3 Passes Human Experts; LLMs Accelerating With Inference Compute Scaling via #TowardsAI →