FastAPI and LLMs: A Great Choice for Serving Language models

Large Language Models (LLMs) are transforming language interactions, offering new opportunities to automate complex tasks. The challenge not only lies in training these models but also in efficiently deploying them in applications that leverage their capabilities. In this context, FastAPI stands out for facilitating integration with a lightweight, fast, and scalable infrastructure. 

Why use FastAPI to host LLMs? 

1. Native Support for Asynchrony: LLMs can require high response times, which may slow down APIs handling large volumes of requests if they don't support asynchrony. FastAPI natively supports asynchrony, allowing multiple requests to be managed simultaneously without affecting performance. 

1. Simplified Scalability: The demand for LLMs requires an infrastructure capable of scaling with increasing workloads. FastAPI easily integrates with tools like Kubernetes and supports optimizations such as WebSockets, which facilitate the scalability of APIs to handle a high volume of requests. Additionally, it supports streaming responses, which is useful in scenarios such as chatbot development. 

1. Integration with Machine Learning Tools: Deploying a machine learning model can be complex, but FastAPI, designed to work with external libraries, simplifies this process. As noted in the FastAPI documentation, “Or in other way, no need for them, import and use the code you need” (FastAPI Dependency Injection). This allows you to load the model before starting the API and use asynchronous endpoints for predictions without needing to modify pre-trained models. Another advantage is that FastAPI is built in Python, the most popular language for data science. 

1. Real-Time Processing and Validation: FastAPI’s asynchronous capabilities also apply to real-time data validation, allowing data to be formatted and checked for quality before reaching the model. This is crucial for LLMs, as the accuracy of responses largely depends on the quality of the input data. 

Creating a FastAPI Endpoint with LLM 

Next, we’ll explain how to create an endpoint with an LLM, where the bot has a predefined prompt that allows it to receive a topic and generate a summary or 'in a nutshell'. Although the endpoint will ultimately call the Together API, having a custom endpoint allows for greater flexibility and security, as well as easier integration with other services or processes. 

Step 1: Install Dependencies 

First, we need to install FastAPI to create the API, Uvicorn to host it, and Together to load the bot model that will generate the summaries. If you want to know more about this choice, you can check out this article "Exploring Open-Source LLMs and Together.ai: Alternatives to OpenAI for Your Business."

pip install fastapi uvicorn together  

Step 2: Import Dependencies and Create FastAPI Instance 

We import the necessary dependencies and create a FastAPI instance. The app variable is an instance of the FastAPI class, which manages the entire API. 

Step 3: Create a Together Client 

We create a Together client using the API key stored in environment variables (though not mandatory, it's a good practice to store keys securely). This client will interact with the language model to generate summaries. 

Step 4: Define the Endpoint 

We define a POST endpoint on the /chat route using the @app.post decorator. The asynchronous function generate_text accepts a message parameter, representing the user's input. Inside this function, we declare an internal asynchronous function called stream_response that handles the generation of the summary. 

Step 5: Bot Configuration 

In this step, we configure the model and prompt for the bot to generate the summary. We use the stream variable to enable real-time response streaming: 

Here, client.chat.completions.create requests the model to summarize the provided message. The stream=True option allows the response to be streamed in real-time. 

Step 6: Store the Real-Time Response 

In this step, we process the real-time response generated: 

We loop through each chunk of the response transmitted by the model. We extract the content of each chunk and store it in the content variable. We use yield to continuously return these content fragments, allowing the response to be streamed to the client in real-time. 

Step 7: Return the Summary 

Finally, we return the generated summary using a StreamingResponse: 

We use StreamingResponse to send the result of the stream_response function as a continuous response to the client. The media type text/event-stream indicates that the response will be streamed in real-time, allowing the client to receive the summary as it is generated. 

Step 8: Test the Endpoint! 

To test the endpoint, follow these steps: 

1. Run the API with the following command: 

uvicorn nutshellbot:app --reload 

1. Open your browser and go to http://localhost:8000/docs, which is the default address for the auto-generated documentation in FastAPI. 

In the documentation interface, you’ll find the /chat endpoint that you can use for testing. 

For example, sending the message "chatbot" will generate this response from the bot. 

IMPORTANT: FastAPI's auto-documentation does not natively support StreamingResponse. However, it is useful for testing the basic functionality of the bot and ensuring the endpoint works correctly. 

Conclusion 

Large Language Models (LLMs) are revolutionizing how we automate complex tasks, and FastAPI emerges as an ideal solution for integrating them efficiently. Its native support for asynchrony and real-time streaming enhances user interaction with applications based on these models.