A Generative AI Buyers Guide

Generative AI (GenAI) represents a source of significant competitive advantage for many organisations. It is unfortunate then that in the rush to invest in GenAI, many organisations are succumbing to marketing hype and relatively small cash inducements from the various vendors to make quick decisions about their GenAI platform strategy. These GenAI investment decisions may have significant implications for an organisation for years to come. What I would encourage any organisation looking to make a decision about which GenAI platform to use, is to pause, consider a few platform options and ask a few questions before making a decision about their GenAI platform of choice. It may seem inconsequential now but it could be one of the biggest technology purchase decisions you ever make!

Over the last year, I've been lucky to go on hundreds of hours of Generative AI (GenAI) training by all the major GenAI providers and have been involved in more than 10 different GenAI pilots across a variety of technology platforms and industries. I have experience with all the major GenAI platform providers including IBM's WatsonX, Microsoft OpenAI / Co-Pilot, AWS Bedrock/Sagemaker, and Google Vertex. I already carry advanced cloud and ML certifications in Microsoft, AWS, Google and IBM and have taught on GenAI. Based on that training and practical experience and looking across all these products, I have the following observations/ insights into what capabilities/ functionality organisations should be considering when investing in a GenAI cloud platform.

What follows here then is a simple list of capabilities that many organisations need and that most GenAI platforms have that you can use to evaluate any proposed GenAI platform you are looking to invest in. Essentially, a shopping list (in no particular order) of GenAI platform capabilities, features and functions that you can consider and ask questions about when making your GenAI platform decision...

1) Local Service Availability

Many of the GenAI cloud services may not be available in a specific country. This has to do with the fact that supporting GenAI requires platform vendors to make significant investments in local infrasructure. As many vendors have rushed to market they have had to make choices about which markets they invest in first. So your preferred GenAI platform may not be available in your market for some time. This can be a concern for those organisations worried about Data Soverignty.

2) Model Availability

All the GenAI platform providers have their own proprietary LLM or multimodal models. All generally offer some open source models as well. There are literally hundreds of thousands of open source models available on the market. Yet some of the platform providers only offer a handful of models through their platforms. Others offer hundreds. There is no doubt that the platform providers are adding to the list of available models every day. But you will want to look at that list to see if your GenAI models of choice are available through the platform provider you are considering.

3) Low Code LLM Testing / Prompt Evaluation Environment

Just like with any AI solution there is a significant amount of experimentation and testing that needs to go on before you can settle on the right solution architecture. GenAI is no different. Decisions about how you design your search, how you to construct your prompting, the Large Language Model (LLM) you use, wether you are going to use Fine Tuning or not and how to set the hyperparameters of that LLM all have to be arrived at through experimentation. Most of the vendors provide low code UI driven environments to allow you to do that experimentation easily before you hard code the solution into your end application. The quality of the UI and it's ease of use are two factors to consider when picking a GenAI platform.

4) Model Evaluation Tools

As mentioned in the previous section selecting a GenAI soluton Architecture requires a high degree of experimentation before you can settle on the right solution architecture. Having built in model evaluation tools that help you pick the right LLM and configure it correctly are important. Whilst all the platform vendors have these tools in some shape or form some are better than others and the quality of the evaluation tooling can have an impact on the quality of your GenAI solution. So comparing evaluation tool sets across the GenAI platforms under consideration can be important.

5) Flexible Development Environment for More Complex Use Cases

Whilst all of the vendors offer you the ability to move straight into production from your low code Gen AI, UI driven environment, most use cases involve more complex prompting or use of other more advanced techniques such as functions, code extensions, etc. Therefore you need to have a solid development environment (usually Jupyter notebooks and Python) to allow you to handle these more complex use cases. So considering the flexibility of the Gen AI Development environment that is part of your platform is an important consideration.

6) Compute/ Fine Tuning Options

Whilst most users of GenAI will start out with simple Prompt Engineering to get the results they want, there is no doubt in my mind that as the user expands their use of GenAI they will start looking at Fine Tuning of the LLMs to produce better results. Fine Tuning can be resource intensive, costly and take a long time. Some vendors have better options than others in terms of Fine Tuning their models. Therefore looking at the tooling the vendor provides around Fine Tuning a model can be important.

7) Retrieval Augmented Generation

One of the biggest challenges with LLMs and GenAI models is their tendency to hallucinate and make things up. This hallucination can be a real problem depending on the use case. One of the ways to reduce hallucination is to "ground" your LLM. In simple terms this means providing the LLM model with your own currated data specific to your use case and telling it to base it's responses off of that data. This process is referred to a Retrieval Augment Generation (RAG). By some estimates 60% of GenAI use cases will involve some form of RAG. Therefore the ability to provision a RAG quickly and easily through your GenAI platform is important. The optionality in terms of search, data source, etc. that your platform provides in terms of building a RAG is also important.

8) Hidden Throttling - reading the fine print in terms of capacity

As previously mentioned providing a cloud based GenAI platform is infrastructure resource intensive. What I have realised in my own personal use of the various GenAI platforms is that several of the platform providers in an effort to enter a market quickly have quietly throttled the throughput capacity of the platforms they provide. This is something that is often buried in the fine print of your user agreement. So you might want to ask what capacity they have and wether they are doing any throttling before pick your GenAI platform.

9) LLM Operations - managing the model's ongoing

Most organisations are still in the pilot/ experimentation stage of their GenAI journey. Assuming they move to the next stage which is to productionise some of their GenAI pilots they are going to need tools and processes to manage their GenAI and LLM solutions ongoing. This is referred to LLM Ops. Much like traditional AI, GenAI models that are currently working can stop working for a variety of reasons including data drift, changes in user behaviour etc. LLM Ops is the process and tooling that you use to manage your GenAI models, identify when the stop working or when their performance degrades and take corrective action to get them working a peak performance again. Therefore if you are serious about GenAI you need to consider how you will do LLM Ops and how your GenAI platform will support this.

10) Governance Tooling - and being regulatory compliant

I've written many articles on the issue of AI regulation and governance. Fact is most countries, with general public support, are ratching up the amount of regulations they have to govern the use of GenAI by organisations and to make sure people are using GenAI safely. I've written a whole article on the topic which you can find here: The Components of a Good GenAI Governance Program: How to manage GenAI risk and be AI regulation compliant... | LinkedIn . All the GenAI platforms provide some form of tooling to support governance and regulatory compliance. Some are better than others and it is something you should consider before picking a platform provider if you want to avoid large $ fines. Your other option is to pick your platform provider but then supplement it with a third party GenAI Governance tool like WatsonX.Governance. See here for details.

Conclusions:

The market for GenAI is fast growing. The adoption of GenAI has been faster than any technology that has come before it. There is no doubt that organisations have a lot of options now in terms of their GenAI platform provider. There is also no doubt that all those providers already offer a lot of functionality in their platforms. The question is how good is that functionality and which functionality is more important to you. Hopefully this article has given you some ideas and questions to ask your potential GenAI platform provider before you make what could be one of your more important technology decisions.

Dr David Goad is the CTO and Head of Advisory for IBM Consulting Australia and New Zealand. He is also Microsoft Regional Director. David is frequently asked to speak at conferences on the topics of Generative AI, AI, IoT, Cloud and Robotic Process Automation. He teaches courses in Digital Strategy and Digital Transformation at a number of universities. David can be reached at david.goad@ibm.com if you have questions about this article.