So, you have an FAQ?

Whether yes or no, this is your way in to using AI.

Do you have an FAQ? You know, that always out of date collection of snippets of info hypothetically of use to your company staff, or your users, or customers, or clients. Where the least favourite task in your company, the most boring of grunt work, the most thankless, the usually left undone task, is updating the FAQ?

If so, then congratulations, you already have the raw material for AI-enhanced greatness. Nowadays, it is easy to create and maintain high quality FAQs, and to do so much more with them. Possibilities for synergies abound, a win-win, a positive feedback loop, real potency.

If not, never fear, for you can soon and easily have an FAQ of your very own, and then the above hyperbole applies.

For the purpose of this article, we will focus on the FAQ as one source (among several) of answers to some of your users’ common website search queries. The same mechanics and benefits apply for queries by clients or customers contacting customer support, or even sharing internal staff knowledge. But at least we can stop with the clunky catch-all phrase, “your company staff, or your users, or customers, or clients”.

So (whisper it), you don’t in fact have an FAQ?

What is an FAQ?

The acronym FAQ stands for Frequently Answered Questions.

It is an accumulation of Things Worth Knowing :™:, stuff your users frequently ask about. It contains routes into your products, content, or expertise. It defines issues for which you already have good answers, and why waste time re-thinking an answer when you can just reply with the canned answer?

Or, often under-emphasised, an FAQ is a repository of what you know about your users’ needs or values. It can be an aspect of what makes your company distinctive and valuable to your users.

Your FAQ can have several distinct parts, such as

Basic Site Navigation - “Where is X functionality?” … Which is like a UX bug report, so maybe should not be in the FAQ and instead fix the site? YMMV.
Basic Topics - “What is an X?”
Basic Help - “How do I X?”
Product inquiry - “Do you have an X?”
Best Practices
…

Who can do FAQ?

Find members of staff with analytical tendencies (they are legion), ideally with opinions about the benefits to society of correct, precise wording. They will understand FAQs, LLMs, and the power they can wield, instinctively. You will unleash an irresistible force within your company. Be warned.

My First FAQ, by hand

FAQs are easy to set up. A simple text document containing a list of question/answer pairs, and, well, that’s it.

Someone asks a question. If the question is covered by the FAQ, reply with that answer. If not, create a new answer (via a local expert), add it to the FAQ, and reply with that. Very quickly, the FAQ builds up.

Actually, that would be it in an ideal world. But in reality, FAQs inevitably go stale, unloved, festooned with cobwebs and regret. Started and fed by someone who once cared, but abandoned when that person leaves. Very rarely do staff volunteer to maintain the FAQ.

You will need a non-life-force-draining way of maintaining the FAQ, keeping it fresh, relevant, and focused. Read on.

My First FAQ, with some AI help

If you have some customer queries, let’s say a few hundred, get access to the free chat webpage of one of the latest big LLM offerings. Easy to do. Manually create a long LLM prompt listing the queries, and asking the LLM to distil the most commonly asked questions. Just the questions. See below for a description of LLMs, and the Case Studies at the end for an example prompt.

You now have a starter for 10 for your FAQ. Whether you agree or disagree with the suggested items, it is something to work with. Next is a wee task to construct (or locate) answers to those questions.

You can get more nuanced about the framing of your FAQ prompt. Ask the LLM to identify the different types of questions, and to group them accordingly, e.g. site navigation “how do I …” questions, factual “what is the X of Y” questions, bug reports, etc.

Once you have seen how the LLM chooses to group the questions, you can modify that high level grouping as you see fit, insert it into a new prompt, and get the LLM to look for questions within those specified groups. The bug reports won’t go in the actual FAQ, of course, but they will be of interest to the support teams.

Even without real customer queries, you can get started if you have a document or two which encapsulates what your company is about. Feed that to an LLM and ask it to generate questions based on the document. See the Case Studies at the end for an example prompt.

So, NOW you have an FAQ, what about those AI-lit uplands you were promised?

Two AI tools we will make use of:

Vector Search

This technique, which pre-dates the recent furore around chatbots, is described in more detail in [Mind The (AI) Gap]. In essence, there are easy to use bits of software which can convert a small block of text (say a question, or a question and answer pair) into a vector (a list of numbers). The magic is that texts with similar meanings have similar vectors. This gives you the ability to match an incoming user query with a range of relevant Q&A items quickly and easily. This works very well indeed, even when different-but-meaningfully-similar texts have no words in common.

Modern search engines now tend to support vector search. And you can cobble together something FAQ-sized yourself with only a little bit of coding (remember Cosine Similarity from school?). This also can be done quite easily for live user queries.

LLMs - Large Language Models (underlying ChatGPT, and friends)

Also described in more detail in [Mind The (AI) Gap]. The basic gist is that if you can clearly describe the kind of info you want extracted from some given text, as if instructing a keen, literal-minded, but slightly dim intern, the LLM can probably take those instructions and the given text, as a prompt, and do a reasonable job of carrying them out effectively.

This is not asking the LLM for facts it has learned (or stolen from creators) during training, or confabulating in a desperate effort to please, but simply using its inherent facility with English text to understand (kinda sorta) the instructions and to look for certain concepts in the given text.

Recent LLMs have enormous context windows, that is to say, the ‘given text’ can be huge, 100’000+ words, entire novels. So huge in fact, that if you gave the same task to a human, it would take them a very long and tedious time to carry it out, plus there is every chance they will miss stuff. The LLM will get you a decent answer back in ~10s, for little to no cost.

LLMs are too slow and expensive for responding to a barrage of concurrent, live user queries on high traffic websites, but ideal for calm, methodical, manual processing offline.

Improving and maintaining the FAQ

Every few hundred new user queries, manually re-interrogate the LLM, specifying the existing FAQ, the list of new user queries, and instructions to identify any user queries not well served by the existing FAQ. You’ll get a snappy reply listing a few new question suggestions for the FAQ. Update the FAQ accordingly. (See the Case Studies at the end.)

You could also track the ‘answered-ness’ of incoming user queries, logging those that did not find good matches in the FAQ. That would give you a dense list of unanswered queries.

As easy as that. It might take a few experiments to settle on the largest effective batch size of user queries to process, but definitely hundreds should be a fine start.

This process can be part automated. Every few hundred user queries, run the LLM query. Generate an email report of the outcome if there are new un-answered questions cropping up. Someone will still have to manually update the FAQ, but this is a lot less burdensome than before.

The current FAQ sets the tone for subsequent question suggestions. You have explicitly defined for the LLM: this is what we care about, and this is how we express it. The FAQ can iterate to be quite different from the initial draft, as you home in on your company’s FAQ voice.

If the FAQ becomes unwieldy, perhaps too many overlapping questions, ask the LLM to re-assess the FAQ, with the aim of streamlining it, and to suggest a new structure.

There is a possibility of using the LLM to generate answers for the new questions, but that is not within the scope of this article. That approach has a variety of pitfalls, not necessarily using the LLM to its best.

Although, there is a possibility to consider. If you use vector search to find pieces of your own content which match the new user question, you can construct a prompt where you give the LLM some or all of that relevant content, and ask it to answer the new user question only using that relevant content. This is an easy first draft of an answer, likely to be fairly good.

Ideally, your FAQ is a high quality, highly useful, comprehensive expression of all the corporate/user knowledge from the expertise within your company. Best not to dilute this by using LLMs to generate poor answers. Questions, yes. Answers, generally no (but see above).

Using the FAQ

The OG use of FAQs

The original uses of your FAQ still stands, manually answering frequently asked questions with a copy-pasted response.

Or as a script for customer support agents.

FAQ in Search

It can occasionally work if you have the FAQ as a source of items in the site keyword search, but the main weakness of keyword search prevails: get the query words ‘wrong’ and you won’t find anything useful. What is worse, you won’t know if there was actually nothing relevant, or the user chose the ‘wrong’ words to look for. Most unsatisfactory.

The game-changer here is you can use text vectors to robustly match an incoming user query with the most relevant items in the FAQ, regardless of specific phrasing. If there is a relevant FAQ item for a user query, vector search will find it, and it will be clear when there is no relevant FAQ item.

Furthermore, you can use the FAQ to shape the user query for a deeper search. This is subtle, but profound. Presumably, the FAQ represents what your company knows about and is good for. You can nudge a user query towards the relevant part of the FAQ, towards what you are good for.

If there is a particularly relevant FAQ item for a given user query, you can embellish the user query with that FAQ item and make a subsequent search into your news articles, products, whatever.

An example might be an investment website, where the user has asked a naive query that matches an FAQ item full of advice and best practices. The subsequent search will be matching on a combination of the user query and the FAQ item, and might find articles extolling the best practices mentioned in the FAQ item, which it would not have matched with the initial user query.

You have a UX choice about whether to present the FAQ item as the main part of the response, supported by associated article/product results, or the FAQ item is supporting those results. Or simply keep the FAQ aspect hidden from the user.

It may be you wish to keep the actual details of your FAQ under wraps, offline, out of reach of the Slurper Up Of All Text (which rhymes with Floogle), in which case you do not need to include the FAQ item in the response at all, but can simply use it to shape the search behind the scenes.

FAQ-based Chatbot

If you did want to go down the route of having a chat interface to an LLM for your users, (Don’t. Really don’t. But if you did…) the FAQ is a very good way of constraining the chat responses to be on-message. It helps alleviate some of the main types of risks associated with allowing external users free access to a delusional chatbot that represents your company.

You can wrap the LLM prompt in robust(ish) guardrails:

checking that the user query is in good faith, and not obviously trying to be problematic, looking for sneaky indirection and other jailbreaks
Checking that the user query can reasonably be answered from the FAQ
Insisting that the query is only answered from the FAQ, and “I don’t know” is the default answer if there is any doubt.
Then and only then attempting to answer the user query from the FAQ, citing which part(s) of the FAQ were relevant.

Turn your site navigation on its head

Make the FAQ the root of your site tree. All topics, products, articles, etc, hang off the appropriate FAQ items.

It is a simple matter to assess each of your topics, products, articles, etc, using vector search, identifying which FAQ items they are most relevant for, and highlighting where they are not relevant for any FAQ item.

So now you have a list of identified topics, products, articles that are not often relevant to your customers. This is a problem, an opportunity, or a non-issue. You decide.

An interesting thought exercise.

SEO?

Do you really want to expose your FAQ to the world? It is, after all, a repository of what you know that your users need or value, and an aspect of what makes your company distinct, valuable.

This is a big question. If you do, it will be taken and used without your control/permission by the giant Slurpers Up Of All Text.

It is not clear what SEO means these days. The recent announcements from the biggest SEO AI player seem to indicate they are now taking it all over and doing their thing with it. Maybe the rest of us will get some crumbs. But …

FAQ items as topics

A user might have a longer term interest in an FAQ item, if it represents a particular area of interest to them.

The user might want to know if the answer is ever changed, or if there are new topics, products, articles relevant to the FAQ items. Or if there are new, related FAQ items.

In short, they might want to save/follow the FAQ item, and be alerted if there are changes.

Engagement to die for.

FAQ items as community focus

Continuing to treat FAQ items as persistent topics, if one user is interested in an FAQ item, the likelihood is so will others be. You could offer comments functionality on each FAQ item, encouraging more in depth discussion around the topic of the item.

Such an engaging situation.

New FAQ items are news

This is in the same territory as SEO. You will lose exclusive access to your FAQ if you do this, but it might be a positive for your users.

Ah, engagement again.

Educating and shaping user behaviour

You can start linking ‘good’ query phrasing to FAQ items, and use it to suggest alternate queries to a user, when their query matches an FAQ item.

Targeted FAQs

You could have various, expertise or persona-based FAQs, each phrased/targeted differently.

One for experts
One for newbies
One for JFDI types
…

Here, LLMs might be usable. Changing the tone of an existing text, changing the level of expertise of the target audience, is something LLMs do well.

And so many more.

Matching your FAQ to world events.
Answering external issues (e.g. orgs in trouble) with your FAQ. How would you tackle other people's problems?
Personalised FAQs - given what we know of a user, create an entire FAQ just for them.
FAQs in different languages
FAQ Tests/Quizzes - How well does a user understand the topic represented by the FAQ?
Matching internal experts and expertise to the FAQ.
Identifying gaps in internal expertise. What don’t you (collectively) know?
The FAQ as a definitive statement of who we are and what we are about.
Onboarding new staff? - read the FAQ.
…

So, why have FAQs so often failed to thrive?

It comes down to multiple failures of framing, including:

Considering FAQs as individual Products, and not as shared Capabilities.
Considering priorities Product First, rather than Capability First.
Everyone knew as fact that maintaining FAQs was the least favourite task in the company, the most boring of grunt work, intensely and monotonously manual, the usually left undone task. There was no respite. No thanks. It was a punishment exercise, borderline constructive dismissal to assign someone this task.
‘Maintaining the FAQ’ was also usually not considered important enough to be an ‘official’ task. Instead this was quietly added to someone’s list of burdens.

In the old days, ie, more than a year ago …

… an FAQ would routinely fail cost/benefit thinking as a stand-alone product. Vague benefits, few defenders, but real, long-term effort/costs.

An FAQ as a contributory part of another individual, potential, unvalidated product, assuming that possibility was even recognised, would fail cost/benefit thinking. Not worth creating an FAQ just for that product idea.

The idea that an FAQ might be a contributory capability to multiple products, across different parts of the business, did not mesh well with how teams were structured, incentivised, and distributed across the business. There might be several, limited FAQs, with overlapping domains, each needing their own resources, each sitting in their own walled garden.

Justifying the cost and effort to set up and maintain, revive, or increase investment in an FAQ was always a tough call. Therefore often a “no”.

This per-product, short-termist approach to capabilities would play out over a sequence of such decisions. A series of “no”s, or walled garden “yes”s, to something that is clearly (seen here, anyway) for the greater, common good. Kind of the opposite of what you would hope for from a strategy.

But, good news!

It is now easy and cheap to have and maintain a fabulous FAQ. That cost argument? Gone!

More than that, it is fun, satisfying, enabling, challenging (in a good way). Boring? Anything but!

The shared, Capabilities First view of FAQs is so potent, so fecund, it shows up the Product First approach for the implicit anti-strategy that it is. So there is a sensible way to plan? Who knew?

[Possibly some personal frustrations being worked out in this section? - Ed]

Wrapping up / Summary

The message of this article is multi-pronged:

Human plus AI is likely to be the future - the best of both worlds.
You can set up and maintain an FAQ much more easily, more enjoyably, with human+AI.
The resultant FAQ will be demonstrably better, more relevant, fresher.
The AI-enhanced uses of FAQs are many and marvellous.
Yet more good uses and products based on FAQs will occur to you along the way.
The FAQ will be far more valuable to you, and clearly worth investing more effort in.
Think “Capability First” for true strategy, and for the win.
As promised: synergies abound, a win-win, a positive feedback loop, real potency.

Case Studies

Generating a new FAQ from a document

Let’s consider this very article about FAQs as all the knowledge in your company.