Speech Recognition in Simulation - Is it Worth the Expense?

Usability and expense are, most often, the factors that drive the reluctance to use speech recognition for human-in-the-loop simulation training. Usability and expense are linked when deciding on the acquisition of a speech-enabled simulator or adding speech recognition to existing training capabilities. It costs a lot of money to buy automatic speech recognition (ASR). Will it work for my situation?

Is it Worth the Expense?

Will it work is a fundamental question. I can use this question first to address the issue of expense.

If the ASR works as intended, is the acquisition worth the cost?

A reason to use speech recognition is to save on the expense of role players. If I reduce the number of staff, associated benefits, training, and vacations, how much will that save?

In understanding my annual savings, how many years will it take to recoup the ASR investment? Is this a price I am willing to pay?

Of course, there are other reasons for adopting speech in the training program. The additional benefits should also be part of the decision process. For example, added availability of simulators and other training resources. ASR enables self-paced learning for parts of the course. The organization can eliminate the need to find and train pseudo pilots. And, ASR provides consistency of simulator behavior for all students. The calculation of tangible financial metrics is less precise in these circumstances. Still, they will provide added value, and an estimate of that value is worthwhile for return on investment deliberations.

If, after deliberation of the cost you believe the return on investment calculation makes sense, the connected issue, will it work? can be addressed.

Will it Work?

To be blunt, that is dependent on the buyer. It also necessitates trust between buyer and vendor. A disciplined approach to defining the requirements for your ASR component is critical. Unfortunately, defining requirements is complicated, and it is common, if not routine, to find vendor loopholes spread liberally across a requirements document. If you are evaluating vendors primarily on price, they have no other option but to price as cheaply as possible, and they will make maximum use of loopholes to do so.

In 2001, the United States Air Force specified that; the ASR should support phrases in FAA 7110.65 and locally adopted terminology. The system shall be 98% accurate. Quite simple for a component that was so critical to the success of the program. And wholly insufficient. Put aside the meaningless accuracy statement (and for more information see this article Speech Accuracy Doesn't Matter); the terminology requirement added little value or assurance to the buyer.

Writing Requirements

When comparing vendor offerings, it is paramount that the requirements allow you to compare like for like. Locally adopted terminology is not quantifiable. Terminology support is the most significant cost driver. If the vendor wants to win on price, they have no option but to limit the number of supported phrases while reporting compliance with qualifying statements. Given how vague this statement is, it is not unreasonable to believe the buyer does not understand the problem with its inclusion. As I have mentioned numerous times in other articles, this terminology requirement, as written, led to a high additional cost to the buyer. The vendor estimated the inclusion of support for 120 phrases. The system in 2020 has support for well over 100,000 phrases.

System Comparison

When buying an ASR-equipped system, you must be able to compare offerings on a like-for-like basis and be able to empirically test the winning system against the defined requirements and vendor response.

A comparison allows a better assessment of price. Buying the cheapest system can be disastrous if you find the winner has done a better job of exploiting loopholes.

System Testing

Testing ensures that you are getting what you thought you were getting. Testing also simplifies the process of assigning blame (and cost) for any shortfalls in system capability.

Preparation

It is hard to write requirements that protect you as a vendor. The outcome of the weak definition of system requirements is the failure of the project and unusable technology.

Speech Recognition in Simulation - Is it Worth the Expense?

Speech recognition is expensive. The expense can be justified, and if the system works as expected, it can quickly pay for itself. The onus for success initially lies with the buyer. If you plan to buy on price, you may want to consider changing that to value. Using value as the deciding criteria means getting the features you want and need for the best price. You cannot do that if your requirements do not permit the comparison of systems and the methodological testing of what you bought. So what is the answer to the question, will it work? It depends on the answer to these questions, Have you done sufficient work on your documentation? And do you understand how to select and test a system?

If you do not know where to start or if you wish a little more guidance, check out this document Writing Requirements for Speech Recognition.