Google Assistant using federated learning on Android to improve ‘Hey Google’ accuracy

Last year, Google introduced the ability to customize how sensitive your Smart Displays and speakers are to the hotword. Assistant is now using federated learning on Android to prevent “Hey Google” misactivations and misses.

For a handful of users, opening Google Assistant settings today reveals a new “Help improve Assistant” option to “save audio so speech technologies can learn over time.” This new capability also appears in the Google app’s General preferences.

Your audio recordings stay on your device while a privacy-preserving technology combines information from you and many other participants to help Assistant learn over time and develop better smart features. 

Off by default, Google wants to refine its hotword detection model in two scenarios:

• On rare occasions, misactivate, or trigger when you didn’t say “Hey Google.”
• In other cases, miss, or not trigger, when you did say “Hey Google.”

The company is using federated learning so that raw data — you saying something close to “Hey Google” — is processed locally. Instead of audio being sent to the cloud, your device only sends a “summary of the model changes to [a] Google server.” This will help “adjust Google Assistant’s triggering logic.”

Near activations happen when the “Hey Google” model detects audio or something you say that almost activates Google Assistant.

Google servers don’t save your voice recordings from near activations, but your device does if federated learning is on. Your device may not indicate anything when it stores these recordings and may store up to 20 recordings per day…

Voice recordings are encrypted, with processing occurring when your phone is idle, charging, and connected to Wi-Fi. It will be deleted after it’s no longer needed or 63 days.

If you disable the setting, local recordings are deleted, though your data is still kept in the cloud as part of the Web & App Activity setting, which can be separately removed or set to auto-delete. 

Besides for Assistant, Google already uses federated learning on Gboard and the Health Studies app. This feature is not widely rolled out today and only appeared on one device/account (Pixel 5 on Android 11).