Nest Hub Max getting camera-powered Look and Talk and Quick Phrases to skip ‘Hey Google’

At I/O 2022, Google is giving a surprising amount of attention to the Assistant Smart Display that it announced three years ago on the very same stage. “Look and Talk” will let you activate Assistant by just gazing at your Nest Hub Max, while Quick Phrases is also coming to give you another way to skip “Hey Google.”

Look and Talk for Nest Hub Max

Humorously codenamed “Blue Steel” after Zoolander, Google today announced “Look and Talk” for the Nest Hub Max.

From up to five feet away, the Smart Display can recognize that you’re looking directly at it and start listening for a command without you having to say “Hey Google.” Google equates this hotword-less interaction to making eye contact with a person in real-life before talking.

Behind-the-scenes, six machine learning models process over 100 signals from the camera and microphone, including: proximity, head orientation, gaze direction, lip movement, context awareness and intent classification. As such, Google can tell “whether you’re actually making eye contact with your device rather than just giving it a passing glance”.

All this processing occurs locally on-device and no video is sent to the cloud. For Look and Talk to work, both Voice Match and Face Match have to be manually enabled. Real Tone, like on the Pixel 6 camera, is leveraged to make sure the feature works “across a range of skin tones.”

Hey Google continues to work, and tapping the screen cancels Look and Talk. This joins how the Nest Hub Max already uses its camera (6.5MP with 127 degree field of view) for quick gestures where you can hold up your palm to the lens to pause/resume media, dismiss timers/alarms, and stop Assistant responses.

Look and Talk is rolling out to the Nest Hub Max for Android users of the Google Home app “this week” in US English. It will be available on iOS in mid-May.

Quick Phrases

Google is bringing Quick Phrases from the Pixel 6 series to the Nest Hub Max this summer in US English. For some pre-selected commands, you won’t have to say the “Hey Google” hotword beforehand. This is meant to make conversations more fluid and also works on-device. Example commands include:

• Set a timer or alarm
• Ask for the time
• Turn the lights on or off
• Cancel a timer or alarm
• Dim or brighten the lights
• Ask for the weather

It requires that Voice Match be enabled, while users have to select which phrases/commands they want the Nest Hub Max to listen for.

Natural Assistant conversations

Lastly, Google previewed today how conversations with Assistant will become more natural. Assistant will essentially ignore “umm,” interruptions, natural pauses, and other self-corrections rather than just going with what you initially said.

To make this happen, we’re building new, more powerful speech and language models that can understand the nuances of human speech — like when someone is pausing, but not finished speaking. And we’re getting closer to the fluidity of real-time conversation with the Tensor chip, which is custom-engineered to handle on-device machine learning tasks super fast.

Google expects to launch these natural conversation speech models for Assistant in early 2023.