There are plenty of virtual assistants that we use through various devices. Siri, Google Assistant, Alexa, and Cortana are the most prominent examples. But what do all of these have in common? You need to talk to them in a fixed syntax to engage with them. This feels unnatural and odd. But, Microsoft is set to revolutionize the way in which we interact with these virtual assistants.

As per a recently published patent application, Microsoft has developed a system that will allow non-verbal engagement with virtual assistants. Imagine that you’re working on your PC, and you wish to listen to some music or open up your browser. As shown in the figure above, you will only need to look at the Cortana icon on your screen, without saying “Hey Cortana”. The system will use gaze detection to determine that you want to engage Cortana. Cortana will wake-up and greet you with phrases like “How may I help you?”. You can then proceed with your desired task.

The system will also consider contextual information from your device. For example, if you are watching a video and you suddenly say “What was that?” referring to something on your screen, the system will analyze your speech through Natural Language Processing (NLP), and given the context of the content playing on your screen, analyze your query to give you relevant results. The system will also be able to know if you are addressing your virtual assistant, or are talking to someone else. For example, while talking over the phone to your friend about a possible outing, you may suddenly ask “What is the weather forecast tomorrow”, and go back to talking with your friend. The system will confirm your question, with your assistant asking you ”Do you want to know the weather for tomorrow?”, and on receiving your confirmation, proceed to tell you the information.

This technology looks like it will make our conversations with virtual assistants seamless and natural. I can’t wait for Microsoft to roll this feature out in a Windows update, and other tech companies to follow suit.

Originally published at medium.com on July 15, 2019.

US20190187787A1