The Power of Voice Interaction in Connected Products

Jennifer Fraser | March 31, 2016 | 5 Min Read

The longer the fridge door is open, the more the animal speaks. I can only assume that my penguin is encouraging me to shut the fridge door, but since I don’t speak Japanese, this is only a guess on my part.

Conversations with my Japanese Fridge Penguin

A good friend gave my son a fridge penguin for Christmas. Yes. A fridge penguin. From what I could gather from the packaging, this fridge penguin is one of a series of animals created to encourage people to conserve energy by not leaving their fridge door open. There is a light sensor built into the animal, so when you open your fridge door and the light in the fridge comes on, the animal speaks to you. The longer the fridge door is open, the more the animal speaks. I can only assume that my penguin is encouraging me to shut the fridge door, but since I don’t speak Japanese, this is only a guess on my part.

Voice Interaction: Trying to get it Right

While my fridge penguin is in no way “connected” to anything, she still got me thinking about Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) in today’s connected products. In addition to everyone talking about “IoT” these days, and posts about designing for the Amazon Echo taking over the Internet, so many organizations are attempting to develop connected products with voice interaction.

Voice interaction is not new and as a mechanism for interfacing with hardware, it is still really challenging in terms of design and development. On the ASR front, there is still a lot of training involved to understand how to speak to the system in a way that it will understand. With NLU, developers still struggle to find a way for technology to interpret what we mean, not exactly what we say. Dialogue with a computer is still a challenge because computers have difficulty recognizing that humans use more than just our voice to communicate (inflection, emphasis, pause, eye contact, etc.). We’ve all experienced the torment of yelling at our GPS system.

If this is the case, then why are so many organizations still clamoring to develop connected products with voice interaction? It really comes down to the fact that we, as humans, have a natural desire to communicate via voice and if a product can get the voice interaction right, it can have a significant impact on behavior change in the users and the subsequent impact on product adoption.

The Human Desire for Conversation

As I explained, the penguin has a simple light sensor, so she speaks when the fridge light turns on. I know and understand this, yet, I find myself talking back to the penguin as if I can engage her in conversation. But, I know that I cannot. I know that she is just continuing to speak to me because I still have the door open while I am inevitably digging around trying to find something in the fridge, but I can’t help myself. Even though I have no idea what she is even saying to me, I can’t help but talk back, apologizing for having the door open and explaining why the door is still open.

Recently, I attended the O’Reilly Design Conference and saw Abi Jones give a talk on “How we talk and how machines listen.” She explained that, unlike human-to-human conversation in which we combine semantic information, syntax, and context in our understanding of speech, human-computer conversation takes place in an alternate reality where people and machines make entirely different sets of assumptions about their respective capabilities and intentions. Abi’s talk helped me to understand that my anthropomorphizing of the penguin had me instinctively trying to follow the cooperative principle for communication. But, as I have explained, the penguin is unaware of her conversational context, which leads to frustration, and brings me to the next point.

Context is Key

User Experience designers have used the “context is key” mantra for decades, but this same mantra becomes even more important when designing for embedded sensors and connected devices. The times that I am tempted to throw my penguin into a snow bank are when I return home from grocery shopping. If I could redesign my fridge penguin, I would include a way to let my penguin know that the fridge door will be open for a while as the groceries are being put away. But, for now, my penguin only has a light sensor and is unaware that the context has changed. So, her ongoing encouragement to shut the fridge door isn’t helpful. In fact, it’s down-right annoying. However, upon reflection, I do realize that I have changed how I put the groceries away because of her.

The Power to Change Behavior

While I understand the importance of feedback loops as a tool for behavior change, I was not expecting that my behavior was going to change as a result of having a talking penguin in my fridge. Even though my fridge penguin does not respond to voice input, I still talk back to a penguin whenever I open the fridge door, and I have found that having the penguin has actually changed how I put away the groceries. I now pre-sort the groceries, piling together all the items that need to go into the fridge as I take them out of my grocery bags. This way, instead of having multiple short conversations with the penguin, I have one long conversation as I put away everything into the fridge at the same time.

So, what has the little fridge penguin highlighted? She was able to demonstrate the importance of recognizing how people interact with each other and that understanding context continues to be key when designing for voice interaction. She reinforces how powerful feedback loops are, including voice interaction and audio feedback, in driving behavior change. If my fridge penguin could impact my behavior without even having an NLU system, imagine what a modern connected product can achieve when properly designed.

Get Email Updates

Get updates and be the first to know when we publish new blog posts, whitepapers, guides, webinars and more!

Suggested Stories

Increasing Patient Engagement Using Behavior Design

Increasing patient engagement is easier said than done. In this video course, you'll learn how thoroughly understanding the behavior of health consumers can allow stakeholders to increase patient engagement.

Read More

Applications of Voice Assistants in Healthcare

Discover how organizations across the continuum of care can leverage the growing consumer demand for voice-enabled devices to achieve an extensive list of objectives from increased patient engagement to improved outcomes and lowered care costs.

Read More

Voice UI Design Best Practices

Voice assistants are poised to transform customer engagement as well as business models. Discover why voice is the next digital frontier – and what you should know about voice-first solutions.

Read More