Macadamian Blog

The Power of Voice Interaction in Connected Products

Jennifer Fraser

The longer the fridge door is open, the more the animal speaks. I can only assume that my penguin is encouraging me to shut the fridge door, but since I don’t speak Japanese, this is only a guess on my part.

Voice interaction in connected products

Conversations with my Japanese Fridge Penguin

A good friend gave my son a fridge penguin for Christmas. Yes. A fridge penguin. From what I could gather from the packaging, this fridge penguin is one of a series of animals created to encourage people to conserve energy by not leaving their fridge door open. There is a light sensor built into the animal, so when you open your fridge door and the light in the fridge comes on, the animal speaks to you. The longer the fridge door is open, the more the animal speaks. I can only assume that my penguin is encouraging me to shut the fridge door, but since I don’t speak Japanese, this is only a guess on my part.

Voice Interaction: Trying to get it Right

While my fridge penguin is in no way “connected” to anything, she still got me thinking about Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) in today’s connected products. In addition to everyone talking about “IoT” these days, and posts about designing for the Amazon Echo taking over the Internet, so many organizations are attempting to develop connected products with voice interaction.

Voice interaction is not new and as a mechanism for interfacing with hardware, it is still really challenging in terms of design and development. On the ASR front, there is still a lot of training involved to understand how to speak to the system in a way that it will understand. With NLU, developers still struggle to find a way for technology to interpret what we mean, not exactly what we say. Dialogue with a computer is still a challenge because computers have difficulty recognizing that humans use more than just our voice to communicate (inflection, emphasis, pause, eye contact, etc.). We’ve all experienced the torment of yelling at our GPS system.

If this is the case, then why are so many organizations still clamoring to develop connected products with voice interaction? It really comes down to the fact that we, as humans, have a natural desire to communicate via voice and if a product can get the voice interaction right, it can have a significant impact on behavior change in the users and subsequent impact on product adoption.

The Human Desire for Conversation

As I explained, the penguin has a simple light sensor, so she speaks when the fridge light turns on. I know and understand this, yet, I find myself talking back to the penguin as if I can engage her in conversation. But, I know that I cannot. I know that she is just continuing to speak to me because I still have the door open while I am inevitably digging around trying to find something in the fridge, but I can’t help myself. Even though I have no idea what she is even saying to me, I can’t help but talk back, apologizing for having the door open and explaining why the door is still open.

Recently, I attended the O’Reilly Design Conference and saw Abi Jones give a talk on “How we talk and how machines listen.” She explained that, unlike human-to-human conversation in which we combine semantic information, syntax, and context in our understanding of speech, human-computer conversation takes place in an alternate reality where people and machines make entirely different sets of assumptions about their respective capabilities and intentions. Abi’s talk helped me to understand that my anthropomorphizing of the penguin had me instinctively trying to follow the cooperative principle for communication. But, as I have explained, the penguin is unaware of her conversational context, which leads to frustration, and brings me to the next point.

Context is Key

User Experience designers have used the “context is key” mantra for decades, but this same mantra becomes even more important when designing for embedded sensors and connected devices. The times that I am tempted to throw my penguin into a snow bank are when I return home from grocery shopping. If I could redesign my fridge penguin, I would include a way to let my penguin know that the fridge door will be open for a while as the groceries are being put away. But, for now, my penguin only has a light sensor and is unaware that the context has changed. So, her ongoing encouragement to shut the fridge door isn’t helpful. In fact, it’s down-right annoying. However, upon reflection, I do realize that I have changed how I put the groceries away because of her.

The Power to Change Behavior

While I understand the importance of feedback loops as a tool for behavior change, I was not expecting that my behavior was going to change as a result of having a talking penguin in my fridge. Even though my fridge penguin does not respond to voice input, I still talk back to a penguin whenever I open the fridge door, and I have found that having the penguin has actually changed how I put away the groceries. I now pre-sort the groceries, piling together all the items that need to go into the fridge as I take them out of my grocery bags. This way, instead of having multiple short conversations with the penguin, I have one long conversation as I put away everything into the fridge at the same time.

So, what has the little fridge penguin highlighted? She was able to demonstrate the importance of recognizing how people interact with each other and that understanding context continues to be key when designing for voice interaction. She reinforces how powerful feedback loops are, including voice interaction and audio feedback, in driving behavior change. If my fridge penguin could impact my behavior without even having an NLU system, imagine what a modern connected product can achieve when properly designed.

Author Overview

Jennifer Fraser

Jennifer is Macadamian’s Director of Design and she brings more than fifteen years of experience working as an interaction designer. Jennifer has experience being thrown in at any phase of a project from discovery, analysis, and definition, to development and delivery. She works closely with clients to understand not only their needs, but also the needs of their customers. Jennifer has a holistic understanding of how to integrate design and development in an agile process, built on years of experience and much head banging (unfortunately not to music). She is a founding member of the Interaction Design Association’s Ottawa chapter and sits on the Design Advisory Board of Algonquin College. Jennifer holds both a Bachelor’s Degree and Master’s in Architecture from Carleton University, which may, or may not, relate to her passion for designing and building cocktails.
  • Diego Ventura

    Thank you Jennifer very well put. I agree context is key. We create Virtual Assistants (VA) and we love to pass as much information about the end user to the VA before the conversation even starts. The tricky part is where to get that information. This is where integration with other systems comes in. Cheers, D.