Automating Alexa Skill Dialog Testing

Hayk Gharagyozyan | October 13, 2017 | 2 Min Read

Alexa Skill testing can be a long and laborious task, having to run through possibly hundreds of dialogs manually. In this post, we will walk you through a concept for testing Alexa skill dialogs without speaking to Alexa and completing tests in a matter of seconds.

While chatting with one of my colleagues about a project she is currently on, she was telling me about an Alexa skill that she is testing and how it supports hundreds of dialogs between the Alexa skill and the user. This got me wondering if there wasn’t a tool that could help automate Alexa skill dialog testing.

After a few searches on the web, I was unable to find anything that could appropriately test skills however, I did find a few technologies that I could use to create a tool myself. Specifically Bespoken and Mocha.

The goal of the tool would be to ease a QAs life and make it possible to test hundreds of Skill invocations in less than a minute without having to speak to Alexa. This solves the recently introduced problem of testing Alexa skill dialogs. With mobile apps, it’s quite easy to test at your desk, but with voice interaction experiences a QA would have to speak to the skill allowed disturbing their colleagues or becoming frustrated when Alexa doesn’t quite catch what was said because of background noise. This solves that without having to sacrifice valuable meeting room space to have a dedicated quiet space to test.

The tool would also help improve the quality of Skills since testing all the dialogs of a complex feature could be achieved within seconds with no human effort required to repeat it after new changes are introduced. You can imagine just how beneficial it would be to use this tool with continuous integration and be informed about a bug immediately.

To test this concept I created a simple, yet practical, skill that will walk a Doctor through the UK’s NHS Summary Flowchart to Aid Diabetes Diagnosis. This would have similar specifications to my colleague’s project and would be complex enough to demonstrate the concept’s viability.

So using bespoken tools with the mocha testing framework I was able to create tests which will test dialogs. Afterwards, I’ve moved them into a separate file with the easy-to-read yaml format to make it possible to modify and define new tests without writing any code.

Here is the testing code.

Here is the yaml file with the test definitions:

And here are the successful test results!

Obviously, this simple testing logic does not cover all complex cases of real-world Alexa skills but it proved that this concept is useful to make Alexa skill testing more efficient. Both Alexa skill and testing logic code can be found in the following repository.

Get Email Updates

Get updates and be the first to know when we publish new blog posts, whitepapers, guides, webinars and more!

Suggested Stories

Guide to Mobile Application Testing Strategy

Application development for mobile devices has increased immensely in the last decade and is expected to continue growing at a profound pace. This white paper will serve as your guide to creating your own test plan, covering a variety of testing strategies and considerations for when they should be used.

Read More

Applications of Voice Assistants in Healthcare

Discover how organizations across the continuum of care can leverage the growing consumer demand for voice-enabled devices to achieve an extensive list of objectives from increased patient engagement to improved outcomes and lowered care costs.

Read More

Voice UI Design Best Practices

Voice assistants are poised to transform customer engagement as well as business models. Discover why voice is the next digital frontier – and what you should know about voice-first solutions.

Read More