Macadamian Blog

Analytics for your Alexa Skills

Christian Nadeau & Timon LeDain

Without solid analytics, it will be almost impossible to understand what is and isn’t working with Alexa Skills. We went on the hunt this summer for an analytics solution that could help us better understand how users were interacting with our new skill and where they were having issues.

Smartly ai Alexa Skills Analytics Dashboard

As with the web and mobile space before it, the success of Alexa skills will hinge on the ability of developers to measure usage, user acquisition and user retention. Without solid analytics, it will be almost impossible to understand what is and isn’t working with your Skill. Voice specific usage analytics will provide a huge amount of value to voice interaction designers looking to understand how the skill is being used and any places users are having trouble successfully completing their tasks.

The space is still very young with a variety of solutions out there vying for the top spot as the predominant analytics tool for voice and AI assistants.

See Also: The Designing & Developing Alexa Skills Collection.

The Problem

Last year, we published a skill called Scryb which allows a user to send a text message to a specified phone number from their Amazon Echo. The skill essentially transcribes what the user says into a message that is then sent to the pre-defined mobile phone number via the Twilio SMS service. After launching Scryb, we quickly realized that other than the usage statistics from Twilio, we had no way of analyzing our skill’s performance. However, the Twilio data highlighted that we had a Skill discovery issue. The first days while the Skill was on the front page, usage was high. Then as we rolled off the first page, usage dropped markedly.

twilio usage statistics

But how is user retention doing? Which intents are used the most? What do users say before sending an SMS message?

We had a fairly stable 10 to 30 messages per day usage rate, but we were unable to tell if those were all from the same users, or if we had a steady stream of new users trying out the skill each day.

So this past summer we went on the hunt for an analytics solution that could help us answer these questions. While searching, we uncovered Smartly.ai (formerly Alexa Designer) and Voice Insights from VoiceLabs. Ultimately, we decided to use Smartly.ai to deliver the Alexa skill usage insights we were looking for.

Overview of Smartly.ai

Using Smartly.ai, we are able to gain invaluable user insights for our Skill.

The smartly.ai dashboard quickly gives us insights into usage statistics. Over the last 30 days, we are able to see the breakdown of new vs. returning users. 27% of users were returning users with the remaining 73% being new users. Not quite the skill discovery issue we thought we had, but more of a user retention issue.

Alexa Skill Analytics User Sessions

In addition to user statistics, it exposes a lot of additional data about the usual intent flow, and all requests that were part of a user’s session. Using the intent flow data we are able to see how users are interacting with our skill. This will give us plenty of data to understand where users are running into problems, as well as inform the decisions we make as we try and improve the user experience.

While the tool is still more primitive than the best mobile app analytics tools, it already provides better insights than just looking at the number of requests, and it was very easy to integrate.

Dashboard experience

Pros

  • Log data is presented immediately, which is great for troubleshooting and confirming data is received.

Cons

  • It wasn’t clear that I was logged in with a demo account at first, and had to redo the Skill creation after logging in
  • The Dashboard is slow to load; often displaying spinners for over 5 seconds
  • Graphs don’t show the usage information in real time

Setup experience

When we tested it, we needed to deploy the skill before completing the setup process, which was painful to work around when you have a multi-stage deployment environment for dev, stage and production, and a few Skills. We ended up creating a different account for each environment in order to avoid grouping the dev, stage, and production analytics in one account.

Connector code

If you build your Skill directly in Smartly.ai, the analytics functionality will work right out of the box. However, if you are like us and created it outside of Smartly.ai, you will need to log in with your Amazon account in order to get your hands on the tracking code to attach to your Lambda function. We chose not to use AWS Lambda for our Skill, so we had to change the sample code to integrate with the Dashboard.

We had a couple issues with the provided code to push the logs:

  • It didn’t pass our code style validation rules for things like camel case names or use of brackets. This also uncovered a line that was there twice
  • The parameters forced the callers to extract value from the response, while it is easier to let the function extract what it needs
  • The slots URL parameter was not URI encoded
  • Had the skillID and skillSecretKey hard coded which makes it hard to handle multiple skills and deployment environment scenarios

Our cleaned up integration code looks like this:

function pushLogs(skillID, skillSecretKey, eventData, response, callback) {

function getReprompt(response) {

 return response.response.reprompt && response.response.reprompt.outputSpeech.text || ‘’;

 }

var speechOutput = response.response.outputSpeech.text || response.response.outputSpeech.ssml;

 var repromptText = getReprompt(response);

 var shouldEndSession = response.response.shouldEndSession;

var req;

 var slots = ‘{}’;

 var intentName;

 var https = require(‘https’);

 var intentRequest = eventData.request;

 var session = eventData.session;

 var intent = intentRequest.intent;

if (intentRequest.type === ‘LaunchRequest’) {

 intentName = intentRequest.type;

 }

 else {

 var slotName = intent.slots;

 var jsonSlots = {};

if (slotName !== undefined) {

for (var key in slotName) {

 if (slotName.hasOwnProperty(key)) {

 var slotValue;

 if (trackerSettings.secretMode) {

 slotValue = ‘XXXX’;

 }

 else {

 slotValue = (intent.slots)[key].value;

 }

if (slotValue && key) {

 jsonSlots[key] = slotValue.replace(/#/g,’’);

 }

 }

 }

 }

slots = JSON.stringify(jsonSlots);

 intentName = intent.name;

 }

var tags = encodeURI(JSON.stringify(trackerSettings));

 var output = encodeURI(speechOutput);

 var lambdaResponse = encodeURI(‘{“outputSpeech”:{“text”:”’ + speechOutput +

 ‘”},”card”:{“type”:”Simple”,”title”:”’ + skillID + ‘”,”content”:”’ +

 speechOutput + ‘”},”reprompt”:{“outputSpeech”:{“text”:”’ +

 repromptText + ‘”}},”shouldEndSession”:”’ + shouldEndSession + ‘”}’);

 var lambdaRequest = encodeURI(‘{“session”:’ + JSON.stringify(session) + ‘,”request”:’ +

 JSON.stringify(intentRequest) + ‘,”version”:”1.0"}’);

 var options = {

 host: ‘www.alexa-designer.com’,

 port: 443,

 path: ‘/api/dashboard/?platform=alexa&user_id=’ + session.user.userId + ‘&session_id=’ +

 session.sessionId + ‘&skill_id=’ + skillID + ‘&input=’ + intentName + ‘&output=’ +

 output + ‘&lambdaResponse=’ + lambdaResponse + ‘&lambdaRequest=’ + lambdaRequest +

 ‘&slots=’ + encodeURI(slots) + ‘&tags=’ + tags,

 method: ‘POST’,

 headers: {

 ‘key’ : skillSecretKey

 }

 };

req = https.request(options, function(res) {

 callback();

 });

 req.end();

}

Overall, more effort is needed to understand how we can increase the usage of our Skills and retention of our users, but with a voice analytic tool like this, we at least have some data to start from.

Insights delivered to your inbox

Subscribe to get the latest insights in IoT, Alexa Skills development and connected products.

Tags:

Author Overview

Christian Nadeau

Christian is a veteran software developer at Macadamian with specialties in .NET ( WPF, Silverlight, Window Phone 8 ), java (J2EE, JBoss), C++ (Qt, BB10). He holds a bachelor's degree in computer engineering from the University of Sherbrooke.
Timon LeDain

Timon is the Director of Macadamian's growing business in the Internet of Things (IoT). Responsible for IoT strategy, partnerships, and platform product management, his areas of focus include healthcare, cleantech, and consumer products. He was previously VP Operations for SDTC, Canada’s largest cleantech fund where he had an opportunity to review hundreds of business plans and guide new management teams on their product commercialization strategies. He was also VP Engineering for March Healthcare, a wholly owned subsidiary of March Networks, where he developed a telehealth platform that was later sold to Intel. Timon is active in his community, serving on the Board of Directors of Big Brothers Big Sisters of Ottawa. In his spare time he enjoys running, biking, skiing, and traveling the globe with his family.