Analytics for your Alexa Skills

Christian Nadeau | January 17, 2017 | 5 Min Read

Without solid analytics, it will be almost impossible to understand what is and isn’t working with Alexa Skills. We went on the hunt this summer for an analytics solution that could help us better understand how users were interacting with our new skill and where they were having issues.

As with the web and mobile space before it, the success of Alexa skills will hinge on the ability of developers to measure usage, user acquisition and user retention. Without solid analytics, it will be almost impossible to understand what is and isn’t working with your Skill. Voice specific usage analytics will provide a huge amount of value to voice interaction designers looking to understand how the skill is being used and any places users are having trouble successfully completing their tasks.

The space is still very young with a variety of solutions out there vying for the top spot as the predominant analytics tool for voice and AI assistants.

The Problem

Last year, we published a skill called Scryb which allows a user to send a text message to a specified phone number from their Amazon Echo. The skill essentially transcribes what the user says into a message that is then sent to the pre-defined mobile phone number via the Twilio SMS service. After launching Scryb, we quickly realized that other than the usage statistics from Twilio, we had no way of analyzing our skill’s performance. However, the Twilio data highlighted that we had a Skill discovery issue. The first days while the Skill was on the front page, usage was high. Then as we rolled off the first page, usage dropped markedly.

But how is user retention doing? Which intents are used the most? What do users say before sending an SMS message?

We had a fairly stable 10 to 30 messages per day usage rate, but we were unable to tell if those were all from the same users, or if we had a steady stream of new users trying out the skill each day.

So this past summer we went on the hunt for an analytics solution that could help us answer these questions. While searching, we uncovered Smartly.ai (formerly Alexa Designer) and Voice Insights from VoiceLabs. Ultimately, we decided to use Smartly.ai to deliver the Alexa skill usage insights we were looking for.

Overview of Smartly.ai

Using Smartly.ai, we are able to gain invaluable user insights for our Skill.

The smartly.ai dashboard quickly gives us insights into usage statistics. Over the last 30 days, we are able to see the breakdown of new vs. returning users. 27% of users were returning users with the remaining 73% being new users. Not quite the skill discovery issue we thought we had, but more of a user retention issue.

In addition to user statistics, it exposes a lot of additional data about the usual intent flow, and all requests that were part of a user’s session. Using the intent flow data we are able to see how users are interacting with our skill. This will give us plenty of data to understand where users are running into problems, as well as inform the decisions we make as we try and improve the user experience.

While the tool is still more primitive than the best mobile app analytics tools, it already provides better insights than just looking at the number of requests, and it was very easy to integrate.

Dashboard experience

Pros

  • Log data is presented immediately, which is great for troubleshooting and confirming data is received.

Cons

  • It wasn’t clear that I was logged in with a demo account at first, and had to redo the Skill creation after logging in
  • The Dashboard is slow to load; often displaying spinners for over 5 seconds
  • Graphs don’t show the usage information in real time

Setup experience

When we tested it, we needed to deploy the skill before completing the setup process, which was painful to work around when you have a multi-stage deployment environment for dev, stage and production, and a few Skills. We ended up creating a different account for each environment in order to avoid grouping the dev, stage, and production analytics in one account.

Connector code

If you build your Skill directly in Smartly.ai, the analytics functionality will work right out of the box. However, if you are like us and created it outside of Smartly.ai, you will need to log in with your Amazon account in order to get your hands on the tracking code to attach to your Lambda function. We chose not to use AWS Lambda for our Skill, so we had to change the sample code to integrate with the Dashboard.

We had a couple issues with the provided code to push the logs:

  • It didn’t pass our code style validation rules for things like camel case names or use of brackets. This also uncovered a line that was there twice
  • The parameters forced the callers to extract value from the response, while it is easier to let the function extract what it needs
  • The slots URL parameter was not URI encoded
  • Had the skillID and skillSecretKey hard coded which makes it hard to handle multiple skills and deployment environment scenarios

Our cleaned up integration code looks like this:

function pushLogs(skillID, skillSecretKey, eventData, response, callback) {

function getReprompt(response) {

 return response.response.reprompt && response.response.reprompt.outputSpeech.text || ‘’;

 }

var speechOutput = response.response.outputSpeech.text || response.response.outputSpeech.ssml;

 var repromptText = getReprompt(response);

 var shouldEndSession = response.response.shouldEndSession;

var req;

 var slots = ‘{}’;

 var intentName;

 var https = require(‘https’);

 var intentRequest = eventData.request;

 var session = eventData.session;

 var intent = intentRequest.intent;

if (intentRequest.type === ‘LaunchRequest’) {

 intentName = intentRequest.type;

 }

 else {

 var slotName = intent.slots;

 var jsonSlots = {};

if (slotName !== undefined) {

for (var key in slotName) {

 if (slotName.hasOwnProperty(key)) {

 var slotValue;

 if (trackerSettings.secretMode) {

 slotValue = ‘XXXX’;

 }

 else {

 slotValue = (intent.slots)[key].value;

 }

if (slotValue && key) {

 jsonSlots[key] = slotValue.replace(/#/g,’’);

 }

 }

 }

 }

slots = JSON.stringify(jsonSlots);

 intentName = intent.name;

 }

var tags = encodeURI(JSON.stringify(trackerSettings));

 var output = encodeURI(speechOutput);

 var lambdaResponse = encodeURI(‘{“outputSpeech”:{“text”:”’ + speechOutput +

 ‘”},”card”:{“type”:”Simple”,”title”:”’ + skillID + ‘”,”content”:”’ +

 speechOutput + ‘”},”reprompt”:{“outputSpeech”:{“text”:”’ +

 repromptText + ‘”}},”shouldEndSession”:”’ + shouldEndSession + ‘”}’);

 var lambdaRequest = encodeURI(‘{“session”:’ + JSON.stringify(session) + ‘,”request”:’ +

 JSON.stringify(intentRequest) + ‘,”version”:”1.0"}’);

 var options = {

 host: ‘www.alexa-designer.com’,

 port: 443,

 path: ‘/api/dashboard/?platform=alexa&user_id=’ + session.user.userId + ‘&session_id=’ +

 session.sessionId + ‘&skill_id=’ + skillID + ‘&input=’ + intentName + ‘&output=’ +

 output + ‘&lambdaResponse=’ + lambdaResponse + ‘&lambdaRequest=’ + lambdaRequest +

 ‘&slots=’ + encodeURI(slots) + ‘&tags=’ + tags,

 method: ‘POST’,

 headers: {

 ‘key’ : skillSecretKey

 }

 };

req = https.request(options, function(res) {

 callback();

 });

 req.end();

}

Overall, more effort is needed to understand how we can increase the usage of our Skills and retention of our users, but with a voice analytic tool like this, we at least have some data to start from.

Get Email Updates

Get updates and be the first to know when we publish new blog posts, whitepapers, guides, webinars and more!

Suggested Stories

Applications of Voice Assistants in Healthcare

Discover how organizations across the continuum of care can leverage the growing consumer demand for voice-enabled devices to achieve an extensive list of objectives from increased patient engagement to improved outcomes and lowered care costs.

Read More

Voice UI Design Best Practices

Voice assistants are poised to transform customer engagement as well as business models. Discover why voice is the next digital frontier – and what you should know about voice-first solutions.

Read More

Building your own IoT Product: Bad radio! (Wi-Fi, Wireless / LTE, BLE, NFC) – Oh, my!

Picking the right radio solution can make or break your product (or seriously cripple customer adoption). You have to build or select components that make sense for your context of use and build solutions around the trade-offs of each radio option.

Read More