Here's a Look at the Data I Collected Since Launching This Learning Project Two Years Ago

Thank you to the over 60,000 people who have Subscribed to This Side Up! 🎉 Want to get updates when a new This Side Up Newsletter post comes out? Select the Subscribe button on this Newsletter to be alerted.

2020 has been a long year. I think we can all agree on that one. Typically at the end of each year I write a reflection post that covers all of the things I've done throughout the year. This year I find myself reflecting more on past years. Reminiscing about travel and the time I got to spend with folks in the learning community. So this year I'm forgoing my typical reflection post and opting to review an older passion project of mine.

Back in 2018 I got to travel to Nashville, TN to facilitate a workshop on xAPI. As a part of the workshop I wanted people to be able to put themselves in the position of someone trying to solve a problem and show an outcome using xAPI. I also wanted participants to be able to interact with an xAPI enabled project because without seeing xAPI in action, it’s hard to picture. So I created a sample learning material to go with the workshop. I have been tracking the interactions on the learning project for two years.

In the workshop participants place themselves in the shoes of a learning manager at a take out food franchise. Participants are provided with a scenario and asked to answer questions to help them create a strategy for the project. Before we jump into the scenario let’s go over a little about xAPI.

What is xAPI or what used to be called Tin Can API?

Why xAPI?

Why should you or shouldn’t you adopt xAPI? When it comes to other specifications (not SCORM but other), it’s a bit of a loaded question. I cannot claim to be technical enough to explain to you the ins and outs of xAPI versus other specifications but here’s what I do know.

xAPI is already being adopted by the industry

xAPI has already been adopted by elearning authoring tool companies. In December 2020, Articulate announced a 2021 feature to be able to add custom xAPI statements into Storyline 360. 

That means for companies who are already using these authoring tools it makes the most sense to go with xAPI versus another specification, especially if the authoring tools are allowing for easy customization of xAPI statements. 

Next, by using xAPI versus other specifications learning professionals will more easily be able to adopt that specification. Learning professionals will become familiar with how to design to collect xAPI data. This means professionals will be able to move from different companies, teams, and organizations without having to learn a new specification. This is mainly important for organizations who do not have the budget to hire technical talent to set up tracking and analyze learning data in another way, let alone provide the time to learning professionals to learn to design for their team’s data collection method.

Lastly, I will say I'm not a huge fan of the question, "xAPI or SCORM?" They are two very different things. The answer depends on your needs and you may use a mix of different solutions for different needs. It's about asking yourself the question, what do you need (now and in the future) and why?

Now that we’ve covered what xAPI is and a few whys, let’s look at a hypothetical scenario where you might use xAPI.

The Presto Pasta workshop scenario

You are a manager in the learning department of a fictional pasta to go franchise company, Presto Pasta.

Presto Pasta has over 100 locations in 3 states across the North East, U.S. and is rapidly growing. One of Presto’s business goals for the year is to increase profit by 5% through their rewards program.

A McKinsey article (Shaukat and Auerbach, 2011) states loyalty programs can generate as much as 20% of a company’s profits.

Presto recently started a rewards program for customers, buy 10 pasta meals and get one free. Since the rewards program started you have been receiving negative reviews on Yelp, social media (Twitter, Instagram, and Facebook), and through your customer service lines (email and chat).

The reviews have been outlining the inconsistency of the rewards program across different stores. Customers are not being told about the rewards card or are not being given a rewards stamp after receiving their meal. You are considering going to an app format for rewards but have not moved over yet. In the meantime, you conduct an analysis to find out why customers are not hearing about or receiving rewards.

After conducting an analysis, you find that over half of your Presto locations have not been mentioning the program or offering a rewards stamp after purchase of a meal. You’ve found that the issue is cashiers are unsure of how to present the offer to customers or enter it into your system, so they did not present it at all. (Some cashiers heard that they only give a stamp after select meals on the menu while others said it was after any meal. Other cashiers had issues entering the free meal into the system, which has caused confusion.)

In response to the issue, you create a Rewards Program Guide to help your employees become more confident about promoting the rewards program. In your pilot program you do not have the Rewards Program Guide assigned to your employees, you communicate the new guide through management and also have it appear on the restaurant iPad as a pop-up at the beginning of each shift. Let's take a look at the guide and come up with some suggestions on how xAPI can be used in the scenario.

See the Rewards Program Guide and view your statements

You can check out the Rewards Program Guide and see your xAPI statements in the xAPI statement viewer (refresh the page once you interact with the guide).

Help solve for the scenario

Based on the Rewards Program Guide and information in the scenario, how will you use xAPI to measure the effectiveness of the learning initiative? Here are some ideas.

1. What information will you collect about your learners (also known as actors)? Hint: It doesn’t just have to be their name and email or interactions.

One attribute you may want to know about your learners is the franchise store location. Knowing the store location will allow you to compare if certain stores are performing better after using the guide. 

You may want to look at when learners looked at the guide. Was it at the start of their shift or after the shift began? Can you see a correlation between when customers sign up for the rewards program and when the learners used the guide?

How many times did they refer to the guide? This could show you that the information is not clear, show trends in how many times the information needs to be reviewed to be retained, or other interesting leads.

2. What is tracked in the Rewards Program Guide? 

Currently the Rewards Program Guide tracks page visits, simulation video plays and completions, and FAQ accordion clicks. You’ll be able to see a learner’s journey through the guide.

What do learners interact with and not interact with, why, and how does this impact your goal? Could it be because they have certain information, could the UX of the guide be confusing? Something else? How long does it take them to find an answer they are looking for?

Before and after getting this information you may want to do user tests to see how users would use the guide and are using the guide.

3. What will you track that does not include the Rewards Program Guide? Hint: You can bring data in from other platforms.

Remember, you have data to indicate that there is a problem, you can visualize your xAPI data and data from other platforms together. In some cases you can convert data from other platforms into xAPI data and visualize it all together in your LRS, if not you could visualize all data in a tool like Tableau.

Collect the data you had from Yelp and social Media outlets (Twitter, Instagram, and Facebook), and through your customer service lines (email and chat). Eventually you may want to bring in data related to profits generated from the rewards program.

4. How will you use all of the data together to help analyze the learning solution?

You may want to look at if you can find a correlation between the usage of the guide and an improvement in customer reviews for those Presto locations reporting low reviews in relation to the rewards. Across the board you would want to look at how the usage of the guide impacts reviews. In the best case scenario you’d be able to calculate an increase in profits due to customers being offered the rewards program more consistently across locations.

Overall, use the data you find on the project to help you dive deeper to improve the solution or identify a different route to take.

What data was collected on the mock project over the past two years?

The Rewards Program Guide sends xAPI data to Yet Analytic’s xAPI LRS Sandbox. The free sandbox is perfect for you to pilot using xAPI or test out a personal project. You can learn more about the free sandbox on Yet’s resources page.  

The data collected over the past two years on the project has been from users visiting my portfolio and loading the project. Even though the data isn’t performance related since it’s based on a fake company, it’s still interesting. 

Over two years there have been 730 visitors generating more than 3,000 statements. The page that gets the most hits is the main page with at least 50% more traffic than other pages. This could be because people hit the main page and leave or visit other pages and return to the main page. 

The second most visited page is the simulation page. The video on the simulation page had more interactions than the simulation page itself. The video interaction counts are based on any time the user selects play even if they pause the video. There were 506 users who landed on the simulation page and the video on the page has been interacted with 509 times. Data shows that 19% of users who visit the simulation page complete the video. Recently I wrote a new script to record video interactions. The new script records video progress and does not record interactions based on every time a user selects play/pause.

Another interesting data point is in regards to accordion interactions. The accordion interactions are recorded on open and close. If I were to redo the interaction I would only record an interaction when the user clicked to open the accordion.

The accordion that is interacted with the most is the first accordion, which makes the most sense. However, the fifth accordion has more interactions than the third, fourth, and sixth. The fifth accordion’s question states, “A customer has a receipt but did not get a stamp. What can I do?” We can hypothesize that one person closed and opened this accordion more times than others or there could potentially be more interest in the answer to this question for some reason. This is why it’s so important to dive deeper into the data.

We can create hypotheses as to why there is an outlier but unless we dive deeper we most likely will not know the exact reason why there is an outlier. Always ask, why is this happening? Are there outliers because of a bug, due to user interest/confusion, or something else?

Want to learn more about xAPI by creating your own xAPI project? Join the free xAPI Cohort. The next session starts February 4, 2021.

Add to the conversation in the comments. What's helped you learn xAPI? How would you approach the above scenario differently or add to it?

-Mel