Understanding UX Metrics, Part 2

Dear Designer,

This edition of Beyond Aesthetics is part two of a series on UX metrics. Read part one here.

Q: What are UX metrics in practice?

Let's review:

• UX metrics are the numbers we use to measure and improve the user experience.
• UX metrics deal exclusively with quantitative data.
• We measure the UX using a mix of behavioral and attitudinal approaches.

Here is the domain of UX metrics again:

Let's start today by looking at some common UX metric examples 🏁

Behavioral UX Metrics, also called "performance metrics," are based on actual design usage (top-right of our matrix above). Here are some common things to measure:

• Task success
• Time on task
• Errors during tasks
• Advanced metrics like Eyetracking & A.I.-assisted emotion detection

As you can see, most of these are task-based, so you either measure a task during a research study or set up an ongoing measurement in your product.

Attitudinal UX Metrics, also called "self-reported metrics," are based on what the user shares about their experiences (bottom-right of our matrix). These are combined with behavioral task metrics to understand the user's perception. Here are common self-reported metrics:

• Post-task ratings like ease-of-use
• Overall UX ratings like SUS
• Self-reported preference
• Open-ended questions

How to do it:

A common scenario would be to simulate a task for a new product feature to get behavioral data. You might have your user complete a task with two different design flows. You could measure the task success, the time it takes, and the error rate for both versions. Before the test, you could ask for preferences or attitudes related to the experience to capture the pre-test sentiment. After the test, you could capture post-test sentiment by having the user rank the experience on "ease-of-use" and "usability" with a 5-point scale. After analyzing this mix of behavioral and attitudinal UX metrics, you should be able to identify the design that best fits your user.

That's a very common use case for UX metrics to determine the usability of a feature. What if you want to check your whole product?

You might start with a System Usability Scale (SUS) or a Net Promoter Score (NPS). There are a lot of these kinds of things, and they all have their own pros and cons. Start with the scale that your team trusts the most. That will get you started, but eventually, you want to create a group of UX metrics custom to your product.

That's the ultimate goal of UX metrics: to have a tailored set of metrics your team trusts and uses to improve the UX. Google did this with their H.E.A.R.T. framework, and you can, too.

Before you fantasize about a fancy UX analytics dashboard command center for your team, you should learn more about metrics. Many metrics that come out of the box in analytics tools aren't that helpful.

Here are 3 more dualities ☯︎☯︎☯︎ to learn before you set up a UX metric for your product.

Vanity ↔ Actionable Metrics ☯︎

Vanity metrics are numbers that make you feel good, but actionable metrics help you take a course of action.

Vanity metrics count up forever uselessly. “Total Users” is an example of a vanity metric. While it might be important to the business, it doesn’t help us learn.

A better metric might be “numbers of users acquired during the last two weeks.” This is already more actionable because it allows us to isolate the effects of our work, making the metric much more helpful.

Actionable metrics should be ratios. Speed (distance over time) is an example of a ratio. Ratios allow you to integrate essential factors easily.

Not sure what to measure? Pick a vital user action, put it in the numerator, and put a time-based number in the denominator…the resulting metric will be far more helpful than that important user action.

7 Vanity Metrics to Watch Out For

These metrics are everywhere, but they’re not helpful. Many analytics tools measure these things out of the box, but that doesn’t mean they’re actionable.

1. The number of page views: Won’t give insight into who or what happened. Count people instead.
2. The number of visits: Could be 1 person visiting 100 times or 10 people for 10 times. Go fish.
3. The number of unique visitors: Better than above but doesn’t tell you much beyond # of eyeballs.
4. Time on site: isn’t nearly as crucial as user behavior on the pages.
5. The number of followers/friends/likes: doesn’t tell you much. Measure engagement in a set time period instead.
6. Emails collected: Same as above: open rates or click-through rates (CTR) would be better.
7. The number of downloads: it’s good to have app downloads, but User Activation, Engagement, and Retention are more helpful metrics.

Let's say you work for a competitor of Eventbrite that allows users to create events. A bad UX metric would be "# of user-created events." A good UX metric might be "user-created events per week." A great UX metric might be "% of users that create 3+ daily events, per week."

Leading ↔ Lagging Metrics ☯︎

Leading Metrics (also known as leading indicators) are early indicators of user behaviors that will ultimately follow, known as Lagging Metrics.

For Example:

Customer complaints about the UI might be examples of Leading Metrics. Later, these UI problems might lead to cancellations or churn, an excellent example of a Lagging Metric. If you don’t act on the UI problems, you may be unable to stop the churn. If you focus your efforts on the Lagging Metric like Churn, you might be acting on the UI issues too late.

[from Leading vs. Lagging Measures in UX by MeasuringU]

Understanding the relationship between Leading and Lagging Metrics will help you focus on the right metrics at the right time.

To better understand the relationship between leading and lagging metrics, you need to understand the final duality ☯︎ of UX metrics.

Correlated ↔ Causal Metrics ☯︎

Correlated metrics help you predict what will happen through the relationship between variables. This relationship can be causal, but not necessarily.

For Example:

Ice cream and sunglasses are correlated, but the relationship isn’t causal. When ice cream sales go up, so do sunglasses sales...but not always. Ice cream and sunglasses have a correlated relationship.

Ice cream and sunglasses sales DO have a causal relationship with the temperature. The temperature is the true cause here. As the temperature increases, sales of both items go up in a direct cause-and-effect connection.

Correlated metrics have a terrible reputation thanks to bloggers and news reports that use them to create questionable, misleading graphs. Here is a comical example from the website Spurious Correlations:

Nicolas Cage (probably) isn't drowning people, but a graph like this can be very convincing. Be critical of all graphs and look for evidence of a causal relationship before you base product decisions on it.

Correlated Metrics aren't bad, though. Correlation can signal that you’re on the right track to Causation. Lots of causal metrics start out as correlated metrics.

So how do you know if something is causal? Experiments are a vital way to isolate variables from correlations and determine if there is causality (learn how to do that in our course on product experiments).

Causal metrics are powerful because you can directly manipulate the cause-and-effect relationship once you discover them.

Before you can say that something "caused" the numbers to increase, you'll need to prove it in a statistically significant experiment. I'd start small with a lo-fi concept test and work towards a live A/B test with a confidence level of at least 90%.

After a few experiments, you may discover that an experiment's leading metrics are causal to lagging business metrics. Causal leading and lagging metrics are very desirable outcomes in UX.

For example, if you can establish a causal relationship between the leading metric of task success and the lagging metric of revenue, you can show the return on investment for UX work (read a case study about a team that connected UX metrics with business metrics here).

If you're the one with expertise in UX metrics, you can be the one to show business leaders how UX meets their goals.

Any UX designer that can do that will always have a job.

Well, that's it for today! 🏁

You just learned Part 2 of UX metrics. 👏👏👏👏👏🏆👏👏👏👏👏

If you learned one thing today, I hope you learned to avoid pools if this guy is nearby:

That's it for this series. Unless...

Do you want a Part 3 on UX metrics? Yes or No ←this links to a survey

I'll use the survey results to write next week's advice letter. Until then!

-Jeff Humble, Designer & Co-Founder @ The Fountain Institute

Check out my masterclass, How to Lead with UX Metrics. It's completely FREE.

This is your Linkedin edition of Beyond Aesthetics, a free newsletter for UX and Product Designers from the Fountain Institute.

For more design education, follow us on Linkedin, Instagram, or YouTube