Incorporating Clutter in the SUPR-Q Measurement Framework

Clutter distracts and detracts from a good user experience.

A cluttered website makes it hard to find information. Clutter gets in the way of completing tasks.

At least that’s what we think about clutter. But is there quantifiable evidence that a cluttered design degrades a website’s user experience?

To answer that question, you need a valid measure of the website user experience and a valid measure of clutter. Fortunately, we have both.

The SUPR-Q® (Standardized User Experience Percentile Rank Questionnaire) is the industry standard measure of website quality. And we recently described how we developed a valid measure of website clutter.

In this article, we show how well our measure of clutter fits within the SUPR-Q framework to model clutter’s impact on the website user experience.

The SUPR-Q

The SUPR-Q measures four website UX factors with eight questions: Usability, Trust, Appearance, and Loyalty. Figure 1 shows a regression model demonstrating how the first three factors of the SUPR-Q influence the Loyalty factor.

In Figure 1, the double-headed arrows indicate correlations among the first three factors, single-headed arrows show beta weights (which indicate the strength of the connection with Loyalty), and the number above the upper-right corner of Loyalty is the coefficient of determination (R2) of the regression (indicating that variation in the first three factors accounts for 46% of the variation on Loyalty ratings).

We developed this measurement model primarily to enhance our understanding of the measurement properties of the SUPR-Q but also to provide a framework for modeling additional consequences and antecedents.

The Perceived Website Clutter Questionnaire (PWCQ)

Figure 2 shows the PWCQ, which is a new standardized questionnaire we developed for the perceived clutter of websites.

Figure 3 illustrates the strength of the connection of the two PWCQ subscales (Content Clutter and Design Clutter) to the rating of Overall Clutter with significant beta weights of .32 and .41 accounting for 44% of the variation in Overall Clutter.

The Data for Measuring Clutter’s Impact on the Website UX

To understand how clutter impacts the website user experience, we pulled data from eight retrospective SUPR-Q® consumer surveys conducted between April 2022 and January 2023. Each survey targeted a specific sector, and in total, we collected 2,761 responses to questions about the UX of 57 websites. The sample had roughly equal representation of gender and age (split at 35 years old).

Table 1 shows the participant gender and age for each survey, with sector names linking to articles with more information about each survey (including the websites selected for the sectors). Participants were members of an online consumer panel, all from the US.

Some survey content differed according to the nature of the sector being investigated, but all surveys included the SUPR-Q, basic demographic items, a measure of brand attitude, and the initial version of the perceived website clutter questionnaire.

To support independent exploratory and confirmatory analysis, we split the sample into two datasets by assigning every other respondent to an exploratory (n = 1,381) or confirmatory (n = 1,380) sample by sector and website in the order in which respondents completed the surveys. These sample sizes ensured that we far exceeded the recommended minimum sample sizes for exploratory and confirmatory factor analysis. We used the confirmatory sample to build a structural equation model (SEM) combining the SUPR-Q and PWCQ (almost like plugging LEGOs together using the components in Figures 1 and 2).

Combining SUPR-Q and PWCQ with a SEM

Figure 4 shows the SEM based on the SUPR-Q measurement framework with the addition of constructs from the PWCQ (Overall Clutter, Content Clutter, Design Clutter) as antecedents before and Brand Attitude as an additional consequence following the SUPR-Q components of Trust, Usability, and Appearance.

Like putting together Legos, there are many ways to combine variables in a model. The model should follow a theory of how each variable correlates and influences another. But unlike with Legos, there isn’t a picture on a box to help you see how well your model fits together. Instead, there are quantitative measures of fit to assess the quality of the model. Following the recommendations of Jackson et al. (2009), we focused on Comparative Fit Index (CFI), Root Mean Square Error of Approximation (RMSEA), and Bayesian Information Criterion (BIC). There are guidelines for good levels of model fit for CFI (> 0.90) and RMSEA (< 0.08), but not for BIC (which is used for the relative comparison of models where smaller is better).

The model shows how the clutter questionnaire’s Content Clutter and Design Clutter factors drive Overall Clutter, which, in turn, drives the SUPR-Q attitudinal factors of Appearance and Usability. The SUPR-Q Trust, Appearance, and Usability factors (attitudinal) directly affect the SUPR-Q Loyalty (behavioral intention) factor and influence Loyalty through their effect on Brand Attitude, ultimately accounting for 62% of the variation in Loyalty ratings.

The link between Overall Clutter and Trust was not significant, but its links with Appearance and Usability were (p < 0.0001). As expected, Overall Clutter had more influence (about five times as much) on ratings of Appearance than Usability (−.11 for Usability, −.56 for Appearance). The links between Overall Clutter and SUPR-Q factors were negative because higher ratings of clutter indicate poorer UX while higher ratings of SUPR-Q factors indicate better UX.

The model has excellent fit statistics (CFI: 0.98, RMSEA: 0.08, BIC: 296) and demonstrated convergent validity (significant beta weights for relationship of Content Clutter and Design Clutter with Overall Clutter) and divergent validity (strongest effect of Overall Clutter on Appearance, no significant effect of Overall Clutter on Trust) for the new PWCQ.

Focusing on the clutter constructs in the model, together Content Clutter and Design Clutter accounted for 44% of the variation in Overall Clutter, with a beta weight of 0.32 for Content Clutter and 0.41 for Design Clutter (as shown in Figure 3). The weights were significant (p < 0.0001) but left 56% of the variation in Overall Clutter unaccounted for, leaving open the possibility of additional, as yet undiscovered, clutter factors that would account for some of its remaining variability.

Summary and Discussion

To understand how clutter impacts the website user experience using data from 1,380 data points from 57 websites, we used structural equation modeling to see how the SUPR-Q, perceived clutter, and brand attitude fit together.

There is strong evidence for the distracting and detracting effects of clutter. The SEM showed that perceived clutter drags down ratings of Usability and Appearance (distracts). That, in turn, drags down ratings of Loyalty measured with behavioral intentions to use and recommend (detracts).

Perceived clutter has its strongest influence on ratings of the Appearance construct. Overall Clutter had more influence (about five times as much) on ratings of Appearance than on ratings of Usability (−.11 for Usability, −.56 for Appearance).

The basic SUPR-Q measurement model is a good framework for the assessment of additional constructs. Adding the constructs of perceived website clutter and brand attitude to the basic SUPR-Q model provided proof of the convergent and divergent validity of the new PWCQ.

The fit statistics of the expanded model were good. In particular, CFI was .98 (beating the standard criterion of .90) and RMSEA was .08 (meeting the standard criterion). The model accounted for 62% of the variation in Loyalty ratings.

There appears to be room for the measurement of additional clutter factors. Content Clutter and Design Clutter accounted for 44% of the variation in Overall Clutter ratings, indicating the possibility of defining or discovering additional clutter factors that could account for some of the remaining 56% of the variation in Overall Clutter.

For more details about this research, see the paper we published in the International Journal of Human-Computer Interaction (Lewis & Sauro, 2024. Measuring the Perceived Clutter of Websites).