If you’ve ever designed a survey or questionnaire you’ve probably wondered how many points the response options should have. Most questionnaires I’ve examined either use five point scales or seven-point scales. Is one better?
7-point scales are slightly better
The short answer is that 7-point scales are a little better than 5-points—but not by much. The psychometric literature suggests that having more scale points is better but there is a diminishing return after around 11 points (Nunnally 1978). Having seven points tends to be a good balance between having enough points of discrimination without having to maintain too many response options. So what are the consequences of this?
Users need choices
If there aren’t enough response options users will be forced to choose the next best alternative and this introduces measurement error. For example, if users think a 5 is too high and a 4 is too low they are forced to settle on an option which is higher or lower than they wanted (assuming they can’t pick a 4.5). If this happens a lot then there aren’t enough scale points.
System Usability Scale—Too Much Error?
The most popular usability questionnaire is the System Usability Scale (SUS) which uses ten five-point scales. Is there too much response error with SUS?
A recent article tested the SUS response error by counting the number of times users couldn’t decide between two points. The users were allowed to “interpolate” or pick between points such as 3.5. In 2.5% of the 858 responses users provided responses between two points (95% CI between 1.6% to 3.9%).
In contrast, when a 7-point version of the SUS was used there was no interpolating for any of the 840 ratings.
While this seems like compelling evidence to always use 7-point over 5 scales there are two tempering factors.
- While there was error in the five point SUS it is unclear how much of an impact this actually has on the final SUS score since the study used different systems for each scale.
- Errors in statistics have a way of cancelling themselves out. It is likely that many responses that are “forced” into higher numbers will be cancelled out by those forced into lower numbers.
There are many things that can go wrong with questions and responses: users misinterpret the question, users select the wrong box or administrators forget to invert the scales. The data suggests this type of error is small but can be addressed in the following four ways.
1. For Single Item Questionnaires Use 7 or More Points
If you have only a single question you will get a lot more benefit by providing more response options: 7, 9 or 11 versus 5. Joe Dumas and I found a single 7-point scale did well.
2. For Multiple Item Questionnaires it matters Less
As you have more items in your questionnaire it matters less how many scales you have. The SUS has 10 questions meaning it has 50 points of discrimination. In fact the standardized SUMI questionnaire has only 3 point scales but it has 50 questions for a total of 150 points of discrimination.
3. Don’t Change Your Scales!
If you or your company has been using questionnaires with 5 points scales don’t change them. Whatever small benefit you get from having more points will be more than offset by losing a valuable benchmark. This is especially the case for SUS which has publically available benchmarks.
4. Use 7 Points for New Scales
If you are designing a new scale and need to decide on the number of response options having 7 will likely give you a small benefit over five. This benefit will likely only be realized if you have fewer response items (less than 10) and you use very large sample sizes.
Keep it in Perspective
In general the effects of usable or unusable applications tend to outweigh the much smaller effects of scale points, labels, scale directions and poorly written questions. So if you find yourself debating whether there should be 5 or 7 points in your rating scales then you’ve probably already spent too much time worrying about it. Focus more effort on finding good benchmarks to compare your results to and on what you’ll do with those ratings when you get them back.
