The latest science on cognitive ability testing

Sometimes, cutting the BS means getting scientific. And that’s what we’re doing this week! 🔬

The way assessment scientists and experts understand cognitive ability tests, and their role in hiring, is changing. This week, we break down the latest science on the effectiveness of cognitive ability testing. We’ll also walk you through some helpful do’s and don’ts for using them. Let’s go!

The latest science 🧪

In 2022, a large and influential meta-analysis summarizing hundreds of existing research papers about the validity of hiring tools was published by researcher and I/O psychologist Paul Sackett and his colleagues.

This study compared some of the most popular hiring tools, such as structured interviews, job knowledge tests, and cognitive abilities tests. 👀

In 2023, Sackett and his team published another large meta-analysis, this time focusing only on the role of cognitive abilities. They summarized more than 130 studies and concluded that the average validity coefficient of cognitive ability tests is lower than they concluded in 2022.

This lowers the rank position of cognitive ability tests down to 7th or 8th place (instead of the 5th place suggested by the 2022 study).

You can see a summary of the key results of the study below. 👇

In short, existing research suggests that structured interviews are the hiring tool with the highest average validity, followed by job knowledge (skills) tests, empirically keyed biodata, and work sample (simulation) tests.

Does this mean cognitive abilities testing is a no-go when hiring? 🤔

Absolutely not. Cognitive ability tests still have a role to play when assessing candidates in the hiring context and are a great tool when used correctly and mindfully.

But what this does show is that they are not the “silver bullet” of every hiring process as was once considered. 

The long and short of it is: Employers should take a nuanced approach to using cognitive ability tests. Wondering where to start? We’ve got you.

Read on for some do’s and don’ts.

Do: Choose your cognitive ability test wisely ✅

Start by thinking about what everyday work will look like for your successful candidate. Unpack the job requirements. Consider the types of tasks they will perform and what it takes to complete them successfully.

Do they have to cross-check information and make sure it is correct? Do they have to solve routine problems with relatively straightforward solutions, or figure out solutions to complex problems that have no precedent? 

There’s a more in-depth discussion on how to understand what your future hire will do in our blog on job analysis, but a lot of it is common sense if you understand the role you’re filling.

➡️ Example: You’re looking to hire a content writer. Would it be better to give them a reading comprehension test, or a mechanical reasoning test?

Don’t: Use cognitive ability tests when a role-specific test is more relevant ❌

Research shows that job knowledge tests like role-specific skills tests or software skills tests are better (on average) than cognitive abilities tests at predicting overall job performance.

So, when thinking about including a cognitive ability test in your assessment battery, you should first consider whether there are other more suitable role-specific tests that could help you identify top candidates for the position.

Note that cognitive ability tests are known to be good at predicting candidates’ learning capabilities, for example, and estimating their growth potential.

This is why they are often considered when hiring for roles that require a lot of on-the-job learning (e.g., fresh graduates hired for a graduate development program) or when recruiting people with little relevant experience.

Don’t: Use cognitive ability tests alone to make hiring decisions ❌

Yes, we say it all the time. But we’re gonna say it again! Multi-measure testing is the most effective way to do skills-based hiring.

And so it follows that you shouldn’t use cognitive ability tests in isolation – rather, they should be used as part of an assessment that covers multiple job-relevant skills.

Fun fact: Employers who use multi-measure testing are also more satisfied with their hires. 🚀

When including more than one test in the assessment, be mindful of their complementarity and avoid skill overlap. Including tests that are too similar to one another will yield little additional information about the candidate.

➡️ Example: Problem-solving and Critical thinking tests measure similar skills. They tap into a similar cognitive ability, and having both adds little additional value to your assessment. Better to just use one – whichever you determine is most relevant.

Do: Place cognitive ability tests early in the hiring process ✅

Cognitive ability tests require deep concentration. The test taker must respond quickly to tough questions under demanding time limits – and so the test has what assessment scientists call a high cognitive load. 

Because of this, it is best to place them at the very beginning of an assessment to ensure candidates approach tests fresh and sharp, and can do their best.

Test order can significantly impact candidate performance on the assessment, as well as the reliability and validity of assessment results (read more about test order here!)

If you’re in the mood for more reading… 🧠

That’s it! If you’re still reading, thanks for bearing with us while we got all scientific. And in case you’re in the mood for even more reading, we’re going to leave a link to our State of Skills-Based Hiring report juuust here! 👇