Language Testing: Part One

This whole text derives from summarising sections of Glenn Fulcher’s (2010) book Practical Language Testing. I was lucky enough to get to know Glenn when he and I both taught at the University of Leicester; he was a highly-esteemed professor, rightly acknowledged as a leading world authority in the field of language testing and assessment, and I was an “associate tutor”, considered by most to be an irksome nuisance. Our difference in status didn’t interfere with our friendship, which I value enormously, and I urge everybody doing an MA TESOL to get familiar with his work.

In the next three posts on this newsletter, I'll look at aspects of language testing and assessment, hopefully giving you some ideas for an assignment.  

The purpose of language testing is to provide information which helps the user to make decisions about possible courses of action. The decisions are diverse, and need to be made very specific for each intended use of a test; to put it another way, the questions of who uses test results and for what purposes are fundamental.

Internally mandated tests

 Decisions are either internal or external to an educational institution. Internal decisions concern the needs of the teachers and learners working within their particular context, for example, placing learners into classes, determining what’s been achieved and diagnosing difficulties that individual learners may have. The tests have three distinguishing characteristics. Firstly, they are predominantly formative, designed to play a role in the teaching and learning process, rather than to certify ultimate achievement. Secondly, they are low-stakes: they don’t have serious consequences. The information from the tests is used to make decisions about immediate learning goals, targets for the next term, and who should be in which class. Mistakes can be easily corrected through dialogue and negotiation. Thirdly, the tests are usually created or selected by the teachers themselves, and the learners may also be given a say in how they prefer to be assessed. 

External Tests

If the test is used to help with external decisions, the decision to test is taken by people who often know little about the local learning ecology, and the motivations for the external tests often appear vague and complex; indeed, policy makers often do not clearly articulate the purpose of the required testing. Most external tests try to measure the proficiency of learners without reference to the context in which they are learning; they are “summative”: they measure proficiency at the end of a period of study, to see if learners have reached a particular standard. Summative tests give scores which carry generalisable meaning; that is, the score can be interpreted to mean something beyond the context in which the learner is tested. Score users of external tests want to know whether the test takers can communicate with people outside their immediate environment, in unfamiliar places, engaging in tasks that have not been directly modelled in the test itself. The greater the claim for generalisability, the more ‘global’ the intention to interpret score meaning. For example, an academic writing task may contain only one or two questions, but the scores are treated as being indicative of ability to write in a wide range of genres, across a number of disciplines.

Generalisability is therefore an important consideration in external tests, and when they are used to certify an ability to perform at a specified level, or to compare and contrast the performance of schools, educational districts, or even countries, they are referred to as “high-stakes tests”. Fulcher (2010) explains: “Failure for individual learners may result in the termination of their studies. Or they may not be able to access certain occupations. For schools, a ‘failure’ may result in a Ministry of Education introducing ‘special measures’, including removal of staff, or direct management from the central authority. At the national level, perceived failure in comparison with other countries could result in the wholesale reform of educational systems as politicians try to avoid the implied impending economic catastrophe”.

Two examples of high-stakes tests are the National College Entrance Test in China (the Gaokao) and South Korea’s college admission test, known as the Suneung. Both tests are of multiple subjects, including maths and English, and both determine the particular college or university each student will attend, which in turn has considerable effects on their longer-term prospects. So both tests are extremely high-stakes and very competitive. During the two days of testing in China, building sites are closed, aircraft flight paths are changed to avoid low-flying aircraft disturbing students, and test centres are provided with their own police guard to reduce traffic noise and maintain security over test papers. Similar measures are taken in South Korea during the Suneung.

Fulcher (2010) draws attention to the ‘rituality’ associated with high-stakes tests, where the event is accompanied by established practices which endow the test with special meaning. The rituals themselves are drawn from the values embedded in the educational and social system, in this case, meritocracy and equality of opportunity. Arriving at a pre-specified place at the same time as others, sitting in a designated seat a regulation distance from other seats, and answering the same questions as other learners in the same time period, are all part of the ritual.

This testing practice is supposedly designed to enable meritocracy, by imposing the same conditions upon all test takers. Cohen and Wollack (2006: 358) define standardised tests as follows: “Tests are standardized when the directions, conditions of administration, and scoring are clearly defined and fixed for all examinees, administrations, and forms”. Thus, any difference between the score of two individuals should directly reflect their ability upon what is being tested; if two individuals have an equal ability on what is being tested, they should get the same score. If one person gets a higher score because they received more time to take the test, or sat so close to a more able student that they could copy, the principles of meritocracy and equality of opportunity would be compromised.   

 Fulcher (2010) points out that the practice of testing and assessment can never be separated from social and political values. If we consider the university tests, there are a limited number of places in universities and there must be some method of judging which applicants to accept.  If the criteria that we use reflect our views about how society is (or should) be organised, what would it say about us if we decided to offer the best places to the children of government officials? Or to those who can pay the highest fees? Thus, testing can be seen as the mechanism by which our social and political values are realised and implemented. If we believe that the purpose of a test like the Gaokao is to provide equality of opportunity, we see meritocratic practices embedded within the testing process. And if the consequences of testing are those that we intend, and our intentions are good, all is well. Unfortunately, whenever tests are used in society, even for well-meaning purposes, there are always unintended consequences.

Unintended Consequences

 The most obvious unintended consequence of high-stakes tests is that students stop learning the language and start to study the test. Mansell (2007) considered the negative washback effects of the United Kingdom’s foreign language General Certificate of Secondary Education examinations. These include: 

•• Memorising unanalysed fragments of text that can be assembled to create a variety of 100-word essays on simple topics.

•• Memorising scripted fragments of speech in relation to common oral interview type questions, and extended chunks for presentation-type tasks.

•• Teaching written responses to questions, followed by oral memorisation drills, for all common topics such as ‘family and friends’, ‘holidays’ or ‘shopping’. 

Associated with this kind of teaching is the publication of test preparation materials on an industrial scale, and the growth of private schools that specialise in test preparation. These ‘cram schools’ claim that they can raise test scores through specialised tuition in short time periods, primarily by practising test-type questions over and over again, and learning test-taking strategies. Parental and peer pressure may make students spend considerable periods of out-of-school time in test preparation classes, the value of which are questionable, as studies of the huge number of cram schools in South Korea show.

 Another unintended consequence of high-stakes testing is the possibility of deteriorating health. Longer hours of study without periods of rest and relaxation, or even time to pursue hobbies or extra-curricular activities, can lead to tiredness and even exhaustion.  Given the pressure to succeed, stress levels can be high, and becoming run-down can add significantly to fears of failure. At its worst, some students become clinically depressed and suicide rates increase. This is not an isolated problem. Mental health and stress-related illnesses have been reported in many countries with high-stakes standards-based tests for high school students.

 The final example of unintended consequences concerns ‘test migration’. Universities in China allocate numbers of places in advance to the various provinces of the country, for which the students in those provinces are competing. In rural provinces students have to get higher scores than their urban counterparts to get into top universities. This has led to the phenomenon of ‘examinee migration’, where families move to provinces where they perceive their children have a better chance of success. This example of ‘unfairness’ is a universal phenomenon, and “fairness” is a concept that is frequently conjured up to both defend and criticise tests.

Fulcher (2010) gives the example of standards-based testing systems that are now operated in many countries around the world. One of the uses of test scores in these systems is to create school league tables. The rhetoric associated with the justification of such tables emphasises ‘openness’ and ‘transparency’ in the accountability of schools and teachers, and the ‘freedom of choice’ that parents have to send their children to a successful school. However, some schools in league tables will necessarily appear at the bottom of the table, and often those at the bottom are situated in areas where families are from lower socioeconomic groups. The ‘catchment area’ of the school is such that the children are likely to be those with fewer life opportunities and experiences on purely financial grounds. There is a resulting pressure upon families to move into the catchment areas of the better schools so that their children are more likely to receive what they perceive to be a better education. The additional demand for houses in these areas pushes up the price of housing, thus reinforcing the lack of mobility of poorer families, and the association between income and education.

Conclusion

To conclude this Introduction, a final quote from Fulcher (2010):

… testing is not just about finding out what learners know and can do. When testing is practised outside the classroom and leaves the control of the teacher, it is part of the technology of how a society makes decisions about access to scarce resources. The decisions to test, how to test and what to test are all dependent upon our philosophy of society and our view of how individuals should be treated. Teachers need to become strong advocates for change and for social justice, rather than bystanders to whom testing ‘happens’.

 In Part 2, I’ll look at the differences between norm- and criterion-referenced tests, standardisation (particularly the CEFR), and the IELTS.

References

 Cohen, A. S. and Wollack, J. A. (2006). Test administration, security, scoring and reporting. In Brennan, R. L. (ed.), Educational Measurement. 4th edition. New York: American Council on Education/Praeger, 355–386.

 Fulcher, G. (2010). Practical Language Testing. Hodder.

 Mansell, W. (2007). Education by Numbers: The Tyranny of Testing. Politico’s Publishing.