Flaws in application of language scores

SERIOUS flaws in the interpretation and use of English proficiency test scores at an unnamed Australian university have been revealed by a study.

Now University of Melbourne graduate school of education lecturer Kieran O'Loughlin is calling for a change in selection policy and procedures across the country on the basis of his recent work on how the International English Language Testing System was used.

"Test scores are used in isolation from other considerations; IELTS itself recommends the scores be looked at in the context of the student's language learning background: how long they have been learning or whether they have studied a course in English before," Dr O'Loughlin said.

"In the US this is commonly done and in some areas of the UK. [As it is] too much weight is placed on the scores."

The study, funded by IELTS, took place during a year at a metropolitan university and examined how the test was used to select international students and plan for their future language learning. It also investigated what administrative and academic staff thought about the test and selection process.

It was conducted in the faculty with most international students. The university required undergraduates to have an IELTS score of 6.5 and postgraduates to have between 6.5 and 8.

"The selection staff have minimal understanding of what the test scores mean" and the system "doesn't promote any understanding of what the scores mean", Dr O'Loughlin said.

For example, in the case of administrative staff, responses to eight questions about the test ranged from 34 per cent correct to 64 per cent, a result Dr O'Loughlin called troubling.

Only 54 per cent of administrative staff and 44 per cent of academic staff believed selection officers had a good understanding of IELTS scores.

Dr O'Loughlin was also concerned about the scores' reliability. "They will always be fallible because of what's happening on the day of the test, such as nerves or fatigue. When we are talking about the difference between 6 and 6.5, that has life-changing consequences for students.

"Also there are education consequences: the extent to which students' test scores might inform subsequent support given to them if they only get 6.5. The university I looked at has since made some inroads in that regard: a 6.5 score triggers educational support."

Dr O'Loughlin's paper is published in Language Assessment Quarterly.