Measuring the immeasurable more and more accurately

The announcement by the Office of Qualifications and Examinations Regulation (Ofqual), the UK schools examination regulator, of the new grade structure for GCSEs is explained on their website by their Chief Regulator Glenys Stacey as follows:

For many people, the move away from traditional grades, A, B,C and so on, may be hard to understand. But it is important. The new qualifications will be significantly different and we need to signal this clearly. It will be fairer to all students that users of the qualification will be able to see immediately whether they did the new or a previous version of the GCSE. The new scale will also allow better discrimination between the higher performing students.

This is a big claim, which is not supported by any evidence I have seen. As Dylan Wiliam pointed out as long ago as 2001, the available data suggest that a student receives the grade that their achievement would merit only around 65% of the time. This is very close to the proportion of the time a random variable (which I think is how an examination mark needs to be treated) with a Normal Distribution falls within one standard deviation either side of its expected value. For mathematics, grade boundaries in 2014 were about 15% apart (80% A*, 65% A, 50% B, etc).

Therefore the narrowing of the grade boundaries the new system ushers in, now helpfully illustrated by Ofqual, will merely introduce more randomness to the grading process amongst higher performing students.

Ofqual

If the distribution of marks really is normal, a replacement of a 15% grade width by one closer to 10% would be expected to reduce that 65% accuracy to closer to 50%, ie you will be as likely to get the wrong grade as the right grade. This does not look like progress to me. Ofqual are, however, undaunted:

We realise introducing the new GCSEs alongside other changes will be challenging for schools, teachers and students. But the prize – qualifications that are better to teach, better to study, better assessed and more respected – will be worth it.

I remain to be convinced.