header
 in partnership with Providing Outreach  in Computer Science Bringing Bayesian solutions to real-world risk problems

# The Schools League Tables Fallacy

Look at the following table showing the scores achieved by a set of schools  (that we have made anonymous by using numbers rather than names). School 38 achieved a significantly higher score than the next best school, and its score is over 52% higher than the lowest ranked school, number 41.

If I told you that these were the available state schools in your borough and that, instead of your child being awarded a place in school 38, he/she was being sent to school 41. You would be pretty upset, wouldn't you? Now scroll down to below the table.

 School Number Score 38 175 43 164 44 163 25 158 31 158 47 158 11 155 23 155 48 155 40 153 7 151 30 151 6 150 9 149 33 149 19 148 10 147 12 147 32 147 2 146 27 146 42 146 28 145 35 145 49 145 45 144 46 143 1 142 18 142 22 141 26 141 4 140 14 140 29 140 39 139 8 138 5 136 17 136 34 136 3 134 24 133 36 131 37 131 15 130 21 130 16 128 13 120 20 116 41 115

In fact the numbers do not represent schools at all. They are simply the numbers used in the UK National Lottery (1 to 49).  And each ‘score’ is the actual number of times that particular numbered ball had been drawn in the 1172 draws of the UK National lottery that had taken place up to 17 March 2007. So the real question is: Do you believe that 38 is a 'better' number than 41? Or, making the analogy with the schools league table even more accurate, do you believe the number 38 is more likely to be drawn next time than the number 41? (after all your interpretation of the schools league table is that if your child attends the school at the top he/she will get better grades than if he/she attends the school at the bottom).

The fact is that the scores are genuinely random. Although the ‘expected’ number of times any one ball should have been drawn is about 144 you can see that there is a wide variation above and below this number (even though that is still the average score). What many people fail to realise is that this kind of variation is inevitable. To see why look at this explanation. If you rolled a die 60 times you would almost certainly not get each of the 6 numbers coming up 10 times. You might get 16 threes and only 6 fours. That does that not make the number three better than the number four. The more times you roll the die, the closer in relative terms will be the frequencies of each number (specifically, if you roll the die n times the frequency of each number will get closer and closer to n divided by 6 as n gets bigger); but in absolute terms the frequencies will not be exactly the same. There will inevitably be some numbers with a higher count than others. And one number will be at the 'top' of the table while another will be 'bottom'.

I am not suggesting that all schools league tables are purely random like this. But, imagine that you had a set of genuinely equal schools and you ranked them according to a suitable criteria like average A-level results. Then in any given year you would expect to see a wide variation like the table above. And you would be completely wrong to assume that the school at the top was any better than the school at the bottom.

In reality, there are inherent quality factors that help determine where a school will come in a league table. But this does not disguise the fact that much of the variation in the results will be down to nothing more than pure and inevitable chance.

footer
Norman Fenton

Return to Main Page Making Sense of Probability: Fallacies, Myths and Puzzles