Troubling TIMSS trends

This post reassesses

Progress against the government’s national performance targets and

The comparative performance of England’s high attaining pupils

following publication of the TIMSS 2015 international comparisons study.

About TIMSS

The results of the 2015 Trends in Maths and Science Study (TIMSS) were published at the end of November 2016.

TIMSS is a quadrennial exercise, first undertaken in 1995 and most recently in 2011.

Four assessments are conducted: maths and science at age 9/10 (Year 5) and maths and science at age 13/14 (Year 9).

Fifty-seven countries participated in TIMSS 2015, though numbers undertaking each assessment varied (49 in Y5 maths; 47 in Y5 science and 39 in each of the Y9 assessments).

The National Report makes much of the 20-year trend since 1995, but there were no Y5 assessments in 1999 and in 1995 they drew pupils from both Y4 and Y5. There is greater continuity in the Y9 assessments.

The National Report describes the TIMSS and PISA studies as ‘complementary’:

‘TIMSS…assessments are more focused on pupils’ knowledge and understanding of curriculum content than PISA, which assesses pupils’ ability to apply their science, maths and reading skills to everyday situations.’ (p25).

But this does not mean that TIMSS results are entirely dependent on curriculum coverage:

‘For example, Singapore was the highest-achieving country for science in year 9, but it had taught only seven of the 23 TIMSS topics by the time these pupils took their TIMSS assessments’. (p28)

TIMSS outcomes in each assessment are reported as:

National mean scores, the overall mean score being set at 500 and

Percentages of participants achieving four attainment benchmarks (Advanced – 625+; High – 550+; Intermediate – 475+; and Low – 400+)

The summary statements describing achievement against the Advanced benchmark are:

Y5 maths – ‘Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning.’

Y9 maths – ‘Students can apply and reason in a variety of problem situations, solve linear equations, and make generalisations.’

Y5 science – ‘Students communicate understanding of life, physical, and Earth sciences and demonstrate some knowledge of the process of scientific enquiry.’

Y9 science – ‘Students communicate understanding of complex concepts related to biology, chemistry, physics and Earth science in practical, abstract, and experimental contexts.’

Results in the same assessment are comparable over time but results in different assessments are not comparable.

This analysis draws on the IEA reports of TIMSS 2015 outcomes and especially the TIMSS National Report for England, prepared by the UCL Institute of Education.

Government targets.

An earlier post – TIMSS PISA PIRLS: Morgan’s targets scrutinised (May 2016) – described the evolution of the government’s 2020 performance targets, as articulated in the Conservative election manifesto and subsequently by former Secretary of State Morgan.

It exposes some confusion over the framing of these targets, relying ultimately on Morgan’s final iteration, as conveyed to the Education Select Committee in April 2016:

’We want to be top in Europe for reading, writing, and maths.’ (Q86 p5)

but extending this to include science, which features in the earlier manifesto version.

The earlier post examined the chances of the government securing best in Europe by 2020, against the TIMSS, PIRLS (a parallel reading assessment next published in 2017) and PISA measures respectively.

It estimated the mean scores required in 2015 to achieve the necessary trajectories, but also the improvements needed to be on target to achieve best in Europe against both high and low achievement benchmarks.

This post revisits progress towards the 2020 targets, to reflect the actual TIMSS 2015 results. It also reviews:

The trend in England’s performance against the TIMSS Advanced benchmark, relative to other benchmarks.

How England’s performance against the TIMSS Advanced benchmark compares with other jurisdictions, especially the highest-performing.

The characteristics of the population achieving the TIMSS Advanced benchmark in England.

Changes in mean scores

The National Report’s focus on 20-year trends since 1995 allows it to factor in substantial improvements on three of the four assessments achieved between 1995 and 2003. Only Y9 maths performance was relatively stable over this period.

But, as noted above, there are reasons not to trust these early results on the Y5 assessments, so I have relied instead on trends over the four assessment cycles since 2003, which gives a markedly different profile.

Chart 1, below, shows the trend in England’s mean scores over this period. The dotted lines indicate what further improvement is necessary in the next cycle to achieve a mean score equivalent to the best by a European participant in TIMSS 2015.

In other words these are the improvements necessary to achieve the government’s target, assuming that there is no further improvement in 2019 by any competitor country.

This might be more realistic than it appears, since the mean score returned by the best-performing European country has actually fallen slightly in three of the four assessments between 2011 and 2015. Only in Y5 maths was there a substantial improvement of eight points, achieved by Northern Ireland.

The same is not true of the highest-performing TIMSS participants located in Asia. The highest recorded mean scores on each assessment have all increased since 2011, three of them by at least seven points.

Chart 1: Trend in TIMSS mean scores, 2003-2015 and improvement required to be best in Europe by 2020 (assuming other countries’ scores are unchanged)

The chart shows that England would have to improve at a much faster rate between 2015 and the next round in 2019 than it has between 2011 and 2015 – and this applies to all four assessments:

In Y5 maths it improved by four points between 2011 and 2015 – and would need to add a further 24 points by 2019, six times the 2011-2015 improvement rate.

In Y9 maths it improved by 11 points – and needs to add a further 20 points by 2019, twice the 2011-2015 improvement rate.

In Y5 science it improved by seven points – and would have to add a further 31 points by 2019, some four-and-a-half times the 2011-15 improvement rate.

In Y9 science it improved by four points – and needs to add a further 14 points by 2019, approximately three-and-a-half times the 2011-2015 improvement rate.

Only the Y9 maths trajectory is remotely feasible – and even then unlikely.

Incidentally, the National Report confirms that none of the improvements in England’s mean scores between 2011 and 2015 is statistically significant.

Unless PISA 2015 and PIRLS 2016 results are startlingly different this record of performance makes it almost certain that the government’s targets, barely mentioned since Morgan’s departure, will be quietly buried.

Comparing England’s performance against TIMSS performance benchmarks in 2011 and 2015

The national report celebrates progress against the Low international benchmark since 2011, but is rather less prepared to contrast this with limited progress against the Advanced benchmark:

In Y5 maths the proportion achieving the Low benchmark increased by three percentage points (statistically significant) while the percentage achieving the Advanced benchmark fell by one percentage point – a four point variance. The Advanced benchmark was the only one registering lower performance than in 2011.

In Y9 maths the proportion achieving the Low benchmark increased by five percentage points (statistically significant) compared with a two percentage point improvement at the Advanced benchmark – a three point variance. Achievement improved on all four benchmarks.

In Y5 science there was a four percentage point improvement at the Low benchmark (statistically significant) and a one percentage point fall at the Advanced benchmark – a five point variance. Achievement improved against all except the Advanced benchmark.

In Y9 science there was a two percentage point increase at the Low benchmark and no change at the Advanced benchmark – a two point variance. Achievement improved against all benchmarks except the Advanced benchmark.

England registered limited improvement on the Advanced benchmark in only one of the four assessments (Y9 maths) and limited decline on two (Y5 maths and Y5 science).

Trends in England’s performance against the Advanced benchmark, 2003-2015

The chart below shows the trend since 2003 in England’s performance against the Advanced benchmark.

The profile for Y9 maths is broadly positive, despite stalling between 2007 and 2011. The profile for Y5 maths was positive until this cycle.

The profiles for both science assessments are disappointing, particularly in Y5 science, where there is a consistent downward trend – and the five percentage point decline is statistically significant.

Chart 2: Percentage achieving each TIMSS Advanced benchmark, England, 2003-2015

England is losing ground on other European countries

Comparing 2015 with 2011, only in Y9 maths did England gain any ground on the best in Europe (one percentage point) and it remains five percentage points behind Kazakhstan.

It lost ground in:

Y5 maths – now four percentage points further behind, needing to catch up 10 percentage points on Northern Ireland.

Y5 science – now one percentage point further behind, needing to catch up 10 percentage points on Russia.

Y9 science – now three percentage points further behind, needing to catch up three percentage points on Slovenia.

Back in 2003 England was matching the best in Europe on three of the assessments, and we were holding that position in Y9 science even in 2011, yet now we are behind on all four assessments. Chart 3, below, shows how gaps between England and the best in Europe have changed over this period.

Chart 3: Percentage point gaps between England and the best in Europe, achievement of the TIMSS Advanced benchmark, 2003-2015

The National Report highlights as one of three significant performance issues the fact that:

‘Far higher proportions achieve the Advanced and High benchmarks in both subjects in the highest performing countries’.

It compares England with Singapore and other high-performing jurisdictions:

In Y5 maths 50% of Singaporeans reached the Advanced benchmark compared with 17% of English pupils (three times as many). Seven countries achieved significantly better than England including Russia and Northern Ireland.

In Y9 Maths 54% of Singaporeans achieved the benchmark compared with 10% of English pupils (five times as many). England was outperformed by the five East Asian jurisdictions, but also by Kazakhstan, Russia, the USA, Israel and Hungary.

In Y5 science 37% of Singaporeans achieved the benchmark, set against 10% of English pupils (almost four times as many). At least twelve countries outperformed England, including Russia, Kazakhstan, Finland, Poland, Sweden and Hungary. (The IEA report adds Slovenia to this list.)

In Y9 science 42% of Singaporeans achieved the benchmark, compared with 14% of English pupils (three times as many). Five countries outperformed England (the IEA report lists six) including Kazakhstan and Slovenia

Characteristics of English pupils achieving the Advanced benchmark

The National Report includes useful data about the characteristics of the English pupil population achieving the Advanced benchmark, summarised in the tables reproduced below.

These give the percentage of pupils with each characteristic achieving the benchmark, as well as the standard error, described in the footnotes as:

‘a statistical term that measures the accuracy with which a sample represents its parent population. The smaller the standard error, the more accurate a statistic is. The reverse is also true.’

The first table relates to both maths assessments; the second to both science assessments.

The principal points of interest are:

Only Y5 maths reveals a substantial gender difference, in favour of boys.

The ratio of non-FSM to FSM pupils is around 3:1 for all the assessments except Y9 maths, where it is 4:1. The National Report notes that England has a larger socio-economic achievement gap on mean scores than most other higher-performing countries (according to a related TIMSS measure -number of books in the home). The National Report identifies the socio-economic gap as a second significant performance issue.

On all four assessments the success rate of Chinese pupils is comfortably higher than those of all the other main ethnic groups. This is particularly pronounced in Y5 maths, where half of all Chinese pupils achieve the benchmark. (The standard errors are relatively high but the finding stands nevertheless.) Chinese pupils match the highest-performing jurisdiction in Y5 maths and almost do so in Y9 science, but they fall far short on the two remaining assessments. White British pupils out-perform Asian pupils on science and are level in Y9 maths, but in Y5 maths this is reversed. Few Black pupils achieved the benchmark.

The gaps between EAL and non-EAL learners were relatively small, and smaller in maths than in science.

Conclusion

On this evidence the government will not achieve its 2020 performance targets. Unless PISA 2015 and PIRLS 2016 results are dramatically better it is destined to fall some way short.

The limited improvement in mean scores since the last assessment cycle raises questions about the impact of the wholesale reforms introduced by the previous coalition government. This is particularly true of the Y5 assessments since the entire educational experience of those pupils took place during this period.

The government argues that 2014 curriculum reforms will deliver the necessary step change:

‘The new more demanding primary maths curriculum began to be taught in schools from September 2014, and we expect future TIMSS surveys to reflect further progress.’

But, given the indirect relationship between TIMSS scores and curriculum coverage, this must be treated with some scepticism.

The questions are more pressing when it comes to achievement against the Advanced benchmark since:

There was zero improvement on three of the four assessments and only a two percentage point improvement on the fourth (Y9 maths).

England is now three percentage points behind the best in Europe in Y9 science, five points behind in Y9 maths and 10 points behind in Y5 maths and science.

England has lost significant ground since 2003 when it was best or equal best in Europe on three of the four assessments.

England is out-performed by three other European countries (four if Israel is included) on the Y9 maths assessment and by six other European countries (seven if Slovenia is included) on the Y5 science assessment.

There is scant evidence here that the most able have benefited from Coalition reforms. What improvement there has been is predominantly at the other end of the attainment distribution, hinting that the ‘long tail’ has been receiving the lion’s share of attention.

England’s problematic socio-economic achievement gap is borne out by the substantial excellence gaps between the proportions of FSM and non-FSM pupils achieving the Advanced benchmark.

This is particularly pronounced in Y9 maths, which might otherwise be viewed as the assessment on which England’s recent record is strongest (although we are out-performed by too many other countries).

We have no trend data – there is nothing comparable in the equivalent report on TIMSS 2011 – and what we have is unaccountably based on the old-style FSM measure rather than ever-6 FSM.

It would be useful to know whether pupil premium is beginning to close these excellence gaps. This hole in the evidence base inclines one to suspect otherwise.

The next post in this series will examine whether or not the PISA 2015 results tell a similar story.

December 2016