When I first became a headteacher 15 years ago people thought I was mad to seek out a challenging headship. I remember one particular deputy almost sneering at me about the school I had chosen to lead. At that time there were very few schools serving disadvantaged communities that were any good – many of them were where poor quality teachers washed up and (this is true, I remember it well!) the dominant culture among headteachers who led these schools was one of excuse-making for poor standards and achievement. I suppose that was why other ambitious deputies thought I was mad and felt able to sneer.
For 10 years though it was exhilirating and exciting because that dominant culture was changed by heads like me. We worked very hard turning around schools and communities, transforming the attitudes towards schools like ours and producing the frst generation of employed young people and higher education students that some areas had seen for decades. We were sometimes unpopular with colleagues as we proved it could be done, but we worked in partnership with our government; many of us became Principals of the first academies – the most challenging communities of all. We had our National College of School Leadership assisting us in establishing networks and helping us learn from each other, many of us became NLEs and were able to work across the system: it felt like there was real system leadership emerging.
I remember the challenges as they emerged around school performance very clearly. First we were ensuring that all children were expected to and could gain qualifications and then we were focusing on the quality of those qualifications. We moved to ensuring that there was a clear focus on English and Maths as they are so vital in opening doors to employability and being able to benefit from further study. Then there was the challenge of ensuring there were no abuses in the system of “mickey mouse” useless qualifications, which needed to be done, as there were undoubted abuses which disadvantaged some students. Each new challenge was embraced by those of us passionate about transforming the life chances of the children raised in our most disadvantaged communities. It is ridiculous for anyone to suggest that those of us who have been doing this are wedded to any form of excuse making or under aspirational culture for our schools and communities, but we are now bitterly disappointed and let down.
When we first heard about the move to “Progress 8” in our accountability we were excited. This was something that we could embrace wholeheartedly; it would be fair. For the progress of every child to be valued and for proper consideraion to be given to progress from starting points was not only a fair and just way of measuring our schools’ performance, but would also enable us to move away from the dubious focus on “cliff edges” and the iniquity of schools forced to pile resources into C/D borderline students to the detriment of others. I wrote about this in November 2013, and I called it my six reasons to smile.
Since then we have seen some things happening which should give pause for thought. We know that there are far more schools who have below average prior attainment that receive an OFSTED inadequate judgment and that where student outcomes are higher it is less likely that the OFSTED judgment will be inadequate. This blog from @kristianstill asks the question, and I think it is a key question: are the evaluative mechanisms for assessing the outcomes of schools that have the highest proportion of higher attaining students sufficiently taxing? Or putting it another way as there are notably fewer schools with students who have lower prior attainment receiving outstanding OFSTED judgments – are we measuring school effectiveness fairly and accurately? Many of us have been increasingly concerned that our judgment-based accountability system (OFSTED) has become increasingly reliant on our data-based accountability system – this in itself begs all kind of questions. But …. the move to Progress 8 will address this, won’t it as the data will be fairer?
Yesterday I was concerned to read this from @dataeducator and asked @drmarkarobinson to do some modelling for me. All my reasons for welcoming Progress 8 are swept away. It is clear from our modelling that all my “reasons for smiling” back in November 2013 have proved to be entirely false. In our early analysis we see clearly that a system which could have provided some accuracy and fairness is now being manipulated to benefit schools with a higher proportion of students with high prior attainment. I understand that the official excuse for this is the changing GCSEs. Really? Given what we know about how students with different levels of prior attainment progress differentially this is smoke and mirors. Students with higher prior attaniment tend to be easier to drive further progress from as they tend to come from families and communities doing a large part of the work for the school. It is ridiculous to deny this. You can see from @DrMarkARobinson’s work Prog 8 models 070315 UPDATE that schools adding valuing around the G-F and F-E will receive signficantly less recognition, whereas those adding value in the C-B, B-A and A-A* will receive significantly more.
Now I am angry – “proper angry”, as we say round here. So here are my five reasons for being angry:
I am angry at the unfairness of “clever” children counting more than the others.
I am angry that the judgement-based accountability system is so firmly tied to a data-based accountability system which is working hard to obfuscate, rather than cast a light on performance.
I am angry about my colleagues who simply cannot take the stress and want to quit. I am angry about how the children and the families in our communities will lose great school leaders and struggle to recruit. I didn’t feel brave back in 2000, but someone embarking on the headships I’ve tackled now would certainly be brave.
I am angry at the loss of the system leaders this may well cause. (I doubt people like me will be labelled “outstanding” again!) The implications for the system are huge. I am angry that headteachers running schools filled with students of high prior attainment will be the ones with the supposed wisdom to help failing schools instad of those of us who know how to do it.
But above all I am angry at an arrogant cadre of people who believe they are born to rule, know better than anyone else, listen only to the few who agree with them and who are causing deep damage to our most vulnerable.
Ros McMullen: Things I've learned and things I'm learning 
Thank you Ros. You say what I feel! I would add, those of us who chose to come to schools that offer low PA challenges are now finding it hard to take our hard earned experience and skills elsewhere. Apparently the school judgement now defines the leader and it doesn’t count a jot how far you have moved the school, reduced exclusion, increased attendance and attainment, if you are still below NAt av. The biggest injustice is still that those with sig- PA are expected to add so much more value than other schools. Where is the parity, the equity in such a system?!
This is a serious problem. Having a leadership judgment determined by a school’s outcomes actively stops the best leaders wanting to work in the toughest schools. It is bonkers and a complete reversal of all the turnarounds we were able to effect in the early part of this century. I took on the 15th “worst” school in the country and got labelled “outstanding” as I was turning it around. Now I may have been inadequate. Bonkers!
About the way Ofsted grades are distributed along the spectrum of prior attainment – I wrote about it https://jtbeducation.wordpress.com/2014/06/29/whats-the-easiest-way-to-a-secondary-ofsted-outstanding/ , but I’m afraid it will only make you more angry. Very clear correlation between Ofsted grades and prior attainment. You can even “predict” your grade fairly accurately using KS2 APS and nature of school (just for fun) – https://www.jtb-education.co.uk/ofstedpredictor.php
Thank you Ros, I share your concern. We are already seeing the impact in terms of recruitment of heads but I fear to about teache recruitment into schools in challenging circumstances. Somebody needs to see the bigger picture and the unintended consequences of continuing to up the data fuelled accountability system.
I really enjoyed this and I totally agree. There is an assumption built into our education system that says teaching in all schools is exactly the same, that it doesn’t matter about the background of the children or their prior attainment, all schools do the same job. Having worked in schools at both ends of the spectrum I know it is not the same job at all, I think your blog highlights this really well. It is time to rethink how schools are judged and measured and bring better equity to the system! Thanks for writing it
This is so frustrating. I can’t profess to fully understand Mark’s spreadsheet but it’s the opposite effect to the one Tim Leunig told us about in 2013; he said he’d be going after the grammar schools that benefitted from the higher inherent rate of progress of higher attainers on entry – that clearly hasn’t found its way into the algorithms. Ludicrous.
Dear Twitter users- please look at and retweet if you agree with the open comment that @dataeducator has made in relation to the changes for 2017 attainment 8 and progress 8 scores (link here: https://twitter.com/DataEducator/status/574714693266309120). I have retweeted this as I think it is very sensible and a fairer approach for moving forward! Tom great to hear your views!
Dear Ros,
This spreadsheet is not a fair reflection of what is happening, and I fear it has mislead you.
Ofqual have argued – and it is their job to make this assessment – that F & G are closer together than (say) A & B. Under the current ABC etc grades, with 1 point per grade, the school who gets a pupil who would typically get a B to an A is therefore not getting a fair reward compared with a school who gets a pupil who typically get a G to an F, because a B to an A is a bigger stretch than a G to an F. We have corrected the accountability point scores to reflect that. Nothing more and nothing less.
Note too that if you have pupils who would typically get an F, and they get a G, the current system penalises you (too) heavily – you lose a mark, even though an F and a G are not that far apart. That is unfair to schools with a lot of pupils who would typically get an F. That excess penalty has now gone, because the school only loses half a mark, not a full mark.
The new system, then, is fairer to schools with lots of pupils who might be expected to get low grades, because it reduces their vulnerability to very small changes in the marks these pupils may get. It is also fairer to good schools who push B students to As, and harder on weak schools that should be getting Bs but whose pupils only get Cs. This is not because these children matter more than others – all children matter equally – it is just that these grades are further apart. Once the new 1-9 grades come in, all grades will be 1 mark apart, because all grades will be equally difficult to get a child +1 or -1.
Finally, “Students with higher prior attaniment tend to be easier to drive further progress from as they tend to come from families and communities doing a large part of the work for the school. It is ridiculous to deny this.” No-one is going to deny this – but because the new Progress 8 system compares these children in school A with similar children in school B it is fair to both schools. What wasn’t fair was having the same “expected levels of progress” for these children as for children with lower prior attainment – for the reasons you set out. That is why we are changing it!
Your reasons for smiling remain just as valid now as when you wrote your piece.
Best wishes
Tim
Dear Trevor,
On Ofsted. For years the government has told Ofsted to look at 5AC and then 5ACEM. In that context it is hardly surprising that Ofsted end up giving better grades to schools with higher performing intakes. If Ofsted are still doing that when Progress 8 becomes the official measure then you will have them bang to rights. But until then, Ofsted, like schools that put their best teachers into the C/D borderline group, are both responding to the incentives that were set by government. Those responses were not (always) helpful or fair to all pupils and all schools, and that is why we have changes the incentives.
Best wishes – Tim
Dear Tom,
On the grammar school issue, we have moved away from the previous KS2-KS4 modelling approach which (inadvertently) gave a small bonus to schools with a lot of L5 pupils. This didn’t make a big difference to many schools, but it did help grammar schools. That has now gone, so the system is fairer in that technical sense. That is in the algorithms – never fear!!
Best wishes – Tim
Dear All,
The bottom lines remain:
1) For the first time ever, if any child gets better grades than expected, the school rises up the league table. That isn’t true now, and I think it is a change for the better.
2) For the first time ever, if any child gets worse grades than expected, the school slips down the league table. That isn’t true now, and I think it is a change for the better.
3) For the first time ever, children are not expected to make the same progress, irrespective of their entry points. That isn’t true now, and I think it is a change for the better.
4) For the first time ever, a school with lots of low prior attainment children has the same chance of being in the top half/quarter/10% etc of the league tables as a grammar school. That isn’t true now, and I think it is a change for the better.
Those are the reasons that made both Ros and I smile, and those reasons remain just as true now as before.
Finally, I am not sure who the “arrogant cadre of people who believe they are born to rule” are. I hope you don’t see me as one of them. Believe me, nothing in my upbringing has ever given me such an idea.
Best wishes to you all, as ever –
Tim
PS Sorry if this string of comments are a bit too forthright. It is late at night, and I thought you would rather have a reply now than tomorrow, or maybe not at all. Hope I judged that correctly.
Hi Tim,
Re: Points 1-3 and probably 4. Value Added allows all these things to happen.
Secondly. How does say a History teacher understand that in 2016 teaching 10 students to a grade E gives 30 points towards attainment 8 but 10 students with the same prior attainment a year later attaining the same E grade equals 20 points. It is the same qualification.
Meanwhile a History teacher somewhere else gets 10 A*s in both years but in 2017 gets 5 more points.
The students are attaining the same outcomes in the same subject in both 2016 and 2017 however the higher ability students are now adding more points to the schools attainment 8 score and the lower ability fewer.
Seems daft that in 2018 the difference between all grades will be one unit, but in 2017 we will have differing units?
data educator: My view (put to me by the head of the highest VA school recently) is that SMTs should encourage teachers to devote 100% of their effort to teaching all children as well as possible. Classroom teachers should not devote any time to thinking about points, accountability, etc. These are things for SMT to think about.
In accountability terms, it would be better if we could move to the 1-9 system immediately, but the extent of change required to do that does not make it feasible. So for a year or two, schools with kids likely to get Fs get slightly more points added or taken away if their pupils get a grade better or worse. I can think of no school for which this is likely to make a meaningful difference to their overall outcomes. From memory only 1.1% of children get F grades. Even if 1 in 10 of your children are pupils expected to get Fs, and even if half of them get one grade more in every subject, and only 10% get a grade less, the overall unfair bonus from us not yet having moved to 1-9 is only +0.04, which is not going to make much difference to the overall picture of the school. I would be really surprised were any school to fall outside +/-0.02 on this, and would expect >90% to be within +/-0.005. There just aren’t that many pupils with KS2 APS that predict an F grade.
Don’t disagree with anything you say, Ros (or anything you say, Tim – you are both looking at different aspects/interpretations of “fairness”) – but two points I would make are (1) it isn’t an issue for English and Maths, which will rightly always have the most focus and are of course 40% of Progress 8, and (2) this is a temporary problem which will go in 2019. I’m not saying that makes it all OK! But it isn’t a “permanent” injustice and I strongly suspect it is an inevitable and if not accidental then at least undesired artefact produced by transitioning from 8 grades to 9.
I attended a significant number of OFQUAL consultation events and they were absolutely explicit that one of their key reasons for wanting to move to a system with n 8 grades was to prevent people trying to align new and old grades in any way, shape or form other than at specific points (eg 4 = C).
They deviated away from this later with new 5 = top but of C / bottom but of B etc.
Tim: 3.7% of children get F grades, and 14.4% fall within the range E to U.
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/398921/SFR2_2015_Subject_Timeseries.xls
Obviously these proportions are distributed differently between schools, and I could not comment easily on the overall impact this will have on these students and these schools, suffice to say that I believe there will be an impact.
It is my belief that the DfE’s first priority should have been to ensure fairness between students and secondly to ensure fairness between the grading systems. I know you say both have been achieved but I am sceptical that in practice this won’t be the case.
What I also take issue with is the lack of transparency and presenting of the rationale for structuring the points in 2017 as they are. For example, schools have no idea why the gap in a legacy GCSE subject between an B and A grade is deemed to be 1.5 points in 2017 compared to 1 point in 2016. It’s is not 50% more difficult to move from a B grade to an A grade in 2017 is it, the qualification is the same?
The guidance document is woefully thin on this point, “the other qualifications will be mapped onto the 1-9 scale”.
If the 2017 points system is the true picture and the gap between a grade F and E is less than a B to A, then why does the 2016 points system also not reflect this situation?
Although all your comments are gratefully received, it seems wrong to be having this conversation in the comments section of a blog. I feel that there should have been open consultation to set the points as they have been. I cannot find evidence from Ofqual that says it is three times easier for a student to move from a grade F to E, or vice versa than it is from B to A. This evidence and the DfEs conclusions should be published in a more transparent manner.
Well it is rather surprising for me to have caused such a storm with a simple blog, and yes it probably is not the right place to get into the detailed debate, but I do want to reply to some of Tim’s observations.
We know that our spreadsheet was an over-simplification but it was drawn to illustrate the impact of progress of children with differing prior attainment. Our concern is that in the coming three years there is a clear incentive not to view the progress of all children as being equally important to the outcome for the school.
Mark and I have looked in detail to the points you raise Tim and it seems to us that a lot of what you offer hinges on the acceptance that it is harder to shift someone from a B to an A than it is from a G to an F. You (and OFQUAL) base this on there being fewer marks separating these grades; however we would dispute that fewer marks between grades means less stretch between them. Indeed the leap from B to A for students with relatively high prior attainment, strong basic skills and supportive middle-class families this “stretch” can be much easier than moving students who have a history of low attainment and huge social problems even a few marks.
Your concept of Progress 8 and the way you outline it still is “fairer”; however, how can it be seen as “fairer” when value added receives less recognition for the children who struggle the most to make progress? My over-riding concern is that this entire methodology for judging school performance is being undermined in its early application. Although we can see that from 2019 things should have reached a stable place, the unfairness and disincentives to work with students of lower prior attainment will already have damaged the credibility of the system. This is a tragedy.
I know that there are certain secondary schools that even successful sponsors refuse to take on as “too high risk” to their reputation – this is going to exacerbate that problem, and once again, the communities that need the best will be left wanting.
The problem seems to have arisen in the first place as a result of the DfE messing up by trying to introduce an excellent progress 8 system alongside the staged introduction of new GCSEs and the move to a 9 point system. I have an opinion about that. It might have been sensible if, having encountered the problem, there was an open discussion on a variety of potential solutions.
The alignment of old grades E, F and G with new grades 1 and 2 is set out in the Ofqual paper available here: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/377771/2014-09-12-board-paper-for-new-gcses-in.pdf
Look at the graphs for English and Maths on p. 7 Since the same kids will get E, F and G one year and 1 and 2 the next year, it makes sense for those grades to get a school the same number of accountability points, yes? The kids have learned the same amount, yes? That is all we have done.
I accept that we have explained this all in a way that has not convinced you, but that really is all we have done, and that still looks like the right decision to me.
(Ofqual consulted on where the grade boundaries should be – there is a discussion of the consultation responses at the back of that document.)
Hope this helps
Tim
This paper from Ofqual really helps to see why the DfE have assigned the points scores that they have to 2017. Many Thanks.
I would still question whether it is fair to assign the 1-9 points scale to the A*-G qualifications and whether this mapping was really necessary to achieve balance between the 1-9 points scale and the 1-8 point scale. This is mainly because the 1-9 subjects will be taken by nearly all students and they largely count in their own little basket. What would the downfall of not mapping have been?
I have modelled some scenarios myself to investigate the impact of using the 2017 points scores:
https://drive.google.com/file/d/0B8cR7OtZSTKlNWhjTnh2RFZOOFU/view?usp=sharing
Whereas the DfE have modelled forward and said, the students getting E, F, G in say History in 2017 – well that would be a 1 and 2 in 2018, so let’s give them a 1, 1.5 and 2.
The reverse thinking would say look these students got a E, F, and G in 2016 that equated to 1, 2 and 3 respectively. It’s the same qualification, to the same grading structure so let’s give them 1, 2 and 3 again. Let’s only change the points when the grades in their subject changes.
Data educator: If (say) in 2017 history is a 1-9 subject and A.N.Other subject is an A-G subject then if we didn’t make the accountability points work across both there would be a big incentive for schools to put people in for either History or ANOther not according to ability or interest, but according to the points that they were likely to get in each subject because they were on different systems. We never want the accountability tail to determine subject choice in this way. As you say, mainly it is ok – but not across the board. Hence we had to have a points system across both, and – on balance – I think our approach is correct.
But in 2017 English and maths are the 1-9 subjects, there is absolutely no incentive not to enter these subjects in favour of A.N.Other subject because they are double weighted in attainment 8 and even have their own separate performance indicator as well. No school in the country is not going to enter English and maths if the student is able to access these qualifications.
FFT have explored the issue here- their conclusions seem sensible. Tim what do you think or is it an ‘Ofqual matter’?
Ok – here goes with an alternative that seems fair. To help illustrate it I will use a single student example but this would then be done for every student in school and averaged (which is what happens anyway).
Steve has an average ks2 point score of 5.2.
‘Same cohort’ data is being used to calculate progress8 for the first few years so, on results day, the DfE look at all students with the same starting point and calculate
A) the average GCSE points score for all the new GCSEs taken by students with that ks2 average (let’s say 7.3 for arguments sake).
B) the average GCSE points score for all the old GCSEs taken by students with that ks2 average (which might be 6.9 equating to a high B).
It is important to note that these figures (7.3 and 6.9) have absolutely no relation to eachother other than 1 point increment equals a 1 grade improvement.
So, as it happens, Steve takes 3 new GCSEs and 5 old style ones. His ‘target’ is therefore 7.3×3 + 6.9 x 5. This equals a total points target of 21.9 + 34.5 = 56.4
Now Steve actually gets the following grades 8, 7, 6 (new) plus B, B, C, C, C (old) equating to 21+ 27 = 48 points.
Ie. Steve has underperformance compared to his equally able peers by 8.4 points. His personal Progress 8 score is -8.4
Crucially if he had taken a different mix of old and new GCSEs then his target would be different to reflect this but still based on exactly comparing his results to that of equally able peers with that mix of old/new GCSEs.
We retain the every grade matters equally core principle of Prog8.
We are genuniely comparing outcomes on a like for like basis irrespective of individual choices and combinations of old/new GCSEs.
Yes attainment8 is a mess but it will be anyway in the transition period and it’s not a floor measure so just hold that back a Bit until it is meaningful.
(Oh I know eng/maths count double but got to the end of my calcs so didn’t want to go back but the model holds even with that change).
Have I missed a flaw here somewhere?
Pingback: Getting from B to A (or G to F) | The Echo Chamber
Pingback: Attainment and Progress 8 – Your Opinion | dataeducator
Reblogged this on The Echo Chamber.