Mind the Gap

Mind the Gap sought to improve the metacognition and academic attainment of pupils in Year 4. There were two aspects to the intervention. The first involved training teachers in how to embed metacognitive approaches in their work, and how to continue to effectively and strategically involve parents. This training took place over a day and was provided by a consultant. The second component focused on parental engagement and offered families the opportunity to participate in a series of facilitated workshops where children and parents work together to create an animated film. Sessions were coordinated by a practitioner who helped participants to think about how they are learning, create learning goals and reflect on their progress; to be metacognitive about the learning process they were engaged in together. The families were offered 2 hours of workshops per week for 5 weeks (10 hours in total).

The project targeted schools in four areas of England: Birmingham, Devon, London and Manchester. It was delivered by the Campaign for Learning, with assessments carried out by Durham University. Delivery started in September 2012 and finished in October 2013.

The project was evaluated using a randomised controlled trial, which compared the interventions to a ‘business-as-usual’ control group. It is important to note that it was eligibility for the animation course, not participation, that was randomised, so the results must be regarded as estimating the effect of being offered the animation course (alone or in combination with teacher training, as appropriate) rather than participating in it.

Key Conclusions

The following conclusions summarise the project outcome

  1. The headline findings provide no evidence of a statistically significant impact of Mind the Gap on attainment. There is not sufficient evidence to conclude that any observed effect was caused by the programme rather than occurring by chance.

  2. The estimate of the programme’s impact on pupils’ metacognition was positive and statistically significant. This improvement in metacognition may in time lead to an impact on academic attainment.

  3. Participating families and staff felt the intervention enhanced home–school relationships and strengthened the learning relationship between children and parents.

  4. To increase the proportion of families who sign up to the animation course, it is important that schools clearly communicate its aims and promote the potential benefits of participation.

  5. The difficulty in recruiting schools suggests that introducing Mind the Gap more generally may be difficult where schools are not committed to parental engagement or where they have difficulty delivering activities out of hours.

What is the impact?

The impact analysis considered the effect of Mind the Gap as a whole, as well as the separate effects of the teacher training component and eligibility for the parental engagement intervention. InCAS tests of reading, general maths and mental arithmetic were administered six months after the intervention. The intervention had no statistically significant effect on the primary outcome, a combined reading and numeracy score, so we do not have sufficient evidence to conclude that any observed effect was caused by the programme rather than occurring by chance. This was also true for the subgroup of pupils eligible for free school meals.

A number of secondary outcomes were also considered. These included reading, general maths and mental arithmetic scores as well as a measure of how children felt about the relationship with their parents (constructed using their responses to questions taken from the Self-Description Questionnaire). These estimates were consistent in showing no statistically significant effect. However, measures of metacognition (elicited from data collected using Pupil Views Templates) provided evidence that eligibility for the animation course increased pupils’ ‘productive thinking’ and ‘metacognitive skilfulness’.

GroupNo. of pupilsEffect size (95 confidence intervals)Estimated months’ progressEvidence strengthCost of approach
Mind the Gap overall effect (all pupils) 492 -0.141 (-0.697, 0.414)-2
Teacher training alone2780.009 (-1.119, 1.137)
Parental engagement offer alone186 -0.252 (-0.576, 0.071)-3
Mind the Gap overall effect (FSM pupils)123-0.265 (-1.332, 0.801) -4

How secure is the finding?

Overall the evaluation findings are judged to be of low security, largely due to the high degree of attrition.

Mind the gap was evaluated using a randomised controlled trial. Participating schools were randomised to a ‘business-as-usual’ control group or an intervention group, which received the teacher training element. Within the intervention group, Year 4 classes were then randomised into a further two groups: one that were eligible to participate in the animation course and a group that were not. This design allowed three effects to be estimated:

  • the overall effect of Mind the Gap
  • the effect of teacher training alone
  • the effect of eligibility for the animation course.

It was eligibility for participation in the animation course that was randomised. Participation in the course was very low: 72% of pupils eligible for the animation course did not attend any sessions. The results must therefore be regarded as estimating the effect of eligibility for the animation course (alone or in combination with teacher training, as appropriate) rather than participation in it.

In practice, implementation difficulties raised concerns about the security of the results. The main problem was the high level of dropout and the fact that this was concentrated among control schools. There was no means of including such schools in the analysis since they provided no outcome data. A comparison of baseline characteristics suggests that there may be some degree of imbalance across experimental arms in the sample used for analysis, but these comparisons themselves are made difficult by the high incidence of missing baseline data. Furthermore, among schools that did not drop out, there were missing outcome data, compounding the problem caused by dropout.

Another limitation is that, in practice, the treatment status of classes within schools did not agree in all cases with the randomised status. Such non-compliance is a common feature of randomised controlled trials and generally one can still adhere to the intent to treat (ITT) principle. However, in several cases the randomised status became meaningless in practice, as the teachers who were randomised (in most cases, classes were identified by their teacher) had left the school. The final analysis used de facto treatment status. This is a deviation from ITT and further reduces the extent to which the impact estimates for the components of Mind the Gap can be viewed as experimental.

Existing evidence suggests that metacognition and parental engagement interventions have a high average impact on attainment. However, these interventions can take many forms and the intervention considered in this project has no directly comparable precedent.

How much does it cost?

For schools that already have the required IT hardware, delivery of the animation course for up to 15 families costs in the region of £1,950. This translates into a per child cost of roughly £130. The cost of teacher training was typically an additional £195 per delegate.