The Literacy Octopus: Communicating and Engaging with Research
The two ‘Literacy Octopus’ trials - named after their multi-armed design - drew on a wide range of evidence-based resources and events designed to support the teaching and learning of literacy at Key Stage 2. They were delivered by four partners with extensive expertise in education research, and included printed and online research summaries, evidence-based practice guides, webinars, face-to-face CPD events, and access to online tools. The trials were funded jointly with the Department for Education and the Mayor’s London Schools Excellence Fund.
A large multi-arm randomised controlled trial, investigating a range of different methods of communicating research to schools and engaging them in research evidence.
Translating evidence-based resources into real change in the classroom is difficult. Simply sending resources to schools is often thought to have no effect, even when those resources are high quality and evidence-based. However, there have been specific examples where putting hard copies of practical guides in the hands of professionals appears to have made a difference to practice, and at minimal cost. So the first trial tested whether sending schools high quality evidence-based resources in a range of different formats could have an impact on pupil outcomes.
The second trial tested whether combining the provision of resources with light-touch support on how to use them would have greater impact. Some schools were simply sent evidence-based resources, while others received the resources along with simple additional support, such as invites to seminars on applying the resources in the classroom. As well as pupil outcomes, this trial also measured teachers’ use of research, to help us understand the impact on teacher behaviour.
Neither trial found evidence of improved literacy attainment at Key Stage 2 and the second trial found no increase in teachers' use of research. These high quality trials tested a wide range of interventions in a large number of schools. The results suggest that, in general, light-touch interventions and resources alone are unlikely to make a difference.
This has important implications for organisations which are using evidence to improve teaching and learning. At EEF we have already started to take a more intensive approach to scale-up, with more structured support for schools and a greater focus on implementation. Our campaigns and Research Schools network are good examples of this. Importantly, we will continue to evaluate the impact of this activity.
The first trial tested the following interventions. The Research Results table below shows the results for each.
- 1A ‘Improving Reading’ guide - Evidence-based printed booklet Improving Reading: A Guide for Teachers. From CEM, part of Durham University.
- 2A Evidence updates & website - Regular printed and electronic materials, including the Better Evidence-based Education magazine and the ‘Best Evidence in Brief’ email, and access to a searchable database, Evidence 4 Impact. From IEE at the University of York.
- 3A Webinar - A link to the archived webinar and materials from a conference on research evidence relating to KS2 literacy. From ResearchEd, in collaboration with NatCen.
- 4A Teaching How2s website - Access to the Teaching How2s website, including evidence-based visual guides on CPD. From Campaign for Learning/Train Visual.
The second trial tested the following interventions. The Research Results table below shows the results for each.
- 1B Evidence updates & website - Regular printed and electronic materials, including the Better Evidence-based Education magazine and the ‘Best Evidence in Brief’ email, and access to a searchable database, Evidence 4 Impact. From IEE at the University of York.
- 2B Evidence updates & website + event - Intervention 1B plus invitation to an evidence-based literacy programmes event.
- 3B Teaching How2s website - Access to Teaching How2s website, including evidence-based visual guides on CPD. From Campaign for Learning/Train Visual
- 4B Teaching How2s website + intro event - Intervention 3B plus invitation to a one-day introduction to using the Teaching How2s website, and updates on using the visual guides.
- 5B ‘Improving Reading’ guide & emails - Hard copy evidence-based KS2 literacy teaching materials, including evidence-based printed booklet Improving Reading: A Guide for Teachers and monthly classroom activity posters. From CEM, part of Durham University.
- 6B ‘Improving Reading’ & emails + CPD session - Intervention 5B plus invitation to one twilight CPD session.
- 7B ‘Improving Reading’ & emails + CPD sessions & tools - Intervention 6B plus invitation to one further twilight CPD session, use of pupil diagnostic tools, and teacher peer observation between sessions.
- 8B Conference - Invitation to free Saturday conference on research evidence relating to Key Stage 2 literacy. From ResearchEd, in collaboration with NatCen.
- 9B Conference + webinars - Intervention 8B plus invitation to two webinars to provide support on applying the research from the conference in schools.
The first trial found no evidence that Literacy Octopus passive dissemination interventions improved pupils’ Key Stage 2 English scores compared with the control group. The five padlock security rating means we have very high confidence in this result.
These findings suggest that simply disseminating research summaries and evidence-based resources to schools is not an effective way for research organisations to support schools to improve pupil outcomes.
It is likely that these materials formed a small part of the information received by schools during this time. It is possible that if schools had support to navigate and prioritise that information, greater impact could be achieved. Alternatively, schools may need more support in transforming such materials into actual change in the classroom.
The evaluation team have analysed the Key Stage 2 results from a further year following the project to explore if there are longer-term effects of the interventions. These results are published in addenda to the main reports.
The second trial found no evidence that any of the interventions improved pupils’ Key Stage 2 English scores. The five padlock security rating means we have high confidence in this result.
There was no evidence of impact on any of the six teacher Research Use Measures used in this trial. However, we have limited confidence in this result given the low response rate to the questionnaires designed to capture these outcomes, and some measures were only moderately reliable.
Schools’ level of engagement varied: six out of ten schools did not engage to the level expected by the providers, although a small proportion engaged to a greater extent than expected (for example by hosting CPD sessions). Reasons for not engaging included lack of time, the timing and location of events, and a preference for face-to-face support rather than online or remote formats only.
Teachers felt research evidence was most effectively communicated when it was interactive, accessible, relevant, included a balanced and credible discussion of the evidence, and focused on how to apply the evidence in practice. Where schools went on to implement changes in light of the interventions, these came about through mechanisms such as in-school collaboration, further enquiry, and trying out, reviewing, adapting, and embedding approaches.
The lack of impact across the different interventions suggests that simply communicating research evidence to schools is not enough to improve outcomes. How easily the presented evidence can be used in practice—and the conditions in schools for implementing evidence-based change—might be just as important. Further research should assess whether interventions can transform evidence into practical action, and develop supportive implementation conditions in schools.
Full project description
The first trial, the ‘Literacy Octopus’ Dissemination Trial, aimed to test the impact on pupil outcomes of disseminating research summaries and evidence-based resources to schools. The materials aimed to support teaching and learning of Key Stage 2 literacy and were created by leading organisations with experience of engaging schools in evidence use.
A sample of 12,500 primary schools was identified by the evaluator, NFER, and randomised into five groups of 2,500. In spring term 2015, each of the four providers sent evidence-based materials to their 2,500 primary schools. The remaining 2,500 schools were in the control group. Given the trial did not require any active engagement or consent from the schools, the materials were simply sent to the schools with a brief note about the trial. The evaluation did not aim to collect information about schools’ use of the materials, but to provide a large-scale study of the impact of disseminating literacy-related evidence-based materials on pupils’ outcomes in KS2 English in summer 2016.
The second trial, the Evidence-based Literacy Support—‘Literacy Octopus’ Trial tested a range of interventions and resources, all of which aimed to engage schools in using evidence-based materials to improve teaching and learning in Key Stage 2 literacy. Four delivery partners provided interventions. These included light-touch, ‘passive’ approaches—such as emailing materials to schools—and more ‘active’ support such as face-to-face events about putting research into practice.
The evaluation was designed to assess the impact of the different interventions on pupil outcomes and teachers’ use of, and engagement with, research. The interventions began in spring 2015 with a range of materials and light-touch support offered through to the following academic year. Key Stage 2 English results from the summer 2016 were used to assess the impact on pupils. This was a large-scale effectiveness trial. The evaluator, NFER, recruited 823 primary schools; 60 were allocated to each of the nine intervention arms and 283 to the control group. The qualitative evaluation included observations of activities, interviews, and case studies.
The trials were funded by the Education Endowment Foundation (EEF), the Department for Education, and the Mayor’s London Schools Excellence Fund as part of a round of funding exploring Research Use in Schools.