SHINE in Secondaries

This is a report of a pilot programme of the SHINE in Secondaries Saturday school transition intervention. The programme aims to improve attainment by focusing on literacy and numeracy and revisiting areas where pupils are struggling through a creative curriculum approach and enrichment opportunities.

The Saturday programme is run by teachers from the school for 25 weeks throughout Year 7. Schools are responsible for developing their own curriculum of activities to suit the abilities and needs of their pupils following guiding principles specified by SHINE.

The programme is designed to run for approximately 60 pupils staffed by four qualified teachers, three teaching assistants and three peer mentors all recruited and employed by the host secondary school.

This evaluation was set up as a pilot study with four schools and 613 pupils participating. This report is one of two studies evaluating SHINE’s Saturday programmes. Hallé SHINE on Manchester evaluation is an EEF-funded randomised controlled trial of the impact of the Saturday school on primary school pupils’ literacy, numeracy and music outcomes.

This study had three aims:

  1. To explore the impact of SHINE in Secondaries on literacy, numeracy and attitudes to school and learning outcomes
  2. To explore the organisational implications and lessons for future wider roll-out
  3. To explore the feasibility of using regression discontinuity design (RDD) as a pilot study

The study was funded by the Education Endowment Foundation as one of 24 projects in a themed round on literacy catch-up at the primary-secondary transition.

Key Conclusions

The following conclusions summarise the project outcome

  1. Impact of the intervention: Very small effect sizes on the primary and secondary outcomes of literacy and numeracy were detected. As this was an exploratory evaluation design and the number of schools involved in the project was small, this made it unlikely that any impact of SHINE could be detected, should the true estimate of effect be small.

  2. Implementation and organisation issues: Feedback from the teachers and pupils suggested that the intervention was perceived to be valuable in providing an opportunity to consolidate literacy and numeracy skills.

  3. Feasibility of evaluation design: The purpose in piloting this design was to explore the feasibility of using this approach when randomly allocating pupils to an intervention group was not possible or desirable. The regression discontinuity design is an appropriate evaluation method for education interventions. However, because it is less powerful than a randomised controlled trial, the number of pupils involved must be two and a half times greater if a small effect is to be detected.

What is the impact?

This pilot was designed to explore the impact of a Saturday school for secondary pupils on literacy and numeracy, as well as the organisational and implementation issues of the intervention. The feasibility of using a non-randomised evaluation approach using a regression discontinuity design (RDD) to evaluate the impact was also tested in this pilot.

The evaluation showed a very small impact on literacy outcomes across all four schools using the Progress in English and Progress in Maths tests. These findings are limited evidence of promise of the intervention. A larger RDD study or randomised controlled trial would be required to investigate the impact of the intervention and the size of the effect (if any) with confidence.

A subgroup analysis of children eligible for free school meals (FSM) was conducted. This showed no evidence of the intervention differentially benefiting pupils eligible for free school meals in any of the schools. These findings should be treated with caution because, due to the small sample size, the study was unable to detect the small effects with confidence.

The pilot demonstrated that it was feasible to design, implement and analyse the RDD successfully; and the design could be used again to evaluate similar interventions where randomisation is not possible or desirable.

The process evaluation involved observations of activities, interviews and focus groups with pupils and teachers. However, due to the very small samples in the evaluation, all findings should be treated with caution.

All teachers interviewed agreed that pupils liked the familiarity and the relaxing atmosphere that SHINE in Secondaries encouraged and that they learnt while being engaged in an enjoyable way. An important outcome reported by the teachers in two schools observed was pupils’ improved social skills, better relationships at all levels and increased confidence through their participation in practical and team-building activities.

The five pupils who were interviewed spoke enthusiastically about their SHINE in Secondaries experiences. All pupils mentioned that they had made good progress in their learning while having fun.

The SHINE project managers, who were also teachers in the school, were considered to be integral to the successful running of the project. All eight teachers interviewed agreed that SHINE in Secondaries gave them the opportunity to think and plan ‘outside of the box’ and they enjoyed teaching around a particular theme, which helped reinforce links with ‘real life’.

It was also felt that the continuity with the school week that the SHINE in Secondaries programme provided for pupils in Year 7 was an excellent feature of the intervention. From our visits, it seemed that all SHINE in Secondaries teachers see all or at least some of the pupils during the week. This allowed the teaching content of the SHINE in Secondaries programme to better target pupil needs and also to build on the pupils’ familiarity with the teaching staff.

How secure is the finding?

This evaluation was a pilot using a regression discontinuity (RD) design. Unlike a randomised controlled trial, where pupils are randomly allocated to an intervention or control group, this approach assigns pupils to intervention and control groups according to their results on a pre-test. We began by looking at the range of pupil outcomes on a Progress in English test. Pupils were assigned to the intervention if they fell below a certain mark (the first ‘cut point’). Pupils were assigned to the control group if they fell above a second ‘cut point’ on the test. Pupils falling between the two cut points were randomly allocated to the intervention or control group, thereby creating a ‘mini’ randomised controlled trial (RCT) within the RDD.

Although a randomised controlled trial design is stronger for inferring causation, for this evaluation it was not deemed to be suitable. The RD design guaranteed all low-attaining pupils would be offered the opportunity to attend SHINE in Secondaries, which was a very important consideration for schools.

Uptake of places at the intervention was 73% of those who were assigned to the intervention group which means that 27% of those counted in the analysis did not attend the intervention. The intention-to-treat analysis provides average impacts of the intervention for those who were offered the intervention, whether or not they actually participated in the intervention. This created a ‘fuzzy’ regression discontinuity and reduces the robustness of the evaluation. However, average individual attendance of those children who accepted places was 68%. This includes any pupil that is considered as taking up their place by attending a minimum of one session. The average attendance is based on the numbers of sessions possible in each school.

How much does it cost?

The funding provided to each school to employ staff and run the programme was £52,000. If the intervention took place with 60 pupils as recommended, this is equivalent to around £870 per pupil per year. Although schools in the evaluation did not pay anything towards the intervention, they expended considerable time, resources and effort in running SHINE in Secondaries. SHINE’s ‘Making it Work’ handbook explains that projects are normally up to 100% funded for the first three years while for years 4 to 6 schools would be expected to fund, or find funding for, at least 50% of the costs, with this rising to at least 70% of costs in years 7 to 9.