Diagnostic Questions

This project and its evaluation were affected by the 2020 partial school closures caused by the Covid-19 pandemic, and the cancellation of GCSE examinations in 2020. As a result, we were not able to use GCSEs in order to estimate the impact of the project on maths attainment. The report will, therefore, feature the findings of the implementation and process evaluation undertaken as part of the trial, in addition to some exploratory analysis on the impacts on teacher workload. 

Diagnostic Questions (DQs) is an online formative assessment programme that allows teachers to set quizzes for students with ease and also provides teachers with feedback about pupils’ misconceptions so that they can be quickly addressed. This project will test the programme in Key Stage 4 maths lessons, with pupils completing the quizzes as homework tasks.

DQs are carefully designed multiple choice questions, which are designed to detect underlying misconceptions that cause pupils to answer questions incorrectly. Teachers can use the system to set quizzes very quickly as all questions are aligned with exam specifications from AQA, Edexcel and OCR. The developers say it can take as little as one hour to set quizzes for the entire year. The system also provides a breakdown of marks to teachers and sophisticated feedback on common misconceptions in their class, meaning that teaches will potentially save considerable time on both setting and marking homework: this potential reduction in teacher workload will be measured as part of the evaluation.

The system includes a parental engagement element with parents alerted to the quizzes that their children need to complete and the marks that they receive. Behavioural Insights Team (BIT) are going to act as advisors for this element of the work, as well as leading the project. The programme is owned by Eedi. 

Why are we funding it?

The EEF’s recent review of marking led us to commit £2 million to building the evidence base around teachers’ marking. This project would be an interesting test of an alternative to time-consuming written marking. In addition, the programme will use lessons from the successful EEF Texting Parents project to communicate to parents.

It is well known that formative feedback can positively impact on attainment and there is a solid rationale for teachers having easy access to information about the misconceptions of their pupils. However, finding programmes that help all teachers to implement the approach is challenging. DQs has not been subject to a robust evaluation, but evaluations of a similar tool in the Netherlands found an impact of three months’ additional progress on students’ outcomes.

How are we evaluating it?

The intervention will be evaluated by a team from Alpha Plus. The evaluation will be an efficacy trial, structured as a 2-arm randomised controlled trial (RCT) with allocation of secondary schools to treatment or a business-as-usual control group. The trial will focus on GCSE maths classes in secondary schools, with pupils receiving the programme across Years 10 and 11. The primary outcome will be GCSE maths results after two years of the programme. There will be an in-depth process evaluation alongside the impact evaluation to understand implementation and necessary conditions to success.

When will the evaluation report be due?

The evaluation report will be published in Spring 2021.