Deciding your outcome measure
Once a question has been defined, the next step is to determine the measure against which success will be judged. Although many different outcomes are important in education, this guide focuses on measuring academic attainment. In this section we discuss different ways of measuring attainment.
When deciding your academic outcome measures there are three main sources from which you can choose
- National assessments
- Standardised tests from reputable suppliers
- Design your own
Above all you are trying to select a measure which is both valid (meaning that it measures what it claims to measure) and reliable (meaning that it is consistent over time and context). Any provider of high-quality standardised tests should have made sure that their tests pass these quality criteria. If you cannot answer these questions satisfactorily then you should probably not use the test:
Tips for determining the quality of an assessment
- Can you define clearly what the assessment measures are?
- Ask other teachers to look at the assessment content, required responses and marking:
- Do they think it looks appropriate?
- Could anything other than what the assessment is supposed to measure influence the outcomes? (e.g. Reading comprehension influencing the results of a science test.)
- Does the assessment cover the full range of measures, in terms of content and level? Are there any gaps? Is it too easy or hard for some?
- Are the outcomes reliable? If the assessment was repeated or marked by someone else, would you get the same results?
If your intervention is short then you might want to use a different assessment from the outcome measure to prevent “practice effects”, e.g.; where students remember answers from previous tests. To avoid these effects, you could use old SAT papers as the pre-test and this year’s exam as the post-test. Alternatively, many standardised tests provided by suppliers have multiple forms.
It is important not to over-burden children with too much testing. Often, you will want to use tests your school already uses as outcome measures. However, when you need a new test it is important to understand the alternatives, and the advantages and disadvantages of other options. For example, be designing your own test, you can tailor your assessment to exactly what you want to measure, but it might not be as reliable as a test designed by a reputable supplier. A full list of advantages and disadvantages is outline in Table 1 below, and a list of suppliers can be accessed here.
|A. National Assessment||Strengths||Limitations|
|Use old or current national test papers (e.g. SAT or GCSE papers) as the pre-test and post-tests.||The best predictor of actual performance is national tests. They are cheap and easy to access and provide good practice for pupils.||They may not be as reliable as some external tests provided by reputable suppliers and it is not possible to tailor them to the needs of specific pupils or the focus of an intervention|
|B. Standardised tests||Strengths||Limitations|
|There many providers of high-quality standardised tests of attainment (see links in the Help content above).||They are likely to provide reliable and valid results. They are often standardised using national populations so you can compare your children's attainment to national norms. They can be highly predictive of performance in national tests and some may provide predictions as well as actual scores. Digital tests can provide instant results and rich data on individual children. ||They may be expensive as many providers are commercial organisations. They may not be aligned with the curriculum, or the specific area in which you are interested. |
|C. Design your own||Strengths||Limitations|
|Design your own completely from scratch or combine sections of other assessments you have used in the past||You can tailor the assessment to suit your own needs including the subject area being assessed and the age and ability range of your pupils. They are cheap to get hold off and your school may already have home-made tests that it uses. ||Home-made tests may not be as reliable as external tests provided by reputable suppliers which will have been thoroughly piloted. You will also not be able to compare your results to national norms. Ideally, you will need to design more than one version of the test in order to account for practice effects. |
Deciding your pre-test measure
Your pre-test measure should be as closely aligned to, or predictive of, your outcome measure as possible. In most cases it will make sense to use the same assessment as the pre-test. However, in some cases this is not possible. For example, if the intervention lasts a whole year, an assessment that would be suitable at the end of the year may be too hard at the beginning, especially at younger ages. In these cases, you will need to use something that is similar, but easier.