Appraising the Quality of Systematic Reviews
This technical brief explains how readers can judge the quality of a systematic review in disability and rehabilitation research. It argues that systematic reviews vary widely in quality, so users of research need a way to separate rigorous reviews from weak ones before relying on their conclusions.
Why appraisal matters
Systematic reviews try to reduce bias in finding, selecting, coding and combining studies. They help practitioners by pre-filtering evidence, show the state of knowledge on a question, and reveal gaps for future research. Even so, they are not a cure-all. Examples cited include four reviews of treatments for self-injurious behavior that reached different conclusions, and a survey of published reviews that found large differences in how transparently they were reported. The criteria discussed apply to most reviews, whether or not they use meta-analysis.
Key criteria
- Protocol: a plan written in advance acts as a road map. Readers should check whether the methods mention one, whether it covers the expected elements, and whether any departures are explained. Dropping a study after seeing results is a warning sign. Fewer than half of the reviews in one sample used a protocol, mostly outside the Cochrane group.
- Question: the topic should be narrowed to a precise question that defines the intervention, outcomes and population.
- Sources: reviewers should search several databases, plus hand-searching of journals and checking reference lists. Relying on one database, or only on published work, introduces database, source-selection and publication bias, since unpublished studies tend to show smaller effects. Grey literature should be sought but examined carefully because it is usually not peer reviewed.
- Scope: geographic, time and language limits should be justified. Geographic limits are seldom appropriate, a starting date should have a sensible rationale, and excluding languages conflicts with the aim of covering all evidence.
- Selection principles: inclusion and exclusion rules should be stated with reasons, ideally with a log of rejected studies, and a sample of the studies, about 20 to 30 percent, should be screened independently by a second rater.
- Data extraction: coding categories should be defined, a second coder should check a sample with agreement reported, coders should be trained, and disagreements resolved by discussion.
Trial quality and tools
A good review also rates the quality of each included study. Some authors restrict combined analysis to high-quality studies in a best-evidence synthesis; others analyze everything and then examine whether quality is related to outcomes. Either way, a quality rating helps readers interpret effect sizes, because a large effect from a weak study deserves caution. Scales such as the PEDro scale can be used.
Several appraisal tools exist. A government-commissioned comparison of twenty tools found only two that met its standards, one being an update of the other. The brief notes that other checklists are available but largely unvalidated, and that the same considerations apply to reviews of development activities, apart from statistical points. The brief also lists registries where reviews can be located, including one the Center was building.
About this summary: this page condenses a longer original publication and does not reproduce it.
This page is a summary. Related material is available from the content areas page.