Around ten percent of most recent Cochrane reviews included outcomes in their literature search strategy and were associated with potentially exaggerated results: A research-on-research study

Abstract

Objectives: To assess the proportion of the recent Cochrane reviews that included outcomes in their literature search strategy, how often they acknowledged these limitations, and how qualitatively different the results of outcomes included and not included in the search strategy were.

Design and setting: We identified all the Cochrane reviews of the interventions published in 2020 that used a search strategy connecting outcome terms with "AND." Reviews were defined as acknowledging the limitations of searching for outcomes if they mentioned them in the discussion. We compared the characteristics of outcomes included and not included in the search strategy.

Results: Of the 523 Cochrane reviews published in 2020, 51 (9.8%) included outcomes in their search strategy. Only one review acknowledged it as a limitation. Forty-seven (92%) assessed outcomes not included in the search strategy. Outcomes included in the search strategies tended to include a larger number of studies and show their effects in favor of the intervention.

Conclusions: Around ten percent of the recent Cochrane reviews included outcomes in their search, which may have resulted in more outcomes significantly in favor of the intervention. Reviewers should be more explicit in acknowledging the potential implications of searching for outcomes.

Keywords: Systematic review; meta-analysis; meta-epidemiology; publication bias; search strategy; selective outcome reporting.