Aim: Prediction intervals are a useful measure of uncertainty for meta-analyses that capture the likely effect size of a new (similar) study based on the included studies. This study aimed to: (i) estimate the proportion of meta-analysis studies that report a prediction interval in sports medicine, using medicine as a comparator; and (ii) estimate the proportion of studies with a discrepancy between the reported confidence interval and a calculated prediction interval.
Methods: We identified all meta-analysis published between 2012 and 2022 in 10 highly ranked sports medicine and eight medical journals. From these articles, 750 were randomly selected from each discipline. Articles that used a random effect meta-analysis model were included. We randomly selected one meta-analysis from each article to extract data from, which included the number of estimates, the pooled effect, and the confidence and prediction interval.
Results: Of the 1500 articles screened, 866 (514 from sports medicine) used a random effect model. The probability of a prediction interval being reported in sports medicine was 1.7% (95% CI = 0.9%, 3.3%). In medicine the probability was 3.9% (95% CI = 2.4%, 6.6%). There were 57% lower odds of a prediction interval being reported in sports medicine (odds ratio = 0.432, 95% CI = 0.178, 0.995). However, the 95% CI was compatible with the difference being practically none. Three of the nine sports medicine studies that reported a prediction interval considered it in their conclusions. A prediction interval was able to be calculated for 220 sports medicine studies. For 60% of these studies, there was a discrepancy in study findings between the reported confidence interval and calculated prediction interval. Prediction intervals were 3.4 times wider than confidence intervals.
Conclusion: Very few meta-analyses report prediction intervals and hence are prone to missing the impact of between-study heterogeneity on the overall conclusions. The widespread misinterpretation of random effect meta-analysis could mean that potentially harmful treatments, or those lacking a sufficient evidence base, are being used in practice. Authors, reviewers, and editors should be aware of the importance of prediction intervals. Journals should consider mandating the inclusion of prediction intervals for all meta-analyses. Changing the default settings of meta-analysis software to include a prediction interval and changes to reporting guidelines could help improve prediction interval reporting rates.
Load the packages required for all analyses ("0_libraries"). All published meta-analysis from a selection of highly ranked sports medicine and medical journals between 2012 and 2022 are identified in files 1.1 ("1.1_find_abstracts_medicine") and 1.2 ("1.2_find_abstracts_sports_medicine"), and subsequently joined in file 2 ("2_join_extracted_data"). From these pool of studies, a random sample of 1500 is taken ("3_random_sample_1500"). Twenty-percent of the 1500 articles were random sampled, as a validation set ("4_select_validation_set"), for which the inclusion and data extraction accuracy is subsequently calculated ("5_calculate_data_extraction_accuracy"). The dataset containing details of the meta-analyses is wrangled and cleaned ("6_data_wrangling_and_excluded_studies"), before undergoing analyses that align with the three aims of the paper (files 7, 8 and 9). Figure 1 is created in ("10_figure1").