You are here

Article Text

Article menu

Download PDFPDF

XML
Evidence based practice
Research
Search for unpublished data by systematic reviewers: an audit
  1. Hedyeh Ziai1,2,
  2. Rujun Zhang1,3,
  3. An-Wen Chan4,5,
  4. Nav Persaud3,6,7
  1. 1 Keenan Research Centre, Li Ka Shing Knowledge Institute, St Michael’s Hospital, Toronto, Canada
  2. 2 Faculty of Medicine, University of Ottawa, Ottawa, Canada
  3. 3 Department of Family and Community Medicine, University of Toronto, Canada
  4. 4 Women’s College Research Institute, Women’s College Hospital, Toronto, Canada
  5. 5 Department of Medicine, University of Toronto, Toronto, Canada
  6. 6 Department of Family and Community Medicine, St Michael’s Hospital, Toronto, Canada
  7. 7 Centre for Urban Health Solutions, Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada
  1. Correspondence to Dr Nav Persaud; nav.persaud{at}utoronto.ca

Abstract

Objectives We audited a selection of systematic reviews published in 2013 and reported on the proportion of reviews that researched for unpublished data, included unpublished data in analysis and assessed for publication bias.

Design Audit of systematic reviews.

Data sources We searched PubMed and Ovid MEDLINE In-Process & Other Non-Indexed Citations between 1 January 2013 and 31 December 2013 for the following journals: Journal of the American Medical Association, The British Medical Journal, Lancet, Annals of Internal Medicine and the Cochrane Database of Systematic Reviews. We also searched the Cochrane Library and included 100 randomly selected Cochrane reviews.

Eligibility criteria Systematic reviews published in 2013 in the selected journals were included. Methodological reviews were excluded.

Data extraction and synthesis Two reviewers independently reviewed each included systematic review. The following data were extracted: whether the review searched for grey literature or unpublished data, the sources searched, whether unpublished data were included in analysis, whether publication bias was assessed and whether there was evidence of publication bias.

Main findings 203 reviews were included for analysis. 36% (73/203) of studies did not describe any attempt to obtain unpublished studies or to search grey literature. 89% (116/130) of studies that sought unpublished data found them. 33% (68/203) of studies included an assessment of publication bias, and 40% (27/68) of these found evidence of publication bias.

Conclusion A significant fraction of systematic reviews included in our study did not search for unpublished data. Publication bias may be present in almost half the published systematic reviews that assessed for it. Exclusion of unpublished data may lead to biased estimates of efficacy or safety in systematic reviews.

  • Review
  • Systematic Publication Bias Unpublished Data

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2017-017737

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • To our knowledge, this is the first study to report the proportion of systematic reviews that searched for and included unpublished data from multiple sources including registries.

    • Two reviewers independently reviewed each included review to minimise bias.

    • Only reports of trials published in selected high-impact journals were reviewed, and this may reduce the external validity of our results.

    Introduction

    Readers of systematic reviews should know whether unpublished data was sought as the conclusions of the review may depend on studies that have not been published. Studies with positive results have a three times higher odds of being published than those with negative or null results.1 The results of studies in favour of a new treatment also have a higher chance of being published than negative studies.2 3 Published studies are likely to have larger treatment effects than unpublished studies.4–6 An analysis of 42 meta-analyses on drug trials found that 92% of the efficacy estimates were altered when unpublished studies that were submitted to regulatory agencies were included.7 For example, agomelatine was reported to be an effective depression treatment by several systematic reviews of only published studies,8 9 but no treatment effect was demonstrated when seven unpublished studies were included.10 11 Other examples of publication bias include oseltamivir for influenza in adults,12 statins for the prevention of venous thromboembolic events,13 quinine for nocturnal leg cramps14 and reboxetine for depression.15

    The Cochrane Handbook for Systematic Reviews of Interventions states that ‘the convincing evidence for the presence of several types of reporting biases demonstrates the need to search comprehensively for studies that meet the eligibility criteria for a Cochrane review’.16 Several studies have demonstrated that widely agreed on reporting standards are not always followed. Following the publication of reporting guidelines, the reporting of randomised control trials (RCTs)17 and abstracts18 remained poor. Audits of adherence to reporting standards might have improved the reporting of RCTs.19 Just as changes to reporting guidelines for RCTs improved RCT reporting, changes to reporting guidelines for systematic reviews may improve systematic review reporting. A recent editorial and other published guidance suggest that data should routinely be sought from regulatory agencies and trial registries.20 21

    We performed an audit of systematic reviews of healthcare interventions published in 2013 to determine the proportion of reviews that reported searching for unpublished data. We also determined how many systematic reviews included unpublished data in the analysis and how many assessed for publication bias.

    Methods

    Eligibility criteria

    Ethics approval was not required as the study was based on published systematic reviews. We included all full-text systematic reviews (with prespecified methods) that were published in 2013 in Journal of the American Medical Association (JAMA), British Medical Journal (BMJ), Lancet and Annals of Internal Medicine, as well as a random subset of 100 reviews published in the Cochrane Database of Systematic Reviews. Methodological reviews were excluded.

    Search strategy

    PubMed and Ovid MEDLINE In-Process & Other Non-Indexed Citations database were used to identify all published systematic reviews between 1 January 2013 and 31 December 2013 for the following journals: JAMA, BMJ, Lancet and Annals of Internal Medicine. The search for each source included the journal name, ‘systematic review’ and the publication date (eg, the search strategy used for JAMA in PubMed was: ‘JAMA’[journal] AND ‘systematic review’[title] AND‘2013’[pdat]). The journals were also manually searched to identify potential relevant papers that were missing from the electronic searches.

    The Cochrane Library was also searched to identify literature published in the Cochrane Database of Systematic Reviews not retrieved by the previous electronic searches. The search was limited to the ‘Cochrane Database of Systematic Reviews’, the publication date was set to ‘2013 to 2013’ and the product type was identified as ‘Review’. A random number sequence generator was used to select and include 100 reviews from the Cochrane Database of Systematic Reviews. Based on our piloting of the data abstraction instrument, we believed that 100 reviews would be sufficient to determine the common approaches to unpublished data. The date of the last search for all databases was 11 February 2014.

    Data extraction

    Unpublished data included complete trials that have never been published as well as specific outcomes that are not reported in published trials. For this study, we considered data appearing in conference proceedings, research reports and dissertations as part of the grey literature, and these were included as a type of unpublished data.

    From each systematic review included in the analysis, the following general characteristics were collected: journal name, title, first author and the date of publication. The following main outcomes were extracted: whether there was any search for unpublished data as described in the methods or evidence of unpublished data inclusion, whether unpublished data were used in meta-analysis, whether the interventions of review were pharmacological in nature and the results of any assessment of publication bias.

    Two authors (HZ, NP or RZ) independently extracted data from the Abstract, Methods and Results sections using a standardised electronic form. Disagreements were discussed until consensus was achieved or resolved by a third individual. The following information was collected from the methods sections of each report: databases searched, electronic search strategy, other sources searched (conference abstracts, unpublished studies, ongoing studies, contact with study authors/experts and so on), any restrictions to publication status, language, whether publication bias was assessed and methods used to assess publication bias. For the results section, we reviewed any analysis of unpublished data and assessment of publication bias. For the discussion section, we assessed whether the authors commented on the presence of publication bias. Additionally, the following terms were searched electronically in the article text to ensure inclusion of all relevant information: ‘unpublish’, ‘publication’, ‘bias’, ‘funnel’, ‘Hegg’, ‘Egger’, ‘gray’, ‘grey’ and ‘reporting’.

    Data synthesis and analysis

    The primary purpose of this study was to determine the proportion of systematic reviews that reported searching for unpublished data. The secondary objective was to determine the proportion of systematic reviews that found unpublished data and included them in the meta-analysis. Pearson χ2 tests were completed to assess for any difference between systematic reviews of pharmacological versus other interventions, and for any difference between reviews published in the Cochrane Database of Systematic Reviews versus the other journals.

    Results

    We identified 104 systematic reviews published in JAMA (n=13), BMJ (n=40), Lancet (n=10) and Annals of Internal Medicine (n=41). The Cochrane Library search strategy yielded 1090 results, and subsequently 100 articles were randomly selected. Among the 204 systematic reviews reviewed in full text, one review was excluded as a methodological review. Our final cohort consisted of 203 reviews. Seventy-one reviews (35%) were pharmaceutical reviews.

    Search for unpublished data

    Of the 203 included systematic reviews, 73 (36%) did not describe in the methods whether there was any search for unpublished or grey literature data and 130 (64%) described some search for unpublished data. Overall, 42% (86/203) of reviews described searching for unpublished completed studies (with or without other types of unpublished data) in their search strategy and 22% (44/203) of reviews described searching the grey literature without specifically describing searching for unpublished studies. No reviews described only searching for unreported data from published studies included in the review in their methods.

    Table 1 summarises the number of reviews that searched each source according to what was reported in the entire report including the results and discussion in addition to the methods section. National and international trial registries were the most significant source of unpublished data collection, followed by data received from contacting individuals or organisation and conference proceedings. Specifically, 42% (85/203) searched national and international trials registers, 41% (84/203) of the reviews contacted individuals or organisations (eg, sponsors or research organisations), 35% (71/203) searched conference proceedings, 8% (16/203) searched pharmaceutical industry trials registers and 2% (4/203) searched subject-specific trials registers. No review included a search for data from regulators via their websites or information requests.

    View this table:
    Table 1

    Sources of data and yield of searches in systematic reviews. Types of unpublished data are in rows, and sources of unpublished data are in columns

    Some reviews identified types of unpublished data that were not described in the search strategy in the Methods section (ie, they included unpublished data in the results section without describing the methodology of searching for unpublished data in the methods). Among the 130 reviews that described searching sources for unpublished data, 55 of them (43%) described only searching one source, 37 (28%) reviews described searching two sources and the remaining 38 (29%) reviews described searching three or more sources. No review described searching all sources for unpublished data. Furthermore, some included systematic reviews did not specify the source of unpublished data.

    In a prespecified analysis of factors associated with performing a search for unpublished studies, we did not find any difference between reviews of pharmacological and non-pharmacological interventions. However, reviews published in the Cochrane Database of Systematic Reviews were significantly more likely to search for unpublished studies than those published in standard journals (p<0.0001).

    The unpublished data discovered by systematic reviews were often not included in the results and analyses. Among the 130 reviews that included a search for unpublished or grey literature data, 89% (n=116) found such data. Of the 116 reviews that searched and found unpublished data, 46 reviews (40%) both identified and included unpublished data. The remaining 70 reviews did not include the supplementary data. Among the 46 studies that included unpublished data, 23 (50%) included data from unpublished studies. Twenty reviews (43%) included unpublished data from published studies and 7% (3/46) included data from abstracts of published papers only.

    Assessment of publication bias

    Thirty-three per cent (68/203) of the reviews included an assessment of publication bias. An additional 27% (55/203) of the reviews planned a publication bias assessment, but these analyses were not reported for a number of reasons: 10 did not find any eligible studies, 42 had insufficient quantity (<10) of selected studies or other reasons and three did not provide an explanation. Thirty-nine per cent (80/203) of the reviews did not describe an intent to assess publication bias.

    Of the 68 systematic reviews that assessed publication bias, 58 reviews performed statistical or graphical analysis, and 10 reviews employed other methods such as a discussion of the likelihood that highly statistically significant results could be explained by publication bias. Thirty-four per cent (20/58) of the non-qualitative assessments and 70% (7/10) of the qualitative assessments were significant for publication bias (table 2). There was a trend towards systematic reviews that searched for unpublished data being less likely to indicate publication bias.

    View this table:
    Table 2

    Results of assessment for publication bias

    Discussion

    Among the 203 systematic reviews published in high-impact general medical journals in 2013, 36% did not describe any attempt to search for unpublished studies, although guidelines recommend searching for unpublished data. Of the 116 reviews that completed a search and found unpublished data or grey literature, 40% included unpublished data for analysis. Thirty-three per cent of the reviews included a publication bias assessment, and 40% of these reviews revealed evidence of publication bias. The quantitative and/or qualitative suggestion of publication bias was more prevalent in reviews that did not search for unpublished data, when compared with those reviews that searched for unpublished data.

    To our knowledge, this is the largest study that has investigated the proportion of systematic reviews that searched for and included unpublished data. A 2017 audit found that 52% of selected systematic reviews did not report a search of trial registries.22 A cross-sectional survey of corresponding authors of Cochrane reviews with 37% response rate found that 76% of Cochrane respondents reported searching for unpublished data, 82% of unpublished data were used in analysis and that the most common source of unpublished data was from contacting the study investigators.23 Over 10% of Cochrane reviews from 2000 to 2006 included unpublished studies.24

    The results of this study highlight inadequacies in identifying unpublished data in systematic reviews. Cochrane reviews might have been more likely to search for unpublished data because of guidance in the Cochrane handbook as well as rigorous protocol review and editorial practices. Clear standards regarding the search for and inclusion of unpublished studies may help. There is some evidence that unpublished data are sometimes misleading25 and that unpublished data may not contain sufficient information to assess methodological quality and in turn may be unreliable. Conference abstracts may not always be reliable; 40%–60% may not have reported the main outcome results in the same way as the final published study.25–30 Further studies will be needed to determine whether including unpublished data improves systematic reviews as their unpublished data may be unreliable.

    Current reporting guidelines for systematic reviews such as Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) require only specification of the data sources searched, but they do not recommend describing whether a search for unpublished data was performed.31 One of the 27 items is on the description of information sources, but the checklist does not mention unpublished data. The PRISMA statement explanation and elaboration indicates that describing the search for unpublished information is ‘useful’ but not mandatory. Specifically, they indicate: ‘Authors should also report if they attempted to acquire any missing information (such as on study methods or results) from investigators or sponsors; it is useful to describe briefly who was contacted and what unpublished information was obtained’.31 Two systematic review quality assessment tools, A Measurement Tool to Assess Systematic Reviews and the Risk of Bias in Systematic Reviews (ROBIS), explicitly include the search for unpublished data.32 33

    The reporting of the search for unpublished data in systematic reviews may be improved by its explicit inclusion as a recommendation in the PRISMA reporting guideline. Similarly, the Consolidated Standards of Reporting Trials (CONSORT) statement was developed in response to concerns about the quality of RCTs.34 35 Evaluations of its effectiveness have suggested that journal adoption of CONSORT is associated with improved quality of reporting of RCTs.17 36–38 Furthermore, the inclusion of unpublished studies in systematic reviews might lead to those primary studies eventually being published through initiatives such as the Restoration Invisible and Abandoned Trials process.26

    To reduce potential reporting bias, it is important to search for unpublished data using several sources: subject matter experts, investigators, commercial sponsors, trial registries, regulatory agency documents and conference proceedings.21 Searching for trial protocols can help to better understand the study methods and identify selective reporting of outcomes within published studies.1 If there are concerns about the quality of data in any included study, whether it is published or not, a sensitivity analysis should be conducted to determine the effect of the suspect data.

    Limitations

    There are some limitations to this study. Only reports of trials published in the JAMA, BMJ, Lancet, Annals of Internal Medicine and the Cochrane Database of Systematic Reviews were reviewed. Systematic reviews published in lower impact factor journals were not included. We also randomly included 100 reviews from a group of 1090 reviews in the Cochrane Library. Therefore, the findings might not be representative of all published systematic reviews in 2013. We report the searches as described in the methods section of the publications, and the actual searches may have been different. We found that two systematic reviews obtained unreported data regarding published studies from registries, although they did not describe searching these registries. Time-lag bias that may have overestimated the under-reporting of unpublished data, although time-lag bias cannot explain the poor descriptions of the search strategy.

    Conclusion

    The search for unpublished data in systematic reviews is still suboptimal. A significant number of systematic reviews published in 2013 did not search for grey literature and unpublished data. Inadequate reporting of unpublished data and grey literature can lead to reporting bias. Almost half of the included reviews that assessed for publication bias suggested the presence of publication bias. Improving reporting guidelines for systematic reviews and better adherence to reporting guidelines may help address this issue.

    References

    1. 1.↵
      2. Chan AW ,
      3. Song F ,
      4. Vickers A , et al
      . Increasing value and reducing waste: addressing inaccessible research. Lancet 2014;383:257–66.doi:10.1016/S0140-6736(13)62296-5
      OpenUrlCrossRefPubMedWeb of Science
    2. 2.↵
      2. Dickersin K
      . How important is publication bias? A synthesis of available data. AIDS Educ Prev 1997;9:15–21.
      OpenUrlPubMedWeb of Science
    3. 3.↵
      2. Benjamin DK ,
      3. Smith PB ,
      4. Murphy MD , et al
      . Peer-reviewed publication of clinical trials completed for pediatric exclusivity. JAMA 2006;296:1266.doi:10.1001/jama.296.10.1266
      OpenUrlCrossRefPubMedWeb of Science
    4. 4.↵
      2. Hopewell S ,
      3. McDonald S ,
      4. Clarke M , et al
      . Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev 2007;18:MR000010.doi:10.1002/14651858.MR000010.pub3
      OpenUrl
    5. 5.↵
      2. McAuley L ,
      3. Pham B ,
      4. Tugwell P , et al
      . Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:e31–1231.doi:10.1016/S0140-6736(00)02786-0
      OpenUrl
    6. 6.↵
      2. Abraham NS ,
      3. Moayyedi P ,
      4. Daniels B , et al
      . Systematic review: the methodological quality of trials affects estimates of treatment efficacy in functional (non-ulcer) dyspepsia. Aliment Pharmacol Ther 2004;19:e41.doi:10.1111/j.1365-2036.2004.01878.x
      OpenUrl
    7. 7.↵
      2. Hart B ,
      3. Lundh A ,
      4. Bero L
      . Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012;344:d7202.doi:10.1136/bmj.d7202
      OpenUrlCrossRefPubMed
    8. 8.↵
      2. Singh SP ,
      3. Singh V ,
      4. Kar N
      . Efficacy of agomelatine in major depressive disorder: meta-analysis and appraisal. Int J Neuropsychopharmacol 2012;15:417–28.doi:10.1017/S1461145711001301
      OpenUrlCrossRefPubMed
    9. 9.↵
      2. Montgomery SA ,
      3. Kasper S
      . Severe depression and antidepressants: focus on a pooled analysis of placebo-controlled studies on agomelatine. Int Clin Psychopharmacol 2007;22:283–91.doi:10.1097/YIC.0b013e3280c56b13
      OpenUrlCrossRefPubMedWeb of Science
    10. 10.↵
      2. Howland RH
      . A benefit-risk assessment of agomelatine in the treatment of major depression. Drug Saf 2011;34:709–31.doi:10.2165/11593960-000000000-00000
      OpenUrlCrossRefPubMedWeb of Science
    11. 11.↵
      2. Koesters M ,
      3. Guaiana G ,
      4. Cipriani A , et al
      . Agomelatine efficacy and acceptability revisited: systematic review and meta-analysis of published and unpublished randomised trials. Br J Psychiatry 2013;203:179–87.doi:10.1192/bjp.bp.112.120196
      OpenUrlAbstract/FREE Full Text
    12. 12.↵
      2. Ebell MH ,
      3. Call M ,
      4. Shinholser J
      . Effectiveness of oseltamivir in adults: a meta-analysis of published and unpublished clinical trials. Fam Pract 2013;30:125–33.doi:10.1093/fampra/cms059
      OpenUrlCrossRefPubMedWeb of Science
    13. 13.↵
      2. Rahimi K ,
      3. Bhala N ,
      4. Kamphuisen P , et al
      . Effect of statins on venous thromboembolic events: a meta-analysis of published and unpublished evidence from randomised controlled trials. PLoS Med 2012;9:e1001310.doi:10.1371/journal.pmed.1001310
      OpenUrlCrossRefPubMed
    14. 14.↵
      2. Man-Son-Hing M ,
      3. Wells G ,
      4. Lau A
      . Quinine for nocturnal leg cramps: a meta-analysis including unpublished data. J Gen Intern Med 1998;13:600–6.
      OpenUrlCrossRefPubMedWeb of Science
    15. 15.↵
      2. Eyding D ,
      3. Lelgemann M ,
      4. Grouven U , et al
      . Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010;341:c4737.doi:10.1136/bmj.c4737
      OpenUrlAbstract/FREE Full Text
    16. 16.↵
      2. Higgins JPT ,
      3. Green S
      ; The Cochrane Collaboration, 2011. Cochrane handbook for systematic reviews of interventions 5.1.0. www.cochrane-handbook.org (accessed Jan 2015).
    17. 17.↵
      2. Hopewell S ,
      3. Dutton S ,
      4. Yu LM , et al
      . The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ 2010;340:c723.doi:10.1136/bmj.c723
      OpenUrlAbstract/FREE Full Text
    18. 18.↵
      2. Ghimire S ,
      3. Kyung E ,
      4. Kang W , et al
      . Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials 2012;13:77–83.doi:10.1186/1745-6215-13-77
      OpenUrlCrossRefPubMed
    19. 19.↵
      2. Schulz KF ,
      3. Altman DG ,
      4. Consort MD
      . Statement: updated guidelines for reporting parallel group randomised trials. BMJ 2010;2010:c332.
      OpenUrl
    20. 20.↵
      2. Schroll J ,
      3. Bero L
      . The Cochrane Library. Regulatory agencies hold the key to improving Cochrane Reviews of drugs. http://www.cochranelibrary.com/editorial/10.1002/14651858.ED000098 (accessed Apr 2015).
    21. 21.↵
      2. Chan AW
      . Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ 2012;344:d8013.doi:10.1136/bmj.d8013
      OpenUrlFREE Full Text
    22. 22.↵
      2. Baudard M ,
      3. Yavchitz A ,
      4. Ravaud P , et al
      . Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ 2017;356:j448.doi:10.1136/bmj.j448
      OpenUrlAbstract/FREE Full Text
    23. 23.↵
      2. Schroll JB ,
      3. Bero L ,
      4. Gøtzsche PC
      . Searching for unpublished data for Cochrane reviews: cross sectional study. BMJ 2013;346:f2231.doi:10.1136/bmj.f2231
      OpenUrlAbstract/FREE Full Text
    24. 24.↵
      2. van Driel ML ,
      3. De Sutter A ,
      4. De Maeseneer J , et al
      . Searching for unpublished trials in Cochrane reviews may not be worth the effort. J Clin Epidemiol 2009;62:838–44.doi:10.1016/j.jclinepi.2008.09.010
      OpenUrlCrossRefPubMedWeb of Science
    25. 25.↵
      2. Saldanha IJ ,
      3. Scherer RW ,
      4. Rodriguez-Barraquer I , et al
      . Dependability of results in conference abstracts of randomized controlled trials in ophthalmology and author financial conflicts of interest as a factor associated with full publication. Trials 2016;17:213.doi:10.1186/s13063-016-1343-z
    26. 26.↵
      2. Bhandari M ,
      3. Devereaux PJ ,
      4. Guyatt GH , et al
      . An observational study of orthopaedic abstracts and subsequent full-text publications. J Bone Joint Surg Am 2002;84-A:615–21.doi:10.2106/00004623-200204000-00017
      OpenUrlAbstract/FREE Full Text
    27. 27.↵
      2. Kleweno CP ,
      3. Bryant WK ,
      4. Jacir AM , et al
      . Discrepancies and rates of publication in orthopaedic sports medicine abstracts. Am J Sports Med 2008;36:1875–9. 14.doi:10.1177/0363546508319054
      OpenUrlCrossRefPubMedWeb of Science
    28. 28.↵
      2. Toma M ,
      3. McAlister FA ,
      4. Bialy L , et al
      . Transition from meeting abstract to full-length journal article for randomized controlled trials. JAMA 2006;295:1281–7. 15.doi:10.1001/jama.295.11.1281
      OpenUrlCrossRefPubMedWeb of Science
    29. 29.↵
      2. Klassen TP ,
      3. Wiebe N ,
      4. Russell K , et al
      . Abstracts of randomized controlled trials presented at the society for pediatric research meeting: an example of publication bias. Arch Pediatr Adolesc Med 2002;156:474–9. 16.doi:10.1001/archpedi.156.5.474
      OpenUrlCrossRefPubMedWeb of Science
    30. 30.↵
      2. Rosmarakis ES ,
      3. Soteriades ES ,
      4. Vergidis PI , et al
      . From conference abstract to full paper: differences between data presented in conferences and journals. Faseb J 2005;19:673–80.doi:10.1096/fj.04-3140lfe
      OpenUrlAbstract/FREE Full Text
    31. 31.↵
      2. Moher D ,
      3. Liberati A ,
      4. Tetzlaff J , et al
      . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9.doi:10.7326/0003-4819-151-4-200908180-00135
      OpenUrlCrossRefPubMedWeb of Science
    32. 32.↵
      2. Whiting P ,
      3. Savović J ,
      4. Higgins JP , et al
      . ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225–34.doi:10.1016/j.jclinepi.2015.06.005
      OpenUrlCrossRefPubMed
    33. 33.↵
      2. Shea BJ ,
      3. Bouter LM ,
      4. Peterson J , et al
      . External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One 2007;2:e1350.doi:10.1371/journal.pone.0001350
    34. 34.↵
      2. Begg C ,
      3. Cho M ,
      4. Eastwood S , et al
      . Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637–9.
      OpenUrlCrossRefPubMedWeb of Science
    35. 35.↵
      2. Moher D ,
      3. Schulz KF ,
      4. Altman D
      . The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001;285:1987–91.doi:10.1001/jama.285.15.1987
      OpenUrlCrossRefPubMedWeb of Science
    36. 36.↵
      2. Plint AC ,
      3. Moher D ,
      4. Morrison A , et al
      . Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006;185:263–7.
      OpenUrlPubMedWeb of Science
    37. 37.↵
      2. Devereaux PJ ,
      3. Manns BJ ,
      4. Ghali WA , et al
      . The reporting of methodological factors in randomized controlled trials and the association with a journal policy to promote adherence to the Consolidated Standards of Reporting Trials (CONSORT) checklist. Control Clin Trials 2002;23:380–8.doi:10.1016/S0197-2456(02)00214-3
      OpenUrlCrossRefPubMedWeb of Science
    38. 38.↵
      2. Turner L ,
      3. Shamseer L ,
      4. Altman DG , et al
      . Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev 2012;1:60.doi:10.1186/2046-4053-1-60
      OpenUrlCrossRefPubMed

    Footnotes

    • Twitter @NavPersaud

    • Contributors HZ contributed to the conception and design of the project and the acquisition, analysis and interpretation of the data; drafted the work and revised it; approved the final version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy and integrity are appropriately investigated and resolved. RZ contributed to the analysis and interpretation of the data; drafted the work and revised it; provided final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy and integrity are appropriately investigated and resolved. A-WC contributed to the conception of the project, reviewed it critically for important intellectual content, provided final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy and integrity are appropriately investigated and resolved. NP contributed to the conception and design of the project; the interpretation of the data; revised the work; approved the final version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy and integrity are appropriately investigated and resolved.

    • Funding There was no specific funding for this project. NP is supported by a Physician Services Incoroporated Graham Farquharson Knowledge Translation Fellowship.

    • Disclaimer The views expressed in the submitted article are those of the authors and not an official position of the institution or funder.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement We have no further data to share.