Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Why do authors derive new cardiovascular clinical prediction rules in the presence of existing rules? A mixed methods study

Abstract

Background

Researchers should examine existing evidence to determine the need for a new study. It is unknown whether developers evaluate existing evidence to justify new cardiovascular clinical prediction rules (CPRs).

Objective

We aimed to assess whether authors of cardiovascular CPRs cited existing CPRs, why some authors did not cite existing CPRs, and why they thought existing CPRs were insufficient.

Method

Derivation studies of cardiovascular CPRs from the International Register of Clinical Prediction Rules for Primary Care were evaluated. We reviewed the introduction sections to determine whether existing CPRs were cited. Using thematic content analysis, the stated reasons for determining existing cardiovascular CPRs insufficient were explored. Study authors were surveyed via e-mail and post. We asked whether they were aware of any existing cardiovascular CPRs at the time of derivation, how they searched for existing CPRs, and whether they thought it was important to cite existing CPRs.

Results

Of 85 derivation studies included, 48 (56.5%) cited existing CPRs, 33 (38.8%) did not cite any CPR, and four (4.7%) declared there was none to cite. Content analysis identified five categories of existing CPRs insufficiency related to: (1) derivation (5 studies; 11.4% of 44), (2) construct (31 studies; 70.5%), (3) performance (10 studies; 22.7%), (4) transferability (13 studies; 29.5%), and (5) evidence (8 studies; 18.2%). Authors of 54 derivation studies (71.1% of 76 authors contacted) responded to the survey. Twenty-five authors (46.3%) reported they were aware of existing CPR at the time of derivation. Twenty-nine authors (53.7%) declared they conducted a systematic search to identify existing CPRs. Most authors (90.7%) indicated citing existing CPRs was important.

Conclusion

Cardiovascular CPRs are often developed without citing existing CPRs although most authors agree it is important. Common justifications for new CPRs concerned construct, including choice of predictor variables or relevance of outcomes. Developers should clearly justify why new CPRs are needed with reference to existing CPRs to avoid unnecessary duplication.

Introduction

Healthcare professionals regularly face diagnostic and prognostic uncertainties. Clinical prediction rules (CPRs) can help address these uncertainties and help make evidence-based decisions using information from individual patients [14]. The following steps are needed to develop a CPR [2, 4, 5]. Firstly, a CPR is constructed in a derivation study using variables predictive of an outcome. Secondly, the generalizability of the CPR should be tested in external validation studies. Lastly, the ability of the CPR to improve clinical outcomes and efficiency should be examined in impact analyses.

Many CPRs have been developed for various cardiovascular problems [68]. These cardiovascular CPRs are better recognized and more commonly used by general practitioners (GPs) compared with CPRs in other clinical areas [9]. However, the development of cardiovascular CPRs has been disproportionately focused on some clinical domains while many have not been externally validated and few have been assessed in impact studies [6, 7, 10]. For example, a systematic search identified 64 similar prognostic CPRs for congestive heart failure and 50 modified CPRs that consisted of many overlapping predictors [11]. At the same time, many developed CPRs are not utilized. A survey of general practitioners from the UK and a systematic review of international clinical guidelines from selected clinical domains in which CPRs were known to have been published demonstrated that in most clinical domains, CPRs were seldom adopted by guidelines or used in practice [9].

Before new research is undertaken, what is known about the subject should be carefully reviewed to identify pertinent research questions, choose optimal study designs and avoid unnecessary duplication [1214]. Researchers developing new CPRs should also clearly establish the need by examining all existing CPRs for the clinical problem. Producing CPRs by analyzing existing data without assessing what is already available may lead to many redundant CPRs. The Transparent Reporting of a multivariable prediction model for individual Prognosis Or Diagnosis (TRIPOD) statement published in 2015 recommends authors present a rationale for developing a new CPR with references to existing CPRs [15]. However, it is not yet known whether developers of cardiovascular CPRs justify need by reviewing existing CPRs.

We examined firstly whether authors cited existing cardiovascular CPRs in derivation studies. To understand why authors proceed to develop a new CPR when previous CPRs exist, we then studied the stated insufficiencies of existing cardiovascular CPRs, according to the authors of derivation studies. Lastly, a survey was conducted to examine why some authors cited existing cardiovascular CPRs and others did not.

Materials and methods

Information source and selection of derivation studies

We used cardiovascular CPRs included in the International Register of Clinical Prediction Rules for Primary Care. The development of the international register and its contents have been reported previously in detail [7]. The international register is composed of 745 articles retrieved by searching Medline (PubMed) for CPR literature published between 1980 and 2009, searching secondary sources, and contacting experts in CPR research [7]. It contains 434 CPRs across 17 clinical domains according to the International Classification of Primary Care, Second Edition (ICPC-2) [7, 16]. The cardiovascular domain includes 138 CPRs for various cardiovascular problems in 17 clinical sub-domains such as acute myocardial infarction (K75), heart failure (K77), and pulmonary embolism (K93).

All articles included in the cardiovascular domain of the international register were considered. Eligibility was not restricted to any derivation year, population, setting or language used. For an article to be eligible, authors must have developed a new cardiovascular CPR from the ground up by selecting a unique set of predictor variables. Articles were excluded if they exclusively updated existing cardiovascular CPRs: (1) the intercept and coefficients of an existing CPR were modified using the calibration intercept and slope (recalibration), (2) the intercept and coefficients of an existing CPR were recalculated (re-estimation), or (3) some predictors of an existing CPR were added, removed, or replaced (modification) [1721]. We also excluded review articles that presented CPRs without describing how they were developed. Articles that lacked an introduction section were also excluded (e.g. conference abstracts).

Descriptive review of derivation studies

Two authors (JB and EW) independently extracted data. Disagreements were adjudicated by discussion, and if agreement could not be reached by a third reviewer. For each eligible derivation study, the following information was recorded: (1) bibliographic data (authors and publication year), (2) the country where the CPR was developed, (3) the type of CPR (diagnostic or prognostic), (4) the sub-domain of the international register that the CPR belongs to, and (5) the citation of existing cardiovascular CPR.

A CPR was classified as a diagnostic rule if it assessed the presence of a condition at the time of the prediction [15, 22]. A CPR was classified as a prognostic rule if it assessed a future occurrence of an outcome after the prediction [15, 22]. We considered a derivation study to be citing an existing CPR when an introduction section included a sentence discussing an existing cardiovascular CPR and a relevant bibliographic reference. The derivation studies were divided into 3 groups according to the citation of existing cardiovascular CPR: derivation studies that (1) cited existing CPR, (2) did not cite any existing CPR, and (3) declared there was no CPR to cite for the cardiovascular condition of interest.

Content analysis

We examined the introduction sections of the derivation studies that cited existing cardiovascular CPRs. A thematic content analysis was conducted to produce a typology of the stated insufficiencies of existing CPRs according to the authors of these derivation studies [23]. The thematic content analysis focused on manifest content rather than latent content because reliably interpreting implicit meaning of content could be difficult. Since there was no prior knowledge about why new cardiovascular CPRs are created, we used an inductive approach to develop a coding scheme by allowing themes to emerge from the examination of data.

Derivation studies that cited existing cardiovascular CPRs were imported to Nvivo for Mac (QSR International Pty Ltd). To become familiarized with data, one of the authors (JB) repeatedly read the introduction section of each derivation study. Then, the author (JB) identified sentences that contained any reason why an existing CPR was insufficient to address the cardiovascular problem. A preliminary code was created using key words from each sentence.

After creating preliminary codes for 30% of randomly sampled derivation studies, a coding scheme was generated by combining preliminary codes with an overlapping theme. The remaining derivation studies were coded according to the coding scheme. A new code was generated when there was content that described an insufficiency of existing cardiovascular CPRs but was not compatible with any existing codes. Previously coded derivation studies were re-coded using the updated coding scheme. Data within each code were reviewed to confirm that they were consistent with the assigned code. The code was divided when an inconsistency was found. Similar codes were organized into categories.

The results of the content analysis are presented by describing each category with associated texts from derivation studies. The number of derivation studies that provided each category of reasons as to why existing cardiovascular CPRs were insufficient is reported.

Survey of authors

A survey was conducted to understand why some authors cited existing cardiovascular CPRs in their derivation studies and others did not. The Medical Sciences Interdivisional Research Ethics Committee (MS IDREC) of the University of Oxford conducted an ethical review and approved the survey (reference number: MS-IDREC-C1-2015-161).

Target survey participants were the corresponding authors of the derivation studies in the descriptive review described above. When a corresponding author could not be reached, we attempted to contact the first author. E-mail and postal addresses of the potential target participants were recorded from each derivation study and were updated by searching in PubMed and the internet.

In November 2015, each target participant was contacted first by sending a personalized e-mail which included a link to an online questionnaire and a participant invitation letter. A second e-mail was sent if the author did not respond within eight weeks. We posted a personalized survey packet to authors whose e-mail address was not known or who did not respond to a third e-mail invitation. Authors with known postal addresses in the United States (US), Canada or United Kingdom (UK) were contacted. The survey packet included a participant invitation letter, a questionnaire, and a stamped return envelope. The participant invitation letter outlined the purpose of the survey, why the author was chosen, the length of survey, implied consent by completing, confidentiality of data collected, and how to raise any concern.

The survey was developed by considering possible reasons why authors did not discuss existing cardiovascular CPRs. These include: (1) authors were not aware of any existing cardiovascular CPR, (2) authors did not search for existing cardiovascular CPRs, and (3) authors did not believe citing exiting cardiovascular CPR was important. The final survey consisted of three closed-ended questions (S1 Appendix). It was piloted among researchers from the Nuffield Department of Primary Care Health Sciences, University of Oxford, who had experience in developing CPRs.

Participants were asked to confirm they had read and understood the information about the survey by checking a box. They were also required to enter a unique alphanumeric code before starting the survey. No incentive was offered for participating.

Statistical analysis

The characteristics of derivation studies were reported by absolute and relative frequencies for categorical variables. For non-normally distributed continuous variables, the characteristics of derivation studies were presented with medians and interquartile ranges. The answers to survey questions were presented with absolute and relative frequencies. We used Fisher’s exact test to assess the null hypothesis that answers to survey questions are same between derivation studies that cited existing CPR, did not cite any existing CPR, and declared there was no CPR to cite. Stata (Release 14. College Station, TX: StataCorp LP) was used for all analyses.

Results

Review of derivation studies

The cardiovascular domain of the international register of CPRs contained 138 CPRs from 131 articles. Of the 131 articles, 46 articles were excluded: 35 articles updated existing CPRs, 10 that presented CPRs in reviews or guidelines without describing how they were developed, and one conference abstract without an introduction section. Eighty-five derivation studies met the inclusion criteria. Of these, 48 derivation studies (56.5%) cited at least one existing CPR, 33 derivation studies (38.8%) did not cite any existing CPR, and 4 derivation studies (4.7%) declared there was no existing CPR to cite. Characteristics of the derivation studies are presented in Table 1.

thumbnail
Table 1. Characteristics of derivation studies according to the citation of existing cardiovascular prediction rule.

https://doi.org/10.1371/journal.pone.0179102.t001

Of 85 derivation studies of cardiovascular CPRs, 12 reported more than one CPR with different number of predictor variables. The median number of predictor variables included in cardiovascular CPRs was 6 (IQR of 4 to 8 when counting the smallest final models and 4.5 to 9 when counting the largest final models). The median publication year for 48 studies that cited existing CPR was 2002 (IQR 1995, 2005) and for 33 studies that did not cite existing CPR was 1999 (IQR 1995, 2002). There was only one derivation study published before 1980 [24]. The majority of derivation studies (57.6%) were published after 2000. The proportion of derivation studies that cited existing cardiovascular CPRs increased from 46.2% between 1980 and 1989, to 50.0% between 1990 and 1999, and to 61.2% between 2000 and 2009.

Fifty derivation studies (58.8%) were conducted in the US or the UK. These studies included 218,447 participants (87.1% of all participants). The median numbers of participants in the derivation studies were 1,253 (IQR 526, 5711). The largest derivation study was conducted in the UK and included 1,897,727 participants [25]. The smallest derivation study was from India and included 43 participants [26].

Content analysis

A coding scheme was created by reviewing the introduction sections of 13 randomly selected derivation studies that cited existing CPRs. This initial coding scheme included 21 codes each representing a unique stated insufficiency of existing CPRs (e.g. important predictors were missed, predictors were dichotomized, predictors were not available routinely). While coding 31 remaining derivation studies, four more codes were identified and the coding scheme was updated. Using the updated coding scheme, all 44 derivation studies were re-coded.

The following five categories of insufficiency of existing CPRs emerged when linked codes were aggregated and organized in a hierarchical structure: (1) derivation related, (2) construct related, (3) performance related, (4) transferability related, and (5) evidence related insufficiencies. In addition, some studies simply mentioned existing cardiovascular CPRs without discussing why they were insufficient. The categories of insufficiencies reported in derivation studies of cardiovascular CPRs are presented in S1 Data.

(1) Derivation related insufficiency.

Five studies (11.4%) stated that existing cardiovascular CPRs had insufficiencies related to their derivation process [24, 2730]. Four studies stated that the derivations were conducted using biased methods (e.g. selection bias) [24, 2830] and one study stated that the existing CPRs were developed without the intention of clinical use [27]. For example, Wang et al. [29] reported that the method used to derive one of the prognostic rules for atrial fibrillation (AF) was flawed:

“They tested this scheme using Medicare claims data from patients who were hospitalized for AF but did not receive anticoagulation therapy. A potential limitation of this approach is selection bias, because clinical features associated with nonuse of warfarin or hospitalization for AF are likely to influence stroke risk. Also, some strokes may be missed by hospital discharge data if they are small, immediately lethal, or improperly coded.”

(2) Construct related insufficiency.

Thirty-one derivation studies (70.4%) stated that existing CPRs had insufficiencies related to their construct [25, 2857]. Twenty derivation studies (45.4%) mentioned problems with predictor variables of existing CPRs including predictor variables were missed, unavailable in daily clinical practice, clinically insensible, inaccurate, irreproducible, not standardized, not clearly defined, or dichotomized [25, 28, 3032, 35, 36, 39, 41, 4346, 4851, 54, 55, 57]. The following quotations illustrate some of the issues related to predictor variables:

“[T]he Framingham algorithm does not include factors such as social deprivation, body mass index, family history of cardiovascular disease, and current treatment with antihypertensives.”

[25]

“A more fundamental limitation is that the MMPS, and other currently available analogous mortality predictors, use clinical information from the first 24 hours or more not available upon presentation to the hospital,… they cannot be used immediately in the real-time clinical setting.”

[35]

Twelve derivation studies argued that the existing CPRs’ outcomes were irrelevant [29, 33, 34, 39, 40, 42, 47, 48, 50, 52, 53, 56]. One example is shown here:

“Available risk scores do not include mortality as an end point although studies have indicated that AF is an independent risk factor for death as well as stroke and therapies for AF may affect mortality.”

[29]

There were five derivation studies that described issues with presentation of CPRs (e.g. does not produce absolute risk) [25, 29, 42, 49, 51]. Eight studies discussed difficulties of using existing CPRs due to complex calculations, need for a calculator or computer, and large numbers of predictor variables [30, 31, 33, 35, 37, 38, 40, 48].

(3) Performance related insufficiency.

Authors of 10 derivation studies (22.7%) stated that existing CPRs performed disappointingly [25, 28, 30, 35, 37, 53, 56, 5860]. Three of these studies discussed inadequate performance of existing CPRs in a subgroup of population (e.g. socially deprived) [25, 35, 60]. Subramaniam et al. [30] justified developing the Hamilton score by highlighting the inaccuracy of Wells score for Deep Vein Thrombosis (DVT):

“The modified Wells score has limitations in discriminating patients likely to have DVT and those unlikely to have DVT. In a study involving 1,096 ambulatory outpatients,… identified… 495 patients (45.2%) as likely to have DVT using the modified Wells score. This is despite the ambulatory population, which is expected to have a lower risk for DVT than hospital inpatients.”

(4) Transferability related insufficiency.

Authors of 13 derivation studies (29.5%) stated that existing CPRs had limitations in their transferability [25, 28, 29, 36, 43, 47, 50, 56, 6165]. Seven derivation studies discussed transferability constraints due to differences in key characteristics of participants such as age, sex, and co-morbidities [25, 29, 36, 43, 61, 63, 65]. Five studies reported that differences in disease spectrum may hinder CPR transferability [29, 36, 50, 56, 65]. Some studies discussed dissimilarities in disease prevalence [25, 47, 61] and setting [28, 62, 64] as potential obstacles. Authors of one derivation study discussed concern about using an existing CPR developed in a different setting:

“These studies have been made of selected groups of patients in rehabilitation units. The results therefore may not be applicable to general hospital stroke admissions.”

[62]

(5) Evidence related insufficiency.

Eight derivation studies (18.2%) cited paucity of evidence in the development of existing CPRs as the reason to create a new rule [25, 27, 31, 37, 47, 53, 57, 61]. Lack of validation and difficulties in updating the CPR in a local population were frequently cited [27, 37, 47, 57, 61]. Uncertainties regarding the impact [25, 53] and lack of uptake in clinical practice [27, 31] were also mentioned.

Lack of validation of existing CPRs was the reason for deriving a new CPR in the following example:

“Although pregnancy is recognized as a risk factor for venous thrombosis, no prospective studies validate the use of current diagnostic strategies for DVT.”

[61]

(6) Simple citation.

Authors of three derivation studies (6.8%) simply cited existing CPRs without clearly explaining why these CPRs were insufficient and a new CPR was needed [26, 66, 67]. For example, authors of the derivation study in the following example just listed available prediction models without stating why they needed to develop a new CPR:

“Two stroke risk prediction models have been published: the Framingham Study model, and the Israeli Ischemic Heart Disease Project (IIHD) model.”

[67]

Survey of authors

Of the 85 derivation studies in the review, we were able to contact the authors of 76 derivation studies by e-mail, post or both (Fig 1). Authors of 54 derivation studies responded (39 by e-mail and 15 by post) giving a response rate of 71.1%. Characteristics of derivation studies where a response from an author was received, authors were contacted but no response was received, and no author could be contacted are presented in S1 Table.

The results of the author survey are presented in Table 2. Of 54 authors who responded, 25 (46.3%) indicated they were aware of existing CPRs that addressed the same clinical problem at the time of derivation. We found that 33.3% of authors who did not cite existing CPR at the time of derivation stated that they were aware of existing CPR at the time of survey. There was no clear association between stated awareness of existing CPRs and citation of these rules (Fisher’s exact p = 0.11). To become aware of existing CPRs, 29 authors (53.7%) conducted either a full or a limited systematic review and 15 authors (27.8%) only used non-systematic methods (searching the team’s reference collection or consulting experts). Nine authors (16.7%) did not search for existing CPRs. Type of search method was not clearly associated with citation of existing CPRs (Fisher’s exact p = 0.19). Most authors (90.7%) thought citing existing CPRs in a derivation study was either very important or somewhat important. There was no clear evidence that authors who indicated it was important to cite existing CPRs were more likely to do so (Fisher’s exact p = 0.71).

thumbnail
Table 2. Survey responses according to the citation of existing cardiovascular prediction rule.

https://doi.org/10.1371/journal.pone.0179102.t002

Discussion

Summary of main findings

The first objective of this study was to assess whether authors cited existing cardiovascular CPRs when developing new cardiovascular rules. We found many authors did not cite existing cardiovascular CPRs with only 56.5% of derivation studies citing at least one existing cardiovascular CPR. Our results also suggest that the citation of existing cardiovascular CPRs has improved over time.

Five categories of existing CPRs’ insufficiency was identified from the derivation studies. These included insufficiencies related to: (1) derivation, (2) construct, (3) performance, (4) transferability, and (5) evidence. The most commonly cited category (70.4%) was insufficiency related to the construct of existing CPRs such as issues in choice of predictor variables, outcomes, clinical usability, or presentation format. The next most common category (29.5%) was the insufficiency of existing CPRs’ transferability due to differences in population or setting of care.

Some authors justified the need for developing new cardiovascular CPRs by highlighting the insufficiencies of existing CPRs. Instead of creating new CPRs, some of these insufficiencies may be addressed by updating existing CPRs. For example, when an existing cardiovascular CPR is missing an important predictor variable, it may be updated by adding the predictor variable and recalculating the regression coefficients and intercept [17, 19, 20]. When an existing CPR does not perform well in a new population or setting, re-calibrating the intercept and slope or re-estimating coefficients may optimize the performance [19, 20]. When using a CPR is difficult because of complex calculations, presenting the CPR in a simpler and more user-friendly format (e.g. score chart) might resolve the problem [20, 68]. Lastly, some authors developed new CPRs mentioning transferability constraints of existing CPRs due to differences in key participant characteristics. The appropriate next step may be to conduct an external validation study to evaluate CPR performance, despite the differences in key participant characteristics [4, 69, 70]. The external validation study will also provide an opportunity to update the CPR in new populations [71]. If a new CPR was developed for every time there is a difference in characteristics, there will be numerous CPRs for the same problem.

We also evaluated why some authors cited existing CPRs in their derivation studies and others did not. However, the results of our survey did not explain the reasons for this difference: neither being aware of existing CPRs, having conducted a systematic review of existing CPRs, nor believing it is important to cite existing CPRs was clearly associated with citing existing cardiovascular CPRs.

One of the most interesting findings of the survey was that most authors believed citing existing CPRs was important with over 90% agreeing it was important to cite existing cardiovascular CPRs in derivation studies. Given this, it is difficult to justify the poor citation rate found in this study.

Comparison with existing literature

The TRIPOD statement recommends authors present a rationale for developing a new CPR with references to existing CPRs [15]. We have shown that citation of existing CPRs is suboptimal in derivation studies of cardiovascular CPRs. This is similar to findings from previous reports that showed existing research evidence is often inadequately examined or ignored when conducting randomized controlled trials (RCTs) [7275]. For example, Clarke and Hopewell [73] have shown that only 68.6% of RCTs published in four major general medicine journals discussed existing evidence in the introduction section by citing other clinical trials (28.6%) or a systematic review (40.0%).

It has been suggested that new research should be justified by discussing systematic reviews of existing evidence [13, 72, 73]. In our study, only three derivation studies (3.5%) cited a systematic review of existing cardiovascular CPRs [25, 38, 45] which is substantially less than citations of systematic reviews in RCTs [73]. Although more than half of authors who responded to our survey indicated they conducted a systematic review to identify existing CPRs, none of the included studies reported this. Ideally, authors of derivation studies should systematically review, discuss, and cite all existing CPRs relevant to clinical context as a part of justifying their research [46]. At the bare minimum, authors should refer to those CPRs of which they are aware or the best known in the field when conducting a systematic review is impractical.

Strengths and limitations

To our knowledge, this is the first study to show that existing CPRs are inadequately cited in the introduction section of published derivation studies. We supplemented the review of derivation studies with a thematic content analysis and further augmented the understanding of problem by an author survey.

We used the International Register of Clinical Prediction Rules for Primary Care as the data source. Although published in 2014, the international register currently only includes CPRs published up to 2009. Therefore, it is important to consider that all derivation studies in our review predate the publication of TRIPOD statement and many of them predate the modern expectation that a systematic review of existing evidence should be done before any new research [13, 15, 72, 73]. In addition, we assessed only CPRs from the cardiovascular domain of the international register. Therefore, our findings may not be generalizable to the CPRs in other clinical areas or more recently developed CPRs.

The small number of the authors surveyed might have made it difficult to detect the associations between citation of existing CPR and the survey answers. We received responses from more than 70% of authors contacted by structuring a simple and short survey. The main tradeoff was that we could not include any open-ended questions that might have generated richer survey data and led to more in-depth analysis of the issue. Recall bias should also be considered when interpreting our survey results because two questions relied on authors’ recollection of information from past CPR development.

Some authors indicated in their survey response that they were unaware of an existing CPR at the time of derivation which was at odds with the review of their derivation studies which demonstrated that existing CPRs were cited. These discrepancies might have arisen from the different interpretations of the term “same clinical problem” in the first survey question. Some of these authors acknowledged there were existing CPRs for the problems in introduction sections but indicated existing CPRs were inapplicable to their populations or practice settings. The differences in setting or population that existing CPRs were developed might have lead some authors to determine these CPRs did not address the “same clinical problem”.

Research and clinical implications

We speculate that researchers keep deriving new CPRs when CPRs already exist for the same clinical problem for the following reasons. Firstly, researchers may be more familiar with methods for developing a new CPR (e.g. logistic regression) than methods needed to update an existing CPR (e.g. recalibration). Secondly, clinical data for developing a CPR are increasingly available from widespread use of electronic health records [76]. Thirdly, researchers may be attracted to creating a new CPR because they view this as more novel and therefore more likely to be published in higher impact journals. Lastly, there were no standardized reporting guidelines for authors to provide a clear rationale for developing a new CPR before the publication of the TRIPOD statement in 2015.

CPRs are notably more common for some conditions (e.g. cardiovascular, respiratory, musculoskeletal) but seldom developed for many clinical areas (e.g. eye, ear, pregnancy related) [7]. Researchers should systematically search for any existing CPRs relevant to their context before deciding to develop a new CPR. When there is no existing CPR for the problem and there is an unmet clinical need, developing a CPR is well justified.

When a systematic evaluation of existing CPRs identifies a well-built CPR with good performance in external validations, researchers should focus on implementing the CPR and assessing the impact in clinical practice. Then clinicians can confidently adopt a CPR that has been externally validated and proven to improve relevant clinical outcomes. For example, the Ottawa ankle rules showed robust predictive performance in wide range of populations and settings [7779] and reduced utilization of radiography without jeopardizing clinical outcomes in impact studies [8083]. Many GPs recognize and use Ottawa ankle rules in clinical practice despite lack of clinical guidelines on use of these CPRs at the point of care [9]. When updated, the International Register of CPRs can be an important source for setting research priorities and identifying CPRs ready to be tested for implementation in clinical practice.

When existing CPRs for the clinical problem are found but some insufficiencies are also identified, researchers should attempt to resolve these insufficiencies by validating and updating the existing CPR. When it is not possible to resolve these insufficiencies, developing a new CPR can be justified. This approach may reduce research waste in CPR development in the following ways. Firstly, the knowledge generated from researchers’ time and efforts in derivation studies is used in planning new research. When the only reason participants were recruited was to develop a new CPR, their contribution is not thrown away. Lastly, this approach may prevent unnecessarily adding new CPRs for the same clinical problem.

Conclusion

This is the first study to demonstrate that existing CPRs are poorly cited in the derivation studies of cardiovascular CPRs. Almost all authors of cardiovascular CPR believe citing existing CPRs is important and the TRIPOD statement also recommends to do so. Authors developing a new cardiovascular CPR should provide a clear rationale with citations to existing CPRs to avoid unnecessary duplication.

Supporting information

S1 Appendix. Questionnaire for authors of cardiovascular prediction rules.

https://doi.org/10.1371/journal.pone.0179102.s001

(TIF)

S1 Data. Insufficiencies of existing cardiovascular prediction rules.

https://doi.org/10.1371/journal.pone.0179102.s002

(XLSX)

S1 Table. Characteristics of derivation studies that authors responded, did not respond and could not be contacted.

https://doi.org/10.1371/journal.pone.0179102.s003

(DOCX)

Acknowledgments

We would like to thank Dr. Clare Bankhead and Prof. Ann Van den Bruel for providing invaluable comments on this project.

Author Contributions

  1. Conceptualization: JB RS RP.
  2. Data curation: JB.
  3. Formal analysis: JB.
  4. Funding acquisition: JB.
  5. Investigation: JB EW RS RP.
  6. Methodology: JB RS RP.
  7. Project administration: JB RS RP.
  8. Resources: JB.
  9. Supervision: RS RP.
  10. Validation: JB EW.
  11. Visualization: JB EW RS RP.
  12. Writing – original draft: JB.
  13. Writing – review & editing: JB EW RS RP.

References

  1. 1. Laupacis A, Sekar N, Stiell IG. Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA: the journal of the American Medical Association. 1997;277(6):488–94. Epub 1997/02/12. pmid:9020274.
  2. 2. McGinn TG, Guyatt GH, Wyer PC, Naylor CD, Stiell IG, Richardson WS. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA: the journal of the American Medical Association. 2000;284(1):79–84. Epub 2000/06/29. pmid:10872017.
  3. 3. Falk G, Fahey T. Clinical prediction rules. Bmj. 2009;339:b2899. Epub 2009/08/12. pmid:19666679.
  4. 4. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Annals of internal medicine. 2006;144(3):201–9. Epub 2006/02/08. pmid:16461965.
  5. 5. Wallace E, Smith SM, Perera-Salazar R, Vaucher P, McCowan C, Collins G, et al. Framework for the impact analysis and implementation of Clinical Prediction Rules (CPRs). BMC medical informatics and decision making. 2011;11:62. Epub 2011/10/18. pmid:21999201.
  6. 6. Damen JA, Hooft L, Schuit E, Debray TP, Collins GS, Tzoulaki I, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. Bmj. 2016;353:i2416. pmid:27184143;
  7. 7. Keogh C, Wallace E, O'Brien KK, Galvin R, Smith SM, Lewis C, et al. Developing an international register of clinical prediction rules for use in primary care: a descriptive analysis. Annals of family medicine. 2014;12(4):359–66. pmid:25024245;
  8. 8. Wessler BS, Lai Yh L, Kramer W, Cangelosi M, Raman G, Lutz JS, et al. Clinical Prediction Models for Cardiovascular Disease: Tufts Predictive Analytics and Comparative Effectiveness Clinical Prediction Model Database. Circ Cardiovasc Qual Outcomes. 2015;8(4):368–75. pmid:26152680;
  9. 9. Pluddemann A, Wallace E, Bankhead C, Keogh C, Van der Windt D, Lasserson D, et al. Clinical prediction rules in practice: review of clinical guidelines and survey of GPs. The British journal of general practice: the journal of the Royal College of General Practitioners. 2014;64(621):e233–42. pmid:24686888;
  10. 10. Counsell C, Dennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis. 2001;12(3):159–70. pmid:11641579.
  11. 11. Rahimi K, Bennett D, Conrad N, Williams TM, Basu J, Dwight J, et al. Risk prediction in patients with heart failure: a systematic review and analysis. JACC Heart Fail. 2014;2(5):440–6. pmid:25194291.
  12. 12. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. pmid:24411644.
  13. 13. Lund H, Brunnhuber K, Juhl C, Robinson K, Leenaars M, Dorch BF, et al. Towards evidence based research. Bmj. 2016;355:i5440. pmid:27797786.
  14. 14. Clarke M. Doing new research? Don't forget the old. PLoS medicine. 2004;1(2):e35. pmid:15578106;
  15. 15. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Annals of internal medicine. 2015;162(1):55–63. pmid:25560714.
  16. 16. World Organization of National Colleges Academies and Academic Associations of General Practitioners/Family Physicians. Classification Committee. ICPC-2: international classification of primary care. 2nd ed. Oxford: Oxford University Press; 1998. x, 190 p. p.
  17. 17. van Houwelingen HC. Validation, calibration, revision and combination of prognostic survival models. Statistics in medicine. 2000;19(24):3401–15. pmid:11122504.
  18. 18. Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y. Updating methods improved the performance of a clinical prediction model in new patients. Journal of clinical epidemiology. 2008;61(1):76–86. pmid:18083464.
  19. 19. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: a review. Journal of clinical epidemiology. 2008;61(11):1085–94. Epub 2009/02/12. pmid:19208371.
  20. 20. Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009. xxviii, 497 p. p.
  21. 21. Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98(9):691–8. pmid:22397946.
  22. 22. Hendriksen JM, Geersing GJ, Moons KG, de Groot JA. Diagnostic and prognostic prediction models. Journal of thrombosis and haemostasis: JTH. 2013;11 Suppl 1:129–41. pmid:23809117
  23. 23. Green J, Thorogood N. Qualitative methods for health research. Third edition. ed. London: SAGE Publications Ltd; 2014. xvii, 342 pages p.
  24. 24. Norris RM, Brandt PW, Caughey DE, Lee AJ, Scott PJ. A new coronary prognostic index. Lancet. 1969;1(7589):274–8. pmid:4178977.
  25. 25. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, May M, Brindle P. Derivation and validation of QRISK, a new cardiovascular disease risk score for the United Kingdom: prospective open cohort study. Bmj. 2007;335(7611):136. pmid:17615182;
  26. 26. Jain AP, Gupta OP, Jajoo UN, Kumar A, Trivedi SK. Evaluation of clinical variables as predicators of left ventricular function in acute myocardial infarction. J Assoc Physicians India. 1993;41(1):17–9. pmid:8340320.
  27. 27. Kernan WN, Horwitz RI, Brass LM, Viscoli CM, Taylor KJ. A prognostic system for transient ischemia or minor stroke. Annals of internal medicine. 1991;114(7):552–7. pmid:2001088.
  28. 28. Wicki J, Perneger TV, Junod AF, Bounameaux H, Perrier A. Assessing clinical probability of pulmonary embolism in the emergency ward: a simple score. Archives of internal medicine. 2001;161(1):92–7. pmid:11146703.
  29. 29. Wang TJ, Massaro JM, Levy D, Vasan RS, Wolf PA, D'Agostino RB, et al. A risk score for predicting stroke or death in individuals with new-onset atrial fibrillation in the community: the Framingham Heart Study. JAMA: the journal of the American Medical Association. 2003;290(8):1049–56. pmid:12941677.
  30. 30. Subramaniam RM, Chou T, Heath R, Allen R. Importance of pretest probability score and D-dimer assay before sonography for lower limb deep venous thrombosis. AJR Am J Roentgenol. 2006;186(1):206–12. pmid:16357403.
  31. 31. Pryor DB, Harrell FE Jr., Lee KL, Califf RM, Rosati RA. Estimating the likelihood of significant coronary artery disease. Am J Med. 1983;75(5):771–80. pmid:6638047.
  32. 32. Tierney WM, Roth BJ, Psaty B, McHenry R, Fitzgerald J, Stump DL, et al. Predictors of myocardial infarction in emergency room patients. Crit Care Med. 1985;13(7):526–31. pmid:4006491.
  33. 33. Mark DB, Hlatky MA, Harrell FE Jr., Lee KL, Califf RM, Pryor DB. Exercise treadmill score for predicting prognosis in coronary artery disease. Annals of internal medicine. 1987;106(6):793–800. pmid:3579066.
  34. 34. McNamara RF, Carleen E, Moss AJ. Estimating left ventricular ejection fraction after myocardial infarction by various clinical parameters. Am J Cardiol. 1988;62(4):192–6. pmid:3400597.
  35. 35. Selker HP, Griffith JL, D'Agostino RB. A time-insensitive predictive instrument for acute myocardial infarction mortality: a multicenter study. Med Care. 1991;29(12):1196–211. pmid:1745078.
  36. 36. Hubbard BL, Gibbons RJ, Lapeyre AC 3rd, Zinsmeister AR, Clements IP. Identification of severe coronary artery disease using simple clinical parameters. Archives of internal medicine. 1992;152(2):309–12. pmid:1739359.
  37. 37. Silver MT, Rose GA, Paul SD, O'Donnell CJ, O'Gara PT, Eagle KA. A clinical rule to predict preserved left ventricular ejection fraction in patients after myocardial infarction. Annals of internal medicine. 1994;121(10):750–6. pmid:7944052.
  38. 38. Morise AP, Haddad WJ, Beckner D. Development and validation of a clinical score to estimate the probability of coronary artery disease in men and women presenting with suspected coronary disease. Am J Med. 1997;102(4):350–6. pmid:9217616.
  39. 39. Selker HP, Griffith JL, Beshansky JR, Schmid CH, Califf RM, D'Agostino RB, et al. Patient-specific predictions of outcomes in myocardial infarction for real-time emergency use: a thrombolytic predictive instrument. Annals of internal medicine. 1997;127(7):538–56. pmid:9313022.
  40. 40. Kidwell CS, Saver JL, Schubert GB, Eckstein M, Starkman S. Design and retrospective analysis of the Los Angeles Prehospital Stroke Screen (LAPSS). Prehosp Emerg Care. 1998;2(4):267–73. pmid:9799012.
  41. 41. Wells PS, Ginsberg JS, Anderson DR, Kearon C, Gent M, Turpie AG, et al. Use of a clinical model for safe management of patients with suspected pulmonary embolism. Annals of internal medicine. 1998;129(12):997–1005. pmid:9867786.
  42. 42. Pocock SJ, McCormack V, Gueyffier F, Boutitie F, Fagard RH, Boissel JP. A score for predicting risk of death from cardiovascular disease in adults with raised blood pressure, based on individual patient data from randomised controlled trials. Bmj. 2001;323(7304):75–81. pmid:11451781;
  43. 43. Stevens RJ, Kothari V, Adler AI, Stratton IM, United Kingdom Prospective Diabetes Study G. The UKPDS risk engine: a model for the risk of coronary heart disease in Type II diabetes (UKPDS 56). Clin Sci (Lond). 2001;101(6):671–9. pmid:11724655.
  44. 44. Wang SJ, Ohno-Machado L, Fraser HS, Kennedy RL. Using patient-reportable clinical history factors to predict myocardial infarction. Comput Biol Med. 2001;31(1):1–13. pmid:11058690.
  45. 45. Farkouh M, Oddone E, Simel D. Improving the clinical examination for a low ankle-brachial index. International Journal of Angiology. 2002;11(1):41–5.
  46. 46. Moons KG, Bots ML, Salonen JT, Elwood PC, Freire de Concalves A, Nikitin Y, et al. Prediction of stroke in the general population in Europe (EUROSTROKE): Is there a role for fibrinogen and electrocardiography? J Epidemiol Community Health. 2002;56 Suppl 1:i30–6. pmid:11815642;
  47. 47. Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, De Backer G, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. European heart journal. 2003;24(11):987–1003. pmid:12788299.
  48. 48. Lee DS, Austin PC, Rouleau JL, Liu PP, Naimark D, Tu JV. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA: the journal of the American Medical Association. 2003;290(19):2581–7. pmid:14625335.
  49. 49. van Walraven C, Hart RG, Wells GA, Petersen P, Koudstaal PJ, Gullov AL, et al. A clinical prediction rule to identify patients with atrial fibrillation and a low risk for stroke while taking aspirin. Archives of internal medicine. 2003;163(8):936–43. pmid:12719203.
  50. 50. Auble TE, Hsieh M, Gardner W, Cooper GF, Stone RA, McCausland JB, et al. A prediction rule to identify low-risk patients with heart failure. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine. 2005;12(6):514–21. pmid:15930402.
  51. 51. Aujesky D, Obrosky DS, Stone RA, Auble TE, Perrier A, Cornuz J, et al. Derivation and validation of a prognostic model for pulmonary embolism. Am J Respir Crit Care Med. 2005;172(8):1041–6. pmid:16020800;
  52. 52. Rothwell PM, Giles MF, Flossmann E, Lovelock CE, Redgrave JN, Warlow CP, et al. A simple score (ABCD) to identify individuals at high early risk of stroke after transient ischaemic attack. Lancet. 2005;366(9479):29–36. pmid:15993230.
  53. 53. Christenson J, Innes G, McKnight D, Thompson CR, Wong H, Yu E, et al. A clinical prediction rule for early discharge of patients with chest pain. Annals of emergency medicine. 2006;47(1):1–10. pmid:16387209.
  54. 54. Lauer MS, Pothier CE, Magid DJ, Smith SS, Kattan MW. An externally validated model for predicting long-term survival after exercise treadmill testing in patients with suspected coronary artery disease and a normal electrocardiogram. Annals of internal medicine. 2007;147(12):821–8. pmid:18087052.
  55. 55. Ridker PM, Buring JE, Rifai N, Cook NR. Development and validation of improved algorithms for the assessment of global cardiovascular risk in women: the Reynolds Risk Score. JAMA: the journal of the American Medical Association. 2007;297(6):611–9. pmid:17299196.
  56. 56. Uresandi F, Otero R, Cayuela A, Cabezudo MA, Jimenez D, Laserna E, et al. [A clinical prediction rule for identifying short-term risk of adverse events in patients with pulmonary thromboembolism]. Arch Bronconeumol. 2007;43(11):617–22. pmid:17983546.
  57. 57. Gagne P, Simon L, Le Pape F, Bressollette L, Mottier D, Le Gal G, et al. [Clinical prediction rule for diagnosing deep vein thrombosis in primary care]. Presse Med. 2009;38(4):525–33. pmid:19062245.
  58. 58. Kline JA, Mitchell AM, Kabrhel C, Richman PB, Courtney DM. Clinical criteria to prevent unnecessary diagnostic testing in emergency department patients with suspected pulmonary embolism. Journal of thrombosis and haemostasis: JTH. 2004;2(8):1247–55. pmid:15304025.
  59. 59. Younge BR, Cook BE Jr., Bartley GB, Hodge DO, Hunder GG. Initiation of glucocorticoid therapy: before or after temporal artery biopsy? Mayo Clin Proc. 2004;79(4):483–91. pmid:15065613.
  60. 60. Woodward M, Brindle P, Tunstall-Pedoe H, estimation Sgor. Adding social deprivation and family history to cardiovascular risk assessment: the ASSIGN score from the Scottish Heart Health Extended Cohort (SHHEC). Heart. 2007;93(2):172–6. pmid:17090561;
  61. 61. Chan WS, Lee A, Spencer FA, Crowther M, Rodger M, Ramsay T, et al. Predicting deep venous thrombosis in pregnancy: out in "LEFt" field? Annals of internal medicine. 2009;151(2):85–92. pmid:19620161.
  62. 62. Allen CM. Predicting the outcome of acute stroke: a prognostic score. J Neurol Neurosurg Psychiatry. 1984;47(5):475–80. pmid:6736978;
  63. 63. Walker NJ, Sites FD, Shofer FS, Hollander JE. Characteristics and outcomes of young adults who present to the emergency department with chest pain. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine. 2001;8(7):703–8. pmid:11435184.
  64. 64. Oudega R, Moons KG, Hoes AW. Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thromb Haemost. 2005;94(1):200–5. pmid:16113804.
  65. 65. Zhang XF, Attia J, D'Este C, Yu XH, Wu XG. A risk score predicted coronary heart disease and stroke in a Chinese cohort. Journal of clinical epidemiology. 2005;58(9):951–8. pmid:16085199.
  66. 66. Stoffers HE, Kester AD, Kaiser V, Rinkens PE, Knottnerus JA. Diagnostic value of signs and symptoms associated with peripheral arterial occlusive disease seen in general practice: a multivariable approach. Medical decision making: an international journal of the Society for Medical Decision Making. 1997;17(1):61–70. pmid:8994152.
  67. 67. Lumley T, Kronmal RA, Cushman M, Manolio TA, Goldstein S. A stroke prediction score in the elderly: validation and Web-based application. Journal of clinical epidemiology. 2002;55(2):129–36. pmid:11809350.
  68. 68. Sanders S, Flaws D, Than M, Pickering JW, Doust J, Glasziou P. Simplification of a scoring system maintained overall accuracy but decreased the proportion classified as low risk. Journal of clinical epidemiology. 2016;69:32–9. pmid:26070978.
  69. 69. Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Annals of internal medicine. 1999;130(6):515–24. pmid:10075620.
  70. 70. Altman DG, Royston P. What do we mean by validating a prognostic model? Statistics in medicine. 2000;19(4):453–73. pmid:10694730.
  71. 71. D'Agostino RB Sr., Grundy S, Sullivan LM, Wilson P, Group CHDRP. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA: the journal of the American Medical Association. 2001;286(2):180–7. pmid:11448281.
  72. 72. Clarke M, Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. Journal of the Royal Society of Medicine. 2007;100(4):187–90. pmid:17404342;
  73. 73. Clarke M, Hopewell S. Many reports of randomised trials still don't begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc. 2013;24(3):145–8.
  74. 74. Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One. 2014;9(7):e102670. pmid:25068257;
  75. 75. Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Annals of internal medicine. 2011;154(1):50–5. pmid:21200038.
  76. 76. Cook JA, Collins GS. The rise of big clinical databases. Br J Surg. 2015;102(2):e93–e101. pmid:25627139.
  77. 77. Bachmann LM, Kolb E, Koller MT, Steurer J, ter Riet G. Accuracy of Ottawa ankle rules to exclude fractures of the ankle and mid-foot: systematic review. Bmj. 2003;326(7386):417. pmid:12595378;
  78. 78. Dowling S, Spooner CH, Liang Y, Dryden DM, Friesen C, Klassen TP, et al. Accuracy of Ottawa Ankle Rules to exclude fractures of the ankle and midfoot in children: a meta-analysis. Academic emergency medicine: official journal of the Society for Academic Emergency Medicine. 2009;16(4):277–87. Epub 2009/02/04. pmid:19187397.
  79. 79. Jenkin M, Sitler MR, Kelly JD. Clinical usefulness of the Ottawa Ankle Rules for detecting fractures of the ankle and midfoot. J Athl Train. 2010;45(5):480–2. pmid:20831394;
  80. 80. Auleley GR, Ravaud P, Giraudeau B, Kerboull L, Nizard R, Massin P, et al. Implementation of the Ottawa ankle rules in France. A multicenter randomized controlled trial. JAMA: the journal of the American Medical Association. 1997;277(24):1935–9. Epub 1997/06/25. pmid:9200633.
  81. 81. Bessen T, Clark R, Shakib S, Hughes G. A multifaceted strategy for implementation of the Ottawa ankle rules in two emergency departments. Bmj. 2009;339:b3056. Epub 2009/08/14. pmid:19675080;
  82. 82. Stiell I, Wells G, Laupacis A, Brison R, Verbeek R, Vandemheen K, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. Bmj. 1995;311(7005):594–7. Epub 1995/09/02. pmid:7663253;
  83. 83. Stiell IG, McKnight RD, Greenberg GH, McDowell I, Nair RC, Wells GA, et al. Implementation of the Ottawa ankle rules. JAMA: the journal of the American Medical Association. 1994;271(11):827–32. Epub 1994/03/16. pmid:8114236.