Abstract
Background Additional diagnostic means could be of added value when evaluating possible acute coronary syndrome (ACS) in primary care.
Aim To determine whether heart-type fatty acid-binding protein (H-FABP)-based point-of-care (POC) biomarker testing, embedded in a clinical decision rule (CDR), is helpful to the GP when evaluating possible ACS.
Design & setting A prospective, non-randomised, double-blinded, diagnostic derivation study was undertaken, with a delayed-type cross-sectional diagnostic model among GPs in the Netherlands and Belgium.
Method Signs and symptoms predicting acute myocardial infarction (AMI) or ACS were identified using both logistic regression analysis, and classification and regression trees (CART). Diagnostic values of the POC H-FABP test (cut-off value 4 ng/ml) alone and as part of a CDR were determined.
Results A total of 303 participants (48.8% male) with chest pain or discomfort who had consulted a GP were enrolled. ACS was found in 32 (10.6%) of these 303 patients. For ACS, sensitivity and negative predictive value (NPV) of the POC H-FABP test was 25.8% (95% confidence interval [CI] = 12.5 to 44.9) and 91.6% (95% CI = 87.6% to 94.5%), respectively. The area under the receiver operating curve of the optimal CDR was 0.78 for ACS.
Conclusion Sensitivity of the current H-FABP POC test (cut-off value 4 ng/ml) as a stand-alone test is poor, either owing to limitations of the marker or of the test device. Usability of a CDR derived from these results is doubtful: the number of ACS cases missed by the GP is reduced but, as a consequence, disproportionally more ACS-negative patients are referred.
How this fits in
Biomarkers of myocardial cell damage are the cornerstone of ruling in and out ACS in secondary care and might play a role in triaging patients presenting with chest pain in primary care. However, sensitivity of a studied PoC H-FABP test (cut-off 4 ng/ml) as a stand-alone test appeared insufficient to support its use for ruling out ACS in primary care. A CDR was not of added value when compared with the current unaided clinical judgment of the GP. Future studies should focus on highly sensitive and adequately specific tests to improve triage of these patients.
Introduction
Background
In primary care, 1.26% of all consultations are about chest pain.1 Among these patients, cases of ACS are largely outnumbered by less severe causes of chest pain.1–5 Yet, to avoid missing ACS, the GP's threshold to refer patients with chest pain is low.1,5–7 To reduce ACS-negative referrals in primary care and thereby diminish costs and patient discomfort — without a rise in missed cases of ACS — a CDR based on a clinical score of signs, symptoms, and POC biomarker testing detecting myocardial cell damage (such as high-sensitive cardiac troponin [hs-cTn] or H-FABP) is desirable.3,8–17 A potential role for H-FABP POC testing was earlier found using a cut-off value of 4 ng/ml, in an emergency department setting.18
Objectives
This study aimed to:
derive signs and symptoms that have diagnostic value in a CDR to predict or exclude ACS and AMI (stage 2 of the 6 major stages in the development and testing of a CDR, as defined by Stiell and Wells, or step 2 and 3 as defined by Van den Bruel et al).19,20
assess the diagnostic value of a PoC H-FABP test for the outcomes: (1) life-threatening disease; (2) ACS (that is, AMI or unstable angina [UA]); and (3) AMI alone.
assess the diagnostic value of a CDR, including signs, symptoms, and POC H-FABP testing for these outcomes (internal validation).
For an extended background of this study and full details on the methodology, see the authors' previously published research protocol and case reports.21,22
Method
Study design
A prospective, non-randomised, double-blinded, diagnostic derivation study with a delayed-type cross-sectional diagnostic model was performed.23
Recruitment: inclusion and exclusion criteria
It was proposed to let 60 GPs in the southeast of the Netherlands and the northeast of Belgium include 10 patients with chest pain each. For inclusion criteria, see Figure 1. However, GPs faced difficulties in performing the necessary study procedures (for example, informed consent, venous sampling) while being urgently consulted for chest pain of recent onset. Therefore, the study was extended to several out-of-hours (OOH) primary care facilities where incidence of chest pain is higher and specialised nurses are available to aid with study procedures.
Data collection
After obtaining verbal informed consent (for patients in Belgium) or a short version of the written informed consent (for patients in the Netherlands), the standardised case report form was completed by the GP by registration of history, physical examination, electrocardiogram (ECG), if performed, and the presumptive diagnosis together with the original decision whether or not to refer to a cardiologist. Then, a POC H-FABP test with a cut-off value of 4 ng/ml (manufactured by FABPulous BV) was performed. Subsequently, the original GP's decision was followed, unless a decision not to refer a patient was followed by a positive H-FABP test. In those cases, GPs were instructed to overrule their original decision and refer the patient, based on the positive test result. Patients who were not referred to secondary care were instructed to have a venous blood sample drawn for measurement of hs-cTn (as a gold standard to rule out AMI in non-referred patients), creatinine, and estimated glomerular filtration rate, within the interval between 3 hours and 3 days after onset of complaints. Within 7 days of the initial consultation, patients (in both Belgium and the Netherlands) were given the opportunity to complete written informed consent after having read an information letter at a more convenient moment. In the event of refusal at this point, patients were excluded.
Final diagnosis and study outcomes: life-threatening disease, AMI, and ACS
After 30 days, all available patient data were collected by the study team. Final diagnoses were established by an expert panel of one independent GP and one independent cardiologist. The expert panel was blinded to the PoC H-FABP test result. The expert panel had access to all other patient data, including troponin measurements. The final diagnosis was based on these results, including the gold standard troponin test results, if available. All outcome diagnoses were determined following current definitions of conditions as cardiac (ischaemic) disease, stable angina (SA), UA, AMI, and ACS, as extensively detailed in the authors' published research protocol.21 Diseases labelled as life-threatening were: ACS (that is, UA and AMI), acute heart failure, pulmonary embolism, aortic dissection, and sudden death. The main study outcomes were: life-threatening causes, ACS, and AMI.
Statistical analysis
To assess the predictive parameters for the main outcomes, χ2 tests or Pearson correlation analyses of the data were used. Using multiple logistic regression analysis, variables significantly related to ACS and AMI were selected for inclusion in CDRs. Subsequently, several combinations of these parameters and the H-FABP test results were combined to formulate CDRs based on scoring systems with straight cut-off values to aid clinical usability. The diagnostic value of the CDR for the study outcomes of ACS and AMI, respectively, was assessed.24,25 Additionally, CART analysis was performed to identify predicting factors and to formulate a CDR. The CART model was validated using bootstrap aggregation. Within this procedure, two-thirds of the samples were randomly chosen with replacement to build the classification model, while the remaining one-third of the samples were used to internally validate the predictive ability of the CART model. The entire procedure was repeated 1000 times to ensure that most of the samples were selected for training and validating the CART model. CART is based on binary recursive partitioning. The advantages of CART include the fact that it is inherently non-parametric and that the interpretation of results in the trees is plain, making a straightforward translation to clinical practice obvious.26 Data were processed using Matlab (version R2017b, for the CART analysis) and IBM SPSS Statistics for Windows (version 23). P values ≤0.05 were considered statistically significant.
Data management
Handling of the personal data complied with the Dutch Personal Data Protection Act and Belgian privacy legislation. The study is in agreement with the current version of the WMA Declaration of Helsinki and is in accordance with the Dutch Medical Research Involving Human Subjects Act (WMO).27
Results
Patient characteristics
A total of 342 patients were selected for inclusion by the participating GPs, and 39 patients were excluded (see Figure 1). Thus, 303 patients were included in the study. Patient characteristics are shown in Table 1.
Final diagnoses
After 30 days, ACS was the final diagnosis in 32 (10.6%) of the included patients, and other life-threatening disease was found in three additional patients (all diagnosed with heart failure). Two patients (0.7%) died shortly after assessment by the GP. Of all patients with chest pain, 75.2% suffered from non-cardiac disease. All final diagnoses, as assessed by the expert panel, are available from the authors on request.
Study objective 1: diagnostic value of the POC H-FABP test
Table 2 shows diagnostic parameters for all patients, patients presenting within 3 hours after onset of complaints, and patients presenting 3–24 hours after onset of complaints for several outcomes of the POC H-FABP as a stand-alone test, without considering additional signs and symptoms. Extended data are available from the authors on request. For ACS, sensitivity of the POC H-FABP test was 25.8% (36.8% for the subgroup of patients with complaints for more than 3 hours). NPV for ACS was 91.6%.
Test failure was reported in 12 out of 303 cases (4.0%). The POC H-FABP test was positive in 16 cases (5.3%), as shown in Figure 1. In five of these cases, the GP had no intention to refer before testing. Following the study protocol, all five patients were referred. In one of these cases, an ACS was found as a final diagnosis. In the remaining four cases, no cardiac diagnosis was identified. Additional comments and details on false positive and false negative test results are available from the authors on request.
Study objective 2: derivation of a CDR, and univariable and multivariable analyses
Correlations between signs and symptoms and outcome measures were tested in a bivariate analysis (available from the authors on request). For ACS, significant associations were found for age, absence of left-sided lateral chest pain, a feeling of pressure on the chest, pain in right arm or throat, palor or sweating on physical examination, several ECG-abnormalities, and a positive H-FABP PoC test. No significant influence was seen for the parameters 'duration of complaints', 'seen by own GP or seen at OOH service', 'sex', and 'history of cardiovascular disease (CVD)' or 'history of type two diabetes mellitus (DM2)'.
In a multiple logistic regression analysis of all signs and symptoms that revealed significant predictive value in the univariable analysis, five highly significant predictors for ACS and AMI were found: ST-depressions, ST-elevations, dyspnoea, a feeling of pressure on the chest, and absence of left-sided lateral chest pain. CART analysis (classification model repeated 1000 times, bootstrap used to validate the model) revealed six successive predictors for ACS: cardiac murmur, age <45 years, left lateral chest pain, age ≥68.5 years, male sex, and ST-elevations (Figure 2). Different combinations of signs and symptoms (with and without POC H-FABP testing) were combined in possible CDRs. Sensitivity and specificity with 95% CIs of these combinations were calculated. CDR 1–4 were based on signs and symptoms derived from logistic regression analysis of the study data: in CDR 1, all relevant signs and symptoms found in logistic regression were combined with the POCT H-FABP result; in CDR 2, only signs and symptoms were included, excluding the POCT H-FABP result; CDR 3 is similar to CDR 1, excluding ECG findings (ECG equipment is not always available in primary care); and CDR 4 is based on relevant signs and symptoms without ECG and POCT H-FABP result. CDR 5 was based on CART analysis of the study data (Table 3).
Study objective 3: diagnostic value of the CDR
Rule-in and rule-out characteristics for the different CDRs and area under the receiver operating curves were calculated (Table 3). Possible relevance of a CDR in ruling in or ruling out ACS or AMI was defined as an increase in probability of presence (ruling in) or absence (ruling out) of disease after using the particular CDR. Area under receiver operating curve for ACS was highest for CDR 1 (0.78). NPV, positive predictive value (PPV), and sensitivity were comparable to regular care (that is, the GP's clinical judgment only). False negativity count for CDR 1 (four missed cases) was lower than the false negativity count of the GP's judgment only (eight missed cases). However, specificity of CDR 1 was lower than the specificity of the GP's clinical judgment, leading to a 47.7% increase in false positive test results and thus an increase in ACS-negative referrals. For CDR 5, based on CART analysis, an NPV of 100% was reached; however, referral rate owing to a positive result of the CDR would increase from 37.0% (GP's judgment) to 81.4%.
Discussion
Summary
The diagnostic value of a POC H-FABP test as a stand-alone test was poor: sensitivity of the test for ACS was 25.8%, leading to a high number of false negative test results. An algorithm of a feeling of pressure on the chest, absence of left-lateral chest pain, dyspnoea, ST depressions, ST elevations, and the H-FABP POC test result (CDR 1) predicted the outcome of ACS reasonably well (sensitivity 87.5%, NPV 97.2%). However, the gain as compared with a GP's clinical judgment was limited: the false negativity rate was 50.0% lower (absolute reduction of 1.3 percentage points) at a price of a considerable 47.7% increase in over-referral of ultimately ACS-negative cases (absolute increase of 13.9 percentage points). Moreover, using CDR 1 would lead to the necessity of performing a POC test and an ECG as part of the CDR in all patients with chest pain. CART analysis led to a CDR with a high NPV, of 100% (compared with a NPV of 95.8% based on a GP's clinical judgment), without the necessity of performing additional tests. However, the referral rate using this CDR would be 81.6%; more than twice as high as a GP's regular referral rate.
Strengths and limitations
This study was performed in daily primary care. The H-FABP POC test was tested in a representative clinical situation. The incidence of ACS was 10.6%, comparable with previous findings in the literature, where incidences of 1.5% to 22% were described.28–33 The study team focused on deriving factors to be included in an algorithm of signs and symptoms combined with POC H-FABP testing predictive of ACS (stage 2 of the aforementioned stages of Stiell and Wells).19 All criteria for a methodologically correct stage 2 (definition of outcomes and predictor variables, generalisability of subject selection, several statistical and methodological demands) are present in this study, except the inter-observer reliability for the clinical findings, which is difficult to measure in an acute setting. External validation of the formulated CDRs has not yet been performed.
During the runtime of the study, problems were faced regarding inclusion of sufficient patients; GPs embraced the study with enthusiasm, but faced serious barriers when actually including patients. The fact that patients with chest pain are usually unannounced, and therefore not scheduled, leads to the overrunning of consultation hours. In addition, the possible life-threatening underlying disease seemed to limit inclusion into the study. Moreover, study procedures such as obtaining (temporary) written informed consent were seen as thresholds. As a result, extension of the study from day care in primary care to OOH services was necessary. Finally, although the aim was to include 600 patients, only 303 patients were included.
Comparison with existing literature
In an earlier study, based on a POC H-FABP test with a cut-off value of 7 ng/ml, performance of the test in clinical situations was insufficient.28 The present study worked with a promising new POC H-FABP test with a cut-off value of 4 ng/ml, based on a clinical study with patients with chest pain in a secondary care emergency setting.18 However, performance of this POC H-FABP test as a stand-alone test was insufficient. Several causes could be underlying.
One possible factor that should be taken into account as a cause of absence of biomarker rise is the primary care spectrum tested in this study, where complaints are less typical and presentation is often early. This could possibly have led to a delayed measurable biomarker rise (underperformance of the marker). This is not in accordance with earlier research.18 A comparison between the H-FABP POC test result and the absolute H-FABP concentration (to be measured by ELISA in stored plasma samples obtained from a selection of patients) is planned. Including patients with UA — a usually biomarker-negative condition — possibly compromises the marker performance. However, subgroup analyses of only patients with AMI revealed similar results and, in primary care, ruling-out strategies should aim at ACS, not AMI alone. Additionally, the test signal could be insufficient; for example, the discolouration of the test could take place at higher cut-off values than the cut-off of 4 ng/ml (underperformance of the POC test). Test calibration by the manufacturer, however, suggests high accuracy of the test. Third, slight discolouration of the POC test might have been under-detected by care providers participating in the study (underperformance of the test operator) but, in the current study, the POC H-FABP performance as a clinical tool was being tested and if under-detection of the test was a problem, the test should be improved to augment usability. Moreover, participating caregivers never complained about the usability of the test during the study.
Implications for research and practice
Neither the currently evaluated H-FABP POC test (FABPulous BV, cut-off value 4 ng/ml) as a stand-alone test, nor a CDR incorporating this H-FABP POC test is of added value when compared with the experienced GP's clinical judgment. Annually, a Dutch GP at least briefly suspects approximately 35 patients with chest pain of ACS.1 For every GP using CDR 1, one missed case of ACS would be prevented over 2 years. However, in the same 2 years, 70 ECGs and 70 POC tests would be necessary and 30, instead of the usual 20, ACS-negative patients would be referred to secondary care facilities to rule out ACS. Future studies should focus on: large primary care populations; fusing existing algorithms into a strong tool; (combinations of) biomarker POC tests, and proving cost-effectiveness of such diagnostic means.19,33–36 Thus, cases of ACS should be adequately detected and referral rates should decrease as compared with current practice, improving patient comfort and lowering cost to society.
Notes
Funding
This study is funded by means of an unrestricted grant by FABPulous BV, the company that develops point-of care H-FABP-tests. For more details, see Competing interests.
Ethical approval
This study protocol has been approved by the ethical review board of Maastricht University for the Netherlands and by the ethical review board of KU Leuven for Belgium. Full research procedures are registered on www.clinicaltrials.gov (reference number: NCT01826994).
Handling of personal data complied with the Dutch Personal Data Protection Act and Belgian privacy legislation. All GPs and out-of-hours services received €40 per included patient as a compensation for their extra workload.
Competing interests
All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: the study is funded by means of an unrestricted grant by FABPulous BV, the company that develops point-of care H-FABP-tests. FABPulous BV agreed not to interfere with data collection, data management and analysis of data. Publication of possible unfavourable outcome of this study was guaranteed. Data gathering, data management and data reporting were performed by the reseachers alone, not by the company. JFCG is chief scientific officer at FABPulous BV. All other authors have no additional possible conflict of interest. The authors alone are responsible for the content and writing of the paper. No support from any other organisation was received for the submitted work.
Provenance
Freely submitted; externally peer reviewed.
- Received April 18, 2019.
- Accepted April 24, 2019.
- Copyright © 2019, The Authors
This article is Open Access: CC BY license (https://creativecommons.org/licenses/by/4.0/)