Abstract
Background Several clinical prediction rules (CPRs) are available for sleep apnoea-hypopnoea syndrome (OSAH), but they are difficult to apply in primary care (PC).
Aim Derivation and validation of a CPR using simple measurements available in PC.
Design & setting A prospective study conducted in health centres from the area of influence of three Spanish hospitals.
Method Patients (aged 18–70 years) who attended for any reason; who presented with at least one of the three key symptoms for OSAH (snoring, breathing pauses while sleeping, and daytime sleepiness); and who were not undergoing non-invasive ventilation or prior treatment with continuous positive airway pressure (CPAP) were included. Anthropometric data, smoking habit, comorbidities, and Epworth test were collected. Patients were subsequently referred to the sleep unit (SU), where the decision was taken whether or not to instigate treatment. A multivariate logistic model was constructed using a sub-sample and scores assigned based on the regression coefficients; the CPR was validated with the remaining sample. Both receiver operating characteristic (ROC) curves were plotted and the sensitivity, specificity, and predictive values calculated.
Results The derivation sample comprised 352 patients, with 260 in the validation sample. The final factors (arterial hypertension [AHT], age, body mass index [BMI], and sex) were used to develop a rule with scores ranging from 0.00–5.50. The cut-off point that optimises the area under the curve (AUC) is ≥2.50 points (AUC = 0.78; sensitivity = 86%; specificity = 54%; positive predictive value [PPV] = 45%; negative predictive value [NPV] = 90%; likelihood ratio [LR] = 0.26). The properties for the validation sample with this cut-off point are as follows: AUC = 0.68; sensitivity = 81%; specificity = 43%; PPV = 61%; NPV = 68%; LR = 0.44.
Conclusion As in similar cases, the specificity is low, meaning that healthy people are referred to a specialist. A negative result rules out the disease in most cases.
How this fits in
OSAH is associated with numerous diseases but only 5–9% of the population with severe OSAH is receiving treatment. Although various CPRs have been proposed, none of them is widely used. This CPR presents a lower sensitivity and specificity than other rules, and is more accessible and easier to use, making this rule an ideal tool for use in PC
Introduction
OSAH affects 2–4% of the population.1 Its prevalence increases with age.2 Several studies have found an association between OSAH and numerous diseases, including hypertension,1,3,4 and cardiovascular5 and cerebrovascular disease.6 OSAH is associated with a worse quality of life,7 and a higher number of work8 and traffic accidents.9,10
Despite the risks associated with OSAH, only 5–9% of the population with severe OSAH is receiving treatment.11 Three key symptoms (snoring, pauses, and daytime sleepiness) have been used in standard clinical practice (SCP) to determine the probable presence of OSAH. However, none of these predicts the disease per se, as they are all very common in the general population.
This situation has outlined the need to develop a tool that discriminates patients presenting a more severe OSAH from those in whom the disease is not present or is unlikely to lead to its complications. Therefore, a tool like a CPR would be most useful for PC physicians, who could apply it to decide whether or not to refer a patient to specialised care.
A CPR is considered to be any decision-making instrument prepared from a minimum of three variables obtained from the case history, physical examination, or simple diagnostic tests.12 Although several attempts have been made to create a CPR,13–16 none of them is used in SCP. In some cases, this is because they use apnoea–hypopnoea index (AHI) values that are of little clinical interest (AHI = 5 or AHI = 10)17 or, in the case of the neck circumference, due to the collinearity with BMI, which is in the final model as current guidelines indicate the use of CPAP for AHI values >30, or 15 if accompanied by related comorbidity or severe symptomatology.
Kushida et al 15reported a high-quality prediction (sensitivity = 98%; specificity = 100%), but this rule is difficult to apply in SCP as the collection of the variables requires meticulous training. Chai-Coetzer et al 17 developed the OSA50 screening tool plus overnight oximetry, which shows a very good NPV, with the drawback for its use in PC being that it requires an at-home overnight test.
This study proposes the derivation and validation of a prediction rule, using measurements easy to obtain in PC, in order to distinguish between patients at high and low risk of suffering OSAH.
Method
Design and patients
The CPR was developed using a sample of patients recruited prospectively between February 2011 and August 2012. The derivation sample comprised patients who attended the PC departments of two urban health centres and were referred to the SU at the Araba University Hospital. Patients of the validation sample were recruited at various health centres and referred to the Hospital de Galdakao-Usansolo and the University Hospital in Valdecilla (Santander).
All subjects aged 18–70 years and presenting at least one of the three key symptoms for OSAH (snoring, daytime sleepiness, or breathing pauses while asleep) were recruited on spontaneous consultation of a PC physician for any reason. Patients with a previous diagnosis of OSAH, prior CPAP treatment or non-invasive mechanic ventilation were excluded. Participants provided consent to participate in the study.
Data collection
After collection of the information in the PC centres, participants were asked to attend the SU at the corresponding hospital. The doctor at the SU, blinded to the information collected in the PC setting, made the diagnostic decisions (polysomnography [PSG], respiratory polygraphy [RP], both, or neither) as well as the therapeutic decisions (CPAP, mandibular advancement device [MAD], postural device, or hygiene or dietary measures), following SCP.
Reliability analysis
The only information obtained in both the PC centre and the SU, the BMI, was used for the inter-observer reliability analysis in order to validate the information collected by the PC physicians.
Variables
The characteristic to be predicted is the clinical decision of the sleep specialist as regards indication for specific diagnostic tests (PSG, RP, or both tests); and indication of treatment (CPAP, MAD, or a postural device).
The predictive variables taken into consideration were sex, age, weight, height, neck circumference, snoring (five or more times per week), breathing pauses while asleep reported by the partner or companion, daytime hypersomnia, accidents in the past year due to drowsiness, morning tiredness, morning sensation of asphyxia, AHT, heart failure, diabetes mellitus (DM), smoking habit, alcohol consumption, and daytime sleepiness according to the Epworth scale.
In addition, BMI, neck circumference, and Epworth scale were collected at the SUs. Along with this information, global AHI, supine AHI, and T90 (the percentage of time during which arterial O2 saturation is <90%) were obtained if available from the PSG and/or RP reports.
Sample size
According to Flahault et al, 18 approximately 298 cases with the disease, or 750 subjects, are required to estimate an expected sensitivity for the test of 0.95 with a confidence limit (95%) of not less than 0.90, considering a prevalence of 40% for the disease in the study population (data from the SU in Vitoria).
Two hundred subjects are required for the validation phase for a power of 90% to confirm the hypothesis that the sensitivity of the CPR in the validation population does not differ from that obtained for the derivation sample by >3%.
Statistical analysis
Treatment of variables
Means and standard deviations were used to describe continuous variables, and frequencies and percentages for categorical variables.
Continuous variables were categorised in order to be applied in a simple manner. Looking at the distribution of results, values close to the median or values close to the tertiles were used as cut-off points.
Characteristics of the patients in the derivation and validation samples were compared using the X2 test or the student’s t-test for continuous variables.
SPSS (version 22) and R freeware (version 3.1.1) were used for all analyses.
Derivation of the prediction rule
According to Kharbanda et al, 19 those variables with <10% lost values were selected as potential predictive variables, which did not exclude any of those from the model. The characteristics of treated patients were compared with those of untreated patients using a univariate logistic regression. A value of P<0.2 indicated that a variable was potentially predictive and should be taken into consideration during the multivariate analysis.
Subsequently, the final multivariate logistic regression model was described using a stepwise variable selection method and the LR test to compare two models. The criterion for introducing variables into the model was P≤0.05.
A backward stepwise model was also prepared using the LR test in order to ensure the most parsimonious rule possible. The criterion for excluding variables from the model was P>0.05. Both variable selection criteria led to the same final model. The goodness-of-fit was evaluated using the Hosmer–Lemeshow test.
The final scores for the prediction rule were obtained from the logistic regression coefficients using the lowest risk category as reference.19
Predictive ability and validation of the rule
The predictive ability of the rule was evaluated for each sample using ROC curves, which present the decision reached with respect to the score. To evaluate the predictive accuracy of the rule for the derivation and validation samples, different cut-off points were established and 2×2 tables were constructed to calculate the following measures: sensitivity, specificity, PPVs, and NPVs (including 95% confidence intervals [CI]). Table 1 shows the CPR in the format it would be used in PC.
Reliability analysis
The kappa statistic was calculated to compare the interobserver reliability for measurement of each of the variables included in the CPR. A kappa value >0.60 was considered to be acceptable.
Results
Informed consent was obtained from 620 patients. Eight were excluded: five because they were aged >75 years; two because they did not have any of the key symptoms; and one for technical reasons. Full information (predictive variables, and indication for diagnostic tests and/or treatment) was available for 278 of the 352 subjects in the derivation sample (79.0%) and 233 of the 260 subjects (89.6%) in the validation sample (Figure 1). The complementary examination was conducted for 226 of the 231 subjects from the validation sample who attended the SU (97.8%), and 122 of them (54.0%) had an indication for treatment with CPAP or another device (P<0.001).
Table 2 shows the characteristics of both samples. Differences can be seen between validation and derivation samples regarding sex, BMI, breathing pauses, daytime hypersomnia, history of accidents due to drowsiness, morning tiredness, sensation of asphyxia, neck circumference, AHT, DM, smoking habit, score on the Epworth scale, and final decision by the SU specialist.
Table 3 shows the result of the univariate analysis, in which the risk factors were compared with the therapeutic decision. Subjects requiring treatment were more likely to be older, male, obese, with larger neck circumferences, and with chronic disease.
The final multivariate model (Hosmer–Lemeshow goodness-of-fit P = 0.903), can be seen in Table 4, with the variables included and the coefficients estimated, along with the scores for the CPR calculated using these coefficients. According to this rule, 1.50 points were awarded each to males, to obese patients, and to those aged >60 years. Subjects aged 46–59 years were awarded 1.25 points, with an additional point for those who were hypertensive. Therefore, the total score for the rule ranges between 0.00 and 5.50 points.
The AUCs for the derivation sample (77.8%) and for the validation sample (68.1%) can be seen plotted in Figure 2.
AUC = area under curve.
The sensitivity, specificity, PPV, and NPV for the different cut-off points on the scale are provided in Table 5. To guarantee a high sensitivity, and therefore the detection of patients who require treatment, cut-off points of 1.50, 2.25, 2.50, or 2.75 are proposed as possible discriminatory values. The rule has a better diagnostic ability for the derivation than for the validation sample. However, selection of the most conservative cut-off point possible (1.50) leads to a sensitivity of 97.5% for the derivation sample, although this implies that only 24 of 233 (10.3%) subjects attending a PC are free of diagnosis.
For the reliability, the kappa index for this measure is 0.834, although it should be noted that this analysis was performed for a total of 334 out of 612 cases (54.6% of the total sample).
Discussion
Summary
This study has derived and validated a CPR for identifying patients with suspected OSAH in PC centres. Four independent predictive factors (AHT, age >46 years, BMI ≥30, and male sex) were selected, and a rule ranging from 0.00 to 5.50 points was generated combining them. Despite being variables 'classically' associated with the presence of OSAH, neck circumference and breathing pauses do not form part of the final model as these variables are confounded by sex and, in the case of the neck circumference, due to the collinearity with BMI, which is in the final model.
Although the final goal of a prediction rule is to obtain the highest possible sensitivity, in this study two cut-off points are proposed for two different scenarios. In this case, selecting a cut-off point of ≥1.50 points (highest sensitivity for the rule with the derivation sample) means that sleep tests must be performed in a large number of healthy patients due to the low specificity. This implies the overuse of already limited resources, in addition to the indirect costs and inconveniences generated to the patient.
The cut-off point that achieves the best results in terms of sensitivity and specificity for the derivation sample is ≥2.50 points, with a lower sensitivity counterbalanced by a marked increase in specificity.
Strengths and limitations
However, selection of this cut-off point for the validation sample leads to markedly different results as both sensitivity and specificity decrease. This nevertheless appears to be the most reasonable option in light of the results. The PPV of the validation sample is also 15–20% higher.
This is likely to be mainly due to the fact that the derivation and validation populations sampled differ in terms of baseline risk profile regarding the disease. Thus, the derivation sample includes a higher proportion of males, with a higher BMI and, in general, a higher number of key symptoms for the disease, and therefore a higher pre-test probability.
The main limitation of this study is the difference between the derivation and validation groups; the values obtained during validation are markedly lower than those obtained during derivation of this rule. However, this situation merely reflects the study's setting, where access to some healthcare services is unequal and depends on the healthcare organisation.
The present authors also decided to limit the study to population those aged <70 years, with the aim of identifying the disease in a population with a low prior probability of having it. It is doubtful that a rule valid for asymptomatic people works the same way in an aged population, with a higher probability and a greater possibility of confounding due to comorbidity. Similar studies have also chosen this limit, notably the aforementioned work by Chai-Coetzer.17
Comparison with existing literature
Although the results obtained do not demonstrate as high a sensitivity and specificity as the prediction rules developed previously,14–17 the present authors believe that the studied rule is easier to apply in PC settings, resulting in a more rational referral to specialised units. This means that the rule is useful despite the drawbacks discussed above.
A recent study conducted in the UK20 surveyed PC physicians regarding the use of prediction rules in clinical practice. The results showed that a high percentage of physicians were unaware of and/or did not use prediction rules widely validated in PC, as they considered these rules to be of little use, or preferred their own medical judgement. This could possibly be explained by the complexity of such rules. Therefore, a rule that is easy to use could easily be implemented in already saturated PC centers.
Implications for practice
In conclusion, this study presents a CPR for diagnosing OSAH which, despite presenting a lower sensitivity and specificity than other such rules, is more accessible and easier to use. This makes it an ideal tool for use in PC, allowing the referral of patients susceptible to presenting the disease to SUs while ensuring a more rational use of the resources available.
Notes
Funding
This study was funded by the Carlos III Health Institute (reference number: PI10/01249).
Ethical approval
Participants provided consent to participate in the study, which was approved by the corresponding Clinical Research Ethics Committees.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors declare that no competing interests exist.
- Received January 30, 2018.
- Accepted February 6, 2018.
- Copyright © The Authors 2018
This article is Open Access: CC BY license (https://creativecommons.org/licenses/by/4.0/)
References
In this issue
Jump to section
More in this TOC Section
Related Articles
Cited By...
