Research

Derivation and validation of a clinical prediction rule for sleep apnoea syndrome for use in primary care

Erika Miranda Serrano, Amanda Lopez-Picado, Aitziber Etxagibel, Alfonso Casi, Laura Cancelo, Jose Ignacio Aguirregomoscorta, Itziar Menéndez, Monica Gonzalez and Felipe Aizpuru
BJGP Open 2018; 2 (2): bjgpopen18X101481. DOI: https://doi.org/10.3399/bjgpopen18X101481

Abstract

Background Several clinical prediction rules (CPRs) are available for sleep apnoea-hypopnoea syndrome (OSAH), but they are difficult to apply in primary care (PC).

Aim Derivation and validation of a CPR using simple measurements available in PC.

Design & setting A prospective study conducted in health centres from the area of influence of three Spanish hospitals.

Method Patients (aged 18–70 years) who attended for any reason; who presented with at least one of the three key symptoms for OSAH (snoring, breathing pauses while sleeping, and daytime sleepiness); and who were not undergoing non-invasive ventilation or prior treatment with continuous positive airway pressure (CPAP) were included. Anthropometric data, smoking habit, comorbidities, and Epworth test were collected. Patients were subsequently referred to the sleep unit (SU), where the decision was taken whether or not to instigate treatment. A multivariate logistic model was constructed using a sub-sample and scores assigned based on the regression coefficients; the CPR was validated with the remaining sample. Both receiver operating characteristic (ROC) curves were plotted and the sensitivity, specificity, and predictive values calculated.

Results The derivation sample comprised 352 patients, with 260 in the validation sample. The final factors (arterial hypertension [AHT], age, body mass index [BMI], and sex) were used to develop a rule with scores ranging from 0.00–5.50. The cut-off point that optimises the area under the curve (AUC) is ≥2.50 points (AUC = 0.78; sensitivity = 86%; specificity = 54%; positive predictive value [PPV] = 45%; negative predictive value [NPV] = 90%; likelihood ratio [LR] = 0.26). The properties for the validation sample with this cut-off point are as follows: AUC = 0.68; sensitivity = 81%; specificity = 43%; PPV = 61%; NPV = 68%; LR = 0.44.

Conclusion As in similar cases, the specificity is low, meaning that healthy people are referred to a specialist. A negative result rules out the disease in most cases.

How this fits in

OSAH is associated with numerous diseases but only 5–9% of the population with severe OSAH is receiving treatment. Although various CPRs have been proposed, none of them is widely used. This CPR presents a lower sensitivity and specificity than other rules, and is more accessible and easier to use, making this rule an ideal tool for use in PC

Introduction

OSAH affects 2–4% of the population.1 Its prevalence increases with age.2 Several studies have found an association between OSAH and numerous diseases, including hypertension,1,3,4 and cardiovascular5 and cerebrovascular disease.6 OSAH is associated with a worse quality of life,7 and a higher number of work8 and traffic accidents.9,10

Despite the risks associated with OSAH, only 5–9% of the population with severe OSAH is receiving treatment.11 Three key symptoms (snoring, pauses, and daytime sleepiness) have been used in standard clinical practice (SCP) to determine the probable presence of OSAH. However, none of these predicts the disease per se, as they are all very common in the general population.

This situation has outlined the need to develop a tool that discriminates patients presenting a more severe OSAH from those in whom the disease is not present or is unlikely to lead to its complications. Therefore, a tool like a CPR would be most useful for PC physicians, who could apply it to decide whether or not to refer a patient to specialised care.

A CPR is considered to be any decision-making instrument prepared from a minimum of three variables obtained from the case history, physical examination, or simple diagnostic tests.12 Although several attempts have been made to create a CPR,13–16 none of them is used in SCP. In some cases, this is because they use apnoea–hypopnoea index (AHI) values that are of little clinical interest (AHI = 5 or AHI = 10)17 or, in the case of the neck circumference, due to the collinearity with BMI, which is in the final model as current guidelines indicate the use of CPAP for AHI values >30, or 15 if accompanied by related comorbidity or severe symptomatology.

Kushida et al 15reported a high-quality prediction (sensitivity = 98%; specificity = 100%), but this rule is difficult to apply in SCP as the collection of the variables requires meticulous training. Chai-Coetzer et al 17 developed the OSA50 screening tool plus overnight oximetry, which shows a very good NPV, with the drawback for its use in PC being that it requires an at-home overnight test.

This study proposes the derivation and validation of a prediction rule, using measurements easy to obtain in PC, in order to distinguish between patients at high and low risk of suffering OSAH.

Method

Design and patients

The CPR was developed using a sample of patients recruited prospectively between February 2011 and August 2012. The derivation sample comprised patients who attended the PC departments of two urban health centres and were referred to the SU at the Araba University Hospital. Patients of the validation sample were recruited at various health centres and referred to the Hospital de Galdakao-Usansolo and the University Hospital in Valdecilla (Santander).

All subjects aged 18–70 years and presenting at least one of the three key symptoms for OSAH (snoring, daytime sleepiness, or breathing pauses while asleep) were recruited on spontaneous consultation of a PC physician for any reason. Patients with a previous diagnosis of OSAH, prior CPAP treatment or non-invasive mechanic ventilation were excluded. Participants provided consent to participate in the study.

Data collection

After collection of the information in the PC centres, participants were asked to attend the SU at the corresponding hospital. The doctor at the SU, blinded to the information collected in the PC setting, made the diagnostic decisions (polysomnography [PSG], respiratory polygraphy [RP], both, or neither) as well as the therapeutic decisions (CPAP, mandibular advancement device [MAD], postural device, or hygiene or dietary measures), following SCP.

Reliability analysis

The only information obtained in both the PC centre and the SU, the BMI, was used for the inter-observer reliability analysis in order to validate the information collected by the PC physicians.

Variables

The characteristic to be predicted is the clinical decision of the sleep specialist as regards indication for specific diagnostic tests (PSG, RP, or both tests); and indication of treatment (CPAP, MAD, or a postural device).

The predictive variables taken into consideration were sex, age, weight, height, neck circumference, snoring (five or more times per week), breathing pauses while asleep reported by the partner or companion, daytime hypersomnia, accidents in the past year due to drowsiness, morning tiredness, morning sensation of asphyxia, AHT, heart failure, diabetes mellitus (DM), smoking habit, alcohol consumption, and daytime sleepiness according to the Epworth scale.

In addition, BMI, neck circumference, and Epworth scale were collected at the SUs. Along with this information, global AHI, supine AHI, and T90 (the percentage of time during which arterial O2 saturation is <90%) were obtained if available from the PSG and/or RP reports.

Sample size

According to Flahault et al, 18 approximately 298 cases with the disease, or 750 subjects, are required to estimate an expected sensitivity for the test of 0.95 with a confidence limit (95%) of not less than 0.90, considering a prevalence of 40% for the disease in the study population (data from the SU in Vitoria).

Two hundred subjects are required for the validation phase for a power of 90% to confirm the hypothesis that the sensitivity of the CPR in the validation population does not differ from that obtained for the derivation sample by >3%.

Statistical analysis

Treatment of variables

Means and standard deviations were used to describe continuous variables, and frequencies and percentages for categorical variables.

Continuous variables were categorised in order to be applied in a simple manner. Looking at the distribution of results, values close to the median or values close to the tertiles were used as cut-off points.

Characteristics of the patients in the derivation and validation samples were compared using the X2 test or the student’s t-test for continuous variables.

SPSS (version 22) and R freeware (version 3.1.1) were used for all analyses.

Derivation of the prediction rule

According to Kharbanda et al, 19 those variables with <10% lost values were selected as potential predictive variables, which did not exclude any of those from the model. The characteristics of treated patients were compared with those of untreated patients using a univariate logistic regression. A value of P<0.2 indicated that a variable was potentially predictive and should be taken into consideration during the multivariate analysis.

Subsequently, the final multivariate logistic regression model was described using a stepwise variable selection method and the LR test to compare two models. The criterion for introducing variables into the model was P≤0.05.

A backward stepwise model was also prepared using the LR test in order to ensure the most parsimonious rule possible. The criterion for excluding variables from the model was P>0.05. Both variable selection criteria led to the same final model. The goodness-of-fit was evaluated using the Hosmer–Lemeshow test.

The final scores for the prediction rule were obtained from the logistic regression coefficients using the lowest risk category as reference.19

Predictive ability and validation of the rule

The predictive ability of the rule was evaluated for each sample using ROC curves, which present the decision reached with respect to the score. To evaluate the predictive accuracy of the rule for the derivation and validation samples, different cut-off points were established and 2×2 tables were constructed to calculate the following measures: sensitivity, specificity, PPVs, and NPVs (including 95% confidence intervals [CI]). Table 1 shows the CPR in the format it would be used in PC.

View this table:
Table 1. Clinical prediction rule in the format it would be used in primary care 

Reliability analysis

The kappa statistic was calculated to compare the interobserver reliability for measurement of each of the variables included in the CPR. A kappa value >0.60 was considered to be acceptable.

Results

Informed consent was obtained from 620 patients. Eight were excluded: five because they were aged >75 years; two because they did not have any of the key symptoms; and one for technical reasons. Full information (predictive variables, and indication for diagnostic tests and/or treatment) was available for 278 of the 352 subjects in the derivation sample (79.0%) and 233 of the 260 subjects (89.6%) in the validation sample (Figure 1). The complementary examination was conducted for 226 of the 231 subjects from the validation sample who attended the SU (97.8%), and 122 of them (54.0%) had an indication for treatment with CPAP or another device (P<0.001).

Figure 1.
Figure 1. Flow chart of inclusion and exclusion criteria

Table 2 shows the characteristics of both samples. Differences can be seen between validation and derivation samples regarding sex, BMI, breathing pauses, daytime hypersomnia, history of accidents due to drowsiness, morning tiredness, sensation of asphyxia, neck circumference, AHT, DM, smoking habit, score on the Epworth scale, and final decision by the SU specialist.

View this table:
Table 2. Comparison of the derivation and validation samples

Table 3 shows the result of the univariate analysis, in which the risk factors were compared with the therapeutic decision. Subjects requiring treatment were more likely to be older, male, obese, with larger neck circumferences, and with chronic disease.

View this table:
Table 3. Derivation sample. Univariate logistic regression for the primary variable 'therapeutic decision'. Selection of variables for the final model.

The final multivariate model (Hosmer–Lemeshow goodness-of-fit P = 0.903), can be seen in Table 4, with the variables included and the coefficients estimated, along with the scores for the CPR calculated using these coefficients. According to this rule, 1.50 points were awarded each to males, to obese patients, and to those aged >60 years. Subjects aged 46–59 years were awarded 1.25 points, with an additional point for those who were hypertensive. Therefore, the total score for the rule ranges between 0.00 and 5.50 points.

View this table:
Table 4. Derivation sample. Final logistic regression and scores for the clinical prediction rule.

The AUCs for the derivation sample (77.8%) and for the validation sample (68.1%) can be seen plotted in Figure 2.

Figure 2.
Figure 2. Receiver operating characteristic curves for the derivation and validation samples.

AUC = area under curve.

The sensitivity, specificity, PPV, and NPV for the different cut-off points on the scale are provided in Table 5. To guarantee a high sensitivity, and therefore the detection of patients who require treatment, cut-off points of 1.50, 2.25, 2.50, or 2.75 are proposed as possible discriminatory values. The rule has a better diagnostic ability for the derivation than for the validation sample. However, selection of the most conservative cut-off point possible (1.50) leads to a sensitivity of 97.5% for the derivation sample, although this implies that only 24 of 233 (10.3%) subjects attending a PC are free of diagnosis.

View this table:
Table 5. Different cut-off points for the clinical prediction rule.

For the reliability, the kappa index for this measure is 0.834, although it should be noted that this analysis was performed for a total of 334 out of 612 cases (54.6% of the total sample).

Discussion

Summary

This study has derived and validated a CPR for identifying patients with suspected OSAH in PC centres. Four independent predictive factors (AHT, age >46 years, BMI ≥30, and male sex) were selected, and a rule ranging from 0.00 to 5.50 points was generated combining them. Despite being variables 'classically' associated with the presence of OSAH, neck circumference and breathing pauses do not form part of the final model as these variables are confounded by sex and, in the case of the neck circumference, due to the collinearity with BMI, which is in the final model.

Although the final goal of a prediction rule is to obtain the highest possible sensitivity, in this study two cut-off points are proposed for two different scenarios. In this case, selecting a cut-off point of ≥1.50 points (highest sensitivity for the rule with the derivation sample) means that sleep tests must be performed in a large number of healthy patients due to the low specificity. This implies the overuse of already limited resources, in addition to the indirect costs and inconveniences generated to the patient.

The cut-off point that achieves the best results in terms of sensitivity and specificity for the derivation sample is ≥2.50 points, with a lower sensitivity counterbalanced by a marked increase in specificity.

Strengths and limitations

However, selection of this cut-off point for the validation sample leads to markedly different results as both sensitivity and specificity decrease. This nevertheless appears to be the most reasonable option in light of the results. The PPV of the validation sample is also 15–20% higher.

This is likely to be mainly due to the fact that the derivation and validation populations sampled differ in terms of baseline risk profile regarding the disease. Thus, the derivation sample includes a higher proportion of males, with a higher BMI and, in general, a higher number of key symptoms for the disease, and therefore a higher pre-test probability.

The main limitation of this study is the difference between the derivation and validation groups; the values obtained during validation are markedly lower than those obtained during derivation of this rule. However, this situation merely reflects the study's setting, where access to some healthcare services is unequal and depends on the healthcare organisation.

The present authors also decided to limit the study to population those aged <70 years, with the aim of identifying the disease in a population with a low prior probability of having it. It is doubtful that a rule valid for asymptomatic people works the same way in an aged population, with a higher probability and a greater possibility of confounding due to comorbidity. Similar studies have also chosen this limit, notably the aforementioned work by Chai-Coetzer.17

Comparison with existing literature

Although the results obtained do not demonstrate as high a sensitivity and specificity as the prediction rules developed previously,14–17 the present authors believe that the studied rule is easier to apply in PC settings, resulting in a more rational referral to specialised units. This means that the rule is useful despite the drawbacks discussed above.

A recent study conducted in the UK20 surveyed PC physicians regarding the use of prediction rules in clinical practice. The results showed that a high percentage of physicians were unaware of and/or did not use prediction rules widely validated in PC, as they considered these rules to be of little use, or preferred their own medical judgement. This could possibly be explained by the complexity of such rules. Therefore, a rule that is easy to use could easily be implemented in already saturated PC centers.

Implications for practice

In conclusion, this study presents a CPR for diagnosing OSAH which, despite presenting a lower sensitivity and specificity than other such rules, is more accessible and easier to use. This makes it an ideal tool for use in PC, allowing the referral of patients susceptible to presenting the disease to SUs while ensuring a more rational use of the resources available.

Notes

Funding

This study was funded by the Carlos III Health Institute (reference number: PI10/01249).

Ethical approval

Participants provided consent to participate in the study, which was approved by the corresponding Clinical Research Ethics Committees.

Provenance

Freely submitted; externally peer reviewed.

Competing interests

The authors declare that no competing interests exist.

  • Received January 30, 2018.
  • Accepted February 6, 2018.

This article is Open Access: CC BY license (https://creativecommons.org/licenses/by/4.0/)

References

  1. 1.↵
    1. Durán J,
    2. Esnaola S,
    3. Rubio R,
    4. et al.
    (2001) Obstructive sleep apnea-hypopnea and related clinical features in a population-based sample of subjects aged 30 to 70 yr. Am J Respir Crit Care Med 163(3 Pt 1):685–689, doi:10.1164/ajrccm.163.3.2005065.
    OpenUrlCrossRefPubMed
  2. 2.↵
    1. Young T,
    2. Palta M,
    3. Dempsey J,
    4. et al.
    (1993) The occurrence of sleep-disordered breathing among middle-aged adults. N Engl J Med 328(17):1230–1235, doi:10.1056/NEJM199304293281704.
    OpenUrlCrossRefPubMed
  3. 3.↵
    1. Nieto FJ,
    2. Young TB,
    3. Lind BK,
    4. et al.
    (2000) Association of sleep-disordered breathing, sleep apnea, and hypertension in a large community-based study. Sleep Heart Health Study. JAMA 283(14):1829–1836, doi:10.1001/jama.283.14.1829.
    OpenUrlCrossRefPubMed
  4. 4.↵
    1. Peppard PE,
    2. Young T,
    3. Palta M,
    4. et al.
    (2000) Prospective study of the association between sleep-disordered breathing and hypertension. New Eng J Med 342(19):1378–1384, doi:10.1056/NEJM200005113421901.
    OpenUrlCrossRefPubMed
  5. 5.↵
    1. Newman AB,
    2. Nieto FJ,
    3. Guidry U,
    4. et al.
    (2001) Relation of sleep-disordered breathing to cardiovascular disease risk factors: the sleep heart health study. Am J Epidemiol 154(1):50–59, doi:10.1093/aje/154.1.50.
    OpenUrlCrossRefPubMed
  6. 6.↵
    1. Parra O,
    2. Arboix A,
    3. Bechich S,
    4. et al.
    (2000) Time course of sleep-related breathing disorders in first-ever stroke or transient ischemic attack. Am J Respir Crit Care Med 161(2 Pt 1):375–380, doi:10.1164/ajrccm.161.2.9903139.
    OpenUrlCrossRefPubMed
  7. 7.↵
    1. Lacasse Y,
    2. Godbout C,
    3. Sériès F
    (2002) Health-related quality of life in obstructive sleep apnoea. Eur Respir J 19(3):499–503, doi:10.1183/09031936.02.00216902.
    OpenUrlAbstract/FREE Full Text
  8. 8.↵
    1. Garbarino S,
    2. Guglielmi O,
    3. Sanna A,
    4. et al.
    (2016) Risk of occupational accidents in workers with obstructive sleep apnea: systematic review and meta-analysis. Sleep 39(6):1211–1218, doi:10.5665/sleep.5834.
    OpenUrlCrossRef
  9. 9.↵
    1. Barbé F,
    2. Pericás J,
    3. Muñoz A,
    4. et al.
    (1998) Automobile accidents in patients with sleep apnea syndrome. An epidemiological and mechanistic study. Am J Respir Crit Care Med 158(1):158, doi:10.1164/ajrccm.158.1.9709135.
    OpenUrlCrossRef
  10. 10.↵
    1. Terán-Santos J,
    2. Jiménez-Gómez A,
    3. Cordero-Guevara J,
    4. et al.
    (1999) The association between sleep apnea and the risk of traffic accidents. Cooperative Group Burgos-Santander. N Engl J Med 340(11):847–851, doi:10.1056/NEJM199903183401104.
    OpenUrlCrossRefPubMed
  11. 11.↵
    1. Durán-Cantolla J,
    2. Mar J,
    3. de La Torre Muñecas G,
    4. et al.
    (2004) [The availability in Spanish public hospitals of resources for diagnosing and treating sleep apnea-hypopnea syndrome] El síndrome de apneas-hipopneas durante el sueño en España. Disponibilidad de recursos para su diagnóstico y tratamiento en los hospitales del Estado español Arch Bronconeumol 40(6):259–267.
    OpenUrlPubMed
  12. 12.↵
    1. James KE,
    2. White RF,
    3. Kraemer HC
    (2005) Repeated split sample validation to assess logistic regression and recursive partitioning: an application to the prediction of cognitive impairment. Stat Med 24(19):3019–3035, doi:10.1002/sim.2154.
    OpenUrlCrossRefPubMed
  13. 13.↵
    1. Viner S,
    2. Szalai JP,
    3. Hoffstein V
    (1991) Are history and physical examination a good screening test for sleep apnea? Ann Intern Med 115(5):356–369, doi:10.7326/0003-4819-115-5-356.
    OpenUrlCrossRefPubMed
  14. 14.↵
    1. Kushida CA,
    2. Efron B,
    3. Guilleminault C
    (1997) A predictive morphometric model for the obstructive sleep apnea syndrome. Ann Intern Med 127(8 Pt 1):581–587, doi:10.7326/0003-4819-127-8_Part_1-199710150-00001.
    OpenUrlCrossRefPubMed
  15. 15.↵
    1. Hoffstein V,
    2. Szalai JP
    (1993) Predictive value of clinical features in diagnosing obstructive sleep apnea. Sleep 16(2):118–122.
    OpenUrlPubMed
  16. 16.↵
    1. Chai-Coetzer CL,
    2. Antic NA,
    3. Rowland LS,
    4. et al.
    (2011) A simplified model of screening questionnaire and home monitoring for obstructive sleep apnoea in primary care. Thorax 66(3):213–219, doi:10.1136/thx.2010.152801.
    OpenUrlAbstract/FREE Full Text
  17. 17.↵
    1. Grupo Español de Sueño [Spanish Sleep Network]
    (2005) [National consensus document on sleep apnea and hypoapnea syndrome] Documento de consenso nacional sobre el sindrome de apneas e hipoapneas del sueño accessed 24 Apr 2018. Grupo Español de Sueño [Spanish Sleep Network] . http://www.sen.es/pdf/2005/consenso_sahs_completo.pdf.
  18. 18.↵
    1. Flahault A,
    2. Cadilhac M,
    3. Thomas G
    (2005) Sample size calculation should be performed for design accuracy in diagnostic test studies. J Clin Epidemiol 58(8):859–862, doi:10.1016/j.jclinepi.2004.12.009.
    OpenUrlCrossRefPubMed
  19. 19.↵
    1. Kharbanda AB,
    2. Taylor GA,
    3. Fishman SJ,
    4. et al.
    (2005) A clinical decision rule to identify children at low risk for appendicitis. Pediatrics 116(3):709–716, doi:10.1542/peds.2005-0094.
    OpenUrlAbstract/FREE Full Text
  20. 20.↵
    1. Plüddemann A,
    2. Wallace E,
    3. Bankhead C,
    4. et al.
    (2014) Clinical prediction rules in practice: review of clinical guidelines and survey of GPs. Br J Gen Pract 64(621):e233–e242, doi:10.3399/bjgp14X677860.
    OpenUrlAbstract/FREE Full Text
Back to top

In this issue

BJGP Open
Vol. 2, Issue 2
July 2018
Download PDF
Download PowerPoint
Article Alerts
Email Article
Citation Tools
Derivation and validation of a clinical prediction rule for sleep apnoea syndrome for use in primary care
Erika Miranda Serrano, Amanda Lopez-Picado, Aitziber Etxagibel, Alfonso Casi, Laura Cancelo, Jose Ignacio Aguirregomoscorta, Itziar Menéndez, Monica Gonzalez, Felipe Aizpuru
BJGP Open 2018; 2 (2): bjgpopen18X101481. DOI: 10.3399/bjgpopen18X101481
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo

Jump to section

Keywords

Intended for Healthcare Professionals

 
 

British Journal of General Practice