Derivation and validation of a clinical prediction rule for sleep apnoea syndrome for use in primary care

Background Several clinical prediction rules (CPRs) are available for sleep apnoea-hypopnoea syndrome (OSAH), but they are difficult to apply in primary care (PC). Aim Derivation and validation of a CPR using simple measurements available in PC. Design & setting A prospective study conducted in health centres from the area of influence of three Spanish hospitals. Method Patients (aged 18–70 years) who attended for any reason; who presented with at least one of the three key symptoms for OSAH (snoring, breathing pauses while sleeping, and daytime sleepiness); and who were not undergoing non-invasive ventilation or prior treatment with continuous positive airway pressure (CPAP) were included. Anthropometric data, smoking habit, comorbidities, and Epworth test were collected. Patients were subsequently referred to the sleep unit (SU), where the decision was taken whether or not to instigate treatment. A multivariate logistic model was constructed using a sub-sample and scores assigned based on the regression coefficients; the CPR was validated with the remaining sample. Both receiver operating characteristic (ROC) curves were plotted and the sensitivity, specificity, and predictive values calculated. Results The derivation sample comprised 352 patients, with 260 in the validation sample. The final factors (arterial hypertension [AHT], age, body mass index [BMI], and sex) were used to develop a rule with scores ranging from 0.00–5.50. The cut-off point that optimises the area under the curve (AUC) is ≥2.50 points (AUC = 0.78; sensitivity = 86%; specificity = 54%; positive predictive value [PPV] = 45%; negative predictive value [NPV] = 90%; likelihood ratio [LR] = 0.26). The properties for the validation sample with this cut-off point are as follows: AUC = 0.68; sensitivity = 81%; specificity = 43%; PPV = 61%; NPV = 68%; LR = 0.44. Conclusion As in similar cases, the specificity is low, meaning that healthy people are referred to a specialist. A negative result rules out the disease in most cases.


Introduction
The incidence of complete Achilles tendon rupture is 18 per 100 000 patient-years 1 and is usually diagnosed clinically by GPs. The extent of clinical misdiagnosis is unknown in Norway, but may be high. 2 This is important as delayed treatment has unfavourable consequences. 1, 3 We report how a GP, with no clinical ultrasound experience, recorded images with a pocket-sized ultrasound device (PSUD) under supervision to confirm a complete Achilles tendon rupture. This could present a new indication for GP ultrasound.

Case report
A 36-year-old man experienced acute pain above the right heel accompanied by an audible snap while sprinting. He immediately had difficulty walking and 3 hours later consulted an on-call GP. Posterior ankle swelling with a tender depression 3 cm proximal to the calcaneum was found. Active plantar flexion against resistance was weak and Simmonds-Thompson test was 'partially positive' on applying a strong calf-squeeze. Based on these findings, calf muscle rupture was diagnosed as the Achilles tendon was thought to be intact. The patient was advised to elevate the foot and wait 2 weeks for improvement. Two days later a second GP, who was aware of a history of an audible snap, considered complete tendon rupture and reexamined the patient. Findings included an absent right heel raise due to weakness, minimal active plantar flexion against gravity and lying prone, significant right ankle swelling without bruising, and an altered angle of declination. Palpation elicited no ankle bony tenderness, yet a painful gap was identified 6 cm proximal from the calcaneal attachment, along the line of the Achilles tendon. Simmonds-Thompson's test was clearly positive. The positive Simmond's triad indicated a clinical diagnosis of complete rupture of the Achilles tendon. A 3.4-8 MHz linear array probe PSUD (VScanÔ dual probe, GE Healthcare), set at a depth of 3.5 cm, was used under the supervision of a rheumatologist experienced in ultrasound. The tendon was enlarged from 1 cm to 6 cm above the calcaneal insertion, where a clear gap was seen ( Figure 1). Two hours later a radiologist-performed ultrasound (LOGIQ E9Ô, GE Healthcare) and reported an enlarged distal tendon and a complete rupture at 5-6 cm from the calcaneal attachment, creating a 2.7 cm blood-filled gap ( Figure 2). Surgical exploration 8 days post-injury found a complete Achilles tendon rupture '5-10 cm above the ankle joint'.

Discussion
Tromsø Hospital serves a large area with a population of approximately 160 000. Between 2010-2014 an average of 21 patients per year were referred by their GP for suspected Achilles rupture.

Introduction
The incidence of complete Achilles tendon rupture is 18 per 100 000 patient-years 1 and is usually diagnosed clinically by GPs. The extent of clinical misdiagnosis is unknown in Norway, but may be high. 2 This is important as delayed treatment has unfavourable consequences. 1, 3 We report how a GP, with no clinical ultrasound experience, recorded images with a pocket-sized ultrasound device (PSUD) under supervision to confirm a complete Achilles tendon rupture. This could present a new indication for GP ultrasound.

Case report
A 36-year-old man experienced acute pain above the right heel accompanied by an audible snap while sprinting. He immediately had difficulty walking and 3 hours later consulted an on-call GP. Posterior ankle swelling with a tender depression 3 cm proximal to the calcaneum was found. Active plantar flexion against resistance was weak and Simmonds-Thompson test was 'partially positive' on applying a strong calf-squeeze. Based on these findings, calf muscle rupture was diagnosed as the Achilles tendon was thought to be intact. The patient was advised to elevate the foot and wait 2 weeks for improvement. Two days later a second GP, who was aware of a history of an audible snap, considered complete tendon rupture and reexamined the patient. Findings included an absent right heel raise due to weakness, minimal active plantar flexion against gravity and lying prone, significant right ankle swelling without bruising, and an altered angle of declination. Palpation elicited no ankle bony tenderness, yet a painful gap was identified 6 cm proximal from the calcaneal attachment, along the line of the Achilles tendon. Simmonds-Thompson's test was clearly positive. The positive Simmond's triad indicated a clinical diagnosis of complete rupture of the Achilles tendon. A 3.4-8 MHz linear array probe PSUD (VScanÔ dual probe, GE Healthcare), set at a depth of 3.5 cm, was used under the supervision of a rheumatologist experienced in ultrasound. The tendon was enlarged from 1 cm to 6 cm above the calcaneal insertion, where a clear gap was seen ( Figure 1). Two hours later a radiologist-performed ultrasound (LOGIQ E9Ô, GE Healthcare) and reported an enlarged distal tendon and a complete rupture at 5-6 cm from the calcaneal attachment, creating a 2.7 cm blood-filled gap ( Figure 2). Surgical exploration 8 days post-injury found a complete Achilles tendon rupture '5-10 cm above the ankle joint'.

Discussion
Tromsø Hospital serves a large area with a population of approximately 160 000. Between 2010-2014 an average of 21 patients per year were referred by their GP for suspected Achilles rupture.

Introduction
The incidence of complete Achilles tendon rupture is 18 per 100 000 patient-years 1 and is usually diagnosed clinically by GPs. The extent of clinical misdiagnosis is unknown in Norway, but may be high. 2 This is important as delayed treatment has unfavourable consequences. 1, 3 We report how a GP, with no clinical ultrasound experience, recorded images with a pocket-sized ultrasound device (PSUD) under supervision to confirm a complete Achilles tendon rupture. This could present a new indication for GP ultrasound.

Case report
A 36-year-old man experienced acute pain above the right heel accompanied by an audible snap while sprinting. He immediately had difficulty walking and 3 hours later consulted an on-call GP. Posterior ankle swelling with a tender depression 3 cm proximal to the calcaneum was found. Active plantar flexion against resistance was weak and Simmonds-Thompson test was 'partially positive' on applying a strong calf-squeeze. Based on these findings, calf muscle rupture was diagnosed as the Achilles tendon was thought to be intact. The patient was advised to elevate the foot and wait 2 weeks for improvement. Two days later a second GP, who was aware of a history of an audible snap, considered complete tendon rupture and reexamined the patient. Findings included an absent right heel raise due to weakness, minimal active plantar flexion against gravity and lying prone, significant right ankle swelling without bruising, and an altered angle of declination. Palpation elicited no ankle bony tenderness, yet a painful gap was identified 6 cm proximal from the calcaneal attachment, along the line of the Achilles tendon. Simmonds-Thompson's test was clearly positive. The positive Simmond's triad indicated a clinical diagnosis of complete rupture of the Achilles tendon. A 3.4-8 MHz linear array probe PSUD (VScanÔ dual probe, GE Healthcare), set at a depth of 3.5 cm, was used under the supervision of a rheumatologist experienced in ultrasound. The tendon was enlarged from 1 cm to 6 cm above the calcaneal insertion, where a clear gap was seen ( Figure 1). Two hours later a radiologist-performed ultrasound (LOGIQ E9Ô, GE Healthcare) and reported an enlarged distal tendon and a complete rupture at 5-6 cm from the calcaneal attachment, creating a 2.7 cm blood-filled gap ( Figure 2). Surgical exploration 8 days post-injury found a complete Achilles tendon rupture '5-10 cm above the ankle joint'.

Introduction
Last summer our small medical team visited the Calais 'Jungle'. Since that time much has changed and the camp is being demolished and by the time this article is read, it will probably be long gone. Some youngsters are finally being brought to the UK under the 'Dubs' amendment. However, once this camp is cleared it will not solve the ongoing flight of refugees from war torn areas: other camps are already appearing.

July 2016
A young Afghan man caught his finger on a sharp point while trying to cross a barbed wire fence. The finger was partially degloved. He attended the local hospital, where they placed a few sutures, but now, 2 weeks later, the skin is necrotic and the underlying tissue looks infected. He is in danger of losing his finger. A middle-aged Sudanese man has been having rigors and is generally unwell. He says it is similar to when he last had malaria.
A young Ukrainian woman complains of lower back pain and urinary frequency. The paths of these three people may never have crossed; yet here they are, denizens of the Calais Jungle. They turn up to a makeshift primary care 'clinic' that we set up in the heart of the unofficial refugee camp one weekend in July 2016.
With only basic medical supplies, we are immediately challenged by what we see. How can we arrange secondary care for the young Afghan in danger of losing his finger? We try to persuade him to return to the original local hospital, but he is reluctant. It was not a good experience for him the first time round.
With the other two patients, it is easier. They can attend the Salam clinic run by a local association during weekdays. Later, we receive word that malaria has been confirmed in our Sudanese patient.
More people arrive, presenting with scabies, rat bites, tinea, chest infections, and wheezing from inhaling smoke from fires lit to cook and keep warm in their tents at night. We examine a severely malnourished 2-year-old boy. We meet several of the camp's 600 unaccompanied children, at grave risk of sexual exploitation. We learn that there is inadequate safeguarding in place to protect them. A young Eritrean man comes in worried about his eye. He has sustained direct ocular trauma from a rubber bullet, and will never see normally again out of that eye. We see haematomas from police batons, and hear about children being exposed to tear gas again and again ( Figure 1).

The reality
These are no ordinary patients. They have travelled far from home to escape war, poverty, and misery. They have endured personal odysseys to get here, experienced untold hardships, and suffered unimaginable privations. Many have survived the loss of their families, torture, and rape. Their journeys over, for the moment at least, they must make their homes in the Calais Jungle. Their new shelters are in many cases mere tarpaulin covers, and their new beds just rugs on the ground. They own next to nothing. There is little for them to do, besides use their ingenuity to cross the English Channel in search of a better life. They are vulnerable to exploitation, crime, injury, and disease. Potentially violent clashes with local police, with other ethnic groups resident in the Jungle, or local far Introduction OSAH affects 2-4% of the population. 1 Its prevalence increases with age. 2 Several studies have found an association between OSAH and numerous diseases, including hypertension, 1,3,4 and cardiovascular 5 and cerebrovascular disease. 6 OSAH is associated with a worse quality of life, 7 and a higher number of work 8 and traffic accidents. 9,10 Despite the risks associated with OSAH, only 5-9% of the population with severe OSAH is receiving treatment. 11 Three key symptoms (snoring, pauses, and daytime sleepiness) have been used in standard clinical practice (SCP) to determine the probable presence of OSAH. However, none of these predicts the disease per se, as they are all very common in the general population.
This situation has outlined the need to develop a tool that discriminates patients presenting a more severe OSAH from those in whom the disease is not present or is unlikely to lead to its complications. Therefore, a tool like a CPR would be most useful for PC physicians, who could apply it to decide whether or not to refer a patient to specialised care.
A CPR is considered to be any decision-making instrument prepared from a minimum of three variables obtained from the case history, physical examination, or simple diagnostic tests. 12 Although several attempts have been made to create a CPR, 13-16 none of them is used in SCP. In some cases, this is because they use apnoea-hypopnoea index (AHI) values that are of little clinical interest (AHI = 5 or AHI = 10) 17 or, in the case of the neck circumference, due to the collinearity with BMI, which is in the final model as current guidelines indicate the use of CPAP for AHI values >30, or 15 if accompanied by related comorbidity or severe symptomatology.
Kushida et al 15 reported a high-quality prediction (sensitivity = 98%; specificity = 100%), but this rule is difficult to apply in SCP as the collection of the variables requires meticulous training. Chai-Coetzer et al 17 developed the OSA50 screening tool plus overnight oximetry, which shows a very good NPV, with the drawback for its use in PC being that it requires an at-home overnight test.
This study proposes the derivation and validation of a prediction rule, using measurements easy to obtain in PC, in order to distinguish between patients at high and low risk of suffering OSAH.

Method Design and patients
The CPR was developed using a sample of patients recruited prospectively between February 2011 and August 2012. The derivation sample comprised patients who attended the PC departments of two urban health centres and were referred to the SU at the Araba University Hospital. Patients of the validation sample were recruited at various health centres and referred to the Hospital de Galdakao-Usansolo and the University Hospital in Valdecilla (Santander).
All subjects aged 18-70 years and presenting at least one of the three key symptoms for OSAH (snoring, daytime sleepiness, or breathing pauses while asleep) were recruited on spontaneous consultation of a PC physician for any reason. Patients with a previous diagnosis of OSAH, prior CPAP treatment or non-invasive mechanic ventilation were excluded. Participants provided consent to participate in the study.

Data collection
After collection of the information in the PC centres, participants were asked to attend the SU at the corresponding hospital. The doctor at the SU, blinded to the information collected in the PC setting,

Reliability analysis
The only information obtained in both the PC centre and the SU, the BMI, was used for the interobserver reliability analysis in order to validate the information collected by the PC physicians.

Variables
The characteristic to be predicted is the clinical decision of the sleep specialist as regards indication for specific diagnostic tests (PSG, RP, or both tests); and indication of treatment (CPAP, MAD, or a postural device).
The predictive variables taken into consideration were sex, age, weight, height, neck circumference, snoring (five or more times per week), breathing pauses while asleep reported by the partner or companion, daytime hypersomnia, accidents in the past year due to drowsiness, morning tiredness, morning sensation of asphyxia, AHT, heart failure, diabetes mellitus (DM), smoking habit, alcohol consumption, and daytime sleepiness according to the Epworth scale.
In addition, BMI, neck circumference, and Epworth scale were collected at the SUs. Along with this information, global AHI, supine AHI, and T90 (the percentage of time during which arterial O 2 saturation is <90%) were obtained if available from the PSG and/or RP reports.

Sample size
According to Flahault et al, 18 approximately 298 cases with the disease, or 750 subjects, are required to estimate an expected sensitivity for the test of 0.95 with a confidence limit (95%) of not less than 0.90, considering a prevalence of 40% for the disease in the study population (data from the SU in Vitoria).
Two hundred subjects are required for the validation phase for a power of 90% to confirm the hypothesis that the sensitivity of the CPR in the validation population does not differ from that obtained for the derivation sample by >3%.

Statistical analysis Treatment of variables
Means and standard deviations were used to describe continuous variables, and frequencies and percentages for categorical variables.
Continuous variables were categorised in order to be applied in a simple manner. Looking at the distribution of results, values close to the median or values close to the tertiles were used as cut-off points.
Characteristics of the patients in the derivation and validation samples were compared using the X 2 test or the student's t-test for continuous variables. SPSS (version 22) and R freeware (version 3.1.1) were used for all analyses.

Derivation of the prediction rule
According to Kharbanda et al, 19 those variables with <10% lost values were selected as potential predictive variables, which did not exclude any of those from the model. The characteristics of treated patients were compared with those of untreated patients using a univariate logistic regression. A value of P<0.2 indicated that a variable was potentially predictive and should be taken into consideration during the multivariate analysis. Subsequently, the final multivariate logistic regression model was described using a stepwise variable selection method and the LR test to compare two models. The criterion for introducing variables into the model was P 0.05. A backward stepwise model was also prepared using the LR test in order to ensure the most parsimonious rule possible. The criterion for excluding variables from the model was P>0.05. Both variable selection criteria led to the same final model. The goodness-of-fit was evaluated using the Hosmer-Lemeshow test.
The final scores for the prediction rule were obtained from the logistic regression coefficients using the lowest risk category as reference. 19 Predictive ability and validation of the rule The predictive ability of the rule was evaluated for each sample using ROC curves, which present the decision reached with respect to the score. To evaluate the predictive accuracy of the rule for the derivation and validation samples, different cut-off points were established and 2Â2 tables were constructed to calculate the following measures: sensitivity, specificity, PPVs, and NPVs (including 95% confidence intervals [CI]). Table 1 shows the CPR in the format it would be used in PC.

Reliability analysis
The kappa statistic was calculated to compare the interobserver reliability for measurement of each of the variables included in the CPR. A kappa value >0.60 was considered to be acceptable.

Results
Informed consent was obtained from 620 patients. Eight were excluded: five because they were aged >75 years; two because they did not have any of the key symptoms; and one for technical reasons. Full information (predictive variables, and indication for diagnostic tests and/or treatment) was available for 278 of the 352 subjects in the derivation sample (79.0%) and 233 of the 260 subjects (89.6%) in the validation sample ( Figure 1). The complementary examination was conducted for 226 of the 231 subjects from the validation sample who attended the SU (97.8%), and 122 of them (54.0%) had an indication for treatment with CPAP or another device (P<0.001). Table 2 shows the characteristics of both samples. Differences can be seen between validation and derivation samples regarding sex, BMI, breathing pauses, daytime hypersomnia, history of accidents due to drowsiness, morning tiredness, sensation of asphyxia, neck circumference, AHT, DM, smoking habit, score on the Epworth scale, and final decision by the SU specialist. Table 3 shows the result of the univariate analysis, in which the risk factors were compared with the therapeutic decision. Subjects requiring treatment were more likely to be older, male, obese, with larger neck circumferences, and with chronic disease. The final multivariate model (Hosmer-Lemeshow goodness-of-fit P = 0.903), can be seen in Table 4, with the variables included and the coefficients estimated, along with the scores for the CPR calculated using these coefficients. According to this rule, 1.50 points were awarded each to males, to obese patients, and to those aged >60 years. Subjects aged 46-59 years were awarded 1.25 points, with an additional point for those who were hypertensive. Therefore, the total score for the rule ranges between 0.00 and 5.50 points.
The AUCs for the derivation sample (77.8%) and for the validation sample (68.1%) can be seen plotted in Figure 2.
The sensitivity, specificity, PPV, and NPV for the different cut-off points on the scale are provided in Table 5. To guarantee a high sensitivity, and therefore the detection of patients who require treatment, cut-off points of 1.50, 2.25, 2.50, or 2.75 are proposed as possible discriminatory values. The rule has a better diagnostic ability for the derivation than for the validation sample. However, selection of the most conservative cut-off point possible (1.50) leads to a sensitivity of 97.5% for the derivation sample, although this implies that only 24 of 233 (10.3%) subjects attending a PC are free of diagnosis.

Research
For the reliability, the kappa index for this measure is 0.834, although it should be noted that this analysis was performed for a total of 334 out of 612 cases (54.6% of the total sample).

Discussion Summary
This study has derived and validated a CPR for identifying patients with suspected OSAH in PC centres. Four independent predictive factors (AHT, age >46 years, BMI !30, and male sex) were selected, and a rule ranging from 0.00 to 5.50 points was generated combining them. Despite being variables 'classically' associated with the presence of OSAH, neck circumference and breathing pauses do not form part of the final model as these variables are confounded by sex and, in the case of the neck circumference, due to the collinearity with BMI, which is in the final model. Although the final goal of a prediction rule is to obtain the highest possible sensitivity, in this study two cut-off points are proposed for two different scenarios. In this case, selecting a cut-off point of !1.50 points (highest sensitivity for the rule with the derivation sample) means that sleep tests must be performed in a large number of healthy patients due to the low specificity. This implies the overuse of already limited resources, in addition to the indirect costs and inconveniences generated to the patient.  The cut-off point that achieves the best results in terms of sensitivity and specificity for the derivation sample is !2.50 points, with a lower sensitivity counterbalanced by a marked increase in specificity.

Strengths and limitations
However, selection of this cut-off point for the validation sample leads to markedly different results as both sensitivity and specificity decrease. This nevertheless appears to be the most reasonable option in light of the results. The PPV of the validation sample is also 15-20% higher. This is likely to be mainly due to the fact that the derivation and validation populations sampled differ in terms of baseline risk profile regarding the disease. Thus, the derivation sample includes a higher proportion of males, with a higher BMI and, in general, a higher number of key symptoms for the disease, and therefore a higher pre-test probability.
The main limitation of this study is the difference between the derivation and validation groups; the values obtained during validation are markedly lower than those obtained during derivation of  this rule. However, this situation merely reflects the study's setting, where access to some healthcare services is unequal and depends on the healthcare organisation. The present authors also decided to limit the study to population those aged <70 years, with the aim of identifying the disease in a population with a low prior probability of having it. It is doubtful that a rule valid for asymptomatic people works the same way in an aged population, with a higher probability and a greater possibility of confounding due to comorbidity. Similar studies have also chosen this limit, notably the aforementioned work by Chai-Coetzer. 17 Comparison with existing literature Although the results obtained do not demonstrate as high a sensitivity and specificity as the prediction rules developed previously, 14-17 the present authors believe that the studied rule is easier to apply in PC settings, resulting in a more rational referral to specialised units. This means that the rule is useful despite the drawbacks discussed above.
A recent study conducted in the UK 20 surveyed PC physicians regarding the use of prediction rules in clinical practice. The results showed that a high percentage of physicians were unaware of and/or did not use prediction rules widely validated in PC, as they considered these rules to be of little use, or preferred their own medical judgement. This could possibly be explained by the complexity of such rules. Therefore, a rule that is easy to use could easily be implemented in already saturated PC centers.

Implications for practice
In conclusion, this study presents a CPR for diagnosing OSAH which, despite presenting a lower sensitivity and specificity than other such rules, is more accessible and easier to use. This makes it an ideal tool for use in PC, allowing the referral of patients susceptible to presenting the disease to SUs while ensuring a more rational use of the resources available.

Funding
This study was funded by the Carlos III Health Institute (reference number: PI10/01249).

Ethical approval
Participants provided consent to participate in the study, which was approved by the corresponding Clinical Research Ethics Committees.

Provenance
Freely submitted; externally peer reviewed.