BACKGROUND

The National Academy of Medicine1 and Agency for Healthcare Quality and Research2, 3 have spotlighted concerning levels of burnout among clinicians and healthcare staff, particularly in primary care. High levels of burnout are concerning not only for clinician and staff well-being. A burntout workforce may adversely affect clinical quality, patient experience, and costs of care.1

Several burnout-related initiatives aim to support the “quadruple aim” of a sustainable clinician and staff work experience in addition to improved patient experience, quality, and lower costs.4 The American Medical Association’s “STEPS Forward” modules offer guidance on practice transformation and clinician and trainee well-being.5 The Society for General Internal Medicine made burnout a theme of its 2017 Annual Meeting.6 The American Board of Family Medicine added work experience questions to its 2016 recertification registration to better understand and track burnout,7 and CEOs of leading healthcare organizations have issued a call to action that argues for regular measurement of physician well-being.8

Within this context of heightened attention to clinician and staff well-being, greater understanding of the instruments used to measure burnout is essential. Two common measures are the Maslach Burnout Inventory (MBI) and a five-choice, single item based on self-defined burnout. The MBI, considered an industry standard, has been fielded across large samples of diverse occupations in multiple countries. It is composed of three dimensions: emotional exhaustion, cynicism (or depersonalization), and personal accomplishment (or professional efficacy). The 16-item General MBI survey uses the terms cynicism and personal accomplishment while the 22-item Health Services Personnel survey uses the analogous terms depersonalization and professional efficacy, with results consistent across survey versions.9 MBI instruments require a license fee to administer.

The self-defined burnout measure is one of the ten survey questions on the “Mini-Z” work experience instrument developed by Linzer and colleagues.10 The item originated from a study of burnout in HMOs11 and has subsequently been included in several major studies.7, 12,13,14 It is free to use.

Estimates of Clinicians and Staff Burnout

The most recent national estimate of burnout from MBI subscales identified high emotional exhaustion among 46.9% of physicians and high cynicism among 34.6% of physicians, with 54.4% of physicians reporting one or both symptoms. Primary care physicians from that study reported even higher burnout levels; more than 60% expressed high emotional exhaustion and/or cynicism.15 Other studies indicate burnout is also high among healthcare staff, with high emotional exhaustion ranging from 30 to 40%.16, 17 However, studies using the self-defined burnout measure have found lower point estimates: about one in four family physicians and general internists report “burning out,” “persistent burnout symptoms,” or “complete burnout.”7, 12, 18 These divergent estimates of burnout prevalence raise questions as to whether study populations differ in their well-being, or how MBI and self-defined measures perform differently.

Published Comparisons of Burnout Measures

Few studies directly compare how the MBI and self-defined burnout measures perform in the same sample of physicians. One physician survey found the self-defined burnout measure strongly correlated with the MBI emotional exhaustion subscale (r = 0.64, p < 0.0001); the cynicism subscale was less strongly correlated (r = 0.324, p value not reported).18 A second study of Australian oncology workers also found the self-defined burnout measure and MBI emotional exhaustion subscale strongly correlated (r = 0.68, p < 0.0001).19 Only one study used an analytic approach other than correlations. That study, a survey of rural physicians and advance practice clinicians, examined whether the self-defined burnout measure predicted high and low MBI subscale burnout categories using multivariate linear mixed models. The self-defined burnout measure significantly predicted high emotional exhaustion but did not predict low emotional exhaustion or any category of cynicism.20

Prior studies have also analyzed sensitivity and specificity for MBI single-item measures (one item from each MBI subscale’s five items). A correlation of 0.76–0.83 was found for the MBI emotional exhaustion single-item vs. subscale. A correlation of 0.61–0.72 was found for the MBI cynicism single-item vs. subscale.21 Compared to the self-defined burnout single-item, the MBI emotional exhaustion single-item had a high correlation (r = 0.79) and sensitivity and specificity over 80%.17

These results suggest that the single-item self-defined burnout measure and MBI subscales have strong agreement. However, no comparison to our knowledge has described sensitivity and specificity for the self-defined burnout measure and MBI subscales. Moreover, we are not aware of any previous study examining concurrent validity of the self-defined and MBI responses with related, validated work environment measures. Given many national surveillance efforts and program evaluations using the self-defined burnout measure, there is need for greater understanding among policy makers, researchers, and healthcare leaders on how results compare with the more established MBI burnout subscales.

OBJECTIVE

We compared the self-defined burnout measure and MBI in the same sample of primary care clinicians and staff. Our aims were to (1) compare the prevalence of burnout from the different measures, (2) test the sensitivity and specificity of the self-defined burnout measure to identify individuals experiencing high burnout compared to standard MBI benchmarks, and (3) determine if the self-defined burnout measure and MBI have similar associations with a clinic team culture survey measure previously found to be significantly associated with MBI scores22 and a workplace atmosphere survey measure previously found to be significantly associated with the self-defined burnout measure.23

METHODS

Design

This study was a cross-sectional survey, approved by the Institutional Review Board of the University of California, San Francisco (protocol numbers 11-08048 and 17-23324).

Participants

We surveyed clinicians and staff working in primary care clinics in three San Francisco area health systems: a university-run clinic network, a network of neighborhood and hospital-based clinics administered by a county health department, and a large private medical group. All clinicians and staff at the university and county clinics and all clinicians at the private group were eligible to participate. Clinicians consist of physicians of family and internal medicine, physician assistants, and nurse practitioners. Staff members include registered nurses, medical assistants, and administrative support. The survey was fielded between November 2016 and January 2017, and was primarily administered electronically. Each person received an e-mail invitation to complete the survey, with up to five reminder e-mails to non-respondents. Paper surveys were administered during staff meetings at some county health network sites based on leadership request. Respondents at two systems were entered into a $25 gift card raffle; the third system elected to give each respondent $50 for participation.

Measures

The survey included the 16-item MBI General Survey subscales for emotional exhaustion and cynicism as well as a single-item, self-defined burnout measure. MBI subscales were each composed of five burnout symptoms. Respondents rated how often they experience each symptom from 0 (never) to 6 (every day), and responses were summed for each subscale (composite score of 0–30 points). High, medium, and low MBI burnout cut points were based on a distribution of the composite score into terciles from a reference population.24 High emotional exhaustion was defined as a composite score greater than or equal to 16; high cynicism was a composite score greater than or equal to 11.9 We primarily analyzed MBI scores based on the presence of high emotional exhaustion or high cynicism, as done in commonly cited national prevalence estimates.15, 25

Self-defined burnout was a single question that assessed burnout on a scale from 1 to 5. Most studies using this measure define high burnout as answering positively to option 3, 4, or 5.7, 12 Response options were as follows: (1) “I enjoy my work. I have no symptoms of burnout”; (2) “Occasionally I am under stress, and I don’t always have as much energy as I once did, but I don’t feel burned out”; (3) “I am definitely burning out and have one or more symptoms of burnout, such as physical and emotional exhaustion”; (4) “The symptoms of burnout that I’m experiencing won’t go away. I think about work frustrations a lot”; and (5) “I feel completely burned out and often wonder if I can go on. I am at the point where I may need some changes or may need to seek some sort of help.”

The survey also included a validated 7-item measure of team culture previously found to be associated with the MBI. Team culture included agreement with statements such as, “The group of staff and providers I work with most regularly work well together as a team” and “I can rely on other people at my clinic to do their jobs well.” Respondents rated each item from 1 (strongly disagree) to 10 (strongly agree). A composite score was calculated as an average across the seven items.22 Workplace atmosphere was assessed on a scale from 1 to 5 in response to: “Which number best describes the atmosphere in your primary work area?” Response anchors included 1 (calm), 3 (busy but reasonable), and 5 (hectic, chaotic).10, 23

Data Analysis

All analyses were conducted using Stata 1326 and stratified by clinician or staff respondent. We stratified clinician and staff analyses to be consistent with other reportings14 and based on prior findings of differences in burnout between clinicians and staff.22 We also conducted sub-analyses by gender and part-time work status to confirm whether results were consistent. Correlations for comparing our results with other studies were calculated using Pearson’s correlation coefficients to measure the association between the self-defined measure with the MBI emotional exhaustion and MBI cynicism subscales.17, 19, 27

We used a self-defined burnout cut point of 3 (definitely burning out) or above to test sensitivity and specificity for detecting respondents with high burnout compared to standard MBI classification. We also explored cut points of 2 (under stress) and 4 (persistent symptoms) to assess sensitivity and specificity trade-offs and produced area under receiver operator curves (AUC). An AUC of 1.0 indicates a perfect diagnostic test; above 0.9 indicates excellent discrimination; 0.8–0.9 is good; 0.7–0.8 is fair; and 0.5–0.7 is non-discriminating to poor discrimination.28, 29

We assessed concurrent validity by calculating the proportion of burned out and not burned out respondents for both the self-defined and MBI measures in connection with (1) strong team culture and (2) hectic or chaotic work atmosphere. Unadjusted odds ratios were calculated to compare associations across the two burnout measures.

RESULTS

The response rate was 74%. Four hundred forty-four of 592 clinicians and 606 of 826 staff responded. Respondents were predominantly female (Table 1). About half had worked with their health system more than 10 years. Almost all staff (90%) worked more than 20 h a week while fewer clinicians (27%) worked six or more half-days in clinic.

Table 1 Characteristics of Survey Respondents and Burnout Levels

High burnout based on the MBI—high emotional exhaustion or high cynicism—was reported by 52% of clinicians (95% CI: 47–57%) and 46% of staff (95% CI: 42–50%). Burnout levels for emotional exhaustion and cynicism subscales individually are reported in Table 1. High self-defined burnout based on a score of 3 (“definitely burning out”) or greater was reported by 29% of clinicians (95% CI: 25–33%) and 31% of staff (95% CI: 28–35%). The lower proportion of self-defined burnout was statistically significant among both clinicians and staff compared to the overall MBI measure and MBI emotional exhaustion subscale (McNemar’s test, p < 0.001) but not compared to the MBI cynicism subscale (Fig. 1).

Figure 1
figure 1

High burnout based on MBI subscales and the self-defined burnout measure.

The correlation between the self-defined burnout measure and MBI exhaustion subscale was 0.63 for both clinicians and staff (p value < 0.001). The correlation between the self-defined burnout measure and MBI cynicism subscale was 0.57 for clinicians and 0.48 for staff (p value < 0.001).

Using the common cut point of 3 or greater for self-defined burnout, sensitivity was 50.4% among clinicians and 58.6% among staff—i.e., the proportion of respondents with MBI-assessed burnout whose self-defined response also identifies burnout. Specificity—the proportion of respondents without MBI-assessed burnout who also did not report self-defined burnout—was 94.7% for clinicians and 92.3% for staff. A higher cut point of 4 on the self-defined burnout measure dropped sensitivity to 11.5% among clinicians and 24.3% among staff and increased specificity to 100% for clinicians and 98.1% for staff. A lower cut point of 2 greatly increased sensitivity to 98.2% among clinicians and 91.4% among staff, however decreased specificity to 41.8% for clinicians and 44.4% for staff (Table 2). The AUC was 0.82 for clinicians and 0.81 for staff (Fig. 2). Additional sensitivity, specificity, and AUC estimates for the individual MBI emotional exhaustion and cynicism subscales are provided as an online appendix. Sub-analyses stratified by gender and clinician half-days per week yielded similar results.

Table 2 Sensitivity and Specificity of Self-Defined Single-Item Burnout Measure Cut Points for Detecting High Burnout as Measured by MBI Subscales
Figure 2
figure 2

Area under the receiver operator curve (AUC) for the MBI (high exhaustion or high cynicism) vs. self-defined burnout measure.

In an assessment of concurrent validity, strong team culture was significantly associated with lower burnout for both the MBI (clinician OR 0.34, 95% CI 0.23–0.51) and self-defined burnout (clinician OR 0.33, 95% CI 0.22–0.51). A hectic or chaotic environment was significantly associated with greater burnout for both the MBI (clinician OR 3.56, 95% CI 2.36–5.36) and self-defined burnout measure (clinician OR 3.07, 95% CI 1.99–4.72) (Table 3).

Table 3 Concurrent Validity: Associations with Team Culture and Workplace Atmosphere for MBI and Self-Defined Burnout Measures

DISCUSSION

The most important finding from our study is that the prevalence of burnout among primary care clinicians and staff differed considerably depending on survey instrument. For example, burnout prevalence among clinicians was more than 50% higher using the MBI compared to the self-defined measure’s cut point of 3 (52 vs. 29%). Similarly, the self-defined measure’s sensitivity to detect individuals with burnout in our sample missed about 50% of clinicians and 41% of staff that MBI symptoms had classified as experiencing burnout. A lower self-defined cut point of 2 increased sensitivity to above 90%, but at the expense of substantially decreased specificity. One explanation for the lower self-defined burnout point estimate may be reluctance to self-identify as burned out given an allusion to depression or ineffectiveness. In contrast, the MBI allows individuals to identify with burnout symptoms without directly identifying as burned out.

Consistent with other studies, we found a strong, significant correlation between self-defined burnout and the MBI exhaustion subscale and a modest, significant correlation between self-defined burnout and the MBI cynicism subscale. Our analyses add to prior studies by reporting an AUC of 0.81–0.82, indicating moderate to good discrimination between the self-defined burnout measure and MBI.29, 30

Our results are also the first to demonstrate concurrent validity for both burnout measures in association with team culture, which had only been examined with the MBI,22 and workplace atmosphere, which had only been examined with the self-defined measure.10, 23 Strong team culture was significantly associated with about one-third lower burnout for both burnout measures. A chaotic workplace atmosphere was significantly associated with about three times higher burnout for both burnout measures. The similar magnitude and variance of associations for both burnout measures suggests that the two burnout measures would perform similarly when exploring other burnout predictors.

Our findings have implications for using and interpreting the self-defined and MBI burnout measures. The self-defined burnout measure appears to be an acceptable alternative to the MBI if primary aims are to track burnout trends within a single population or measure work environment factors that predict burnout. However, our results indicate it would be inappropriate to directly contrast high burnout estimates from the self-defined measures and MBI subscale measures.

Researchers and health system leaders should use caution when comparing burnout prevalence across different populations or studies. For example, a recently published study using the self-defined measure concluded that physician burnout may be decreasing in the USA.7 Differences may actually be due to the measurement instrument used and the self-defined measure’s low sensitivity relative to the MBI.

The National Academy of Medicine notes, “[Burnout] terminology and measurement tools used vary substantially across studies…hampering efforts to quantitatively summarize outcomes (for example through meta-analyses), and slowing the rate of advancement in the field.”1 Similar to other researchers who have identified need for greater consistency in defining and reporting burnout,31, 32 our study underlines these challenges to compare and pool findings across studies when different burnout measures are used.

The self-defined burnout measure has several attractive qualities. It does not require a license fee, has low response burden, and may have more face validity to healthcare workers than a multi-item scale score.24 One drawback of the self-defined burnout measure is limited validity testing in contrast to MBI measures, which have been associated with outcomes such as clinical diagnosis of depression.33 Our study also demonstrates the limitations due to the ordinal nature of the self-defined measure. Forty to fifty percent of respondents in our sample selected the level 2 category, a skewed response that creates a large step-off effect. Consequently, the self-defined cut point cannot be smoothly titrated to achieve an optimal balance of sensitivity and specificity relative to the MBI (Fig. 2). This aspect of the self-defined measure may also reduce its predictive and discriminant utility when analyzing burnout gradients rather than yes/no classifications.

Our study has several limitations. First, we studied clinicians and staff in three large health systems in a single region, which may limit generalizability. Yet burnout prevalence in our sample was similar to national samples of family physicians and general internists,7, 15 suggesting that respondents’ work experience resembles that of other settings. Second, our sample had a high proportion of women and part-time clinicians. Sensitivity/specificity sub-analyses stratified by gender and sessions per week did not meaningfully differ from the full sample. Third, as with any survey, response bias may influence the validity of the results. Our response rate of 74% is much higher than for most surveys of healthcare workers, mitigating potential non-response bias. Last, our study relied on survey measures of burnout and did not assess well-being with direct observation or qualitative methods.

The self-defined burnout measure and MBI each have advantages and disadvantages. We do not conclude from our study that there is necessarily a preferred burnout survey instrument. However, researchers and health system leaders addressing burnout must be aware of the measures’ different properties and lack of equivalency for assessing burnout prevalence. We recommend that organizations such as the National Academy of Medicine and Agency for Healthcare Research and Quality take the lead in developing and promoting national guidelines that establish greater consistency across burnout survey efforts. The commitment by the American Academy of Family Physician to offer members free MBI survey access is one example of constructive action by a national organization.34 Greater consistency and clarity in reporting how burnout is defined is essential to support meaningful comparisons across health systems and to enhance understanding of burnout consequences and interventions.