Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

The psychometric properties of the ‘safety attitudes questionnaire’ in out-of-hours primary care services in the Netherlands

  • Marleen Smits ,

    marleen.smits@radboudumc.nl

    Affiliation Radboud Scientific Center for Quality of Healthcare (IQ healthcare), Radboud Institute for Health Sciences, Radboud university medical center, Nijmegen, The Netherlands

  • Ellen Keizer,

    Affiliation Radboud Scientific Center for Quality of Healthcare (IQ healthcare), Radboud Institute for Health Sciences, Radboud university medical center, Nijmegen, The Netherlands

  • Paul Giesen,

    Affiliation Radboud Scientific Center for Quality of Healthcare (IQ healthcare), Radboud Institute for Health Sciences, Radboud university medical center, Nijmegen, The Netherlands

  • Ellen Catharina Tveter Deilkås,

    Affiliations Health Services Research Unit, Akershus University Hospital, Lørenskog, Norway, The Norwegian Directorate of Health, Oslo, Norway

  • Dag Hofoss,

    Affiliation Institute of Health and Society, University of Oslo, Oslo, Norway

  • Gunnar Tschudi Bondevik

    Affiliations Department of Global Public Health and Primary Care, University of Bergen, Bergen, Norway, Uni Research Health, National Centre for Emergency Primary Health Care, Bergen, Norway

Abstract

Background

The Safety Attitudes Questionnaire (SAQ) is one of the most widely used instruments to assess safety culture among healthcare providers. The ambulatory version of the SAQ (SAQ-AV) can be used in the primary care setting. Our study objective was to examine the underlying factors and psychometric properties of the Dutch translation of the SAQ-AV in out-of-hours primary care services.

Design

Cross-sectional observational study using a web-survey.

Setting

Sixteen out-of-hours general practitioner cooperatives and two call centers in the Netherlands.

Participants

Primary healthcare providers in out-of-hours services.

Main outcome measures

Item-descriptive statistics, factor loadings, Cronbach’s alpha scores, corrected item-total correlations, scale correlations.

Results

The questionnaire was answered by 853 (43.2%) healthcare professionals. In the factor analyses, 784 respondents were included; mainly general practitioners (N = 470) and triage nurses (N = 189). Items were included in the analyses based on question type and results from previous studies. Five factors were drawn with reliability scores between .49 and .86 and a good construct validity. The five factors covered 27 of the 62 questionnaire items, with three to five items per factor.

Conclusions

The Dutch translation of the SAQ-AV, with five factors, seems to be a reliable tool for measuring patient safety culture and guide quality improvement interventions in out-of-hours primary care services. The Dutch factor structure differed from the original SAQ-AV and other translated versions. In future studies, the questionnaire should be validated further by examining if there is a relationship between the responses on the SAQ-AV, patient experiences, and the occurrence of adverse events.

Introduction

To improve quality and safety in healthcare, organisations have to create a positive patient safety culture. Patient safety culture is how leader and staff interaction, attitudes, routines and practices protect patients from adverse events in healthcare [1]. The phenomenon exists in groups of people working together—in natural social units like for example hospital wards and ambulatory clinics- and not in single individuals alone [2,3]. Over the last years, the focus in patient safety research has mainly been upon hospital care [4]. Most patients, however, receive their healthcare in primary care settings, particularly in countries with a strong primary care system [5]. Primary care differs from hospital care in terms of organisational structure, administrative and clinical processes and the reasons for encounter. Therefore, also patient safety culture dimensions could differ between the settings [6].

If healthcare organisations want to improve patient safety, it is important to know more about the patient safety culture. Several instruments are available to assess safety culture [712]. A widely used instrument to measure patient safety culture is the Safety Attitudes Questionnaire (SAQ) [13]. It can be used in different healthcare setting [14]. Measurements of safety culture, which is an aspect of the organisational culture, are referred to as climates [1]. Previous research has shown that SAQ climate scores correlate with patient outcomes [7, 1516]. Moreover, the instrument may identify possible weaknesses in a clinical setting and this can stimulate quality improvement interventions [17,18]. From the original SAQ, a questionnaire for measuring safety culture in outpatient settings was developed, adjusted to and tested in the primary care setting [9,19]. This ambulatory version of the SAQ (SAQ-AV) was used in an international study entitled Patient Safety Culture in European Out-of-hours services (SAFE-EUR-OOH) which was led by a coordinating research group from Norway [19]. The study was a project of the European research network for out-of-hours primary health care (EurOOHnet) [20]. We translated the SAQ-AV into Dutch and adjusted it for specific application in out-of-hours primary care services, also called general practitioner (GP) cooperatives, in the Netherlands (see Table 1 for general characteristics of Dutch GP cooperatives).

thumbnail
Table 1. Features of general practitioner (GP) cooperatives in the Netherlands [21].

https://doi.org/10.1371/journal.pone.0172390.t001

Patient safety is of particular importance in GP cooperatives, because of a high patient throughput, diversity of urgent clinical conditions presented, identification of medical urgency during telephone contacts, and limited knowledge of the medical history of the patient. In addition, the GPs work in shifts and have to collaborate with other healthcare providers, which increases the risk of errors caused by discontinuity in information transfer [22, 23].

After translating a questionnaire into another language and applying it in a different setting, it is important to test the validity of the questionnaire in the new context. In addition, if the psychometric properties of the Dutch version of the SAQ-AV are comparable to the original questionnaire, cross-country comparisons can be performed to gain more insight into similarities and differences in patient safety culture between countries. The aim of this study was to examine the underlying factors and psychometric properties of the Dutch translation of the SAQ-AV in GP cooperatives.

Materials and methods

Setting

The study was performed in a convenience sample of 16 out-of-hours GP cooperatives and two call centers in the Netherlands (see Table 1). The two call centers performed the telephone triage of all calls to seven of the 16 GP cooperatives. The GP cooperatives were spread over the East, South and West of the Netherlands and varied in size and urbanisation grade. They served a total population of 2.050.000 inhabitants and employed a total of 2015 healthcare professionals, of whom 76.2% GP’s, 15.9% triage nurses and 7.9% other personnel. Locum doctors who had worked less than five shifts during the past year, were excluded from the study beforehand.

As part of the international SAFE-EUR-OOH project, the study was also performed in Norway (coordinating country), Slovenia, Italy and Croatia. The translation and data collection procedures were equal in each country.

Translation procedure

The SAQ-AV questionnaire was translated following modified principles adapted from Beaton et al [24]. Initially, the original English items [9] were translated into Dutch using a professional Dutch native translator. Next, an expert panel of two GPs, two triage nurses and two researchers adapted the initial translated version to the out-of-hours primary care setting in the Netherlands (for example “office” was changed into “GP cooperative” and “e.g. biopsy” into “e.g. surgical procedure”). This slightly adapted version of the questionnaire was translated back into English by a second independent professional English native translator, who was blinded to the original version. Based on this back-translated version, the expert panel made some adjustments in order to clarify misunderstandings. The prefinal Dutch version was tested in a small group of primary healthcare providers. This did not result in any further adjustments. Pre-tests showed that it took approximately 15 minutes to complete the questionnaire.

Questionnaire

Background variables: Work-related information, e.g. the respondent’s profession, years of experience, number of working hours a week.

Items on patient safety culture: The SAQ-AV contains 62 items on patient safety culture. Respondents rate their agreement using a 5-point Likert scale: 1 = disagree strongly, 2 = disagree slightly, 3 = neutral, 4 = agree slightly, 5 = agree strongly. For all questions, “Not applicable” was included as a response category. The original SAQ-AV described six factors covering 30 of the 62 items: Teamwork climate, Safety climate, Working conditions, Job satisfaction, Perceptions of management and Stress recognition with Cronbach’s alpha scores between 0.68 and 0.86 [9].

Data collection and procedure

The key contact persons of the GP cooperatives provided the e-mail addresses of all professionals having direct patient contact in their clinical work. In January and February 2015, the SAQ-AV was distributed by a link in an e-mail to 2015 primary healthcare providers in these 16 GP cooperatives and two call centers. In the preceding month, the contact person in each GP cooperative informed the staff about the study during work meetings, on the intranet, with posters and by email. Data were collected electronically using the program Qualtrics, whereby the participants responded anonymously. All questions were obligatory to answer. This data collection program automatically sent a reminder to those who had not responded after two weeks and after one month. After three weeks, an additional reminder was sent to the contact persons of the GP cooperatives, asking them to motivate the clinical staff to participate in the study. After the study, each of the participating GP cooperatives received a feedback report with the results of their unit, including a comparison of their results with the mean results of the total group. In this way, the healthcare providers were encouraged to focus on specific factors related to patient safety, and to discuss possible strategies for improvement within their clinical setting.

Data screening and pre-analyses

Completeness of the data was checked, resulting in an exclusion of 69 respondents, because they had completed less than half of all safety culture items—they all prematurely ended the questionnaire. There were no variables with 65% or more answers in one category, thus no floor or ceiling effects.

We checked whether the inter-item correlations were sufficient, by an examination of the correlation matrix. Questions belonging to the same underlying dimension will correlate, as they measure the same aspect of patient safety culture. Items that do not correlate, or correlate with only a few other variables are not suitable for factor analysis. Bartlett’s test demonstrated that the inter-item correlations were sufficient: χ2 = 7478.3; df = 351; p < .001. We also checked whether the opposite occurred: too high correlations between the items. Ideally, every aspect of patient safety culture uniquely contributes towards the concept of patient safety culture. No correlations exceeded the boundary score of 0.7 [25].

In addition, The Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) was determined. This value can range from 0 to 1. A value near 1 indicates that there is hardly any spread in the correlation pattern, enabling reliable and distinctive dimensions by factor analysis. The KMO-score was 0.9; far above Kaiser’s criterion of 0.5 [26]. The pre-analyses demonstrated that the data could be used for factor analysis.

To enable future comparisons of patient safety culture in out-of-hours setting across countries, we tried to fit the factor structure of the Norwegian questionnaire responses [19] (by confirmatory factor analysis program AMOS—not reported in this article). This did not confirm that the factor structure of the Norwegian questionnaire was also present in the Dutch data. The data were therefore studied with exploratory factor analysis to check whether the items form different factors in the Dutch out-of-hours primary care setting.

Statistical analyses

The Qualtrics file with anonymous SAQ-AV data was converted into an SPSS (Statistical Package for the Social Sciences) file for further analysis (IBM SPSS 22). The response category “Not applicable” was treated as a missing value in the data analyses (0%-2% missing values per item). Since the questionnaire contains positively as well as negatively worded items, the negatively formulated items were first recoded to make sure that a higher score always meant a more positive response. For each item, the mean and standard deviation were calculated.

We performed an exploratory factor analysis (Principal Components) with Varimax rotation. To find the most appropriate factor model, we explored the factor structures of different sets of items. Some items were permanently excluded from these exploratory factor analyses: 1) items consisting of general statements that are not necessarily applicable to the specific situation of the respondent, e.g “Truly professional personnel can leave personal problems behind when working” (10 items); 2) items for which it was unclear if a higher score meant a more positive or negative safety culture, e.g. “I have made errors that had the potential to harm patients” (two items), and 3) an item that was ambiguous according to two respondents: “Office management does not knowingly compromise the safety of patients” (one item). As an exploratory factor analysis with the remaining 49 items did not yield an interpretable model, we started with a set of items that were either present in the factors structures of Norway [19] or Slovenia (both SAFE-EUR-OOH study countries) [6]. In subsequent steps, we deleted items that did not fit in the Dutch factor structure and added items that were not present in the Norwegian or Slovenian structures, but which reasonably belonged to one of the factors in our structure, based on the items’ content.

In the exploratory factor analyses, missing values were deleted pairwise. When establishing the number of factors, the Eigen value (Eigen value>1: Kaiser’s criterion) was taken into account, beside the extent of explained variance, the shape of the scree plot and the possibility of interpreting the factors. Kaiser’s criterion is reliable in a sample of more than 250 respondents and when the average communality equals or is larger than 0.6. The shape of the scree plot gives reliable information when the sample is larger than 200 respondents [25]. The data satisfied these conditions.

The internal consistency of the factors was calculated with Cronbach’s alpha (α). If different items are supposed to measure the same concept, the internal consistency (reliability) should be greater than or equal to 0.7 [27]. For each item, the correlation between the item and the total score was calculated (corrected item-total correlations). In a reliable scale, all items should correlate with the total (r > 0.3) [25].

Finally, the construct validity was studied by calculating scale scores for every factor and subsequently calculating Pearson correlation coefficients between the scale scores. The construct validity of each factor is reflected in scale scores that are moderately related. High correlations, however, would indicate that factors measure the same concept and these factors may be combined and/or some items could be removed. For each factor, also the mean and standard deviation were calculated.

Ethical considerations

This study was based on data regarding patient safety culture among healthcare providers. Participation was voluntary. All participants received written information about the purpose of the study, and that the data were collected anonymously and treated in confidence. The Ethical Research Committee concluded that this study does not fall within the remit of the Dutch Medical Research Involving Human Subjects Act [Wet Mensgebonden Onderzoek] (file number 2014–299).

Results

Respondents

Of the 2015 employed healthcare professionals, 1974 correctly received an invitation to complete the questionnaire on a working email address, of which 853 (43.2%) answered the questionnaire (Fig 1).

Analyses were performed on the 784 respondents who answered more than half of all safety culture items. Of the respondents, 526 (68.6%) were female and 233 (30.4%) were aged between 41 and 50 years. Most of the respondents were GPs (61.2%; N = 470) or triage nurses (24.6%; N = 189). The majority had been working at the current GP cooperative for 11 to 20 years (36.0%; N = 276) (Table 2).

Exploratory factor analysis

Five factors were drawn by exploratory factor analysis, covering 27 items of the questionnaire and jointly explaining 52.4% of the variance in the responses: Perceptions of management, Job satisfaction, Teamwork climate, Safety climate and Communication openness. The factors consisted of three to nine items. Cronbach’s alpha scores varied between 0.49 for Communication openness and 0.86 for Perceptions of management. Table 3 shows the mean scores, standard deviations, factor loadings and corrected item-total correlations of each item in the factor structure. The mean scores of the items varied between 3.05 (for item 39r ‘I am frequently unable to express disagreement with staff physicians/intensivists in this office’) and 4.44 (for item 52r ‘I feel frustrated by my job’). For one item (39r), the corrected item-total correlation was below 0.3 (r = 0.235).

thumbnail
Table 3. Mean scores, standard deviations, factor loadings and corrected item-total correlations of the 27 items in the factor structure and Cronbach’s alpha for the five factors.

https://doi.org/10.1371/journal.pone.0172390.t003

Construct validity

For each of the five factors, scale scores were calculated by obtaining the mean of the item scores within one factor for every respondent. Next, correlations between the scale scores were calculated. Table 4 shows the mean scale scores with standard deviations, and the correlations between the factors.

thumbnail
Table 4. Mean scores, standard deviations and intercorrelations of the five factors.

https://doi.org/10.1371/journal.pone.0172390.t004

The highest correlations were those between Perceptions of management and Teamwork climate (r = 0.63) and between Perceptions of management and Safety climate (r = 0.63), but no correlation was exceptionally high.

Discussion

Main findings

We investigated the underlying factors and psychometric properties of the Dutch translation of the SAQ-AV in out-of-hours primary care services (GP cooperatives). With exploratory factor analyses, five factors were drawn with reliability scores between .49 and .86: Perceptions of management, Job satisfaction, Teamwork climate, Safety climate and Communication openness. The five-factor model of the Dutch SAQ-AV covered 27 items. Cronbach’s alpha for the factor Communication Openness and the item-total correlations of the items within this factor were low, indicating a problem with the factor. This could be related to the negative wording of all items within this factor which could have had an impact on the variability of the responses (e.g. respondents being reluctant to express negative opinions). However, as the alpha value is influenced by the number of items in a scale [28], the low value of Cronbach’s alpha could also be a consequence of the inclusion of only three items. The factor was not removed from the model based on content considerations. The construct validity was satisfactory for all factors; the moderate correlations of the factors show that there are no two factors measuring the same construct.

Comparison with other studies

The Norwegian factor structure consisted of five factors covering 30 items, without the factor Communication openness and including the factor Working conditions [19]. The Slovenian structure consisted of the same five factors as the Dutch structure covering 22 items, but with a lot of differences in the items within these factors [6]. At item level, there were more similarities with the Norwegian structure: 17 items fall in the same factors as in Norway whereas 12 items fall in the same factors as in Slovenia. The differences in factor structures between countries make cross-country comparisons of patient safety culture challenging. The structure differences may reflect cross-national variation in the nature and structure of out-of-hours primary care, or mean that item wordings trigger different connotations in the different languages. In a study in the hospital setting, using the Hospital Survey on Patient Safety Culture, more comparable factor structures across countries were found [29].

Strengths and limitations

The questionnaire was translated using an extensive forward-backward translation procedure and experts checking the relevance of the questions for the Dutch GP cooperative setting. For the factor analysis, we used a large sample of cases. The GP cooperatives were spread across the country and varied in size and degree of urbanisation, contributing to the representativeness of the sample. The participating GP cooperatives together served 13% of the Dutch population.

A limitation of the study is the moderate response rate (43%). We could not perform a non-response bias analysis, but the results indicate that GPs were somewhat underrepresented. Of the respondents, 61% were GPs, but among the invited employees 76% were GPs. Triage nurses were overrepresented—of the respondents 25% were triage nurses, but among the invited employees 16% were triage nurses. This finding is in accordance with the Norwegian study [30].

In order to identify a factor structure it was necessary to remove many items that did not contribute to the measurements. The items may still be valuable in local discussions and interpretations of the results, amongst respondents working on improving their safety culture. But there might also be issues regarding the underlying construction of the total questionnaire. More research into this subject is recommended, for example on how useful these additional items are and whether patient safety culture is better measured with only the items that belong to the factor structure.

Conclusions

The Dutch translation of the SAQ-AV, with five factors, may be a useful tool for measuring patient safety culture and guide quality improvement interventions in out-of-hours primary care services. It is interesting to gain insight into the factor structures of the SAQ-AV in other countries. Possibilities for comparisons of factor scores across countries seem to be challenging, but comparisons on item level are still an option.

Future studies should examine variation in safety culture between GP cooperatives, and differences in responses between GPs, triage nurses and other professionals. Furthermore, the SAQ should be validated further by examining whether there is an association between patient safety culture, patient experiences, and the occurrence of adverse events.

Supporting information

Acknowledgments

We thank all the participants in the study.

Author Contributions

  1. Conceptualization: ECTD DH GTB.
  2. Data curation: MS EK.
  3. Formal analysis: MS EK ECTD DH GTB.
  4. Funding acquisition: GTB.
  5. Investigation: MS GTB.
  6. Methodology: ECTD DH GTB.
  7. Software: GTB.
  8. Validation: MS EK PG.
  9. Writing – original draft: MS EK.
  10. Writing – review & editing: PG ECTD DH GTB.

References

  1. 1. Deilkås ET. Patient safety culture—opportunities for healthcare management, PhD thesis. University of Oslo. 2010.
  2. 2. Smits M, Wagner C, Spreeuwenberg P, van der Wal G, Groenewegen PP. Measuring patient safety culture: an assessment of the clustering of responses at unit level and hospital level. Qual Saf Health Care. 2009;18: 292–296. pmid:19651934
  3. 3. Deilkås ET, Hofoss D. Patient safety culture lives in departments and wards: Multilevel partitioning of variance in patient safety culture. BMC Health Serv Res. 2010;10: 85. pmid:20356351
  4. 4. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in/hospital adverse events: a systematic review. Qual Saf Health Care. 2008;17: 216–223. pmid:18519629
  5. 5. van den Berg MJ, van Loenen T, Westert GP. Accessible and continuous primary care may help reduce rates of emergency department use. An international survey in 34 countries. Fam Pract. 2016;33: 42–50. pmid:26511726
  6. 6. Klemenc-Ketis Z, Maletic M, Stropnik V, Deilkås E, Hofoss D, Bondevik GT. The safety attitudes questionnaire—ambulatory version psychometric properties of the Slovenian version for the out-of-hours primary care setting. BMC Health Serv Res. 2017;17:36. pmid:28086919
  7. 7. Colla JB, Bracken AC, Kinney LM, Weeks WB. Measuring patient safety climate: a review of surveys. Qual Saf Health Care. 2005;14: 364–366. pmid:16195571
  8. 8. Flin R, Burns C, Mearns K, Yule S, Robertson EM. Measuring safety climate in health care. Qual Saf Health Care. 2006;15: 109–115. pmid:16585110
  9. 9. Modak I, Sexton JB, Lux TR, Helmreich RL, Thomas EJ. Measuring safety culture in the ambulatory setting: the safety attitudes questionnaire–ambulatory version. J Gen Intern Med. 2007;22: 1–5.
  10. 10. Singer SJ, Gaba DM, Geppert JJ, Sinaiko AD, Howard SK, Park KC. The culture of safety: results of an organization-wide survey in 15 California hospitals. Qual Saf Health Care. 2003;12: 112–118. pmid:12679507
  11. 11. Weingart SN, Farbstein K, Davis RB, Phillips RS. Using a multihospital survey to examine the safety culture. Jt Comm J Qual Saf. 2004;30: 125–132. pmid:15032069
  12. 12. Sorra JS, Nieva VF. Hospital Survey on Patient Safety Culture, (Prepared by Westat, Under Contract No. 290-96-0004). AHRQ Publication No. 04–0041. Rockville, MD: Agency for Healthcare Research and Quality, 2004. http://proqualis.net/sites/proqualis.net/files/User%20guide%20HSOPSC.pdf. Accessed 23 January 2017.
  13. 13. Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The safety attitudes questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6: 44. pmid:16584553
  14. 14. Gallego B, Westbrook MT, Dunn AG, Braithwaite J. Investigating patient safety culture across a health system: multilevel modelling of differences associated with service types and staff demographics. Int J Qual Health Care. 2012;24: 311–320. pmid:22687703
  15. 15. Deilkås ET, Hofoss D. Psychometric properties of the Norwegian version of the Safety Attitudes Questionnaire (SAQ), generic version (short form 2006). BMC Health Serv Res. 2008;8: 191. pmid:18808693
  16. 16. DiCuccio MH. The Relationship Between Patient Safety Culture and Patient Outcomes: A Systematic Review. J Patient Saf. 2015;11: 135–142. pmid:24583952
  17. 17. Thomas EJ, Sexton JB, Neilands TB, Frankel A, Helmreich RL. The effect of executive walk rounds on nurse safety climate attitudes. A randomized trial of clinical units. BMC Health Serv Res. 2005;5: 28. pmid:15823204
  18. 18. Pronovost PJ, Berenholtz SM, Goeschel CA, Needham DM, Sexton JB, Thompson DA, et al. Creating high reliability in health care organizations. Health Serv Res. 2006;41:1599–1617. pmid:16898981
  19. 19. Bondevik GT, Hofoss D, Holm Hansen E, Deilkås EC. The Safety Attitudes Questionnaire—Ambulatory Version: psychometric properties of the Norwegian translated version for the primary care setting. BMC Health Serv Res. 2014;14: 139. pmid:24678764
  20. 20. Huibers L, Phillips H, Giesen P, Remmen R, Christensen MB, Bondevik GT. EurOOHnet—The European research network for out-of-hours primary health care. Eur J Gen Pract. 2014;20: 229–232. pmid:24219341
  21. 21. Giesen P, Smits M, Huibers L, Grol R, Wensing M. Quality of after-hours primary care: a narrative review of the Dutch solution. Ann Intern Med. 2011;155: 108–113.
  22. 22. Giesen P, Ferwerda R, Tijsen R, Mokkink H, Drijver R, van den Bosch W, et al. Safety of telephone triage in general practitioner cooperatives: do triage nurses correctly estimate urgency? Qual Saf Health Care. 2007;16: 181–184. pmid:17545343
  23. 23. Smits M, Huibers L, Kerssemeijer B, de Feijter E, Wensing M, Giesen P. Patient safety in out-of-hours primary care: a review of patient records. BMC Health Serv Res. 2010;10: 335. pmid:21143949
  24. 24. Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25: 3186–3191. pmid:11124735
  25. 25. Field A. Discovering Statistics using IBM SPSS Statistics. London: SAGE Publications; 2013.
  26. 26. Dziuban CD, Shirkey EC. When is a correlation matrix appropriate for factor analysis? Psychological Bulletin. 1974;81: 358–361.
  27. 27. Nunnally J. Assessment of Reliability. In: Psychometric Theory (2nd ed.). New York: McGraw-Hill; 1978.
  28. 28. Zeller RA, Carmines EG. Measurement in the social sciences: the link between theory and data. Cambridge: Cambridge University Press; 1980; p 56–58.
  29. 29. Wagner C, Smits M, Sorra J, Huang C. Assessing patient safety culture in hospitals across countries. Int J Qual Health Care. 2013;25: 213–221. pmid:23571748
  30. 30. Bondevik GT, Hofoss D, Holm Hansen E, Deilkås ECT. Patient Safety Culture in Norwegian primary care—a study in out-of-hours casualty clinics and GP practices. Scan J Prim Health Care. 2014;32: 132–138.