Challenges in measuring interprofessional–interorganisational collaboration with a questionnaire

Background Collaboration between medical professionals from separate organisations is necessary to deliver good patient care. This care is influenced by professionals’ perceptions about their collaboration. Until now, no instrument to measure such perceptions was available in the Netherlands. A questionnaire developed and validated in Spain was translated to assess perceptions about clinicians’ collaboration in primary and secondary care in the Dutch setting. Aim Validation in the Dutch setting of a Spanish questionnaire that aimed to assess perceptions of clinicians about interorganisational collaboration. Design & setting After translation, cultural adaptation, and pre-testing, the questionnaire was sent to GPs and secondary care clinicians (SCCs) in three regions in the Netherlands. The responses of 445 responders were used to assess the validity and reliability of the questionnaire. Method A confirmatory factor analysis (CFA) and an exploratory factor analysis (EFA) were performed to study the construct validity of the hypothesised factor model underlying the questionnaire. Test-retest reliability was evaluated using weighted Kappa statistics. Results Results of the CFA indicated poor fit of the hypothesised factor structure. EFA, executed separately for each region, showed a highly unstable factor structure. The test-retest reliability analysis demonstrated low re-test reliability. Conclusion The underlying factor structure of a Spanish questionnaire could not be reproduced. The construct validity and reliability of this questionnaire were insufficient to warrant use in the Dutch setting. This study demonstrates the need for evaluating validity and reliability of questionnaires in local settings.


Introduction
The incidence of complete Achilles tendon rupture is 18 per 100 000 patient-years 1 and is usually diagnosed clinically by GPs. The extent of clinical misdiagnosis is unknown in Norway, but may be high. 2 This is important as delayed treatment has unfavourable consequences. 1, 3 We report how a GP, with no clinical ultrasound experience, recorded images with a pocket-sized ultrasound device (PSUD) under supervision to confirm a complete Achilles tendon rupture. This could present a new indication for GP ultrasound.

Case report
A 36-year-old man experienced acute pain above the right heel accompanied by an audible snap while sprinting. He immediately had difficulty walking and 3 hours later consulted an on-call GP. Posterior ankle swelling with a tender depression 3 cm proximal to the calcaneum was found. Active plantar flexion against resistance was weak and Simmonds-Thompson test was 'partially positive' on applying a strong calf-squeeze. Based on these findings, calf muscle rupture was diagnosed as the Achilles tendon was thought to be intact. The patient was advised to elevate the foot and wait 2 weeks for improvement. Two days later a second GP, who was aware of a history of an audible snap, considered complete tendon rupture and reexamined the patient. Findings included an absent right heel raise due to weakness, minimal active plantar flexion against gravity and lying prone, significant right ankle swelling without bruising, and an altered angle of declination. Palpation elicited no ankle bony tenderness, yet a painful gap was identified 6 cm proximal from the calcaneal attachment, along the line of the Achilles tendon. Simmonds-Thompson's test was clearly positive. The positive Simmond's triad indicated a clinical diagnosis of complete rupture of the Achilles tendon. A 3.4-8 MHz linear array probe PSUD (VScanÔ dual probe, GE Healthcare), set at a depth of 3.5 cm, was used under the supervision of a rheumatologist experienced in ultrasound. The tendon was enlarged from 1 cm to 6 cm above the calcaneal insertion, where a clear gap was seen ( Figure 1). Two hours later a radiologist-performed ultrasound (LOGIQ E9Ô, GE Healthcare) and reported an enlarged distal tendon and a complete rupture at 5-6 cm from the calcaneal attachment, creating a 2.7 cm blood-filled gap ( Figure 2). Surgical exploration 8 days post-injury found a complete Achilles tendon rupture '5-10 cm above the ankle joint'.

Discussion
Tromsø Hospital serves a large area with a population of approximately 160 000. Between 2010-2014 an average of 21 patients per year were referred by their GP for suspected Achilles rupture.

Introduction
The incidence of complete Achilles tendon rupture is 18 per 100 000 patient-years 1 and is usually diagnosed clinically by GPs. The extent of clinical misdiagnosis is unknown in Norway, but may be high. 2 This is important as delayed treatment has unfavourable consequences. 1, 3 We report how a GP, with no clinical ultrasound experience, recorded images with a pocket-sized ultrasound device (PSUD) under supervision to confirm a complete Achilles tendon rupture. This could present a new indication for GP ultrasound.

Case report
A 36-year-old man experienced acute pain above the right heel accompanied by an audible snap while sprinting. He immediately had difficulty walking and 3 hours later consulted an on-call GP. Posterior ankle swelling with a tender depression 3 cm proximal to the calcaneum was found. Active plantar flexion against resistance was weak and Simmonds-Thompson test was 'partially positive' on applying a strong calf-squeeze. Based on these findings, calf muscle rupture was diagnosed as the Achilles tendon was thought to be intact. The patient was advised to elevate the foot and wait 2 weeks for improvement. Two days later a second GP, who was aware of a history of an audible snap, considered complete tendon rupture and reexamined the patient. Findings included an absent right heel raise due to weakness, minimal active plantar flexion against gravity and lying prone, significant right ankle swelling without bruising, and an altered angle of declination. Palpation elicited no ankle bony tenderness, yet a painful gap was identified 6 cm proximal from the calcaneal attachment, along the line of the Achilles tendon. Simmonds-Thompson's test was clearly positive. The positive Simmond's triad indicated a clinical diagnosis of complete rupture of the Achilles tendon. A 3.4-8 MHz linear array probe PSUD (VScanÔ dual probe, GE Healthcare), set at a depth of 3.5 cm, was used under the supervision of a rheumatologist experienced in ultrasound. The tendon was enlarged from 1 cm to 6 cm above the calcaneal insertion, where a clear gap was seen ( Figure 1). Two hours later a radiologist-performed ultrasound (LOGIQ E9Ô, GE Healthcare) and reported an enlarged distal tendon and a complete rupture at 5-6 cm from the calcaneal attachment, creating a 2.7 cm blood-filled gap ( Figure 2). Surgical exploration 8 days post-injury found a complete Achilles tendon rupture '5-10 cm above the ankle joint'.

Discussion
Tromsø Hospital serves a large area with a population of approximately 160 000. Between 2010-2014 an average of 21 patients per year were referred by their GP for suspected Achilles rupture.

Introduction
The incidence of complete Achilles tendon rupture is 18 per 100 000 patient-years 1 and is usually diagnosed clinically by GPs. The extent of clinical misdiagnosis is unknown in Norway, but may be high. 2 This is important as delayed treatment has unfavourable consequences. 1, 3 We report how a GP, with no clinical ultrasound experience, recorded images with a pocket-sized ultrasound device (PSUD) under supervision to confirm a complete Achilles tendon rupture. This could present a new indication for GP ultrasound.

Case report
A 36-year-old man experienced acute pain above the right heel accompanied by an audible snap while sprinting. He immediately had difficulty walking and 3 hours later consulted an on-call GP. Posterior ankle swelling with a tender depression 3 cm proximal to the calcaneum was found. Active plantar flexion against resistance was weak and Simmonds-Thompson test was 'partially positive' on applying a strong calf-squeeze. Based on these findings, calf muscle rupture was diagnosed as the Achilles tendon was thought to be intact. The patient was advised to elevate the foot and wait 2 weeks for improvement. Two days later a second GP, who was aware of a history of an audible snap, considered complete tendon rupture and reexamined the patient. Findings included an absent right heel raise due to weakness, minimal active plantar flexion against gravity and lying prone, significant right ankle swelling without bruising, and an altered angle of declination. Palpation elicited no ankle bony tenderness, yet a painful gap was identified 6 cm proximal from the calcaneal attachment, along the line of the Achilles tendon. Simmonds-Thompson's test was clearly positive. The positive Simmond's triad indicated a clinical diagnosis of complete rupture of the Achilles tendon. A 3.4-8 MHz linear array probe PSUD (VScanÔ dual probe, GE Healthcare), set at a depth of 3.5 cm, was used under the supervision of a rheumatologist experienced in ultrasound. The tendon was enlarged from 1 cm to 6 cm above the calcaneal insertion, where a clear gap was seen ( Figure 1). Two hours later a radiologist-performed ultrasound (LOGIQ E9Ô, GE Healthcare) and reported an enlarged distal tendon and a complete rupture at 5-6 cm from the calcaneal attachment, creating a 2.7 cm blood-filled gap ( Figure 2). Surgical exploration 8 days post-injury found a complete Achilles tendon rupture '5-10 cm above the ankle joint'.

Introduction
Last summer our small medical team visited the Calais 'Jungle'. Since that time much has changed and the camp is being demolished and by the time this article is read, it will probably be long gone. Some youngsters are finally being brought to the UK under the 'Dubs' amendment. However, once this camp is cleared it will not solve the ongoing flight of refugees from war torn areas: other camps are already appearing.

July 2016
A young Afghan man caught his finger on a sharp point while trying to cross a barbed wire fence. The finger was partially degloved. He attended the local hospital, where they placed a few sutures, but now, 2 weeks later, the skin is necrotic and the underlying tissue looks infected. He is in danger of losing his finger. A middle-aged Sudanese man has been having rigors and is generally unwell. He says it is similar to when he last had malaria.
A young Ukrainian woman complains of lower back pain and urinary frequency. The paths of these three people may never have crossed; yet here they are, denizens of the Calais Jungle. They turn up to a makeshift primary care 'clinic' that we set up in the heart of the unofficial refugee camp one weekend in July 2016.
With only basic medical supplies, we are immediately challenged by what we see. How can we arrange secondary care for the young Afghan in danger of losing his finger? We try to persuade him to return to the original local hospital, but he is reluctant. It was not a good experience for him the first time round.
With the other two patients, it is easier. They can attend the Salam clinic run by a local association during weekdays. Later, we receive word that malaria has been confirmed in our Sudanese patient.
More people arrive, presenting with scabies, rat bites, tinea, chest infections, and wheezing from inhaling smoke from fires lit to cook and keep warm in their tents at night. We examine a severely malnourished 2-year-old boy. We meet several of the camp's 600 unaccompanied children, at grave risk of sexual exploitation. We learn that there is inadequate safeguarding in place to protect them. A young Eritrean man comes in worried about his eye. He has sustained direct ocular trauma from a rubber bullet, and will never see normally again out of that eye. We see haematomas from police batons, and hear about children being exposed to tear gas again and again ( Figure 1).

The reality
These are no ordinary patients. They have travelled far from home to escape war, poverty, and misery. They have endured personal odysseys to get here, experienced untold hardships, and suffered unimaginable privations. Many have survived the loss of their families, torture, and rape. Their journeys over, for the moment at least, they must make their homes in the Calais Jungle. Their new shelters are in many cases mere tarpaulin covers, and their new beds just rugs on the ground. They own next to nothing. There is little for them to do, besides use their ingenuity to cross the English Channel in search of a better life. They are vulnerable to exploitation, crime, injury, and disease. Potentially violent clashes with local police, with other ethnic groups resident in the Jungle, or local far

Introduction
Interprofessional collaboration is necessary for patient-centred care, especially in the complex context of an ageing population in which multimorbidity is common. 1 Patients who suffer from multimorbidity often receive care from multiple healthcare professionals, and frequently undergo transitions between healthcare organisations and settings. When professionals from different organisations contribute to patient care, ensuring continuity of care is important. [2][3][4][5][6] The transitions of patients, in particular when moving from the primary care setting to secondary care, benefit significantly from good collaboration between GPs and SCCs, and between the organisations in which they work. 6,7 Even when a healthcare professional functions well on their own, patient care can be suboptimal when professionals work in a fragmented healthcare system with many boundaries between primary care, outpatient settings, and hospitals. [7][8][9][10][11][12][13][14] In the Netherlands, as in many other countries, various initiatives are undertaken to improve collaborative patient care across organisations. 10,15 Whether these efforts to improve the conditions for collaboration on an individual and an organisational level are successful often remains unclear because adequate instruments to measure collaboration are lacking. Only a few questionnaires have been developed which aim to measure the level of collaboration of professionals in a single organisation or a single team. [16][17][18][19][20] Questionnaires that measure conditions for collaboration across the borders of organisations are scarce, with a few exceptions. 21,22 The Dutch questionnaire developed and studied by Berendsen et al quantifies how SCCs and GPs value their mutual collaboration. Nũ no-Solinís et al's Spanish questionnaire measures the conditions for collaboration, while taking the influence of the work environment into account. 21,22 This article describes a study of the validity and test-retest reliability of Nũ no-Solinís et al's questionnaire in the Dutch healthcare context. The original questionnaire was developed based on a theoretical model for interprofessional collaboration by D'Amour. 22-25 D'Amour described collaboration as 'the structuring of collective action through sharing of information and decisionmaking in clinical processes between professionals in different organisations,' 23-24 and distinguished four dimensions of interprofessional collaboration. Two are relational dimensions: shared goals and vision, and internalisation; and two organisational dimensions: governance, and formalisation. Out of four dimensions, 10 interrelating indicators were formulated and evaluated in a multiple-case study [23][24] (see Box 1).

Factors
Dimensions Indicators In Spain, these 10 indicators have been used in the development of a questionnaire which measures clinicians' perceptions about conditions for interprofessional collaboration. 22 A study of the validity of the 10-item Spanish questionnaire showed a two-factor structure with promising levels of model fit. 22 The present study aims to re-study the validity of the questionnaire developed in Spain, in order to test its appropriateness for measuring conditions for interprofessional collaboration in the Dutch healthcare setting. Data were collected in three geographic regions in the Netherlands where GPs and SCCs work in a hospital context to deliver collaborative patient care.

Method Content validity and face validity
To test face-validity, 28 professionals were asked by mail to look at the questionnaire, translated from English to Dutch (full questionnaire available in English and Dutch from the authors, on request). These professionals (SCCs n = 20, GPs n = 8) were asked whether they could answer the questions and whether they considered the questions relevant for measuring collaboration in their working situation. Their main comment was about long sentences. In the subsequent translation from Spanish to Dutch, the length of the sentences was a major focus of attention.
The content validity was considered adequate because it was based on the diverse and thorough studies of D'Amour and the questionnaire-building process in Spain, together with the reactions of intended responders.

Preparation of the questionnaire
Two bilingual translators forward-translated the original Spanish questionnaire into Dutch, and some adjustments were made to the questionnaire to take account of the Dutch healthcare setting. The structure of the 10-item Spanish questionnaire was preserved including, after each question, the phrase 'Please rate the current situation in your organisation with respect to the other level of care on a scale of 1 to 5.' The response options describe desirable attributes with five distinct descriptions for each item on a 5-point Likert scale, all ranging from 1 (none of the attribute) to 5 (lots of the attribute), as demonstrated in the example in Box 2. A backward translation from Dutch to Spanish was done. This translation into Spanish was reviewed and approved by one of the developers of the questionnaire.
After translation, the questionnaire was pre-tested, to check whether the translation introduced errors after the test for face validity, using the thinking aloud method. [25][26] Pre-testing was done by five GPs (from city and urbanised rural areas) and five SCCs from different specialties, all native Box 2. Example of two items of the 10-item questionnaire to assess the interprofessional collaboration of two different levels of care

SHARED GOALS
The existence of explicit shared goals facilitates collaboration and coordination between primary and secondary care settings. Please rate the current situation in your organisation with respect to the other level of care on a scale of 1 to 5: 1. Common goals are missing 2. There are hardly any shared goals 3.There are some common goals 4. There are quite a lot of common goals 5. Nearly all aspects of care are covered by shared goals 2. PATIENT-CENTRED APPROACH When priority is given to the interests and preferences of the patient, this favours collaboration and coordination between professionals working in the primary and secondary care setting. Please rate the current situation in your organisation with respect to the other level of care on a scale of 1 to 5: 1. In the interaction between levels of care, the interests and preferences of patients are not taken into account 2. In the interaction between levels of care, the interests and preferences of patients are taken into account on few occasions 3. In the interaction between levels of care, the interests and preferences of patients are sometimes taken into account 4. In the interaction between levels of care, the interests and preferences of patients are often taken into account 5. In the interaction between levels of care, the interests and preferences of patients are always taken into account Dutch speakers. Each of them individually read each question aloud. [25][26] A researcher then asked them how they interpreted the questions, and what they thought about the ease of comprehension. None of these professionals indicated relevant aspects of collaboration conditions that were missing in the questionnaire. All comments were noted and discussed within the research team. An example of change of a phrase is 'professionals working in the different levels' being changed to 'the primary and secondary care setting. ' After discussing all the comments in the team, a final version of the questionnaire was established.

Data collection and sample
For validity testing, an invitation to complete the online questionnaire was sent to all GPs and SCCs (N = 1369) practising in one of three selected geographic regions in the Netherlands. Each of these regions contained one large hospital (a top, non-academic, clinical teaching hospital) with a central position in the region. Invitations were sent by e-mail with a link providing direct access to the questionnaire. In two regions, a reminder was sent to all non-responders 2 weeks after the first invitation. In the third region, all responders received the questionnaire again after 2 weeks to evaluate the test-retest reliability. The data were collected between October 2015 and March 2016.

Data analysis
The approach to studying the (construct) validity of the translated questionnaire was similar to the approach used to study the validity of the original questionnaire. 22 First, the construct validity of the questionnaire was examined using a CFA. Because of the ordinal structure of the data, this analysis was conducted by fitting the factor model on the polychoric correlation matrix 27 of the item responses. The factor model is estimated by maximum likelihood on the data from the three regions separately, to avoid regional effects that may contaminate the correlation analyses and to explore the stability of the factor structure across the regions. Nũ no-Solinís et al's correlated two-factor model was assumed, with the uncorrelated error terms. Initially, measurement invariance across regions was not assumed. The fit of the confirmatory factor models was evaluated using common fit statistics: root mean square error of approximation, standardised root mean square residual, and comparative fit index, with usual cut-off points.
Next, an EFA with oblimin rotation (allowing the factors to be correlated) was employed, because the fit of the CFA models was insufficient. The approach taken was, again, similar to the EFA approach taken by Nũ no-Solinís et al. 22 The Bayesian Information Criterion (BIC) was used to determine the optimal number of factors. 28 The lower the BIC, the better the fit of the factor model to the data.
Finally, test-retest reliability was evaluated on item-level by squared weighted Kappa (SW Kappa) statistics, 29 taking an SW Kappa of >0.70 as the threshold criterion for reliability.

Response
From the 1369 (582 GPs and 787 SCCs) invitees, 458 doctors responded: 206 GPs and 252 SCCs. Thirteen questionnaires were incomplete. There was a total of 445 fully completed questionnaires, representing a response rate of 33% (Table 1).

CFAs
The CFA explored the two-factor structure of the questionnaire (interpersonal relationships and organisational setting) in the Dutch healthcare setting, as assumed and tested by Nuñ o-Solinís et al. 22 The fit indices, with their corresponding suggested thresholds levels for sufficient fit to the data, are listed in Table 2. None of the fit indices met the norm criteria for a sufficient fit in each of the three regions. The questions X1-X4 aimed to measure the factor 'interpersonal relationships' and the questions X5-X10 aimed to measure 'organisational setting'. Figure 1 details the estimated CFA diagram 30 assuming a two-factor structure, the estimated standardised loadings, and corresponding maximum likelihood standard errors. The estimated factor loadings show notable variation between regions and items. High factor loadings (>0.7) were rare (n = 4/30). The estimated factor correlations are 0.942 (region 1), 0.754 (region 2), 1.000 (region 3), and 0.923 (total). In all, the estimated factor structures varied considerably between the three regions and, in each of the regions, a lack of support was shown for the assumed two-factor structure. Exploration of modification indices (that is, suggestions for model improvements, such as allowing correlation of error variances) did not yield consistent suggestions for improvements that could improve model fit across the regions (results not shown).

EFAs
To explore alternative factor structures, EFAs were carried out. EFA models with one, two, three, four, five, and six factors were fitted for each of the regions. Similar to the CFA, the EFA with oblimin rotation showed unstable factor structures across regions. Based on the BIC, a four-factor structure was found to be optimal in region 1; five-factor in region 2; and another five-factor in region 3 (other criteria to determine the optimal number of factors were also checked but did not resolve the instability). The factor loadings for the four and five-factor models were not stable across regions (results not shown). Further explorations with alternative orthogonal rotation strategies and factor selection criteria did not provide a solution to the instability of the factor structure.

Test-retest reliability
Test-retest reliability analyses were performed with the data gathered in region 3 (N = 90, GPs = 51, SCCs = 39). The test-retest was measured with the SW Kappa agreement, and is shown in Table 3. The test-retest reliability of this questionnaire was shown to be insufficient for each item (SW Kappa >0.70 is sufficient).

Discussion Summary
In the present study, an evaluation was conducted of the validity and test-retest reliability of a carefully translated and culturally-adapted questionnaire, which aimed to measure interprofessional collaboration conditions between clinicians at different levels of care. In contrast to the original questionnaire, the questionnaire showed poor validity in the Dutch setting, and the additional test-retest reliability was also insufficient. The data collected in three different Dutch regions allowed for a cross-regional comparison of validity, by comparing confirmatory factor models with sufficient  sample sizes in each region. This study provides clear evidence that the hypothesised two-factor structure that underlies this questionnaire cannot be confirmed in any of the regions. Consequently, the authors advise against the use of this questionnaire to measure conditions for interprofessional collaboration in the Dutch healthcare context. One explanation for the difference between validity of the Spanish and Dutch questionnaire may be the differences in organisation of the healthcare system. In the Spanish region where the questionnaire was validated, clinical professionals (doctors and nurses) are part of one integrated care delivery organisation, consisting of a regional hospital (specialised care level), and health centre (primary care level), and are contracted with one provider. In the Netherlands, primary and secondary care takes place in separate organisations and is contracted with different providers. Dutch primary care consists of organisations varying in form and size. Between the Dutch regions, cultural and organisational differences exist and the questionnaire is not stable in its two-factor structure. Different factors were found with the EFA for the regions. The questionnaire appears to be culturallysensitive.

Strengths and limitations
A strength of this study is that statistical analyses were carried out on a larger sample than has been used in Spain. The questionnaire has been tested in three different Dutch regions, and hence was applied in different organisational circumstances within the healthcare system. The analyses were performed by region and can be perceived as three separate studies. The sample size in one of the regions was smaller than recommended. 30 In the other regions, a sample size was obtained that was close to the recommended sample size. Finally, the analysis of the remaining region and the analysis on the entire data set were well above the recommended sample size. Another strength of this study is the measurement of the test-retest reliability. To test whether measurement results are reproducible in test-retest situations is important. 28,31 No test-retest reliability analysis had been conducted for the original questionnaire. A possible explanation of the poor test-retest results could be current collaborative experiences influencing the opinions of the responding medical professionals.
A limitation of this study is that the response rate (33%), although higher than in the original study, was limited.

Comparison with existing literature
The strong theoretical base of the questionnaire, which was derived from several qualitative studies by D'Amour et al, supports the importance of the 10 dimensions related to interprofessional collaboration. 5,24,27 But to measure the relational and organisational items with a questionnaire comprising 10 questions seems difficult. One study which measured how GPs and SCCs rate their collaboration did not take into account the organisation they worked in. 21

Implications for research and practice
Even though the surveyed healthcare professionals considered the questions based on these dimensions relevant for their work in the Dutch healthcare context, the questionnaire proved not to be valid in the Netherlands. The construct validity and test-retest reliability of this translated questionnaire were insufficient to warrant use in the Dutch setting. To develop a useful questionnaire about collaboration, more questions about each of the indicators are probably needed, which means constructing a new questionnaire and testing it with an approach similar to the one presented here. Alternatively, use of qualitative methods such as interviews could lead to more insight into interorganisational collaboration. Even though a pre-testing process might indicate a high face-validity, it is not a guarantee as to other aspects of validity. Even between regions in the same country, cultural differences can exist.
Importantly, this study demonstrates that even when the face-validity of a translated questionnaire seems to be good, thanks to a pre-testing process with sufficient opportunities for cultural adaptation, other aspects of validity must be tested before applying the instrument in new settings. Even between regions, there can be cultural differences. The authors advise caution when using a questionnaire in another cultural setting, population, or context without testing it. Before choosing a questionnaire for a small-scale study in general practice, the authors recommend checking that it is validated for that setting.

Funding
The authors thank the Hein Hogerzeil Foundation, which has contributed to funding for this research.
Ethical approval Approval for this study was obtained by the Medical Ethics Review Committee (MERC) University Medical Center Utrecht.

Provenance
Freely submitted; externally peer reviewed.