BJGP

Background. The potential of the electronic health record to support safety-netting has been recognised and a number of electronic safety-netting (E-SN) tools developed. Aim. To establish the most important features of E-SN tools. Design and setting. User experience interviews carried out with primary care staff who had trialled the EMIS E-SN toolkit for suspected cancer, and a Delphi study with primary care staff involved in safety-netting in any capacity. Method. The user experience interviews were carried out remotely. An electronic modified Delphi approach was used to measure consensus on tool features. Results. Thirteen user experience interviews were carried out and features of E-SN tools seen as important formed the majority of the features included in the Delphi study. Three rounds of Delphi survey were administered. Sixteen (64%) respondents completed all three rounds, and 28 out of 44 (64%) features reached consensus. Primary care staff also preferred tools that are general in scope. Conclusion. Primary care staff indicated that tools which were not specific to cancer or any other disease and had features that promoted their flexible, efficient, and integrated use were important. However, when the important features were discussed with our PPI group they expressed disappointment that features they believed would make E-SN tools robust and provide a safety-net that is difficult to fall through did not reach consensus. The successful adoption of E-SN tools will rely on an evidence base of their effectiveness. Efforts should be made to assess the impact of these tools on patient outcomes.


Background
Safety-netting is an important diagnostic strategy in primary care to monitor patients with symptoms that could indicate cancer or another serious illness until they are explained or resolved (1). Originally conceptualised as a communication skill, Neighbour outlined three questions a general practitioner (GP) might ask before the close of every consultation; If I'm right, what do I expect to happen? How will I know if I am wrong? What would I do then? (2). Over the past three and a half decades, safety-netting has evolved from an exercise for GPs when forming their initial diagnosis and considering the consequences if it is correct or not, to include multiple aspects of diagnostic management, particularly where there is uncertainty. These aspects include clinicianpatient communication (what to expect), information provision (what to look out for and where to seek further help), record keeping, and follow-up (1,3). If done well, safety-netting can facilitate the safe self-care of patients with self-limiting illness, the earlier identification of serious disease, avoidance of emergency presentations, and improvement of patient outcomes.
As a disease that may manifest slowly and shares many symptoms with benign conditions, safetynetting when cancer is a possibility has been a focus for both research and guidance (4)(5)(6)(7)(8)(9)(10)(11). Despite the widespread acceptance of the central role that safety-netting plays in avoiding delays to cancer diagnosis, research has also reported that it is inconsistently used and poorly recorded (12), and while the electronic health record (EHR) could support safety-netting, its use has not been maximised. Tools to move safety-netting 'online' have been created, either directly embedding safety-netting codes or functions into the EHR or facilitating safety-netting through EHR integrations with external digital tools, such as text messaging (8) and email (3). The purpose of these tools includes to help with GPs' record-keeping, providing patient information, reminders about followup, and improved patient safety by automating, standardising, and centralising safety-netting actions where possible (3).
Electronic safety-netting (E-SN) tools have been adopted without an evidence base, however, and there have been few evaluations of their impact on patient safety (3). Where E-SN tools have been evaluated, evaluations have stopped short of quantifying any impact they have on outcomes due to insufficient data (13), and have reported conflicting results (Frontier Economics report: C the Signs evaluation: report for RM Partners (private report received on request, 2021), (14)). This qualitative interview and Delphi study is part of the Cancer Safety NETting 2 (CASNET2) study which aims to fill this gap by evaluating the EMIS Safety-Netting toolkit (15,16). E-SN tools have the potential to facilitate the standardisation of safety-netting in primary care but these tools must address the needs of primary care staff who use them, and have a demonstrable, positive impact on patient care. Our quantitative evaluation is described in detail separately (15) (trial registration number ISRCTN15913081). This paper presents an evaluation of the E-SN toolkit in the CASNET2 study including an interview study of user experiences, and a Delphi study to elicit preferences for E-SN system functionality so that the toolkit can continue to be improved and refined. The objectives of these two studies were, respectively, to explore what determines a GP's use of the E-SN toolkit including the barriers and facilitators to the use of E-SN systems, and to identify and prioritise additional desirable features for the CASNET2 E-SN toolkit and E-SN tools generally, and features for enhancement.

Methods
This two part evaluation is centred on primary care clinicians' views of the CASNET2 tool and E-SN tools more broadly. Patients were not included as participants in the user experience interviews because this is a clinician-facing tool and a large proportion of its functions are administrative, so patients are unlikely to know that it is being used in their care. The evaluation was supported by PPI involvement throughout.

User experience interviews
The procedure for recruiting GP practices to the CASNET2 study, their randomisation, and training on the toolkit is described in detail elsewhere (15). Staff from all practices that had trialled the E-SN toolkit as part of the CASNET2 study were invited to participate in a user experience interview. Interviews were partly informed by Normalisation Process Theory (NPT) which aims to identify factors that facilitate or block the adoption of an intervention into usual (or normal) practice (17). The four NPT domains were used to shape the questions in the semi-structured interview guide which allowed for guided questions to be asked on what work was done to implement the toolkit while also allowing the participants to express their preferences and thoughts towards it, for example; -Coherence -how teams were made aware of the toolkit's uses and whether the training provided was effective, -Cognitive Participation -whether information sharing and uptake was consistent and what was done to encourage uptake, -Collective Action -what using the toolkit was like, was there anything about it that was a barrier or facilitator to its use, how burdensome administrative teams found reporting with the toolkit, -Reflexive monitoring -whether the toolkit had an impact on practice processes, whether an effect of using the toolkit was seen.
All staff who volunteered were interviewed. The interviews were conducted online using Microsoft Teams by the lead author (CFS) from October to December 2021 and lasted 24-55 minutes. Interviews were conducted online to facilitate staff from across England to be interviewed and to reduce the burden on practices managing the after effects of the acute period of the Covid-19 pandemic.
The interviews began with a discussion about the interviewee's general practice and role. The discussion then turned to how staff were trained on the toolkit, how it was received, and the features that the interviewees liked, disliked, and felt were missing. These broad themes were also pre-specified for the data analysis (Supplementary Figure S1 (page 1)). The interviews were transcribed verbatim and analysed using a framework analysis by CFS. Framework analysis allows for both inductive and deductive thematic analysis of qualitative data and was selected as we had a set of pre-defined research questions to answer but also wanted to allow unanticipated themes to be generated (18). Weekly meetings were held at which CFS updated the research team on progress and discussed the developing findings. Following the interviews, the features brought up, along with others mentioned in the literature, were presented to the study team. This generated the statements included in the first round of Delphi survey.

Delphi survey study
We used a three round modified Delphi method for this study. The Delphi method aims to reach a consensus of expert opinion on a topic through iterative rounds of questioning and feedback and commonly involves 10 to 50 expert panel members (19).

Sample
Any individual working in primary care who was involved in the safety-netting of patients in any capacity was eligible to take part. The invitation to participate in the Delphi survey was disseminated through a number of channels; invitations to practices participating in the CASNET2 study, the Society for Academic Primary (20)(21)(22)(23)(24)(25)(26)(27)(28). The initial items for the survey were developed by CFS and shared with the study team who amended and added items from their experience as clinicians and researchers. All survey rounds were distributed via email.
Forty items detailing features of E-SN tools were grouped into seven themes: Information Entry; Integration with Other Systems; Communicating with Patients; Warning Notifications; Task Setting and Follow-up; Responsibility for Safety-netting; and Closing the Safety-netting Record and Auditing. The Delphi panel was asked to rate how important they believed each of these features were to a tool that would facilitate effective safety-netting on a five-point Likert scale where 1 indicated not at all important and 5 indicated very important. Additionally, the panel were asked to rank how useful an e-safety-netting tool would be for different types of presentations (e.g. disease specific versus general) and for different patient groups (e.g. those with an urgent referral versus those who do not meet the referral criteria yet). The opportunity to leave free text comments was provided.

Analysis
The proportion of the panel rating each feature as important (Likert scale rating 4 or 5) or unimportant (Likert scale 1 or 2) was calculated. Consensus was defined as at least 75% of the panel rating a feature as important or very important (consensus important), or unimportant or not at all important (consensus unimportant). If consensus was achieved it was ranked as adequate (75-79%), strong (80-84%), very strong (85-89%), or overwhelming (90-100%). Free-text comments were analysed for new features or amendments to existing features. This was repeated for each survey round.

Second and third survey rounds
In the second and third survey rounds, individuals who had responded to the previous round were emailed the subsequent survey. Features remained in their themes but were grouped as those that had achieved consensus and those that had not. Recipients were given a copy of their previous responses for reference. If features were added or amended an explanation was given. Where a feature had not achieved consensus, a summary of how the panel had voted and a selection of anonymous comments given in the previous round was presented (where available). The panel were then asked to rate the importance of the feature again. Any features that had not reached consensus after the third round were defined as not having achieved consensus.

PPI group involvement
Five members of the public were recruited through cancer and health research networks in early 2020 to the CASNET2 PPI group. The first PPI group meeting was scheduled for March 2020 but due to the Covid-19 pandemic, which delayed the start of the CASNET2 study as a whole, the first meeting was held online in September 2020. During the early stages of the study the group met online approximately every 6-8 months and this has increased as results and next steps have become ready for discussion. Two PPI meetings were held following the completion of the user interviews and Delphi survey studies. At the meeting following the user experience interviews the results were fed back to the group and discussed, and the process of a Delphi survey study explained. During this meeting the group were asked to comment on what the interviewees had said and indicate what features they thought were important/less important before the Delphi items were drafted. Following the Delphi survey, a meeting was held at which the features that achieved and did not achieve consensus were discussed in depth. The group shared what they thought the implications could be for patient care if E-SN tools were adopted that were designed in line with the important and unimportant features. We present a summary of this meeting in the Discussion section.

User experience interviews
The demographic characteristics of all 13 primary care staff who volunteered and took part in our user experience interviews are given in Table 1. The majority of the interviewees thought the CASNET2 toolkit had been useful and suggested that they would like to continue using it after the trial ended. The interviewees said they believed the toolkit was useful to standardise safety-netting practice, if used consistently. Although all of the interviewees made suggestions about ways the toolkit could be improved, a small number said that significant changes would need to be made before they would be willing to incorporate it into their practice permanently. The suggestions for areas of improvement could be broadly separated into improving the user interface and providing the user with information. Improvements to the user interface included reducing the number of 'clicks' needed to complete forms, increasing the extent to which the form auto-populates, and improving its integration while reducing any overlap with other systems. In terms of the information, the interviewees wanted the toolkit to provide visual alerts when dates associated with an instance of safety-netting were breached, guidance on appropriate safety-netting timelines (e.g. how long to wait for contact from secondary care, or for a symptom to resolve), and the ability to see how, or if, use of the toolkit was benefitting the patient. A summary of the interview themes with illustrative quotes is available in Supplementary Figure S1 (page 1).

First round Delphi survey
The first round survey was sent to 25 primary care staff who had responded through an expression of interest form. It was circulated in February 2022 and received 20 responses (80%). The characteristics of those responding to the first round are given in Table 2 In round 1, 12 of the 40 features (30%) reached consensus. For all but one the consensus was that the features were important, and in one there was consensus that it was unimportant. In light of the free-text comments we amended two statements for clarity and added three additional features. A selection of the free-text comments is available in Supplementary Figure S5, page 51.

Results from survey round 2
Eighteen responses were received to the second survey round and an additional 10 features achieved consensus. Following the free-text responses one statement was amended for clarity and one feature was added.

Results from survey round 3
Sixteen responses (64% responding to all 3 rounds) were received for the third and final round and an additional 6 features reached consensus. By the end of round 3, 28 out of 44 features (64%) had reached consensus. Over half of the statements that did not reach consensus were features that could support informational continuity of care, for example, 'Creates alert if patient attends repeatedly in a short timeframe' (number 23) and 'Pop-up message boxes on opening patient record if follow-up date has passed' (number 32). Supplementary Figure S6 (page 55) presents the feature statements and the results of all three survey rounds.
Two additional questions were asked in all three survey rounds regarding the scope of E-SN tools ( Figure 1) and the patient groups for which tools were considered most useful (Figure 2). We included these questions to try to find a consensus on the circumstances in which and patients for whom E-SN tools were considered useful. Our participants responded that tools that were general, i.e. could be used with any patient that the clinician chose to safety-net and could facilitate safetynetting for a range of patient presentations, were most useful. Additionally, tools that could facilitate safety-netting for patients that have access, personal, or social problems were ranked highly.  These patients were described as, for example, those who had not had their symptoms for long enough to meet the referral threshold. Scope of e-safety-netting tools

Summary
In this study, our Delphi panel agreed on 28 features of E-SN tools that they believed to be either important or unimportant for effective safety-netting. The features that reached consensus were generally those that could support the efficient and flexible use of a tool, would support decision making and communication with patients and colleagues, and would allow for the impact of the E-SN tool to be assessed. Several items relating to task setting and follow up did not reach consensus. The panel also preferred safety-netting tools that were not targeted at specific diseases, body systems, or symptoms.
Generally the features that reached consensus quickly were those that had a bearing on the clinicians' time, for example use of auto-population. In contrast those that took two or three rounds of Delphi to reach consensus were generally more relevant to the safety implications of the E-SN tool. Further, the majority of features that did not reach consensus were those that provided information and supported informational continuity of care. This implies that concerns over burden of work and timekeeping may have been more salient to our participants than the mitigation of risk. While this is understandable given the current pressures on primary care, and tools which reduce administrative burden are needed, it is equally important that tools should be effective at improving patient safety.

PPI feedback
We presented the outcomes of the Delphi survey to our PPI group to understand what the implications for patients might be if E-SN tools were designed to incorporate the Delphi study priorities. The agreed statements were uncontroversial and accepted with the exception of one. The PPI group were disappointed that the statement 'Use of tool is compulsory for all patients' (Statement 1, Supplementary Figure S6, page 55) was deemed unimportant. The PPI group suggested that making tools optional would erode their use over time and contradicted the message that safety-netting was an important part of the consultation.
For the statements that were not agreed, there was tension between clinicians' desire for E-SN tools not to introduce "superfluous" information or create additional tasks, while the PPI group thought these features contributed to a more effective safety-net. The PPI group raised concerns that the rationale for not rating some statements as important could not be applied to all patients. For example, Delphi participants reported that the clinician would notice that a patient had attended multiple times within a short period or with the same symptoms when reviewing the patient's notes rather than requiring a notification, and that repeat consultations are rarely because of serious illness (Supplementary Figure S5, page 51). The PPI group, however, said that this assumes the clinician has time to thoroughly review the patient's notes before the consultation or that the patient is able to reconsult with the same clinician, neither of which may be possible currently. The PPI group stated that even if repeat attendances for the same symptom are rarely related to serious undiagnosed illness, the rare case is exactly what safety-netting is for. Another feature that was not rated important was the provision of evidence-based guidance for the expected duration of nonspecific symptoms to guide self-management advice. The PPI group agreed that standardisation could be difficult for some non-specific symptoms, but stressed that where evidence was available an attempt should be made as standardisation reduces variation in advice and reassures patients, echoing previous research (29).

Strengths and limitations
The majority of the Delphi features were derived from interviews with primary care staff who were trialling an E-SN tool. As such, the statements were based on their recent experiences of using an E-SN tool, increasing the likelihood that the features included were relevant to current primary care practice. The Delphi panel was balanced in terms of those who held academic positions and those who did not. It is conceivable that academic primary care clinicians could hold different views to those focussed solely on clinical work so the even split is a strength. The panel also included clinicians with experience of a range of current E-SN tools, ensuring that the strengths and weaknesses of a range of tools were reflected in our results.
Our user experience interviews only recruited 13 primary care staff despite a concerted effort by the Royal College of General Practice Research and Surveillance Centre Practice Liaison Officers administering the CASNET2 study. This may be because we recruited in April to November 2021 while UK government restrictions were still in place and the acute phase of the Covid-19 pandemic was coming to an end. It has been well documented that the pandemic placed huge pressures on primary care which may have prevented staff from taking part in non-essential activities such as our interviews (30,31). We did, however, have a narrow remit for these interviews, to discuss the work done to implement and the experience of using the E-SN toolkit, with a specific group of participants, primary care staff who had used the toolkit, and so the informational power (32) may have compensated somewhat for the small sample size. Although we accepted any individual working in primary care involved in safety-netting, the majority of our participants were GPs. It is clear from our user interviews and the comments provided in the Delphi study that aspects of safety-netting and follow-up are carried out by non-clinical staff but we have not captured their preferences in this study. Additionally, we did not include professionals from outside of primary care, for example patient safety experts or clinicians from acute settings. As such the results of the Delphi may be limited by and reflect current primary care safety-netting practice. Patients' views also did not directly contribute to E-SN tool features. Our PPI group have contributed throughout the CASNET2 study helping to set and prioritise outcomes for the quantitative evaluation of the E-SN toolkit, commenting on the findings of the user experience interviews, and by supporting or opposing the findings of this Delphi study. As such, we believe that their views have been and will continue to be meaningfully incorporated into the continuing development of the E-SN toolkit. Finally, our panel size was at the small end of what is considered acceptable for Delphi studies. However, small but homogeneous Delphi panels can produce high quality consensus (19).

Comparison with existing literature
The adoption of EHR systems, which has happened alongside the expansion of safety-netting, should have provided new ways to support safety-netting, for example through electronic alerts (33) but the literature has shown that safety-netting is often kept 'offline' (12). We have found clinicians to be in favour of electronic safety-netting in this and in previous work (30). However, like previous research (27,28,34), we were also told that higher volumes of tasks and alerts leads to clinicians ignoring them, and heard calls to simplify, automate, and "stop the pop" i.e. limit pop-up notifications. The free-text responses in this study support previous studies that describe disagreement around what safety-netting is, how it should be used, and whose responsibility it is to ensure advice is followed (6,7). Furthermore, a number of our participants commented that followup was a separate issue to safety-netting despite previous research that has not made this distinction (12).
A recently published framework outlined nine principles that denote a high-quality E-SN tool (3). The features that reached consensus reflected many of the principles outlined in this framework (Table  3) in that E-SN tools should facilitate shared responsibility for safety-netting, should integrate well with the EHR and automate data capture, facilitate quality improvement, and allow information sharing with the patient. This framework, also includes principles such as that the tool should be used with all patients not only those thought to be high risk, and that it should track patient consulting patterns and create alerts. Similar features did not reach consensus in the Delphi study perhaps highlighting a gap between what might objectively support safety-netting and what will become too burdensome within the consultation. Ways to reduce the burden of such features on staff should be investigated.

Principles of E-SN tools (reproduced from (3))
Features reaching consensus that met the principle Features not reaching consensus that met the principle All patients registered will be e-safetynetted.
Scope and patient groups where a tool was considered most useful.
1. Use of tool is compulsory for all patients (consensus was reached but it was that this item was not important). 9. Use of tool compulsory to create a consistent record of when a patient is safety-netted. All clinicians and primary care staff are responsible for e-safety-netting. 38. Centralised record of safety-netting seen by all practice staff to share the responsibility. Limit burden and cognitive bias by using automatic functions, where possible.
2. Auto population of fields when information is already in the health record. 6. Automatic triggering of e-safety-netting tool based on coding. 29. Automatically sends message to administration team when follow up timelines are breached. 35. Automatically creates diary entry for follow-up. Support diagnostic processes before, during, and after consultations.
12. Links to patient information leaflets. 14. Links to the up to date referral criteria for 2ww pathways provided in the form. 16 21. Creates an alert on opening patient record if patient has not been investigated or a result has not been returned. 22. Alerts take the form of an automatically generated urgent task (e.g. to follow up with the patient). Use simple processes that make it easy to access and transfer complex information.
11. Integrates with other systems to avoid duplication of work. 13. Test results are automatically incorporated so clinician doesn't have to look elsewhere. 33. Guidance on the duration vague symptoms should be managed by the patient before they are advised to re-contact the GP. Spread responsibilities and roles within primary care that have an overall impact on the whole patient pathway. 36. Ability to assign ownership for safety-netting actions. 37. Ability to electronically 'hand-over' responsibility for the safety netting to a colleague. 38. Centralised record of safety-netting seen by all practice staff to share the responsibility. 34. Automatically sends message (e.g. email) to responsible clinician when follow-up timelines are breached. 39. Automatically sends tasks to relevant staff members. Support senior leadership to optimize safety strategies within a regular quality improvement program.
41. Ability to audit use of the e-safety-netting tool.
44. Automatically produces reports of breached follow-up timelines.
Allow for patient interaction and feedback.
20. Ability to automatically send test or investigation results to the patient. Table 3. The Delphi items* reaching and not reaching consensus that met the nine principles of high quality safety-netting tools *A small number of items did not reflect the principles and are not included here

Implications for research and practice
This research has highlighted features of E-SN tools that clinicians believe to be important or desirable and how our PPI group believe these and the omitted features will impact the provision of safety-netting. E-SN tools will continue to evolve and future tools should incorporate these preferences and measure whether user experience and patient safety is improved. Widespread adoption of E-SN tools will be facilitated, and may be dependant, on strong underpinning evidence as highlighted by our Delphi participants and in previous research (28). Assessing the impact of tools on patient outcomes, such as rates of appropriate reconsultation, the number of consultations before referral for two-week-wait investigation, and the number of missed tests or missing test results is an important part of this evidence base. Additionally, gathering patient feedback on the experience of being safety-netted, what in the interaction works well, and what needs to be improved is important for the provision of safety-netting advice generally. Standardised and objective frameworks should also be used to identify high quality E-SN tools. Both of these topics are under-researched with, to our knowledge, only one as yet untested quality framework (3), and one study exploring the impact of E-SN tools on patient outcomes that is yet to report findings (15).
There was a tension between the preference of GPs to have safety-netting tools that are simple, flexible, and unobtrusive, and our PPI group who wanted safety-netting to be standardised and to use more alerts and warning notifications that aim to make the safety-net difficult to fall through. As E-SN tools continue to develop, patient preferences should be taken into account and new ways need to be found that can support the universal and robust provision of safety-netting that patients want, without placing too great a burden on clinicians. An example of how this may be achieved is through the targeted and intelligent use of technology to ensure tool functions are meaningful. Targeted and intelligent technology could, for example, identify the subtle increases in healthcare contacts, which have been shown to occur in the months leading up to a cancer diagnosis (35)(36)(37)(38). For example, E-SN tools could monitor consultation patterns and create an alert if the pattern for the individual patient changes over time. This approach may reduce inappropriate alerts for patients who usually attend frequently while also providing alerts if the individual's presentation patterns change.
The increasing use of the EHR has presented an opportunity to move safety-netting from a verbal exchange, perhaps recorded in a note, to an online systems-based approach. Successful E-SN will depend on the development of tools that are intuitive to use, provide a level of informational continuity even if there is no relational continuity, and are supported by evidence of improved patient outcomes. This research has established consensus among primary care clinicians on the features of E-SN that are important. Future research should incorporate these features into E-SN tools and gather evidence of their impact and the benefits to patients.

Funding
This work was funded through the Cancer Research UK Early Diagnosis Advisory Group (grant number C48270/A27880).

Ethical approval
Ethical approval was not required for this study as it was deemed to be a service evaluation by the University of Oxford Medical Sciences Interdivisional Research Ethics Committee and Clinical Trials and Research Governance unit.