Evaluating the benefits of machine learning for diagnosing deep vein thrombosis compared with gold standard ultrasound: a feasibility study

Patient recruitment

Patients with suspected DVT requiring an ultrasound scan (USS) were recruited consecutively at a hospital in Berlin, Germany, over a 3.5-month period in 2022. Forty-six per cent of the participants were inpatients, 2% had directly scheduled scan appointments, and the majority (52%) were referred from the primary care sector.

A statistical dichotomous endpoint power calculation was conducted by an Imperial College London statistician. The calculation was based on the understanding that approximately 15% of patients with symptoms indicative of DVT would test positive. To achieve a statistical power of 80%, a minimum of seven patients with positive DVT cases were needed to be included in the study.

Patients were eligible for inclusion if they were aged ≥18 years, demonstrated the capacity to consent, exhibited symptoms suggestive of DVT, and the DVT diagnostic algorithm indicated the necessity of a diagnostic USS.¹²

Each consenting patient underwent two distinct USS:

• index: AI-guided POCUS conducted by HCPs without formal DVT diagnostic ultrasound training; and

• reference: the second scan, a formal duplex scan, which was conducted by a specialist blinded to the index scan results.

Index test

The AI-guided scans were conducted using ThinkSono Guidance, an app developed by the company ThinkSono GmbH. This app, installed on a smartphone, was connected to a Clarius L7 HD3 linear handheld ultrasound probe. The app had, at the time of the study, a Class 1 CE certificate (Supplementary Information S1) and was used under its intended purpose (Supplementary Information S2).

Prior validation of the ThinkSono Guidance has been documented in previous studies.^6,7 The app guides users through a two-region POCUS, directing compressions in the groin areas and in the knee pit. If the procedure identifies insufficient compression, improper vessel alignment in the image, or probe displacement during compression, the application suggests redoing the compression. Users retain the choice to override these suggestions. After successfully gathering all compression points, sequences of compression (lasting 10 seconds each) are uploaded to a cloud-based dashboard for remote assessment.

Four sequences are recorded per scan (Figure 1a–g). USS were conducted by a non-specialist without formal experience in performing DVT diagnostic scans, following a 1-hour training session with ThinkSono Guidance.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1. A–G Workflow of ThinkSono Guidance (it guides the operator through anatomical landmarks and indicates when to compress. ThinkSono Guidance analyses the images, and provides a remote diagnosis)

AI algorithm

The ThinkSono Guidance app utilises a convolutional neural network for vessel analysis, supplemented by auxiliary branches for anatomical location prediction. Trained on prospectively collected data, the model employs manual vein and artery delineations along with image-level labels for anatomical locations. It operates on B-mode ultrasound images resampled to 128 × 128 pixels, producing segmentation masks and categorical location labels within 25 milliseconds.

The U-Net architecture serves as the model backbone, augmented by a landmark prediction branch. Three models, each tailored to specific anatomical regions, are trained with identical architecture. Further details and illustrations are provided in the figures and tables within the article.

The U-Net architecture serves as the model backbone, augmented by a landmark prediction branch. Three models, each tailored to specific anatomical regions, are trained with identical architecture. Further details and illustrations are provided in the attached figures.

Reference standard

The DVT diagnostic scan, recommended by NICE guidelines, includes compressions at multiple points on the proximal leg, coupled with duplex.¹² Compressions are typically applied every few centimetres from the groin to the knee pit along the large veins. Duplex is employed on the large veins to demonstrate normal blood movement. A qualified physician, unaware of the index scan outcomes, provided a report after completion of the reference scans, indicating the presence or absence of DVT.

Image review

The ultrasound sequences from the index scan were assessed by five qualified remote experts using the secure cloud dashboard. These experts were kept unaware of all patient attributes and indications, only being informed if the examination was conducted on the left or right extremity.

Remote specialists assessed image quality using the American College of Emergency Physicians (ACEP) score, which is rated on a scale from 1–5. An overall examination score of ≥3 was deemed to indicate sufficient image quality (Table 1). Furthermore, each image scoring ≥3 was categorised as having 'compressible' veins, 'incompressible' veins, or requiring another USS ('indeterminate').

Patient triaging

To mimic clinical triage methods, a retrospective analysis combined the classifications into high-risk and low-risk groups. This process aimed to reflect the proportion of scans that could potentially be managed in primary care without the need for referral to a formal USS. The scans were evaluated for image quality and compressibility assessment, as summarised in Table 2.

Statistical analysis

This study sought to assess the feasibility of utilising AI-guided POCUS to guide healthcare professionals (HCPs) without formal ultrasound training in acquiring ultrasound images suitable for remote interpretation. The evaluation included an analysis of image quality, the accuracy of remote diagnoses, and the efficiency of the AI-guided POCUS application as a primary care triage tool by categorising scans into high- and low-risk groups, potentially reducing the necessity for scans in secondary care.

For the reported confidence intervals (95% CI), statistical methods recommended for diagnostic test accuracy analysis were applied. This approach follows standard practices used in previous studies, including the development phase of ThinkSono Guidance (formerly AutoDVT), as outlined in the study by Prof. Kainz¹³. These methods were used to calculate the sensitivity and specificity of the AI-guided POCUS, contributing to the overall assessment of its diagnostic performance.

The analysis was conducted using Google Sheets, and the outcomes were displayed with variability divergence. The image quality assessment employed a majority voting process with five experts assigning an ACEP score of ≥3 for the uploaded scans. Remote diagnoses relied on the unanimous agreement of at least three experts regarding vein compressibility, and the results were compared with formal USS, considering sensitivity and specificity. The triaging into low- and high-risk categories was performed through majority voting by the remote specialists.

For the reported confidence intervals (95% CI), statistical methods recommended for diagnostic test accuracy analysis were applied. This approach follows standard practices used in previous studies, including the development phase of ThinkSono Guidance (formerly AutoDVT)¹³. These methods were used to calculate the sensitivity and specificity of the AI-guided POCUS, contributing to the overall assessment of its diagnostic performance.