Artificial intelligence automation of echocardiographic measurements
1Department of Cardiology, Smidt Heart Institute, Cedars-Sinai Medical Center, Los Angeles, CA
2Division of Cardiology, Department of Medicine, Stanford University, Palo Alto, CA
3School of Medicine, University of California, Los Angeles, CA
4Department of Bioengineering, University of California Los Angeles, Los Angeles, CA
5Division of Research, Kaiser Permanente Northern California, Pleasanton, CA
6School of Medicine, Brown University, Providence, RI
7Department of Computer Science, Stanford University, Stanford, CA
8Department of Biomedical Data Science, Stanford University
9Department of Electrical Engineering, Stanford University
10Department of Cardiology, Kaiser Permanente Santa Clara Medical Center, Santa Clara, CA
*Correspondence: David.ouyang@kp.org 4480 Hacienda Drive, Pleasanton, CA 94588ABSTRACT
Background
Accurate measurement of echocardiographic parameters is crucial for the diagnosis of cardiovascular disease and tracking of change over time, however manual assessment is time-consuming and can be imprecise. Artificial intelligence (AI) has the potential to reduce clinician burden by automating the time-intensive task of comprehensive measurement of echocardiographic parameters.
Methods
We developed and validated open-sourced deep learning semantic segmentation models for the automated measurement of 18 anatomic and Doppler measurements in echocardiography. The outputs of segmentation models were compared to sonographer measurements from two institutions to access accuracy and precision.
Results
We utilized 877,983 echocardiographic measurements from 155,215 studies from Cedars-Sinai Medical Center (CSMC) to develop EchoNet-Measurements, an open-source deep learning model for echocardiographic annotation. The models demonstrated a good correlation when compared with sonographer measurements from held-out data from CSMC and an independent external validation dataset from Stanford Healthcare (SHC). Measurements across all nine B-mode and nine Doppler measurements had high accuracy (an overall R2 of 0.967 (0.965 – 0.970) in the held-out CSMC dataset and 0.987 (0.984 – 0.989) in the SHC dataset). When evaluated end-to-end on a temporally distinct 2,103 studies at CSMC, EchoNet-Measurements performed well an overall R2 of 0.981 (0.976 – 0.984). Performance was consistent across patient characteristics including sex and atrial fibrillation status.
Conclusion
EchoNet-Measurement achieves high accuracy in automated echocardiographic measurement that is comparable to expert sonographers. This open-source model provides the foundation for future developments in AI applied to echocardiography.
Clinical Perspective
What Is New?
- We developed EchoNet-Measurements, the first publicly available deep learning framework for comprehensive automated echocardiographic measurements.
- We assessed the performance of EchoNet-Measurements, showing good precision and accuracy compared to human sonographers and cardiologists across multiple healthcare systems.
What Are the Clinical Implications?
- Deep-learning automated echocardiographic measurements can be conducted in a fraction of a second, reducing the time burden on sonographers and standardizing measurements, and potentially enhance reproducibility and diagnostic reliability.
- This open-source model provides broad opportunities for widespread adoption in both clinical use and research, including in resource-limited settings.
Article notes
Competing Interest Statement
YS reports support from the KAKENHI (Japan Society for the Promotion of Science: 24K10526), and honoraria for consulting from m3.com inc. DO reports support from the National Institute of Health (NIH; NHLBI R00HL157421, R01HL173487 and R01HL173526) and Alexion, and consulting or honoraria for lectures from EchoIQ, Ultromics, Pfizer, InVision, the Korean Society of Echocardiography, and the Japanese Society of Echocardiography.
Funding Statement
Disclosures This work is funded by NHLBI R00HL157421, R01HL173487 and R01HL173526.
Introduction
Accurate assessment and diagnosis of cardiovascular disease requires detailed and reproducible assessment of cardiac dimensions and function. Echocardiography, the most common and readily available modality of cardiac imaging, enables frequent and rapid evaluation and surveillance of disease. However, echocardiographic measurements require detailed manual measurements and even expert evaluation has considerable variability1,2. Given the large number of relevant and important echocardiographic parameters, accurate echocardiographic measurement is a highly time-consuming and places a significant burden on clinical practice of echocardiography3.
The advent of deep learning models with computer vision has enabled fully automated image analysis and processing across a wide range of medical imaging4–10. Isolated task specific annotations in echocardiography have already been proposed in echocardiography11,12, including for both wall thickness13 and left ventricular ejection fraction14,15, however there are many additional measurements that have been tackled with deep learning models. Furthermore, commercially available models are not open-source, thus limiting the ability to build upon prior developments.
Automation through AI has the potential for more precision as well as more rapid turn-around8,11,13,16–18. In this study, we aimed to develop and validate a deep-learning framework (EchoNet-Measurements) for automated measurement of wide range of echocardiographic parameters and measurements. Using a pipeline of view classification and automated segmentation, EchoNet-Measurements was trained on the largest largest-scale echocardiography annotation dataset to enable automated comprehensive AI measurements.
Methods
Data Curation
For training and held-out test dataset, we identified 155,215 studies from 78,037 patients who received a clinical transthoracic echocardiogram at CSMC between April 2009 and June 2022 (Figure 1). A total of 18 key transthoracic echocardiographic (TTE) parameters were used for model training. Imaging data is saved in Digital Imaging and Communications in Medicine (DICOM) format and annotated by expert sonographers and cardiologists as part of routine practice.
Important linear measurements from B-mode echocardiographic images and videos included intraventricular septum (IVS), left ventricular internal diameter (LVID), left ventricular posterior wall (LVPW) diameter, left atrium diameter, right ventricular basal diameter, ascending aorta diameter, aortic root diameter, pulmonary artery diameter, and inferior vena cava (IVC). Important Doppler measurements include tricuspid regurgitation maximal velocity (TR Vmax), aortic valve maximal velocity (AV Vmax), mitral valve maximal velocity (MV Vmax), left ventricular outflow tract maximum velocity (LVOT Vmax) septal e’, and lateral e’, mitral valve peak E velocity, E/A, tricuspid annular plane systolic excursion (TAPSE).
Additionally, EchoNet-Measurements was evaluated on 1,158 studies from 1,144 patients from SHC who underwent echocardiography between January 2013 and August 2018 for external validation. Furthermore, to assess the end-to-end processing of EchoNet-Measurements for full TTE studies, 2,103 studies from 2,039 patients from June 2022 to November 2024 at CSMC were used as a temporal split validation dataset. All echocardiography studies in the CSMC development cohort dataset and the SHC cohort were performed using Philips EPIQ 7 or iE33 ultrasound machines. All echocardiography studies in the temporal split dataset at CSMC were conducted using Philips EPIQ 7 or EPIQ CVx ultrasound systems. Approval for this study was obtained from the Cedars-Sinai Medical Center and Stanford Healthcare Institutional Review Boards, and the requirement for informed consent was waived for retrospective data analysis without patient contact.
Model Development
For EchoNet-Measurement training and testing, we used a DeepLabv3 architecture19 and trained with a binary cross-entropy with logits loss. The CSMC data was split by patient in a ratio of 8:1:1 for model training, validation, and held-out testing. Task specific models were trained for each measurement. For all models, an Adam optimizer with a learning rate of 0.001 was used, and the model was trained for up to 100 epochs with early stopping after 10 epochs based on the validation loss and a batch size of 24. The weights from the epoch with the minimum loss were used for the held-out test dataset and external validation. All performance metrics were analyzed using data from held-out data that was not involved in model training. An image quality classifier was also developed to identify low quality images and videos (examples in Supplemental Figures 1) which were excluded from downstream analysis. The image quality control model used a video-based convolutional neural network (R2+1D)14 to classify low-quality videos and an image-based model (DenseNet)20 to classify low-quality Doppler images21 or exclude inaccurate tissue doppler annotations (remove a’ annotation when e’ is desired). (Supplemental Methods).
Analysis of Cases with High Discrepancy between AI and clinicians
From the 90th percentile or higher of mean absolute error (MAE) between deep learning measurements and sonographer measurements, a total of 180 subsample data (randomly picked 10 images from each measurement variable) from the CSMC dataset were manually reviewed by two board-certified cardiologists (Y.S. and C.B.R). This analysis was conducted to identify the causes of significant differences between the deep-learning model measurements and those of the sonographers. We identified as if any of the following criteria applies: (1) the sonographer measurement (ground truth) was preferred over automatic measurements by the deep learning model, (2) the automatic measurements by the deep learning model were preferred over sonographer measurements, (3) images that both sonographer measurement and DL-model predictions are within a clinically-acceptable range but with a measurement variability, and (4) predictions on a severely low-quality images.
Statistical analysis
MAE, coefficient of determination (R2) and intraclass correlation coefficients (ICC) between clinically measured parameters and parameters predicted by the developed deep-learning model were calculated in the held-out dataset and the external dataset. Bland-Altman plots where the average of the two measurements was plotted against the difference were used to check the agreement between the actual and predicted measurements. Stratified analyses were done by image-quality and patient characteristics including patient sex and a history of atrial fibrillation. All 95% confidence intervals were calculated with 10,000 bootstrapping samples. Data analysis was performed using both Python (version 3.10.12) and R (version 4.2.2) programming languages. This study was carried out following the CONSORT-AI guideline22.
Code and Data Availability
The code, model weights, and demonstration video are available at https://github.com/echonet/measurement/. The patient data is not publicly available due to their potentially identifiable nature.
RESULTS
Model Development and Validation Cohorts
A total of 877,983 annotated measurements were identified from 155,215 CSMC echocardiography studies from 78,037 patients. The patient cohorts (mean age: 65.0 ± 17.4 years, female: 46.2% in CSMC derivation cohort) exhibited similar patient age, the proportion of female, race, the mean left ventricular ejection fraction and comorbidities across datasets. (Table 1 and Supplementary Table 1-3). The number of training and validation examples for each parameter are summarized in Table 2.
Assessment of EchoNet-Measurement’s Performance
In the CSMC test dataset, the EchoNet-Measurements model demonstrated a strong agreement between automated deep learning measurements and sonographer measurements with an overall R2 of 0.967 (0.965 – 0.970) in the held-out dataset. (ranging from a R2 = 0.27 (0.22-0.32) (ICC: 0.62 (0.60 – 0.63), MAE: 0.425 cm (0.404-0.446)) for ascending aorta diameter to R2 = 0.76 (ICC: 0.87 (0.86-0.87), MAE: 0.387cm (0.382-0.392)) for left ventricular internal diameter (LVID) (Figure 2)). Similarly for the Doppler images, the model demonstrated a high agreement between automated deep learning measurements and sonographer measurement, ranging from R2 = 0.55 (0.52-0.57) (ICC: 0.82 (0.66-0.89), MAE: 50.97 cm/2 (49.9-52.1)) for MR Vmax to R2 = 0.94 (ICC: 0.97 (0.97 – 0.97), MAE: 0.051 cm (0.050-0.052)) for TAPSE (Table 3).
Representative images comparing sonographer annotations and deep learning annotations are shown in Figure 3 and Figure 4 for linear and Doppler measurements respectively. Representative generated videos in linear measurement group are shown in Supplemental Video 1. A demonstration video for the high-throughput analysis of TR Vmax using the input of multiple DICOM files is shown in Supplemental Video 2. Image quality greatly affected the performance of automated measurements, with the high-quality image group consistently showed a significant improvement in MAE and coefficient of determination compared to low-quality echocardiography images (Supplementary Table 4 and Supplementary Figure 2-3). The model performance was consistent across patient characteristics including sex and atrial fibrillation status in all measurement parameters (Supplemental Table 5-6)
On the temporal split dataset from CSMC (June 2022 - November 2024), the model showed robust predictive accuracy (an overall R2 of 0.981 (0.976-0.984)) in end-to-end analysis (e.g., an R2 of 0.962 (0.95-0.97), an ICC of 0.981(0.98-0.98) and an MAE of 0.062 mm (95% CI 0.06-0.07) for TAPSE, and R2 of 0.748 (0.69-0.80), an ICC of 0.855 (0.77-0.90) and an MAE of 0.204mm (95% CI 0.19-0.22) for IVC diameter) (Figure 5 and Supplemental Table 1).
In the external SHC dataset with image-level ground truth measurement value, EchoNet-Measurement’s performance was robust (Figure 5 and Supplemental Table 2) with an overall R2 of 0.987 (0.984 – 0.989) in the SHC dataset. The average R2 for linear measurements was 0.961 (0.95-0.97) with MAE ranging from 0.106mm (0.09-0.12) for IVS diameter to 0.489mm (0.34-0.66) for pulmonary artery diameter. The model’s performance was also consistent and robust across all Doppler parameters with an overall R2 of 0.990 (0.988 – 0.991) for linear measurements (e.g., MAE of 0.148 cm (0.11-0.19) in TAPSE and 7.89 mmHg (7.01-8.82) in LVOT Vmax). The model performance was similarly robust in external dataset with study-level ground truth label for both linear measurement and all Doppler parameters with an overall R2 of 0.942 (0.936-0.947) and 0.983 (0.979-0.986), respectively (Supplemental Table 3, Supplemental Figure 4)
Analysis of High Measurement Discrepancies Cases
Among echocardiography images with absolute differences between sonographer measurement and deep-learning measurement in the 90th percentile or higher, a subset of 180 images (randomly selected 10 images from each measurement variable) were manually reviewed. Of a total of 180 images, 91 (50.6%) were categorized as the cases where sonographer annotations were preferred over automatic measurements by the deep learning model, 20 images (11.1%) was classified as the images where the deep-learning model was preferred over sonographer measurements, and 54 images (30.0%) were categorized the images that both sonographer annotations and deep-learning model predictions were clinically acceptable but demonstrating a measurement variabilities. In the remaining 15 images (8.3%), echocardiography images were severely noisy or blurred, seen as inappropriate images for deep learning-based measurements. Representative images are described in Supplemental Figure 5 and the number of images in each category is described in Supplemental Table 7.
DISCUSSION
In this study, we developed EchoNet-Measurements, a deep learning model for the automatic measurement of 18 echocardiographic parameters, and showed the model performed well in multiple geographically and temporally distinct cohorts. Using the largest yet real-world dataset of sonographer clinical annotations, this open-source model demonstrated favorable measurement accuracy comparable to that of expert sonographers. Further, we have released the code and weights of these models publicly, along with demo user interface, to facilitate further research in the field.
Comprehensive measurement is needed since clinical echocardiography involves the measurement of many parameters that holistically assess patient status. For example, diagnosis for heart failure with preserved ejection fraction (HFpEF) requires a combination of multiple echocardiographic parameters23,24—including left atrial volume index (LAVI), lateral e’, septal e’, E/A ratio, right ventricular systolic pressure), and TR velocity. Additionally, measuring the IVC diameter and its collapsibility is useful for estimating left ventricular filling pressures and right atrial pressure, which are critical for the serial surveillance of heart failure status.25,26 One of the primary benefits of measurement automation is the reduction of examination time and variability between sonographers, leading to more consistent result.27,28
Our study has both strengths and limitations. Previous studies have already focused on automating the measurement of limited echocardiographic parameters using deep learning segmentation models11,16,17, but we note that many prior models do not share code or model weights - limiting utility for reproducible research purposes and future development. EchoNet-Measurements is trained on data from only one medical center, which might capture biases and quirks associated with the dataset, however we demonstrate favorable results compared to previous deep-learning models13,17,18,29. Furthermore, one of the strengths of our model is the size of the training data, which is the largest yet dataset of sonographer annotations. In multiple prior works both within medicine30 and outside31, the scale of training data has a strong impact on downstream model performance. Another features of our work is the public availability of our model and demo interface, which enables the creation of clinical datasets and supports reproducible clinical research.
Conclusion
In this study, we developed and validated EchoNet-Measurements, an open-source deep learning framework for the automated assessment of a wide range of echocardiographic parameters. Trained on the largest dataset to date of clinical annotations, the model demonstrated robust performance across both linear and Doppler measurements.
Supporting information
Data Availability
Code and Data Availability The code, model weights, and demonstration video are available at https://github.com/echonet/measurement/. The patient data is not publicly available due to their potentially identifiable nature.
Untitled section
Abbreviations and Acronyms
- CSMC
- Cedars-Sinai Medical Center
- SHC
- Stanford Healthcare
- DICOM
- Digital Imaging and Communications in Medicine
- LVOT Vmax
- Left Ventricular Outflow Tract Maximum Velocity
- TR Vmax
- Tricuspid Regurgitation Maximum Velocity
- IVS
- Intraventricular Septum
- LVID
- Left Ventricular Internal Diameter
- LVPW
- Left Ventricular Posterior Wall
- TAPSE
- Tricuspid Annular Plane Systolic Excursion
- MAE
- Mean Absolute Error
- AV Vmax
- Aortic Valve Maximum Velocity
- MV Vmax
- Mitral Valve Maximum Velocity
- E/A
- Ratio of Early (E) to Late (A) Diastolic Mitral Flow Velocities
- IVC
- Inferior Vena Cava
Supplemental Materials
Supplemental Video 1: Beat-to-beat Linear Measurement Parameters Prediction Legend: This video demonstrates (A through I) the deep learning model’s beat-to-beat predictions for multiple linear measurement parameters across echocardiographic images. For a detailed explanation of echocardiography parameter abbreviations, refer to Supplemental Table 1.
Supplemental Video 2: Demo for High-Throughput TR Vmax Measurement and Annotation Pipeline for CW Doppler Waveform Prediction
Legend: This interface developed using Gradio demonstrates a process for uploading multiple DICOM images containing TR Vmax measurements and performing high-throughput annotations using an automated pipeline. Users can upload multiple DICOM files and these are then processed to annotate the predicted points and calculate the TR Vmax values. The outputs including annotated Doppler images and extracted TR Vmax values are displayed in a results table and can be exported as a CSV file.
Supplemental Methods
For the development of the quality-control model for the linear measurement group, a video-based convolutional neural network (R2+1D) was used with standardized input videos of 112 × 112 pixel for linear measurement group. For the Doppler image quality-control model, an image-based model (DenseNet) was employed with 426 × 1024 still image inputs. A dataset of 31,506 manually curated videos (9.8% were classified as poor quality videos) from 21,035 patients and 29,232 still images (21.5% were classified as poor images or noisy images) from 20,951 patients in CSMC was split 8:1:1 ratio by patient medical record number and all videos and images were classified as low-quality or high-quality images (representative ground truth images in Supplemental Figure 2). Image quality was evaluated by two cardiologists (Y.S and C.B.R). The model was trained to minimize binary cross-entropy loss using the AdamW optimizer. The initial learning rate was set to 1×10-5, with a batch size of 10 for the video model and 24 for the image model, over the course of 50 epochs with an early stopping setting. Model performance was evaluated in the held-out dataset, achieving an AUROC of 0.984 in the linear measurement group and an AUROC of 0.912 in the Doppler measurement group.
Further, a classification model was developed using a similar procedure to exclude cases where annotated point was plotted on a’ during septal or lateral e’ prediction. Datasets were manually reviewed and finally datasets annotated with only e’ (n=7962) or only a’ (n=2128) were prepared. A classification model was then built using DenseNet and abovementioned hyperparameters and image sizes. This model achieved an AUROC of 0.965 on a test dataset split by patient in an 8:1:1 ratio, and the optimal cutoff value was determined using the Youden Index. Following the inference of septal e’ and lateral e’ prediction, data plotted as a’ were removed by passing them through this model.
Disclosures
This work is funded by NHLBI R00HL157421, R01HL173487 and R01HL173526. YS reports support from the KAKENHI (Japan Society for the Promotion of Science: 24K10526), and honoraria for consulting from m3.com inc. DO reports support from the National Institute of Health (NIH; NHLBI R00HL157421, R01HL173487 and R01HL173526) and Alexion, and consulting or honoraria for lectures from EchoIQ, Ultromics, Pfizer, InVision, the Korean Society of Echocardiography, and the Japanese Society of Echocardiography.