Article Text
PDF
Abstract
Introduction The utility of perioperative point-of-care ultrasound (P-POCUS) is rapidly growing. The successful implementation of a comprehensive P-POCUS curriculum, Focused PeriOperative Risk Evaluation Sonography Involving Gastro-abdominal, Hemodynamic, and Trans-thoracic Ultrasound (FORESIGHT), has been demonstrated. This project sought to further evaluate the utility of P-POCUS with the following aims: (1) to assess the ability to train the FORESIGHT curriculum via a free, open-access, online platform; (2) to launch a P-POCUS clinical service as a quality improvement (QI) initiative; (3) to evaluate the diagnostic accuracy of the P-POCUS examinations to formal diagnostic studies; and (4) to compare the P-POCUS diagnostic accuracy with the diagnostic accuracy of traditional assessment (TA).
Methods This study was launched as a QI project for the implementation of a P-POCUS service. A group of attending and resident anesthesiologists completed P-POCUS training supported by an online curriculum. After training, a P-POCUS service was launched. The P-POCUS service was available for any perioperative event, and specific triggers were also identified. All examinations were documented on a validated datasheet. The diagnostic accuracy of the two index tests, P-POCUS and TA, were compared with formal diagnostic testing. TA was defined as a combination of the anesthesiologist’s bedside assessment and physical examination. The primary outcome marker was a comparison in the accuracy of new diagnosis detected by P-POCUS service versus the TA performed by the primary anesthesiologist.
Results A total of 686 P-POCUS examinations were performed with 466 examinations having formal diagnostic studies for comparison. Of these, 92 examinations were detected as having new diagnoses. Performance for detection of a new diagnosis demonstrated a statistically higher sensitivity for the P-POCUS examinations (p<0.0001). Performance comparison of all P-POCUS examinations that were matched to formal diagnostic studies (n=466) also demonstrated a significantly higher sensitivity. These findings were consistent across cardiovascular, pulmonary and abdominal P-POCUS categories (p<0.01). Additionally, multiple pathologies demonstrated complete agreement between the P-POCUS examination and the formal study.
Conclusion A P-POCUS service can be developed after training facilitated by an online curriculum. P-POCUS examinations can be performed by anesthesiologists with a high degree of accuracy to formal studies, which is superior to TA.
- point-of-care ultrasound
- perioperative medicine
- bedside assessment
- online ultrasound education
- quality improvement
- diagnostic accuracy of point of care ultrasound
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, an indication of whether changes were made, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- point-of-care ultrasound
- perioperative medicine
- bedside assessment
- online ultrasound education
- quality improvement
- diagnostic accuracy of point of care ultrasound
Introduction
Perhaps the most important aspect of physicians’ role is their diagnostic capabilities. If we cannot diagnose pathology with accuracy, we cannot effectively treat and may cause harm. At the bedside, the tools available for the traditional assessment (TA) of patient pathology are often the history and physical examination. Recently, a new tool, point-of-care ultrasound (POCUS), has emerged as a modality to improve bedside assessment. POCUS refers to the use of portable ultrasonography at the patient’s bedside for diagnostic and therapeutic purposes.1 This modality has demonstrated utility for nearly every type of physical examination component, including cardiovascular, pulmonary, airway, neurological and abdominal.
Indeed, several specialties have embraced POCUS and have defined formalized educational and certification pathways.2 For example, in the USA, emergency medicine has adopted POCUS training as a ‘core competency’ and has published guidelines regarding scope of practice, training/proficiency and value/reimbursement.3 The American Medical Association has also passed policies supporting the right to use ultrasound in accordance with specialty-specific standards.1 In addition, the Society of Critical Care Medicine offers training programs and certifications for POCUS.
Regarding the perioperative setting, a rapidly growing body of evidence is emerging supporting the utility of POCUS. Perioperative point-of-care ultrasound (P-POCUS) has demonstrated utility for cardiovascular evaluation,4–7 pulmonary/airway8–12 and abdominal/gastric13 14 examinations. These works, among others, emphasize the concept that P-POCUS is a useful tool for all anesthesiologists. This was nicely summarized in a recent consensus paper reporting a ‘call to action’ on this topic.2 Further support for P-POCUS integration has also recently been provided by the Accreditation Council for Graduate Medical Education (ACGME), with new 2018 program requirements listing ‘competency in using surface cardiac and pulmonary ultrasound to evaluate organ function and pathology’.15
In 2015, one such curriculum, termed Focused PeriOperative Risk Evaluation Sonography Involving Gastro-abdominal, Hemodynamic, and Trans-thoracic Ultrasound (FORESIGHT), was published as an effective strategy for P-POCUS education.16 Results from the study demonstrated utility via all levels of the Kirkpatrick assessment tool.16 Importantly, however, while this study suggested a positive clinical impact, no comparisons were performed on the diagnostic accuracy between the P-POCUS examinations and formal diagnostic studies. Moreover, the curriculum used for study was not open source and thus was hard to reproduce or expand.
This project sought to address these limitations with the following aims: (1) to train the FORESIGHT curriculum at another academic center via a free, open-access, online platform; (2) to launch a P-POCUS clinical service as a quality improvement (QI) initiative; (3) to evaluate the diagnostic accuracy, through sensitivity, of the P-POCUS examinations to formal diagnostic studies; and (4) to compare the P-POCUS diagnostic accuracy, through sensitivity, with the diagnostic accuracy of TA.
Methods
This study was approved by our institutional review board (IRB) (IRB number 5170140) as a retrospective cohort study evaluating all POCUS examinations performed by the P-POCUS service, which was launched as a prospective QI initiative. The IRB waived the requirement for written informed consent as the P-POCUS service was launched as a QI initiative. Given this study design, it is reported following the Standards for Quality Improvement Reporting Excellence guidelines. Data for this study were collected from November 2016 to February 2018. All examinations were performed at a single-center, tertiary care hospital.
Development of the training curriculum
Prior to initiation of the P-POCUS service, an educational curriculum was developed to train both attending anesthesiologists and residents. This curriculum was based on FORESIGHT, a previously validated perioperative ultrasound educational curriculum.16 Briefly, the FORESIGHT curriculum was developed in compliance with ACGME core competency requirements to evaluate cardiovascular function, volume status, pulmonary function, presence of abdomen fluid, gastric content, position of the endotracheal tube and presence of increased intracranial pressure (figure 1).16
Clinical objectives of whole-body comprehensive ultrasound examination: FORESIGHT examination. FORESIGHT, Focused PeriOperative Risk Evaluation Sonography Involving Gastro-abdominal, Hemodynamic and Trans-thoracic Ultrasound; ICP, intracranial pressure; IVC, inferior vena cava; L, left; LV, left ventricle; LVOT, left ventricular outflow tract; R, right.
For this initiative, the authors converted the FORESIGHT curriculum content to a free-to-access, open-source, online platform (www.foresightultrasound.com). After the educational content was developed, attending faculty who had volunteered to participate underwent a training phase in which they reviewed all online material over a period of 9 months (March 2016 to November 2016). Additionally, these faculty attended 1-hour weekly sessions over this time period. These sessions were focused on hands-on training via an ultrasound simulator (Vimedix; CAE Healthcare, Sarasota, Florida, USA) and human models. Minimal time was spent reviewing the concepts of the components of the FORESIGHT examination; rather the majority of time was spent practicing image acquisition and reviewing pathology via stored images and the ultrasound simulator. Completion of training was determined by verification of online content review and performance of at least 50 FORESIGHT examinations, as previously supported.16
Designing the intervention: P-POCUS service
On completion of the training phase, 10 attending anesthesiologists met the completion criteria, and the P-POCUS service was launched in November 2016. At the time of launch, 10 resident anesthesiologists had completed training and an additional 14 residents (n=24) completed the course within the first year of the QI initiative. The P-POCUS service was designed to provide consultation weekdays from 06:30 to 15:00. Additional examinations were performed outside of this time period when a trained faculty member was available. Consultation was available for any patient; however, common scenarios included unknown or concerning medical history, as well as acute events categorized by organ system (cardiovascular, pulmonary and abdominal). Specific acute events were as follows:
Cardiovascular: hemodynamic instability, chest pain, poor functional status, shortness of breath, trauma, volume status and ECG changes.
Pulmonary: shortness of breath, trauma, difficult ventilation, difficult airway, increased oxygen requirements and significant smoking history.
Abdominal: questionable fasting status, history of gastroparesis, trauma, risk of intra-abdominal bleeding, severe nausea and vomiting, high gastric output and bladder evaluation.
Specifically, General Electric provided three Vscan Dual Probe devices and provided support for database management. P-POCUS service used the Vscan devices primarily, and other ultrasound equipment was used if additional examination techniques were required.
Process measures
We conducted a retrospective review study to evaluate the implementation of the P-POCUS service as a QI initiative from November 2016 to February 2018. Patients who underwent perioperative assessment with TA and who also underwent a P-POCUS examination were included in the study. Importantly, the P-POCUS and TA examinations were performed independently by different anesthesiologists for all patients included in this study.
All P-POCUS examinations were documented via a previously validated report form.16 This report was used by the P-POCUS service to (1) capture the preliminary diagnoses known to the primary anesthesia team, (2) detail the trigger and the findings of the P-POCUS examination and (3) survey the primary anesthesiology if the P-POCUS examination offered new information as well on how it impacted management. All P-POCUS examinations included in the study were performed or reviewed by attending anesthesiologists on the P-POCUS service. Importantly, both the POCUS and the TA had to occur at the same perioperative setting (preoperative area and intraoperative and postoperative recovery) within 1 hour of each other to be included in the primary marker comparison.
Patients’ electronic medical records (EMRs) were reviewed to capture the perioperative course and the TA findings. In addition, the EMR was reviewed to capture all formal diagnostic studies performed after both the TA and POCUS examinations. Both TA and POCUS examination findings were compared with these formal diagnostic studies for accuracy comparison. Formal diagnostic studies included all patient imaging studies in which findings were reported by the specialists credentialed at the study institution (cardiology and radiology) within 24 hours after undergoing the TA and POCUS examinations. Formal diagnostic studies included radiological imagining, formal diagnostic ultrasound and echocardiography studies. In addition, documentation of removal of more than 200 mL during intraoperative gastric tube suctioning was also included for the diagnosis of significant gastric volume. Also, ECG reports performed within 24 hours of index tests that were confirmed by cardiology were used for the formal diagnosis of regional wall motion abnormalities if the patient had a subsequent formal echocardiogram. A separate team, which did not perform either the TA or POCUS examinations, reviewed the EMR data.
Outcome measures
This study sought to evaluate the diagnosis accuracy between two index tests (P-POCUS and TA) to a reference test (formal diagnostic studies). The primary marker was a comparison in the accuracy of new pathology diagnosis between the two index tests, with a hypothesis that P-POCUS examinations would have a higher degree of accuracy to formal diagnostic studies. The authors believe that the detection of new pathology diagnosis is an effective way of comparing the two index examinations without potential bias of the patient’s known medical history. Importantly, the two providers performing the two index tests were instructed to communicate only the examination type (cardiac, pulmonary or abdominal) that demonstrated pathology until findings of both index tests were captured.
Secondary outcome measures included (1) description of the P-POCUS examinations performed; (2) comparison of performance between P-POCUS and TA examinations to formal diagnostic studies for all examinations; (3) survey of the clinical impact of the P-POCUS examination; (4) further comparison of performance between P-POCUS and TA by cardiovascular, pulmonary and abdominal examinations; and (5) details of the accuracy of the most common pathologies detected by P-POCUS examination.
Analysis
For this study, accuracy was determined as the amount of agreement between the results of an index test and that of a reference test through a comparison of sensitivities, as previously supported.17 A McNemar test was used to evaluate the differences in accuracy of reported new diagnoses between the two index tests (P-POCUS and TA) and the reference standard (formal diagnostic studies from the EMR). Sensitivity and specificity, as well as other performance measures with corresponding 95% CIs, were computed for the two index tests (POCUS/TA) against the reference standard (EMR diagnostic studies). McNemar exact test estimates and corresponding ORs and 95% CIs were reported for each index against the reference standard; further, the McNemar test compared both index tests when both POCUS and TA results were conclusive. Distribution of P-POCUS examination categories performed, pathologies reported by the P-POCUS examinations and locations of P-POCUS examinations were also described. The Standards for Reporting of Diagnostic Accuracy18 19 was used to construct tables and to report diagnostic measures. All statistical analysis was conducted in R V.3.4.0.
Sample size
Sample size was computed assuming our primary marker was evaluation of diagnosis sensitivity between POCUS to TA examinations. We realized that to perform this study, we would encounter two main factors that would require a larger enrollment: (1) detection of new diagnosis by either of the two index tests and (2) rate at which a formal diagnostic result is present in the EMR. We previously reported a rate of new diagnosis with a P-POCUS service of 24%.16 Given an a priori assumption that 50% of the index tests with new diagnosis detected would have a formal diagnostic study, we assumed a combined prevalence of 12%. Given this prevalence, we calculated a minimum total sample size of 672 examinations would be required to achieve a power of 0.80 in order to detect a change in sensitivity of 0.40–0.70, based on a significance level of 0.05.
Results
A total of 686 P-POCUS examinations were evaluated by both P-POCUS service and perioperative TA (table 1). Out of this total, 466 (68%) had formal diagnostic studies available for comparison, and of those, 92 studies (13%) were reported as having a new diagnosis by either index test (TA or P-POCUS) (table 1). Cardiovascular examination was the most frequent POCUS examination, followed by pulmonary and then abdominal examinations. The majority of P-POCUS examinations were performed in the preoperative area, followed by the intraoperative setting. Significant medical history was the main reason for the P-POCUS examination, followed by unknown medical history. Regarding acute events that triggered P-POCUS examination, pulmonary acute events were the highest, followed by cardiovascular and abdominal (table 1). The primary anesthesiologist reported a positive impact on perioperative management after receiving the information from the P-POCUS examination approximately 44% of the time. For this subgroup, when asked why P-POCUS examination impacted management, there were similar frequencies between new diagnosis, verification of current diagnosis and reassurance of normal findings (table 1). Full details regarding the P-POCUS examinations are listed in table 1.
Perioperative POCUS service summary data
Primary outcome
Performance of the two index tests for detection of a new diagnosis, confirmed by a formal diagnostic study, demonstrated a statistically higher accuracy for the P-POCUS examinations versus TA (p<0.000, table 2). This was determined by comparison of sensitivities of the two index tests to formal diagnostic study results.
Performance evaluation: sensitivity comparison of POCUS to TA in new diagnosis examinations
Secondary outcome
Comparisons of all P-POCUS examinations in which formal diagnostic studies were available (n=466) showed a significantly higher degree of accuracy between the P-POCUS examinations and formal diagnostic studies (p<0.01, table 3). Similar results were demonstrated for subset comparisons by organ system (cardiovascular, pulmonary and abdominal) as well (table 3). Details of the congruency between POCUS examinations and formal diagnostic studies for the most commonly observed pathologies are shown in figure 2. Complete agreement between POCUS examination and formal diagnostic studies was demonstrated for the following pathologies: pericardial effusion, hypertrophic cardiomyopathy, regional wall motion abnormalities, right atrial enlargement, pneumothorax, endotracheal tube localization and intra-abdominal free fluid.
Performance evaluation: diagnostic measures of POCUS and TA studies
Accuracy rates for most common pathologies in all patients evaluated by the point-of-care ultrasound service. ETT, endotracheal tube; LA, left atrium; LUQ, left upper quadrant; RA, right atrium; RV, right ventricle; RUQ, right upper quadrant; US, ultrasound; WNL, within normal limits.
Discussion
Diagnostic performance
This study further supports the impact of POCUS for the perioperative setting by demonstrating an improved diagnostic accuracy, as determined by sensitivity comparison, with POCUS versus TA. We sought to evaluate new diagnosis accuracy between POCUS and TA, given the likely impact of the patients’ known medical histories on both index tests. Indeed, the impact of known medical history influencing physical examination results has been demonstrated.20 Thus, by assessing the accuracy in the detection of confirmed new diagnoses as our primary outcome, we are able to more directly determine the ability of P-POCUS to improve perioperative bedside assessment.
Our results also suggest that P-POCUS examinations demonstrated a high degree of accuracy when separated by organ system (cardiovascular, pulmonary and abdominal) to formal diagnostic studies. Further review demonstrates the strongest level of sensitivity with the cardiovascular category, followed by abdominal and then pulmonary. A likely explanation in the variability of performance of the three main P-POCUS categories may possibly relate to the complexity of performing a complete ultrasound examination for each of these categories. Indeed, performing a complete pulmonary POCUS examination can include up to 16 probe placement positions, which indicates the higher likelihood of missing gross pathology when compared with the four to six probe placement positions for cardiovascular POCUS. Overall, our results show an improved ability of P-POCUS to detect new diagnosis and greater ability to evaluate known pathology compared with TA. These findings support the utility of P-POCUS examinations if implemented under a structured educational curriculum.
Educational/training
Prior to the launch of this P-POCUS service, the authors sought to improve the ability to educate on the topic. The authors transitioned a previously validated curriculum to an online platform. All materials were created to be copyright free, and the website was developed under a Creative Commons license to promote free medical education on this topic. This study suggests that the online platform, in combination with hands-on training sessions was sufficient at training anesthesiologists to perform cardiovascular, pulmonary and abdominal P-POCUS examinations with a high degree of accuracy.
Impact on perioperative care
While this study was powered to evaluate the accuracy of POCUS examinations, our results show a positive impact on perioperative patient care. The primary anesthesiology team reported significant benefits from the P-POCUS examination. Interestingly, the utility of the service was nearly equally split between providing a new diagnosis, verifying current state of known diagnoses and reassurance of normal findings. Additionally, there are instances in which the P-POCUS examination demonstrated utility by determining an improvement or decline in a known pathology as well. In addition, a review of the various pathologies detected by our P-POCUS examinations demonstrated some interesting observations (figure 2). Under abdominal examinations, P-POCUS demonstrated no errors in the assessment of the gastric antral size and detection of abdominal free fluid. Under pulmonary examinations, P-POCUS demonstrated no errors for the evaluation of pneumothorax; however, this was a small sample size. Similar results were found for detection of an appropriate ET position. Assessment for pleural effusion and air-space disease, however, did demonstrate to be more difficult. Finally, under cardiovascular examinations, there was complete agreement in the detection of regional wall motion abnormalities, as well as pericardial effusion; there were no false negatives for the detection of systolic dysfunction or cardiac structure enlargement.
Limitations
This study sought to evaluate the ability of a P-POCUS service to detect new perioperative diagnosis over what is detected by routine perioperative physical examination and bedside assessment (TA). However, this study does have several limitations. While this study involved a prospective QI project in which the index tests occurred prospectively, comparisons were performed to reference tests that were captured from retrospective review of EMR data. Also, all clinical findings discovered by the P-POCUS examination were relayed to the primary anesthesiology team, but there was no formal protocolization for confirmatory testing. As described, the authors evaluated the new diagnosis performance, through sensitivity, between the two index tests as the primary marker to prevent potential bias of known pathologies. Given the projected low prevalence of new diagnosis detected, all P-POCUS examination types were combined for the primary marker of this study. Similarly, this study demonstrated small amounts of false negatives, which are known to bias the magnitude of sensitivity and specificity. There was no charge to the patient for the POCUS examinations, and evaluation for any potential economic impact was beyond the scope of the study. Finally, the 10 experienced attending faculty instructors were required to have completed the online training and 50 FORESIGHT examinations and did not have further formal certification or credentialing. However, it is important to note that there is no current standard P-POCUS certification pathway.
Conclusion
This project demonstrates that a P-POCUS service can be developed after implementation of a validated online training curriculum. Anesthesiologists trained by the FORESIGHT curriculum were able to demonstrate the positive impact on perioperative care through the improved diagnostic ability of P-POCUS when compared with formal diagnostic studies across cardiovascular, pulmonary and abdominal systems.
Acknowledgments
The authors thank Gina Brown, MD, Department of Anesthesiology, Loma Linda University Medical Center (gfbwt2@gmail.com).
Footnotes
Contributors All authors have reviewed this manuscript and have contributed as described. DR, RL, ID, JG and RM contributed to the concept design. BA, DR and AR contributed to the statistical analysis. DR, AR, JG, ID, RL, DW, JY, MA and GS contributed to the data collection. DR and RM did the planning. DR, JG, RL, ID, GS and RM contributed to the administrative requirements. DR, AR, JG, ID, RL, DW, JY, MA, BA, GS and RM contributed to the critical review of the manuscript. DR, AR and BA contributed to the interpretation of data. DR and BA drafted the tables and figures. DR, AR and BA drafted the manuscript.
Funding This project was partially supported by General Electric Healthcare (Fairfield, Connecticut, USA) as an investigator-initiated sponsored project.
Competing interests DR has recieved funded research from General Electric on point-of-care ultrasound and is a consultant for Fujifilm Sonosite
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.