Impact of Digital Ambient EXperience on Pediatric Subspecialist Documentation Burden
Evaluation of AI-Generated Documentation Software to Improve Pediatric Physician
Study Overview
Detailed Description
Background: While electronic health record (EHR) systems have contributed to advances in patient safety and quality of care, they have also been associated with a significant increase in documentation burden, contributing to burnout among clinicians [1]. This is particularly true for physicians with insufficient time for documentation. In some cases, it has resulted in a reduction in appointment slots to allow for additional documentation time, which in turn decreases patient access to care and physician productivity [2,3].
Microsoft Nuance© has recently announced general availability of a new generative artificial intelligence (AI) solution called Digital Ambient eXperience (DAX) DAX CoPilot [4], in which the visit is recorded with patient/parental consent, but a note is generated through the AI along with a visit transcript in near-real time that the provider can use and edit as they see fit. In addition, it allows providers to continue to use their documentation templates while adding the generative AI to "smart sections" within their note. This approach has the potential to substantially reduce documentation burden while maintaining documentation preferences of many providers.
This randomized quality improvement pilot project aims to assess whether the implementation of generative AI software for documentation, Microsoft Nuance's DAX Copilot, can enhance physician documentation efficiency and reduce burnout.
Objectives
Quality Improvement Global Aim: To increase provider documentation efficiency and reduce provider burnout related to documentation burden.
Children's Operational Goal: Determine if the cost of DAX Copilot or related vendor software is justified by reduction in proxies for physician burnout and/or could be offset by seeing more patients in the same time period to improve revenue and patient access.
Goals of the Proposed Work:
- Determine the influence of DAX Copilot on proxies for physician burnout including pajama time, time in notes, and subjective measures of EHR efficiency.
Methods: This is a randomized quality improvement project that will assess changes to proxies for provider documentation efficiency and burnout through a difference-in-differences design.
Project Participants
The project will recruit 20 providers who meet the following inclusion criteria:
- Practices at Children's in a specialty supported by DAX Copilot product and with available documentation efficiency metrics from Epic's Signal product.
- >0.5 clinical full time equivalent (cFTE)
- 2 or more half days per week on average seeing outpatients as the primary provider (not overseeing trainees or APPs).
- Agrees to use the Children's EHR mobile application (Haiku) on their personal device.
- Agrees to offer use of the DAX Copilot generative AI software for all patient visit encounters for the duration of the project period
Sufficient and stable EHR data on documentation efficiency from Epic's Signal product, defined as:
- Having populated data available for at least 40 of the last 52 weeks.
- Having stable metrics for pajama time and time in notes in the last 6 months as determined by two physician informaticists based on visual inspection.
- >75% of ambulatory documentation completed by the provider themselves (as opposed to taking over a note of a trainee or advanced practice provider).
Participants will be identified based on characteristics such as specialty, cFTE, proportion and location of outpatient work, and Epic's Ambient Opportunity Index from Signal, which is based on normalized scores for proportion of same-day charts closed, pajama type, and characters manually typed.
Randomization will be conducted in blocks of two at the specialty level to ensure equal representation of specialties into one of the two following groups for a 3 month pilot:
- Use of DAX Copilot generative AI software (AI Group)
- Continuation of current documentation workflow (Control Group)
Outcomes
Our primary outcomes to be obtained through Epic's Signal product will be:
- "Pajama Time", defined as the average number of minutes per scheduled day spent in charting activities outside 7 AM to 5:30 PM on weekdays, time outside scheduled hours on weekends, and time on unscheduled holidays. This metric is associated with the exhaustion subscale of the Maslach Burnout Inventory [5].
"Time in Notes per Appointment" in minutes
Additional outcome metrics will include:
- Progress Note Length (characters)
- Note Contribution (written by provider vs. others)
- Time to Appointment Closure
- Proportion of notes completed using DAX Copilot generative AI software
- Average patient volume per week
- Pre- and post-project user responses on a modified KLAS EHR Efficiency and Satisfaction survey, a validated benchmarking tool.
- Pre- and post-project patient experience scores through routinely capture NRC surveys.
Data Collection
Data will be collected from Epic© Signal and through surveys. The data will include:
- Demographic information of project participants
- Data related to documentation efficiency as outlined in the quality metrics above
Statistical Analysis Our primary analysis will be a difference-in-differences analysis for each outcome. For example, the difference between the provider's average pajama time before and after the intervention period will be calculated for all participants. We will then determine how this average differs in the AI group and in the control group to assess the difference-in-differences.
Additional analyses will include adjusted or stratified difference-in-differences analyses based on provider characteristics listed above. We will also calculate descriptive statistics to compare the outcomes and covariates between the two groups. Depending on the nature of the data, we may use run charts, t-tests, ANOVA, or other appropriate statistical methods to assess the impact of generative AI documentation software.
References
- Budd, J. Burnout Related to Electronic Health Record Use in Primary Care. J Prim Care Community Health 14, 21501319231166921 (2023).
- Colicchio, T.K., Cimino, J.J. & Del Fiol, G. Unintended Consequences of Nationwide Electronic Health Record Adoption: Challenges and Opportunities in the Post-Meaningful Use Era. J Med Internet Res 21, e13313 (2019).
- Gaffney, A., et al. Medical Documentation Burden Among US Office-Based Physicians in 2019: A National Study. JAMA Intern Med 182, 564-566 (2022).
- Microsoft+Nuance. Automatically document care with Dragon Ambient eXperience (DAX). https://www.nuance.com/asset/en_us/collateral/healthcare/data-sheet/ds-ambient-clinical-intelligence-en-us.pdf. (2023).
- Adler-Milstein, J. & Wang, M.D. The impact of transitioning from availability of outside records within electronic health records to integration of local and outside records within electronic health records. J Am Med Inform Assoc 27, 606-612 (2020).
Study Type
Enrollment (Actual)
Phase
- Not Applicable
Contacts and Locations
Study Locations
-
-
Georgia
-
Atlanta, Georgia, United States, 30329
- Children's Healthcare of Atlanta
-
-
Participation Criteria
Eligibility Criteria
Ages Eligible for Study
- Adult
- Older Adult
Accepts Healthy Volunteers
Description
Inclusion Criteria:
Providers:
- Practices at Children's in a specialty supported by DAX Copilot product and with available documentation efficiency metrics from Epic's Signal product.
- >0.5 clinical full time equivalent (cFTE)
- 2 or more half days per week on average seeing outpatients as the primary provider (not overseeing trainees or APPs).
- Agrees to use the Children's EHR mobile application (Haiku) on their personal device.
- Agrees to offer use of the DAX Copilot generative AI software for all eligible patient visit encounters for the duration of the project period
Sufficient and stable EHR data on documentation efficiency from Epic's Signal product, defined as:
- Having populated data available for at least 40 of the last 52 weeks.
- Having stable metrics for pajama time and time in notes in the last 6 months as determined by two physician informaticists based on visual inspection.
- >75% of ambulatory documentation completed by the provider themselves (as opposed to taking over a note of a trainee or advanced practice provider).
Visits: In person office visits in which interactions occur in English. For example, a visit with an interpreter present who translates verbally into English during the visit would be eligible. A visit in which all interactions occur in another language (i.e. the provider speaks the language the family uses as well) would not be eligible as the AI has not been developed yet for other languages.
Exclusion Criteria:
- Poor WiFi and cellular coverage in the clinic.
Study Plan
How is the study designed?
Design Details
- Primary Purpose: Health Services Research
- Allocation: Randomized
- Interventional Model: Parallel Assignment
- Masking: None (Open Label)
Arms and Interventions
Participant Group / Arm |
Intervention / Treatment |
|---|---|
|
No Intervention: Control
|
|
|
Experimental: Intervention
- Enrolled in DAX Copilot with appropriate training and asked to use for every eligible visit where the family does not decline use for 3 months.
|
|
What is the study measuring?
Primary Outcome Measures
Outcome Measure |
Measure Description |
Time Frame |
|---|---|---|
|
Pajama Time
Time Frame: 12 months prior to study through end of study period.
|
Per Epic Signal: Average number of minutes per scheduled day spent in charting activities outside 7 AM to 5:30 PM on weekdays, time outside scheduled hours on weekends, and time on unscheduled holidays. Numerator: Minutes spent in charting activities outside 7 AM to 5:30 PM on weekdays and outside scheduled hours on weekends. Denominator: Scheduled days where time was spent in the system within the reporting period. |
12 months prior to study through end of study period.
|
|
Time in Notes per Appointment
Time Frame: 12 months prior to study through end of study period.
|
Per Epic Signal: Time spent writing notes per appointment. Numerator: Minutes providers spent in a notes activity or navigator section within the reporting period. Denominator: Appointments within the reporting period. Excludes data from days where the User Action Log data was not submitted. |
12 months prior to study through end of study period.
|
Secondary Outcome Measures
Outcome Measure |
Measure Description |
Time Frame |
|---|---|---|
|
Progress Note Length (characters)
Time Frame: 12 months prior to study through end of study period.
|
Per Epic Signal: Average number of characters in a Progress Note. Numerator: Characters, excluding white spaces, carriage returns, and tabs, in all Progress Notes providers wrote within the reporting period (IDN 34013). Denominator: Progress Notes providers wrote within the reporting period (IDN 34013). |
12 months prior to study through end of study period.
|
|
Note Contribution
Time Frame: 12 months prior to study through end of study period.
|
Per Epic Signal: Percentage of text written by the given source. Numerator: Characters written by the given source in all notes except letters and patient instructions. While you were affected by reportable issue 6823364, this excluded white space characters such as spaces, carriage returns, and tabs. Denominator: Characters in all notes, except letters and patient instructions, signed within the reporting period. While you were affected by reportable issue 6823364, this excluded white space characters such as spaces, carriage returns, and tabs |
12 months prior to study through end of study period.
|
|
Time to Appointment Closure
Time Frame: 12 months prior to study through end of study period.
|
Time from appointment date to appointment closure in the electronic health record.
|
12 months prior to study through end of study period.
|
|
Proportion of notes completed using DAX copilot generative AI software
Time Frame: 12 months prior to study through end of study period.
|
Numerator: Notes with at least 1 smartlink from DAX copilot used in the final signed note.
Denominator: All progress notes signed for ambulatory patients for that provider during the study period.
|
12 months prior to study through end of study period.
|
|
Patient Volume
Time Frame: 12 months prior to study through end of study period.
|
Average patient volume per week
|
12 months prior to study through end of study period.
|
|
Electronic Health Record satisfaction
Time Frame: Pre-test within 2 months prior to study period and post-test within 2 months after study end.
|
Pre- and post-project user responses on a modified KLAS EHR Efficiency and Satisfaction survey, a validated benchmarking tool.
|
Pre-test within 2 months prior to study period and post-test within 2 months after study end.
|
|
Patient Experience
Time Frame: 12 months prior to study through end of study period.
|
Patient experience overall score and sub-scores routinely obtained via NRC for patients seen by the provider.
|
12 months prior to study through end of study period.
|
Collaborators and Investigators
Sponsor
Investigators
- Principal Investigator: Stella Shin, MD, Children's Healthcare of Atlanta
- Study Director: Evan Orenstein, MD, Children's Healthcare of Atlanta
Study record dates
Study Major Dates
Study Start (Actual)
Primary Completion (Actual)
Study Completion (Actual)
Study Registration Dates
First Submitted
First Submitted That Met QC Criteria
First Posted (Actual)
Study Record Updates
Last Update Posted (Actual)
Last Update Submitted That Met QC Criteria
Last Verified
More Information
Terms related to this study
Keywords
Additional Relevant MeSH Terms
Other Study ID Numbers
- STUDY00001962
Plan for Individual participant data (IPD)
Plan to Share Individual Participant Data (IPD)?
IPD Plan Description
Drug and device information, study documents
Studies a U.S. FDA-regulated drug product
Studies a U.S. FDA-regulated device product
product manufactured in and exported from the U.S.
This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.