Can a Tailored Quality Assurance Program Help General Practitioners Maintain POCUS Scanning Competence? A Cohort Study From Denmark

Testing of a quality assurance program for general practitioners with an online multiple-choice quiz including still-pictures and video sequences of 10 application specific POCUS examinations. Through 15-20 questions, the quiz tests participants (1) knowledge about indications for performing POCUS examinations, (2) applied knowledge of ultrasound equipment, (3) ability to optimize images, (4) ability to recognize and present structures, (5) ability to interpret images, (6) ability to describe and document findings and (7) medical decision-making. Following the quiz, the participants are offered guidance to improve their performance.

Study Overview

• Status: Completed
• Conditions: Point-of-care Ultrasonography (POCUS)
• Intervention / Treatment: Other: quality assurance program

Detailed Description

Introduction Point-of-care ultrasound (POCUS) examinations are increasingly used in general practice . General practitioners (GPs) can learn to perform ultrasound examinations by attending ultrasound courses, but after course attendance they are left to maintain and develop scanning competence by themselves . Maintaining scanning competence in general practice is challenged by (1) a low frequency of performed examinations as not all patients seen in general practice present conditions suited for POCUS , (2) a low pre-test probability of disease in general practice which challenges the ability to recognise different pathologies, (3) the lack of supervision as GPs often work alone in small offices , and (4) the short consultations leaving little room for practice.

An ultrasound course tailored for Danish office-based GPs has been developed offering continuous practice under supervision during the three months that the GPs are enrolled in the course. Previous research have shown that this ultrasound course leads to scanning competence within the ten scanning modalities that are included in the course curriculum. The course is designed to make the participants move towards self-directed learning , but we do not know if scanning competence obtained on the course can be maintained over time.

Within some medical specialities, the number of specific POCUS examination performed have been used as an indicator for scanning competence in re-certification . However, little is known about the number of performed POCUS examinations needed to secure skills. In general practice, continuous medical education (CME) programs have been found to increase GPs knowledge and skills in other areas , but as POCUS is just a smaller part of the clinical work in general practice, CME programs for GPs cannot be too time-consuming.

Specific objectives The overarching aim of this study is to investigate whether a quality assurance program, tailored for GPs who use POCUS as part of their patient examination in general practice, can lead to maintained scanning competence 12 months after baseline.

Trial design This is a cohort study designed as a follow-up for a hybrid effectiveness-implementation study (ClinicalTrials.gov Identifier: NCT05274581). In the previous study, 18 GPs attended an ultrasound course targeted to office-based GPs. The ultrasound course consisted of three teaching seminars over three months (Marts - June 2022), a curriculum of 10 POCUS applications, an online learning platform providing educational support before, during and after the teaching sessions (see Figure 1). At the end of the ultrasound course (June 2022) and again 3 months after the ultrasound course (September 2022), all participants had their scanning competences within the 10 POCUS applications assessed by external experts. These experts were blinded to the participants previous experience and learning process. The assessment of scanning competence was done using the OSAUS assessment tool . Following this study participants have implemented POCUS in their clinical practice, where they use POCUS as part of the patient examination.

Study settings This study will be conducted in office-based general practice in Denmark. GPs in Denmark are self-employed and work in office-based general practice clinics. Denmark has a public health care system where almost all patients are listed with a GP for primary health care. Consultations and treatments are free-of-charge for patients. GPs act as gatekeepers for other primary care healthcare providers and secondary care specialists. GPs are paid through a combination of remuneration and fee-for-service financed through taxes8. There is no fee for performing POCUS in primary care and GPs must cover expenses for the ultrasound device and their ultrasound education themselves. However, during this study participating GPs will have a fee for performing POCUS examinations. POCUS scanning-fees will be registered in the medical record system in the practice.

The study will be coordinated from CAM AAU and data will be collected by the research team and by the participating general practitioners in their clinics. All study data will be stored at a secure server at Aalborg University.

The quality assurance program will be developed and delivered by research team, who are experienced GPs, POCUS users and/or teachers. The competence assessment (OSAUS score) will be delivered by a team of POCUS experts (SKA, OG, MBJ, CS and LP), who have extensive POCUS experience.

Outcome measures Background characteristics of participating GPs At baseline the following participant characteristics will be collected: Age (years), gender (M, F, other), previous use of ultrasound (number of months with regular use), previous ultrasound courses of minimum 1 day duration (yes/no), scanner type (low range, mid range, high end), type of practice (collaboration, partnership, solo), location of practice (urban, rural, mixed), number of patients assigned to the practice, and number of GPs working in the practice.

In addition, the results from the OSAUS assessment of scanning competences conducted on all participants in September or October 2022 will be included.

The OSAUS assessment tool has been developed and validated as a generic tool for assessing scanning competence14. The OSAUS scale consists of seven items: 'indication for the examination', 'applied knowledge of ultrasound equipment', 'image optimization', 'systematic examination', 'interpretation of images', 'documentation of the examination' and 'medical decision-making' and each item is rated using a five-point Likert-scale with descriptions of performance ranging from very poor (score = 1) to excellent (score = 5). All items are weighted equally, as high inter-item correlation have been found previously . Hence, for each scanning modality a total score from 7 to 35 points may be achieved.

The OSAUS ultrasound competence assessment tool have been developed as a generic tool to assess ultrasound competence across medical specialities. The assessment tool was developed through a Delphi study14 and it has been used to assess clinicians' ability to transfer learning from an ultrasonography course into diagnostic performance on patients. Furthermore, the tool has been found valid and reliable to distinguish between novice POCUS users and experts .

For the primary outcome(P1) and secondary outcomes (S1-S2), the Objective Structured Assessment of Ultrasound Skills (OSAUS) score after 12 months is used. For the primary outcome (P2), the OSAUS score at baseline and at 12 months is used.

Primary outcomes:

(P1): The summarized OSAUS score after 12 months for all ten modalities will be calculated as percentage of maximum score. To summarize normal and non-normal distributed data similarly mean(SD) as well as median(IQR) will be presented together with minimum and maximum scores.

Primary outcome 2 (P2):

The summarized OSAUS score at baseline and at 12 months for each participating GP will be compared and the proportion of GPs, with an OSAUS score at 12 months larger than or equal to the ultrasound competence score at baseline, will be reported.

Secondary outcomes (S1): For GPs, participating in a POCUS quality assurance program, the OSAUS scores and the item scores after 12 months, for each of the ten scanning modalities included in the curriculum will be calculated and presented as median, IQR, minimum and maximum score.

(S2): The proportion of GPs, who achieve a minimum OSAUS score of three for all seven domains for each of the ten scanning modalities after 12 months will be calculated. As variation in scores can occur between the four expert assessors. The scores will be normalized by multiplying a given accessors score with (0.2 x sum of scores of all five accessors/the given accessors mean score) prior to calculating the proportion of successful GPs.

For the secondary outcome S3, the GPs assessment of their own scanning competence will be used. The GPs will declare to which degree they have POCUS scanning competence within a specific scanning modality to perform the scan un-supervised in general practice (To a very high degree, to a high degree, to some degree, to a lesser degree, not at all, unsure).

(S3): The proportion of GPs, who rate themselves as competent to perform POCUS un-supervised in general practice, after the educational period (three months after baseline), will be calculated for each of the ten scanning modalities.

For the secondary outcome S4, the fee-specific codes registered by the GP during consultation in general practice will be used. Following the use of POCUS in the general practice consultations the GPs will follow normal procedures for registration of activities in general practice using remuneration codes and fee-specific codes for using POCUS in the medical record system. Prior to this study, 10 fee-specific codes for the 10 different POCUS examinations in the curriculum will be installed in the medical record system. The Primary Sector Data Provider Platform (PLSP) will develop an algorithm that allows for the following data extraction on each participating GP in a given time frame: (1) number of POCUS examinations performed and (2) number of consultations. PLSP will deliver monthly aggregated data for each participating GP from baseline to 12 months after baseline.

(S4): The number of performed POCUS examinations by each GP each month during the study (months 1-12 after baseline) will be summarized to demonstrate variation over time.

For the secondary outcome S5 and S6, the results of distributed online quizzes to the participants will be used. During the study, four online quizzes will be send to participants by email (Months 3, 6, 9, and 12). A reminder will follow after two weeks to participants who fail to respond the quiz. The results of these quizzes will be summarized as the total number of completed quizzes for each participant (S5) and the test score for each participant at months 3, 6, 9, and 12 (S6).

For the secondary outcomes S7, S8 and S9, a linear regression model to test associations between OSAUS score after 12 months and the total number of performed POCUS examination for each scanning modality (S7), the number of completed quizzes in the quality assurance program (S8), and the tests scores in the quality assurance program (S9) will be used. Changes in OSAUS scores between baseline and 12 months (S10) are calculated using linear regression. Scores at baseline and after 12 months will also be compared by a paired t-test.

Sample size The potential study population is the 18 GPs who participated in the ultrasound course organized by PLO-e. A participation rate of 80% corresponding to 15 GP is expected.

With 15 participants the total number of possible completed 7-item OSAUS questionnaires will be 15 for each modality and 150 for the combined estimate after 12 months. This is considered sufficient to estimate mean estimates and provide confidence intervals with acceptable spread.

Recruitment All participants, who attended the PLO-e ultrasound course were invited to participate in the study. The ultrasound course was listed in PLO-e course catalogue for 2022 and as such offered to all GP working in primary care in Denmark.

Blinding Expert assessors performing the competence assessment (primary outcomes) have a medical background and are considered experts in the field. They have not been teaching participants in the training program. They will be blinded to the participants prior experience with POCUS, the number of performed POCUS examinations, and any other elements in the participants' learning process.

The principal investigator (SKA) and MBJ acts as expert assessors in this study and as such, they will be blinded to the background information on participants, the data collection and the analysis of data. CAA will be in charge of all participant correspondence, AR will be cleaning the data set and analyzing the data.

Data collection, management, and analysis

Data collection

Multiple Choice Questionnaire (MCQ) MCQ's will be distributed by email to participating GPs at 3, 6, 9, and 12 months. The email includes a link to an online questionnaire prepared and collected using Forms (Microsoft Office, Redmond, USA). A reminder will follow after two weeks to participants failing to complete the MCQ after the initial invitation. Data will be transferred and saved on a secure server at Aalborg University.

Assessment of scanning competence The assessment of participants scanning competence using the OSAUS score will be performed in the same manner using the same expert assessors at baseline and after 12 months.

During the assessment, only the participant and the assessor are present. Each assessment will follow the same structure, where experts will assess the participants by asking the following questions:

- In which clinical scenarios would you perform this POCUS examination (Item 1 in the OSAUS)

The experts will ask participants to demonstrate the POCUS examination (for maximum five minutes) to assess the following:

• Applied knowledge of the ultrasound equipment (Item 2 in the OSAUS)
• Image optimization (Item 3 in the OSAUS)
• Systematic examination (Item 4 in the OSAUS)
• Interpretation of images (Item 5 in the OSAUS)

The experts will present the participants with a picture of common pathology and ask the participants the following questions:

• How would you interpret these ultrasound findings? (Item 5 in the OSAUS)
• If you were to describe this examination in the medical record, what would you write? (Item 6 in the OSAUS)
• What would you do if you found it? (Item 7 in the OSAUS)

Following the assessment, the participant will receive feedback on the level of performance and educational advice. The total duration of the assessment is expected to last 90 minutes for each participant. The assessor completes an OSAUS score for each of the ten examinations performed by the participant.

Education of POCUS expert assessors The POCUS expert assessors all have prior experience using the OSAUS score. Still, prior to the assessment of the participants' performance in this study, the POCUS experts participate in a 90-minutes online training session with CAA who has prior experience with the education of a team of assessors. During this training session, the POCUS experts are presented to the OSAUS score and the procedure for assessing the participants in this study. The POCUS experts are instructed to rate according to the standard expected from a GP, who is capable of performing POCUS unsupervised in general practice.

Self-rated scanning competence In relation to the expert assessment of scanning competence, participants were asked to fill out a questionnaire where they declare whether or not they have POCUS scanning competence within this scanning modality to perform the scan un-supervised in general practice. The expert assessors were blinded to this declaration.

Evaluation questionnaire After 12 months following the final MCQ-quiz and the final evaluation of scanning competence, we will circulate an evaluation questionnaire for participants. This questionnaire will be developed based on collected experiences during the study.

Data management The OSAUS assessment will be filled out by the expert assessors using paper versions of the registration sheet. The registration sheets and questionnaires will be collected by CAA and checked for completeness. CAA will bring the registration sheets to CAM AAU, where they will be safely stored in a locked cupboard. Two research assistants will independently impute the results in a Microsoft Excel fil along with the results from the MCQ-quiz. The two datasets will then be compared for inconsistencies and these will be resolved by involving a third party.

Hereafter, all digital study related data will be pseudo-anonymized and stored on secure server on Aalborg University and handled according to the General Data Protection Regulation.

Statistical methods for analysing primary and secondary outcomes. All statistical analyses will be performed using STATA version 17 (StataCorp, Texas, USA) and analyzed according to a predefined statistical analysis plan, which will be uploaded prior to the data collection for the primary outcome.

Sub-analyses will be performed to explore the importance of different educational elements in the intervention. These sub-analyses will be defined in the statistical analysis plan.

A predictive mean matching (pmm) imputation method will be used to fill in missing values by the command "mi impute pmm" in Stata , for all OSAUS scores after 3 and 6 months. Predictive mean matching (PMM) is a partially parametric method that matches the missing value to the observed value with the closest predicted mean .

Data monitoring

Harms

The participating GPs will be obliged to report any Suspected Unexpected Serious Adverse Reactions (SUSARs), Serious Adverse Events (SAEs) and Adverse Events (AEs) related or possibly related to the use of POCUS to the adverse event committee.

Ethics and dissemination The study will be performed in accordance with the Declaration of Helsinki. The project was notified to The North Denmark Region Committee on Health Research Ethics (Den Videnskabsetiske Komité for Region Nordjylland, reference number 2022-000764) who responded that according to Danish Law (komitélovens § 14, stk. 2), no ethical approval is needed for this project.

Prior to the collection of register data, we will seek permission from the Danish Regions The project has been registered and conducted according to the regulations of the Danish Data Protection Agency (registration number ID-242-2).

Access to data During the study, only CAA and a research assistant will have access to the data. After the pseudo-anonymization, the following authors will have access to collected data stored at the secure server at Aalborg University (SKA, MBJ and CAA).

Dissemination policy Positive, negative or inconclusive results of the study will be published in a peer-reviewed journal. The project group will also present results at conferences.

• Study Type: Observational
• Enrollment (Actual): 16

Contacts and Locations

Study Locations

• Denmark
  • Aalborg, Denmark, 9220
    • Center for General Practice at Aalborg University

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes

• Sampling Method: Non-Probability Sample

Study Population

Eighteen GPs working in office-based general practice in Denmark, who have attended the PLO-e ultrasound course, will be invited to participate in the study. To participate a GPs must fulfill the inclusion criteria and not the exclusion criteria

Description

Inclusion criteria:

1. GP, i.e. postgraduate medical doctor with a specialization in general practice.
2. Work in office-based general practice in Denmark
3. Access to an ultrasound device in the practice during the study period

Exclusion criteria:

1. GPs with a possible conflict of interest
2. No signed informed consent to participate.

Study Plan

How is the study designed?

Design Details

• Observational Models: Cohort
• Time Perspectives: Prospective

Number of groups / cohorts

1

Cohorts and Interventions

Group / Cohort

Intervention / Treatment

Course participants; Participants will be enrolled in a quality assurance program with an online multiple-choice quiz including still-pictures and video sequences of 10 application specific POCUS examinations. Through 15-20 questions, the quiz tests participants (1) knowledge about indications for performing POCUS examinations, (2) applied knowledge of ultrasound equipment, (3) ability to optimize images, (4) ability to recognize and present structures, (5) ability to interpret images, (6) ability to describe and document findings and (7) medical decision-making. Following the quiz, the participants are offered guidance to improve their performance.

Other: quality assurance program; The program consists of an online quiz using a multiple choice questionnaire (MCQ) design. The participants are expected to complete the online MCQ at 3, 6, 9, and 12 months following baseline. The MCQ includes still-pictures and video sequences of the 10 application specific POCUS examinations from the course curriculum. Through 15-20 questions, the quiz tests the OSAUS dimentions Each question has four possible answers of which up to three answers are correct. Each MCQ will have ten questions pertaining to pathologies within the ten applications of the curriculum. Further, there will be five to ten questions relating to image recognition, image optimisation, ultrasound artifacts. Once each MCQ is finalized, the participating GP's will receive feed-back on the answers with suggestions for improvement and guidance toward continuing self-directed learning.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
The total ultrasound competence score at 12 months	Primary outcomes (p1): For GPs, participating in a POCUS quality assurance program, the ultrasound competence score (summarized OSAUS score across ten scanning modalities)	12 months after baseline.
Change in ultrasound competence score after 12 months	Primary outcome 2 (P2): The proportion of GPs, participating in a POCUS quality assurance program, with an ultrasound competence score (summarized OSAUS score across ten scanning modalities) at 12 months larger than or equal to the ultrasound competence score at baseline.	Baseline and 12 months after baseline

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
The OSAUS scores and the item scores at 12 months	Secondary outcome 1 (S1); For GPs, participating in a POCUS quality assurance program, the OSAUS scores and the item scores, 12 months after baseline, for each of the ten scanning modalities included in the curriculum.	12 months after baseline
An OSAUS scores of 3 or more at 12 months	Secondary outcome 2 (S2); The proportion of GPs, participating in a POCUS quality assurance program, who have an ultrasound competence score (OSAUS score) of three or above, 12 months after baseline, for all of the seven OSAUS items of each of the ten scanning modalities included in the curriculum.	12 months after baseline
Self-rated ultrasound competence at 12 months	Secondary outcome 3 (S3): The proportion of GPs, participating in a POCUS quality assurance program, who rate themselves to be competent to perform un-supervised POCUS in general practice, 12 months after baseline, for each of the ten scanning modalities included in the curriculum.	12 months after baseline
The number of performed POCUS examinations	Secondary outcome 4 (S4): The number of performed POCUS examinations by each GP each month during the study	months 1-12 after baseline
Participation in the POCUS quality assurance program	Secondary outcome 5 (S5): The proportion of GPs, participating in a POCUS quality assurance program, who complete the online quiz at months 3, 6, 9 and 12 after baseline.	months 3, 6, 9 and 12 after baseline.
test scores in the POCUS quality assurance program	Secondary outcome 6 (S6): For GPs, participating in a POCUS quality assurance program, the test scores in the online quiz at months 3, 6, 9 and 12 after baseline.	months 3, 6, 9 and 12 after baseline.
Association between OSAUS score and number of performed POCUS examinations	Secondary outcome 7 (S7): For GPs, participating in a POCUS quality assurance program, the association between OSAUS score at 12 months and total number of performed POCUS examinations for each of the ten scanning modalities included in the curriculum.	12 months after baseline
Association between OSAUS score and number of completes quizzes	Secondary outcome 8 (S8): For GPs, participating in a POCUS quality assurance program, the association between OSAUS score at 12 months and the number of completes quizzes in the quality assurance program.	12 months after baseline
Association between OSAUS score and test score in the quizzes	Secondary outcome 9 (S9): For GPs, participating in a POCUS quality assurance program, the association between OSAUS score at 12 months and the test score in the quizzes in the quality assurance program.	12 months after baseline
Association between OSAUS score at baseline and OSAUS scores at 12 months	Secondary outcome 10 (S10): For GPs, participating in a POCUS quality assurance program, the association between OSAUS score at 12 months and the OSAUS score at baseline.	12 months after baseline and baseline
narrative evaluation of the program after 12 months	Secondary outcome 11 (S11): For GPs, participating in a POCUS quality assurance program, their narrative evaluation of the program after 12 months measured in questionaanires	12 months after baseline

Collaborators and Investigators

• Sponsor: Aalborg University
• Investigators: Study Chair: Janus L Thomsen, Professor, Center for General Practice at Aalborg University

Study record dates

Study Major Dates

• Study Start (Actual): November 15, 2022
• Primary Completion (Actual): November 30, 2023
• Study Completion (Actual): January 25, 2024

Study Registration Dates

• First Submitted: November 15, 2022
• First Submitted That Met QC Criteria: November 15, 2022
• First Posted (Actual): November 23, 2022

Study Record Updates

• Last Update Posted (Estimated): February 28, 2024
• Last Update Submitted That Met QC Criteria: February 26, 2024
• Last Verified: February 1, 2024

More Information

Terms related to this study

• Keywords: ultrasonography; CME; POCUS; point-of-care ultrasound
• Other Study ID Numbers: ID 242-5

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: UNDECIDED
• IPD Plan Description: It is unsure if data can be anonymized sufficiently to be made publicly available.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No