Do Script Concordance Tests Correlate With Family Medicine Standardized Tests and Failing Rotation Grades?

Delayed or poor identification and remediation of clinical reasoning difficulties can lead to clinician underperformance and can ultimately compromise patient care. To date, no research has been done to see if SCTs correlate with current grading standards (ITE, ABFM certification exam score, ACGME milestones, or failing clinical rotation grades) in Family Medicine Residents. If investigators can identify that an SCT correlates with current standardized testing of Family medicine residents, it could be possible to identify struggling learners prior to poor scores on the ITE, ACGME milestones, or clinical rotations. If a learner does poorly on an SCT early in the academic year the learner can begin a remediation plan to improve their deficits before receiving a failing grade, poor ITE or ACGME milestone scores.

Study Overview

• Status: Active, not recruiting
• Conditions: Knowledge, Attitudes, Practice

Detailed Description

This study is a longitudinal observation study. A panel of experts (numbering 15-20) consisting of Family Medicine attendings who have been out of residency greater than 3 years will complete the SCT. They will be provided an Informational Consent. Their willingness to be a panel member is their implied consent to be part of the research study.

The basic idea behind SCT is to compare residents' performance with a group of persons who are representatives of the profession (or the specialty) to which they wish to belong. Therefore, panels are made up of physicians with good overall clinical experience in the field rather than experts from narrow parts of the field. Panel composition also depends on the assessment goal.

This study will determine if a Script Concordance Test (SCT) correlates with Family Medicine In Training Exam (ITE) scores, American Board of Family Medicine (ABFM) certification exam scores, Accreditation Council for Graduate Medical Education (ACGME) medical knowledge and patient care milestones, and failing family medicine inpatient clinical rotation grades.

The SCT will be given to PGY1, 2, and 3 residents each year during the Residents three year residency, in June or July.

• Study Type: Observational
• Enrollment (Actual): 79

Contacts and Locations

Study Locations

• United States
  • Nevada
    • Las Vegas, Nevada, United States, 89191
      • Mike O'Callaghan Military Medical Center

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No

• Sampling Method: Non-Probability Sample

Study Population

Family medicine residents at 99MDG

Description

**Patients must be able to get care at Nellis Air Force Base (a military installation) in order to participate in this study**

Inclusion Criteria:

--Family Medicine resident at 99MDG

Exclusion Criteria:

-- N/A

Study Plan

How is the study designed?

Design Details

• Observational Models: Cohort
• Time Perspectives: Prospective

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
SCT scores compared to ITE scores	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 1
SCT scores compared to ITE scores	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 2
SCT scores compared to ITE scores	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 3
SCT scores compared to ABFM scores	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 1
SCT scores compared to ABFM scores	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 2
SCT scores compared to ABFM scores	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 3
SCT scores compared to family medicine rotation grades	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 1
SCT scores compared to family medicine rotation grades	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 2
SCT scores compared to family medicine rotation grades	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 3
SCT scores compared to ACGME milestones.	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 1
SCT scores compared to ACGME milestones.	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 2
SCT scores compared to ACGME milestones.	SCT scores, ranging from 0 to 100 percent, are longitudinal and will be recorded for all PGY1, 2, and 3 subjects enrolled in the study each year of their residency in June or July.	year 3
Binomial SCT classification derived from each interval SCT scores compared to ACGME milestones.	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 1
Binomial SCT classification derived from each interval SCT scores compared to ACGME milestones.	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 2
Binomial SCT classification derived from each interval SCT scores compared to ACGME milestones.	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 3
Binomial SCT classification derived from each interval SCT scores compared to family medicine rotation grades.	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 1
Binomial SCT classification derived from each interval SCT scores compared to family medicine rotation grades.	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 2
Binomial SCT classification derived from each interval SCT scores compared to family medicine rotation grades.	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 3
Binomial SCT classification derived from each interval SCT scores compared to ITE	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 1
Binomial SCT classification derived from each interval SCT scores compared to ITE	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 2
Binomial SCT classification derived from each interval SCT scores compared to ITE	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 3
Binomial SCT classification derived from each interval SCT scores compared to ABFM	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 1
Binomial SCT classification derived from each interval SCT scores compared to ABFM	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 2
Binomial SCT classification derived from each interval SCT scores compared to ABFM	A "fail" classification is defined as an SCT score 1 SD below the respective PGY mean. All other SCT scores will be classified as "pass".	year 3

Collaborators and Investigators

• Sponsor: Pamela Hughes
• Investigators: Study Director: Paul F Crawford, MD, US Air Force

Publications and helpful links

General Publications

• Gagnon R, Charlin B, Coletti M, Sauve E, van der Vleuten C. Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test? Med Educ. 2005 Mar;39(3):284-91. doi: 10.1111/j.1365-2929.2005.02092.x.
• Fournier JP, Demeester A, Charlin B. Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak. 2008 May 6;8:18. doi: 10.1186/1472-6947-8-18.
• O'Neill TR, Li Z, Peabody MR, Lybarger M, Royal K, Puffer JC. The Predictive Validity of the ABFM's In-Training Examination. Fam Med. 2015 May;47(5):349-56.

Helpful Links

• about SCT

Study record dates

Study Major Dates

• Study Start (Actual): June 1, 2019
• Primary Completion (Estimated): June 1, 2026
• Study Completion (Estimated): June 1, 2026

Study Registration Dates

• First Submitted: May 1, 2019
• First Submitted That Met QC Criteria: May 2, 2019
• First Posted (Actual): May 6, 2019

Study Record Updates

• Last Update Posted (Actual): July 23, 2025
• Last Update Submitted That Met QC Criteria: July 22, 2025
• Last Verified: July 1, 2025

More Information

Terms related to this study

• Other Study ID Numbers: FWH20190010H

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: We do not plan on sharing data

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No