Inter-rater Reliability of Preoperative Mortality Risk Calculators

Inter-rater Reliability of Preoperative Mortality Risk Calculators Used for High-risk Noncardiac Surgical Patients

The goal of this study is to investigate the inter-rater variability of mortality risk calculations for high-risk non-cardiac surgical patients. For this purpose patient information from health care files were used. Five anesthesiologists calculated the mortality risks with three calculators for 34 high-risk non-cardiac surgical patients. The patients were discussed in a preoperative multidisciplinary discussion in a large teaching hospital in The Netherlands.

Study Overview

• Status: Completed
• Conditions: Reliability
• Intervention / Treatment: Other: no intervention

Detailed Description

Identifying high-risk patients for perioperative treatment and decision-making remains a challenge due to difficulty in reliably estimating morbidity and mortality risks. Assessing surgical risk helps allocating resources, obtaining informed consent, and making shared decisions with a multidisciplinary team (MDT). Multimorbidity is increasing globally in an ageing population with a growing burden of chronic diseases. It has been shown that high-risk non-cardiac surgical patients suffer disproportionally from perioperative complications. Preoperative mortality risk calculators are available and may help to identify high-risk non-cardiac surgical patients already before surgery, triggering efforts to lower the burden of possible complications, e.g., by extended monitoring or specified treatments. A systematic review suggested a significant risk of bias in developing current preoperative risk calculators due to lack of external validation, highlighting the need for enhanced performance and reliability to ensure their effectiveness in clinical practice. Low reliability and performance may, despite their general availability, be why that preoperative risk calculators are not yet consistently used in clinical practice. For daily clinical use, good predictive performance, low inter rater variability and user friendliness are essential. Discrepancies in predictor measurements can cause miscalibration, changes in discriminatory ability, and overall accuracy, leading to clinically relevant variability in risk calculator results. Previous studies have shown that physicians must trust a mortality risk calculator before utilization. High-risk patients suffer especially from complications, and it has been shown that the complications often result in death perioperatively. Therefore, adequate preoperative calculation of mortality risks and early recognition of high-risk non-cardiac surgical patients could benefit from reliable preoperative risk calculation. These high-risk patients can then be discussed in a preoperative multidisciplinary discussion to lower complications and perioperative deaths.

The current study evaluated the inter-rater reliability of calculating preoperative mortality risk scores for high-risk non-cardiac surgical patients in clinical practice among five anesthesiologists. The hypothesis was that the available risk calculators would show moderate to good reliability. For this purpose, five anesthesiologists used the following three risk calculators: the preoperative score to predict postoperative morbidity [POSPOM, the American College of Surgeons surgical risk calculator, and the surgical outcome risk tool [SORT].

• Study Type: Observational
• Enrollment (Actual): 34

Contacts and Locations

Study Locations

• Netherlands
  • Gelderland
    • Arnhem, Gelderland, Netherlands, 6800 TA
      • Rijnstate Hospital

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: N/A

• Sampling Method: Non-Probability Sample

Study Population

HIgh-risk non-cardiac surgical patients discussed in an MDT meeting.

Description

Inclusion Criteria:

• Patients discussed in a preoperative multidisciplinary team (MDT) meeting.

Exclusion Criteria:

• No complete patient file to calculate the mortality risks

Study Plan

How is the study designed?

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
inter-rater reliability of three risk calculators: POSPOM, SRC and SORT	Five anesthesiologists calculated mortality risks retrospectively for 34 patients. The intra-class correlation coefficient (ICC) was calculated per risk calculator	After calculation of the mortality risks (1 month)

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
mortality risk calculations or 34 patients	the POSPOM, SRC and SORT mortality risk calculators were used for (retrospective) 30-day mortality risk calculation of 34 patients	1 month

Collaborators and Investigators

• Sponsor: Rijnstate Hospital
• Collaborators: Utrecht University; Amsterdam University Medical Centers (UMC), Location Academic Medical Center (AMC)
• Investigators: Principal Investigator: Jacqueline E. Vernooij, MD, MA, Rijnstate Hospital

Study record dates

Study Major Dates

• Study Start (Actual): February 1, 2019
• Primary Completion (Actual): August 1, 2021
• Study Completion (Actual): August 1, 2021

Study Registration Dates

• First Submitted: April 25, 2024
• First Submitted That Met QC Criteria: May 8, 2024
• First Posted (Actual): May 13, 2024

Study Record Updates

• Last Update Posted (Actual): May 13, 2024
• Last Update Submitted That Met QC Criteria: May 8, 2024
• Last Verified: May 1, 2024

More Information

Terms related to this study

• Keywords: preoperative; high-risk patient
• Other Study ID Numbers: 2022-2161

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: UNDECIDED

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No