Reference effect measures for quantifying, comparing and visualizing variation from random and fixed effects in non-normal multilevel models, with applications to site variation in medical procedure use and outcomes

Abstract

Background: Multilevel models for non-normal outcomes are widely used in medical and health sciences research. While methods for interpreting fixed effects are well-developed, methods to quantify and interpret random cluster variation and compare it with other sources of variation are less established. Random cluster variation, sometimes referred to as general contextual effects (GCE), may be the main focus of a study; therefore, easily interpretable methods are needed to quantify GCE. We propose a Reference Effect Measure (REM) approach to 1) quantify GCE and compare it to individual subject and cluster covariate effects, and 2) quantify relative magnitudes of GCE and variation from sets of measured factors.

Methods: To illustrate REM, we consider a two-level mixed logistic model with patients clustered within hospitals and a random intercept for hospitals. We compare patients at hospitals at given percentiles of the estimated random effect distribution to patients at a median or 'reference' hospital. These estimates are then compared numerically and graphically to individual fixed effects to quantify GCE in the context of effects of other measured variables (aim 1). We then extend this approach by comparing variation from the random effect distribution to variation from sets of fixed effects to understand their magnitudes relative to overall outcome variation (aim 2).

Results: Using an example of initiation of rhythm control treatment in atrial fibrillation (AF) patients within the Veterans Affairs (VA), we use REM to demonstrate that random variation across hospitals (GCE) in initiation of treatment is substantially greater than that due to most individual patient factors, and explains at least as much variation in treatment initiation as do all patient factors combined. These results are contrasted with a relatively small GCE compared with patient factors in 1 year mortality following hospitalization for AF patients.

Conclusions: REM provides a means of quantifying random effect variation (GCE) with multilevel data and can be used to explore drivers of outcome variation. This method is easily interpretable and can be presented visually. REM offers a simple, interpretable approach for evaluating questions of growing importance in the study of health care systems.

Keywords: Facility variation; Generalized linear mixed model; Hierarchical model; Hospital variation; Interval odds ratio; Median odds ratio; Random effect.

Conflict of interest statement

Ethics approved and consent to participate

Study was reviewed by the Colorado Multiple Institutional Review Board, ID #12–0347. The Review Board provided a waiver of consent and approval for study since all data was previously collected for clinical purposes as part of routine healthcare and patients were given unique anonymous identifiers to reduce risk of breach of confidentiality.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interest.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Fig. 1

Example 1 Forest Plot. Forest plot showing odds ratios and 95% CIs for individual patient and site fixed effects, and REM ranges for unexplained hospital variation in use of rhythm control treatment for AF patients. Levels of shading represent 97.5, 90, 80, 70, 60, 50 percentiles (and corresponding lower percentiles)

Fig. 2

Example 1 REM Plot. 95% REM ranges and REM(0.75) for all patient risks, site characteristics and unmeasured site variation. Also indicated are individual risk effects and 95% REM ranges for age and time trend. Levels of shading represent 97.5, 90, 80, 70, 60, 50 percentiles (and corresponding lower percentiles)