iqLearn: Interactive Q-Learning in R

Kristin A Linn, Eric B Laber, Leonard A Stefanski, Kristin A Linn, Eric B Laber, Leonard A Stefanski

Abstract

Chronic illness treatment strategies must adapt to the evolving health status of the patient receiving treatment. Data-driven dynamic treatment regimes can offer guidance for clinicians and intervention scientists on how to treat patients over time in order to bring about the most favorable clinical outcome on average. Methods for estimating optimal dynamic treatment regimes, such as Q-learning, typically require modeling nonsmooth, nonmonotone transformations of data. Thus, building well-fitting models can be challenging and in some cases may result in a poor estimate of the optimal treatment regime. Interactive Q-learning (IQ-learning) is an alternative to Q-learning that only requires modeling smooth, monotone transformations of the data. The R package iqLearn provides functions for implementing both the IQ-learning and Q-learning algorithms. We demonstrate how to estimate a two-stage optimal treatment policy with iqLearn using a generated data set bmiData which mimics a two-stage randomized body mass index reduction trial with binary treatments at each stage.

Keywords: Q-learning; SMART design; dynamic programming; dynamic treatment regimes; interactive Q-learning.

Figures

Figure 1
Figure 1
SMART design toy example with two randomized stages and two treatment options at each stage. Patients progress from left to right and are randomized to one of two treatment options just prior to Stages 1 and 2. Randomizations are represented by gold circles; treatments are displayed in blue boxes.
Figure 2
Figure 2
Q-learning algorithm.
Figure 3
Figure 3
IQ-learning algorithm.
Figure 4
Figure 4
Residual diagnostic plots from the second-stage regression in IQ-learning.
Figure 5
Figure 5
Residual diagnostic plots from the regression model for the main effect term.
Figure 6
Figure 6
Residual diagnostic plots from the linear regression model for the contrast function mean.
Figure 7
Figure 7
Residual diagnostic plots from the log-linear variance model.
Figure 8
Figure 8
Normal QQ-plot of the standardized residuals obtained from the contrast mean and variance modeling steps.

Source: PubMed

Sponsors et collaborateurs

Les conditions médicales

Interventions en matière de drogue

3
S'abonner