Incentives & Motivation for Behavior Change:

Incentives & Motivation for Behavior Change: Survey-embedded Experiments

In a series of controlled, randomized experiments, we will systematically manipulate exposure to health-related messages and/or survey methods to examine the effects on behavioral intention.

There are various strategies used to influence health-related decision making and the effects of health behavior have had mixed results. In particular, incentive-based interventions have often failed to increase healthy behavior. We will examine 1) the role of behavioral motivation to increase sleep or exercise and 2) current levels of sleep or exercise when predicting who is interested in a mock RCT invitation to increase each behavior using financial or social incentives.

In addition to the above focus on sleep and exercise, we will also examine another important health behavior: vaccination. Embedded within experiments studying effects of incentives on vaccination decisions, will conduct methodological tests. In particular, we will estimate the effects of using different methods of measuring the study outcome (vaccine intention).

Study Overview

• Status: Completed
• Conditions: Behavior; Intention
• Intervention / Treatment: Behavioral: mandate public; Behavioral: employer mandate; Behavioral: incentive reward; Behavioral: cash large; Behavioral: penalty; Behavioral: incentive lottery small; Behavioral: incentive lottery medium; Behavioral: incentive lottery large; Behavioral: cash small; Behavioral: cash medium; Behavioral: invitation for sleep financial incentive; Behavioral: invitation for sleep social incentive study; Behavioral: invitation for exercise financial incentive study; Behavioral: invitation for exercise social incentive study

Detailed Description

Incentives for Sleep and Exercise:

This experiment will estimate enrollment bias for randomized clinical trials offering to incentivize behavior change. In this experiment, we will test whether those who are most motivated to change behavior are also most likely to enroll in a (hypothetical) RCT when offering financial or social incentives for behavior change.

We hypothesize that those most likely to enroll are already motivated to change their behavior prior to enrollment, which could be bias trials towards the null. We will test this hypothesis by estimating if motivation to change a behavior predicts interest in joining RCT targeting that behavior. We will also test if baseline behavior predicts interest in joining these RCT.

We will conduct this experiment using mock invitations to learn about and potentially join a RCT. The study outcome will be responses to this invitation. We will not offer invitations to an actual trial, but the stimuli (mock invitations to a "ghost" trial) and task (response to the invitation) fundamentally resemble a trial's counterparts. The invitations will specify an opportunity to earn financial or social incentives for improving a healthy behavior.

We will invite participants to earn financial or non-financial incentives for increasing their sleep or exercise. In this study, the primary outcome will measure if they were "not interested," "slightly interested," or "very interested" in participating.

Prior to receiving their invitation, they will complete an online questionnaire measuring their motivation to increase each behavior, plus their recent behavior and socio-demographics.

Separate analyses will be conducted for financial and social incentives and for sleep vs exercise trial invitations. We will report point estimates and 95% confidence intervals (CI) for behavior and motivation.

The analyses will examine if their interest in joining the RCT is predicted by 1) their baseline behavior (i.e., amount of sleep or exercise), or 2) their motivation to change the specific behavior. As noted above, we hypothesize the their level of motivation to change a specific behavior will predict interest in a trial targeting that behavior.

Testing Relative Large and Small Vaccination Incentives:

Using a separate sample, this experiment will test whether policies offering large or small financial incentives are likely to strengthen COVID-19 vaccine intention. This experiment will randomize individuals to one of four study arms that include 1) a control condition, 2) an educational message, 3) a message about the relatively large financial incentive, or 4) a message about the relatively small financial incentive. The goal of this study is to estimate if either type of incentive policy is likely to have negative effects on vaccine intention, as some experts have warned.

When analyzing the effects of relatively large and small financial incentives on vaccination intentions, we will report point estimates and 95% CIs for the overall sample and demographic sub-groups. We will also report summary statistics for all the overall sample and sub-populations. We will test whether, compared to a control condition, either of the financial incentives increase, decrease, or have no effect on the percentage who want to vaccinate. In a fourth study arm, subjects will receive an educational message that will also be compared to the control condition.

Testing Vaccine Incentives Plus Different Measures of Vaccine Intention:

In a related experiment, we will separately test the effects of 10 experimental conditions, with a counter-balanced experimental manipulation using an FDA approval message, plus a control condition. The goal of this study is to compare the effects of a wider variety of vaccine interventions that experts have proposed, including incentives and mandates.

In addition, we will also randomize individuals to questionnaires using different methods of measuring vaccine intention, the study outcome.

Comparing different methods of estimating vaccine intention: Embedded within the experiment testing different proposed vaccine policies, we will test if methodological differences in the response option for the primary outcome effect the percent reporting "yes". To do this, we will test 2 (Yes and No) vs 3 (Yes, No and Unsure) level response options and randomly order both sets.

This methodological experiment will examine whether the proportion responding "yes" to the same question (about whether they want to vaccinate soon) varies depending on the order of response options and whether they include a maybe/unsure option. We will run cross tabs and chi-square tests for the 2 vs 3 response levels and the order. The instrumentation tests will be conducted for COVID-19 vaccination boosters, the initial shots, plus vaccination against influenza.

When testing the effects of potential vaccine policies, the control group, with no vaccine policy presented will be compared to: cash incentives for $1000, $200, or $100; a $1,000 tax credit; lotteries for $100,000, $200,000, or $1 million; $1,000 tax on the unvaccinated; and mandates by employers or airlines, bars, and restaurants. The main outcome is whether they would want to get vaccinated soon given the hypothetical vaccine policy.

(Those assigned to the employer mandate condition will be excluded from analyses if they report being unlikely to have an employer.)

OLS specification will be our main result and the other measures are provided as robustness checks.

The OLS model can exclude all the demographic controls and run the binary dependent variable on the treatment variables. (Note that this approach is legitimate because the treatments are being randomized across respondents.) The treatments include the financial policies (incentives and penalties of different amounts and types) and mandates (of different types) being noted in a message.

Type of model: We will perform pairwise t tests of percent of respondents answering "Yes" comparing those treated with an incentive to the control group. We will perform these pairwise tests on subsets by race, gender, income, education, and other socio-demographics.

Additionally, we will conduct these pairwise tests on by type of treatment. Comparing lottery to cash incentive, comparing positive incentive vs. penalty, comparing size of incentive, and comparing employer mandates against the control.

We will also conduct regression analyses on the pooled dataset where the left-hand side observations are individual responses where those answering "Yes" will be coded as 1 and those answering, "No" or "Unsure" will be coded zero. We will include a set of controls (race, gender, income, education, etc) as well as an indicator variable reflecting whether the respondent received a treatment. Regression models will include ordinary least squares, probit, nearest neighbor matching, and propensity score matching. We will also run these regressions where the treatment variable is split up into several indicator variables reflecting the type of treatment provided as well as an indicator for FDA approval.

We will estimate a model-alternatively using ordinary least squares and logistic regression-with a binary-outcome dependent variable (equal to one if the respondent wanted to be vaccinated, and otherwise equal to zero). For explanatory variables, we include dummy variables for each of the ten treatment arms.

Criteria for statistical significance: We will use .05 as our threshold for statistical significance.

Sample size calculation for the survey experiment comparing 10 different vaccine policies: We estimate that if the final sample sizes for each condition include at least 300 subjects, we can detect about 5% or larger difference. We plan to double the allocation for the control and $1000 conditions to allow for planned comparisons.

All experiments: Each subject will be randomized to a condition. Participants will be randomized to reduce the chance that observed effects are due to unmeasured factors. In addition, all study procedures were automated, which improves the control over how the experiment is conducted, allowing all procedures to be consistently standardized.

The studies will enroll national, theory-based samples recruited through MTurk and/or Prolific platforms. To reduce enrollment bias, recruitment and enrollment materials will describe the research in vague terms (e.g., "we are interested in learning your opinions and preferences related to health). Each experiment will also measure socio-demographic variables for descriptive purposes.

Recommended data cleaning procedures for each experiment: Attention checks can identify those who should be excluded from the main analyses. (Regardless of performance on the attention check, all participants will be compensated for their time.) Analyses will exclude those with duplicate IDs or a high fraud score, We will conduct analyses that include and exclude those who finished the fastest (fastest 5%).

Replication studies will include the same study design and procedures.

• Study Type: Interventional
• Enrollment (Actual): 4000
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Pennsylvania
    • Philadelphia, Pennsylvania, United States, 19104
      • Center for Mental Health. Perelman School of Medicine

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Adults (18 years or older)
• residing in the US
• unvaccinated for COVID-19

Exclusion Criteria:

• children
• those living outside the US
• vaccination

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Other
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

14

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: specific vaccine policy 1; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: mandate public; mandate by airlines, bars, restaurants
Experimental: specific vaccine policy 2; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: employer mandate; your employer requires vaccination
Experimental: specific vaccine policy 3; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: incentive reward; $1,000 tax credit
Experimental: Specific vaccine policy 4; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: cash large; large quarenteed cash amount
Experimental: specific vaccine policy 5; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: penalty; $1000 tax penalty
Experimental: specific vaccine policy 6; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: incentive lottery small; 100,000 smaller monetary amount
Experimental: specific vaccine policy 7; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: incentive lottery medium; 200,000 mid monetary amount
Experimental: specific vaccine policy 8; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: incentive lottery large; 1 million dollar lottery
Experimental: specific vaccine policy 9; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: cash small; smaller guaranteed cash amount
Experimental: specific vaccine policy 10; Those in this arm will learn about a specific financial incentive or mandate policy.	Behavioral: cash medium; mid sized quarenteed cash amount
Experimental: sleep financial incentive; Those in this arm will invite adults to join an RCT that uses financial incentives to reward those who increase their sleep.	Behavioral: invitation for sleep financial incentive; Financial incentives earned when increasing sleep
Experimental: sleep social incentive; Those in this arm will invite adults to join an RCT that uses social (gamification) incentives to reward those who increase their sleep.	Behavioral: invitation for sleep social incentive study; social incentives earned when increasing sleep
Experimental: exercise financial incentive; Those in this arm will invite adults to join an RCT that uses financial incentives to reward those who increase their exercise.	Behavioral: invitation for exercise financial incentive study; financial incentives earned when increasing exercise behaviors
Experimental: exercise social incentive; Those in this arm will invite adults to join an RCT that uses social incentives to reward those who increase their exercise.	Behavioral: invitation for exercise social incentive study; social incentives earned when increasing exercise behaviors

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
measure of whether they are interested in RCT enrollment (for sleep or exercise)	subjects will select a response option to indicate if they are "not interested," "slightly interested," or "very interested"	through the duration of the experiment: less than 1 day
measure of decision (vaccine intention)	subjects select a response option from a 2 ("yes" or "no") or 3 (yes, no, unsure) level response set	through the duration of the experiment: less than 1 day

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
likelihood of behavior	measures their perceived likelihood of performing the behavior	through the duration of the experiment: less than 1 day

Collaborators and Investigators

• Sponsor: University of Pennsylvania
• Investigators: Principal Investigator: Jessica Fishman, PhD, University of Pennsylvania

Study record dates

Study Major Dates

• Study Start (Actual): February 7, 2021
• Primary Completion (Actual): December 30, 2023
• Study Completion (Actual): December 30, 2023

Study Registration Dates

• First Submitted: February 2, 2021
• First Submitted That Met QC Criteria: February 5, 2021
• First Posted (Actual): February 10, 2021

Study Record Updates

• Last Update Posted (Estimated): February 15, 2024
• Last Update Submitted That Met QC Criteria: February 13, 2024
• Last Verified: February 1, 2024

More Information

Terms related to this study

• Keywords: sleep; decision; RCT enrollment; incentives; mandates; message effects; intention; methods; instrumentation effects; vaccine hesitancy; exercise physical activity behaviors
• Other Study ID Numbers: 834772

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: no plans to share IPD at this time

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No