A Bayesian decision theoretic model of sequential experimentation with delayed response [by Stephen Chick]

When is it cost-effective to stop a sequential clinical trial?

Interest in sequential clinical trial design is growing, as health care funders and pharmaceutical companies strive to bring beneficial therapies to patients sooner and stop researching inferior ones earlier. But when is it `cost-effective’ to stop a sequential clinical trial? Might cost-effectiveness criteria help investigators decide whether or not to conduct a sequential trial, or one of a fixed sample size, at design stage?

In A Bayesian Decision Theoretic Model of Sequential Experimentation with Delayed Response (JRSSB, 2017, doi:10.1111/rssb.12225), we model the optimal stopping of a sequential clinical trial which accounts for:

  1. the cost of carrying out research, of treatment and of switching health technologies;
  2. benefits accruing to trial participants and the wider study population;
  3. delay in observing the primary end point.

We seek the policy which maximises the expected benefits of health technology adoption, minus the cost of carrying out the trial itself.

We do this for a two-armed clinical trial in which realisations on the primary outcome start to arrive while the investigator has the option to commit further patients to the trial. The incremental net monetary benefit of the new health technology versus standard X is assumed to have a normal distribution with unknown mean W and known variance σ² (we also model unknown sampling variance). We place a Gaussian prior on W and update its expected value sequentially as realisations arrive with delay.

The key stage of the trial is when realisations are arriving from subjects `in the pipeline’ at the same time as the investigator is deciding whether or not to randomise further subjects to the two arms. Our proposed trial design is therefore fully sequential, in contrast to designs which comprise a small number of stages. As each realisation arrives, Bayesian updating is used to form a predictive distribution which values the expected reward from stopping immediately, following up the remaining `pipeline subjects’ and choosing the best technology. This expected reward is compared with that from making another pairwise allocation at defined sampling cost, and acting optimally thereafter. Dynamic programming yields the optimal policy in (sample size x posterior mean) space, based on a diffusion process approximation, defining upper and lower stopping boundaries for the trial’s optimal stopping problem. The optimal trial design depends on the extent of the delay, and the resulting value function may be used to define three optimal decisions at trial design stage: `do not experiment’/`fixed sample size experiment’/`sequential experiment’.

Monte Carlo simulations show that, as expected, the policy outperforms alternative, non-sequential, trial designs in terms of the expected benefit of health technology adoption, net of trial costs. But they also show that the expected sample size of the optimal Bayes sequential policy can be greater than, equal to, or less than, that of comparator designs. Why? Because the optimal policy achieves the sample size which appropriately balances the expected benefit to patients with the cost of learning during the trial. The simulations also show that the policy performs well when judged according to the probability of correctly selecting the best technology. And they show how the policy and its performance change with changes in key parameters such as the rate of recruitment, the size of the population to benefit and the cost of conducting the trial.

With increasing policy interest in `value-based’ health care, we hope that our work will prompt both applied and more theoretically-inclined researchers to dig deeper. Please check out our paper, supplementary~material and code and contribute to our B’Log!

Author: sechick

I'm a professor of Technology and Operations Management at INSEAD based in Fontainebleau, France. A lot of my research is in math modeling in health care and Bayesian statistics for sequential learning and stochastic optimization. A lot of my teaching is in business model innovation and operations strategy in care delivery.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s