DSMB Blinding: Yes or No?

1

I conducted an (unscientific) poll yesterday on Twitter regarding whether people felt that DSMB members in RCT should be blinded to the treatment assignment. I was surprised to see that the majority of voters said “Blinded” when most clinical trial savvy folks that I know are very strongly of the opinion that the DSMB should be privy to the treatment assignments. I have created a Twitter thread, which I will reference here, but would also like to open for discussion in this forum. Here is my Twitter thread from today:

(THREAD) follow-up on whether DSMB should be blinded to treatment assignments in RCT

I’m a little surprised at how many people voted that DSMB should be blinded. Final results at 390 votes were 63% for keeping DSMB blinded– i.e. just “A/B” in the report vs 37% that treatment arms should be labeled – i.e. “Active/Placebo”

The comments on the poll were mixed – but as the day went on, they included an increasing proportion of incredulous “Wait, why in the world would the DSMB be blinded at the interim analyses?”

I’m with those folks. I do NOT believe that the DSMB should be blinded to treatment assignment as they review results & deliberate whether the trial should continue. I was surprised to see so many votes for a blinded DSMB. Let’s dig into this.

I suspect that a lot of votes for blinded DSMB were derived from the belief that MOAR BLINDING is always better. Like @JasonConnorPhD said here:

Blinding of the recruiters to treatment assignment is important because we don’t want their knowledge of what the next assignment could be to influence who they try to enroll in the trial (i.e. “the next assignment is for the Intervention…this person looks pretty sick…they should get the Intervention”)

Blinding of the participants to treatment assignment is important because we do not want their knowledge of what they are getting to influence their behavior during the trial or any subjective, self-reported outcomes

Blinding of the assessors to treatment assignment is important for similar reasons: we do not want the assessors’ knowledge of what the participant is getting to influence their assessment of outcomes (i.e. probing one group harder to ask about AE’s; reading an imaging study differently)

But none of those apply to the DSMB. They are not involved in participant recruitment or outcome assessment. They cannot introduce bias to the data as it is collected. They are presented interim results and charged with the decision of whether the trial should continue.

So why should they be blinded, then? From that camp, @ErickRScott specifically comments that the DSMB could allow bias to creep into their decision-making if they know which treatment arm is which:

First counterpoint: keeping the DSMB blinded isn’t strictly protective against bias. It just allows a different bias to creep in; a blinded DSMB will (probably) take some guess about which arm is which.

I’d argue that labeling the arms is more likely to be protective against bias (since they know which is which – there is no speculation) than presenting them blinded.

Second counterpoint: trials may generally be stopped for three broad reasons - efficacy, safety, or futility. Perhaps Erick can explain, but I do not think the DSMB can make a properly informed decision about each of these without knowledge of the treatment arms.

EFFICACY: the early evidence is overwhelmingly in favor of the Experimental treatment arm

The bar for efficacy stopping is generally very high, for two reasons: 1) early results can be very noisy, and trials stopped early for efficacy tend to produce overly optimistic estimates of treatment benefit;

  1. alpha-spending to control the risk of falsely concluding that the treatment is efficacious due to multiple looks at the data. As @JasonConnorPhD reminds us, there is no alpha-spend from the DMC looking at efficacy as long as locked into prespecified efficacy stopping rules.

However, even if DSMB is locked into stopping rules & told that the efficacy threshold has not been met, knowledge of the efficacy results is still important for the DSMB in context of the other possibilities below…

SAFETY: the early evidence is suggestive of potential harm from the Experimental treatment.

Back to what I just said above: if the DSMB is locked into prespecified rules about efficacy stopping, why do they need to know anything more than “the stopping rule has not been met” for efficacy?

Well, because they’re also charged with monitoring safety and weighing the risk/benefit of continuing the trial.

If there is a mild/moderate safety concern (suppose, increased risk of one adverse event) it is relevant for the DSMB to know whether interim results are suggestive of efficacy and to what degree, because that should be taken into account in decision to stop for safety

FUTILITY: often confused, this is different from safety stopping; futility stopping occurs not necessarily because the Experimental treatment is harming patients, but interim data show little benefit and it is unlikely that the trial will conclude that Experimental treatment is beneficial

What it means to stop a trial for “futility” is confusing to a lot of people. One might argue that if there is no safety concern, there’s no “harm” in allowing the trial to carry on to its originally planned sample size, even if the interim results are looking pretty null.

Well, you’re still exposing patients to participation in a research study where the likelihood of benefit is fairly low. I don’t think it’s unethical to keep enrollment open until the originally planned sample size, but YMMV.

It’s also dependent on who is funding the trial – naturally, sponsors will not want to burn additional resources on trials that are now unlikely to end in a positive result; if interim analysis shows low probability of proving treatment benefit, they may prefer to kill the trial there and devote resources to other, more promising treatments.

In summary, the risk/benefit of whether to stop trials is usually not “symmetric” as @JasonConnorPhD puts it.

There is a difference between stopping for efficacy, safety, and futility. The level of evidence required to stop a trial for efficacy is higher than stopping for safety concerns.

The @NEJM article brought up by @thebyrdlab, @DonDonRic and others has a nice passage on this:

https://www.nejm.org/doi/full/10.1056/NEJM199805073381911

“Masked monitoring is thought to increase the objectivity of monitors by making them less prone to bias. What is overlooked is what masking does to degrade the competency of the monitors.”

“The assumption underlying masked monitoring is that recommendations for a change in the study protocol can be made independently of the direction of a treatment difference, but this assumption is false.”

“More evidence is required to stop a trial because of a benefit than because of harm. Trials are performed to assess safety and efficacy, not to “prove” harm…it is unreasonable to make the monitors behave as if they were indifferent to the direction of a treatment difference.”

One of the questions, from @DrSarahMcNab:

There’s your answer, Dr. McNab, at least the best way I can think of to explain it.

Most guidance on DMC’s and DSMB’s is pretty well unanimous on this issue:

One of the early replies, from @numbersman77:

Referenced this publication in Clinical Trials:

“DMCs should have access to unblinded efficacy and safety data throughout the trial to enable informed judgments about risks and benefits”

@MarionKCampbell brings up the DAMOCLES charter, which makes a similar recommendation:

The Clinical Trials Transformation Initiative recommendations for Data Monitoring Committees agrees:

@thebyrdlab also provided this excellent videocast from NIH:

https://videocast.nih.gov/Summary.asp?File=19411&bhcp=1

Special note: while the DSMB should not be blinded to the treatment groups at the interim analysis, maintaining confidentiality of all interim results is important (whether the arms are labeled or not)

Dissemination of interim results (even blinded results) can compromise recruitment and/or influence patient and physician behaviors

That’s all I’ve got for now.

I look forward to ongoing discussion in this forum. Perhaps I can even learn a few more things :slight_smile:

4 Likes
2

This is a great post! Thank you for the detailed insights, cautions and links.

I’m also in the non-blinding camp, but as long as the stopping rules are followed and they don’t specifically require which arm to differ, I do not see any harm keeping it blinded if necessary. I still don’t see a strong argument against one way or the other. I believe the stopping rules is the key component and the rest is more philosophical consideration - i.e. how knowing the tx arms will affect committee’s decision in really close call situations.

3

In my own experience you’re emphasizing the stopping rule (which is really a “stopping suggestion”) too much. And the key objective of a DMC is safety monitoring which requires unblinding without doubt IMHO.

2 Likes
4

I agree. but regardless what arm it is, if one of the arms seems to be more harmful, wouldn’t we would stop it anyways? in that case, it wouldn’t matter whether we unblinded or not.

5

It matters a lot because of the risk-benefit tradeoff. The bottom line for me is that there are 4 statisticians in the world who know more about this topic than anyone else (statistician or not) and have written books solely about DMCs. They argue convincingly that one should always unblind the committee.

2 Likes
6

makes sense. could you mind giving the references to at least a couple of these books? i would love to know how they think about it. thanks much.

7

See Ellenberg, Fleming, and DeMets and Herson.

3 Likes
8

One thing to clarify about the risk-benefit tradeoff which I think leads to some of this confusion. I would love for someone with more DSMB experience to chime in but this is my best effort to explain:

“Shouldn’t we stop the trial if either arm seems to be at greater risk than the other arm?”

This makes some degree of intuitive sense. I think this may be an acceptable line of thinking in the case of comparative-effectiveness research of 2 approved-and-already-in-use treatments. But I do not think it makes sense in a trial where one arm is clearly the “experimental” arm (some new agent or treatment that is not yet well studied) and the other clearly the “control” (whether that is placebo, or standard of care).

EDITED FOR CLARITY:

In that case, where the “control” is already an approved/accepted therapy that has been through development and several rounds of trials (or a placebo, which clearly cannot cause “harm”) it seems quite relevant to know which is the “experimental” treatment in the assessment of efficacy and safety. The threshold to stop a trial for “safety concerns” is generally lower than the threshold to stop a trial for “efficacy” (i.e. experimental treatment results are so superior that it is no longer ethical to randomize patients to standard of care).

If there is an difference in the incidence of an adverse event (not necessarily the primary outcome) in one of the treatment arms versus the other, I think it’s quite different for decision making about whether to continue the trial if the higher incidence is occurring in the “experimental” arm (in this case - one might argue that patients are being harmed by their participation in the trial because they are receiving something they would not be getting outside the trial) versus if the higher incidence is occurring the “standard of care” arm (since that is the therapy that the patients would be getting if they were not enrolling in the trial, it is hard to argue that patients are being harmed by their participation in the trial).

Does that make sense?

I am happy to come back and modify this post if I have misstated anything here, but that is one thing which I feel has been underappreciated by those arguing “If there’s a signal in either direction, shouldn’t the trial be stopped regardless of which arm is which?”

3 Likes
9

Well said. Risk is an asymmetric concept and because of that efficacy must ultimately also be. For that reason, unblinding of treatment assignment is essential for proper review.

1 Like
10

i voted against blinding in the twitter poll. I’m sure i’ve read examples where the dsmb published their thoughts/justifications for stopping a large trial based on unblinded analyses. Sometimes there is information from another ongoing study that may be relevant and encourage unblinding too. In any case, the dsmb have a very tough job even when the data are unblinded

2 Likes
11

off topic, but what about triple blinding? this paper defines it as “blinding of data managers and biostatisticians”: CCT journal The authors claim that it is “rarely utilized” and “generally less critical”. In the industry, however, they always developed the programming code on blinded data because the code were written before database lock, so that when lock occurred everything was ready to do. I’m guessing that in academia it may be true that “blinding of data managers and biostatisticians … is rarely utilized” which is worrying because their analysis plans also tend to allow more room to move…

1 Like
12

It’s very rarely used, for data quality control issues as much as anything else. And if the DMC reporting statistician is blinded, they cannot produce the needed reports.

13

for code development there is really no value in using the actual treatment groups and it’s a tiny bit of code to create nominal, spurious trt labels. And i assume code development occurs before breaking the blind, thus i don’t see how triple blinding wouldn’t be employed almost always, and i think it adds value. But yes: dsmb is quite a different thing, i meant in general

2 Likes