When Clinical Trials Don’t Match Their Data

dart off-centerClinical trials (research studies in people) can help find better treatment, care and prevention for patients. When done well, they answer important questions posed in the protocol (study plan) before the trial starts. Unfortunately, those answers are often not published. Some also don’t have the right study designs, which means the studies won’t tell us what we truly want to know.

The Problem?

…”the international Cochrane Collaboration commonly exclude from evidence 50% to 75% of published studies…”

– Soumerai, Starr, and Majumdar, The Health Care Blog

Why? They don’t meet the basic research design standards that Cochrane relies on to produce reliable results. 1/2 to 3/4 of our clinical trials? Big problem! But it’s only one of the issues.

Why Does This Happen?

Many misalignments help foster these problems. In their article, Soumerai et al show how clinical trialists start with errors that get multiplied as they gather steam, especially when those who will profit spin stories that get picked up by the news media.

“We will show how single, flawed studies, combined with widespread news media attention and advocacy by special interests, can lead to ineffective or unsafe policies.”

– Soumerai, Starr, and Majumdar

John Ioannidis at Stanford has written about this for years, and was recently interviewed by Gary Schwitzer from HealthNewsReview.org (podcast available). John published a seminal paper in 2005 called “Why Most Published Research Findings Are False.” Sorry to say that 10 years later, it is still true and will be until incentives change.

The COMPARE Project has taken on a simple (but not easy) goal – to find out which clinical trials reported on the outcomes (endpoints) that they stated in the protocol (study plan). As of 12/10/2015, the 58 trials they checked had these depressing results:

  • 8 trials (14%) reported the stated outcomes and they didn’t add later outcomes.
    That means they did what they said they would do. Thank you.
  • In the 50 trials (86%) that did NOT report on the stated outcomes,
    • They did NOT report on 306 of the original outcomes.
    • They added 304 different outcomes after they saw the data.
Photo credit: stevendepolo via Foter.com / CC BY

Photo credit: stevendepolo via Foter.com / CC BY

That’s like waiting until everyone turns over their cards before you decide how you want to play! There can be valid reasons for some, but certainly not all, of these changes. Retraction Watch has a good post about this.

To follow up, Ben Goldacre and his team send letters to journals who published the trial results to alert them to errors so they can issue a correction.

…”we entrust journals with an incredibly important job, a huge portion of the knowledge management in medicine and science.  We should be able to expect that journals routinely police something as unambiguous as outcome switching, during the peer review and editing process. Where there are slip-ups, we should be able to expect that those errors are corrected swiftly.”

– Ben Goldacre

Why Does This Matter?

Patients rely on doctors for accurate information that can be used to make critical decisions. Physicians also want this, and often base their decisions that treatments off of these clinical trials. If the evidence isn’t real, it could actually lead to wrong decisions that might actually create harm to patients. And no one wants that, do they?

What are publishers doing about this?

The COMPARE Project is relatively recent, so maybe it just takes publishers longer than expected to publish the team’s letters and then issue corrections. Right?

Old letters in string - deskI mean, it only makes sense that such esteemed journals as the Journal of the American Medical Association (JAMA), The Lancet, the Annals of Internal Medicine (Annals), and the British Medical Journal (BMJ) would take this seriously and correct errors quickly. And they would commit to checking something simple like this for each clinical trial BEFORE it is published. Right?

Millions of patients who count on the accuracy of their publications certainly hope so.

All content © 2015 by Deborah Collyar unless otherwise specified. All rights reserved. Permission is granted to use short quotes provided a link back to this page and proper attribution is given to me as the original author.

2 thoughts on “When Clinical Trials Don’t Match Their Data

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.