Clinical trials (research studies in people) can help find better treatment, care and prevention for patients. When done well, they answer important questions posed in the protocol (study plan) before the trial starts. Unfortunately, those answers are often not published. Some also don’t have the right study designs, which means the studies won’t tell us what we truly want to know.
…”the international Cochrane Collaboration commonly exclude from evidence 50% to 75% of published studies…”
– Soumerai, Starr, and Majumdar, The Health Care Blog
Why? They don’t meet the basic research design standards that Cochrane relies on to produce reliable results. 1/2 to 3/4 of our clinical trials? Big problem! But it’s only one of the issues.
Why Does This Happen?
Many misalignments help foster these problems. In their article, Soumerai et al show how clinical trialists start with errors that get multiplied as they gather steam, especially when those who will profit spin stories that get picked up by the news media.
“We will show how single, flawed studies, combined with widespread news media attention and advocacy by special interests, can lead to ineffective or unsafe policies.”
– Soumerai, Starr, and Majumdar
John Ioannidis at Stanford has written about this for years, and was recently interviewed by Gary Schwitzer from HealthNewsReview.org (podcast available). John published a seminal paper in 2005 called “Why Most Published Research Findings Are False.” Sorry to say that 10 years later, it is still true and will be until incentives change.
The COMPARE Project has taken on a simple (but not easy) goal – to find out which clinical trials reported on the outcomes (endpoints) that they stated in the protocol (study plan). As of 12/10/2015, the 58 trials they checked had these depressing results:
- 8 trials (14%) reported the stated outcomes and they didn’t add later outcomes.
That means they did what they said they would do. Thank you.
- In the 50 trials (86%) that did NOT report on the stated outcomes,
- They did NOT report on 306 of the original outcomes.
- They added 304 different outcomes after they saw the data.
That’s like waiting until everyone turns over their cards before you decide how you want to play! There can be valid reasons for some, but certainly not all, of these changes. Retraction Watch has a good post about this.
…”we entrust journals with an incredibly important job, a huge portion of the knowledge management in medicine and science. We should be able to expect that journals routinely police something as unambiguous as outcome switching, during the peer review and editing process. Where there are slip-ups, we should be able to expect that those errors are corrected swiftly.”
– Ben Goldacre
Why Does This Matter?
Patients rely on doctors for accurate information that can be used to make critical decisions. Physicians also want this, and often base their decisions that treatments off of these clinical trials. If the evidence isn’t real, it could actually lead to wrong decisions that might actually create harm to patients. And no one wants that, do they?
What are publishers doing about this?
The COMPARE Project is relatively recent, so maybe it just takes publishers longer than expected to publish the team’s letters and then issue corrections. Right?
I mean, it only makes sense that such esteemed journals as the Journal of the American Medical Association (JAMA), The Lancet, the Annals of Internal Medicine (Annals), and the British Medical Journal (BMJ) would take this seriously and correct errors quickly. And they would commit to checking something simple like this for each clinical trial BEFORE it is published. Right?
Millions of patients who count on the accuracy of their publications certainly hope so.
All content © 2015 by Deborah Collyar unless otherwise specified. All rights reserved. Permission is granted to use short quotes provided a link back to this page and proper attribution is given to me as the original author.