According to this, a randomised double-blind placebo-controlled trial into the effects of chiropractic treatment on infantile colic is to take place. Some will say “too little too late”, given that chiropractors have been claiming to “treat” or “help with” colic for some time now. I have a different concern. The notes on the planned trial state that “subgroup analyses will be performed to identify possible subgroups of infants who will benefit the most from the treatment”. My concern is that, even in the event that the trial does not show a benefit for chiropractic treatment of colic, advocates may claim that subgroup analysis shows that there is a benefit for some children and therefore chiropractic works as a treatment for colic. [The remainder of this post will consist mainly of copying and pasting from here and one or two other sources.]
Ben Goldacre wrote in April about a trial of fish oil pills. Here are some quotes from the piece that I found interesting:
The fish oil pills did nothing, as in many previous studies, so this trial has not been press released by the company, nor has it been covered in the media. The funders of this study, Equazen, will doubtless have been disappointed with a negative result. The authors of the study may have been disappointed too. But there was some light on the horizon. They looked at the data more closely, and found that some children did, in fact, respond: “a subgroup of 26% responded with more than 25% reduction of ADHD symptoms and a drop of CGI scores to the near-normal range.”
Interestingly it turns out that you can show significant benefits, using a subgroup analysis, even in a fake trial, where the intervention consists of doing absolutely nothing whatsoever. 30 years ago Lee et al published the classic cautionary paper on this topic in the journal Circulation* […] They were not disappointed. Overall, as expected, there was no difference in survival between the two groups. But in a subgroup of 397 patients (characterized by “three-vessel disease” and “abnormal left ventricular contraction”) the survival of Treatment 1 patients was significantly different from that of Treatment 2 patients. This was entirely by chance.
You can also find spurious subgroup effects in real trials, if you do an analysis that’s foolish enough. Close analysis of the ECST trial found that the efficacy of a procedure called endarterectomy depended on which day of the week you were born on. Base your clinical decisions on that: I dare you.
At one point in the article, Goldacre points out that:
Subgroup analyses are widely derided in academia, and for very good reasons.
This article in the NEJM, meanwhile, begins by telling us that:
Subgroup analyses are an important part of the analysis of a comparative clinical trial. However, they are commonly overinterpreted and can lead to further research that is misguided or, worse, to suboptimal patient care.
I look forward to reading the trial of chiropractic for colic and hope that the subgroup analyses will not be overinterpreted and lead to misguided research or suboptimal patient care.
*This [PDF] is the Circulation paper that is mentioned in the Bad Science article.