While there are conventional journals that will publish research on alternative medicine, some researchers into alternative medicine seem to prefer to publish their research in alternative journals. Some time ago, I spotted something rather interesting when reading up on food supplement research. It involved two reviews – one published in a conventional journal, and one in an alternative journal. Here are the PDFs: Annals Internal Med EPO review and AltMed EPO review.
There is one paper that was cited by both reviews, but the authors of the two reviews seemed to come to different conclusions about what this paper meant. Here is a copy of the original PDF: EPO paper 1994. The author of the review published in Alternative Medicine Review stated that:
A single six-month, randomized, double-blind, placebo-controlled study examined EPO for menopausal hot flashes and night sweating in 56 women. There was a statistically significant reduction in nighttime flushing in the treatment group compared to the control group, but not of daytime hot flashes. Further studies are warranted to examine the possible therapeutic effects of evening primrose oil for hot flashes.
The review is quite clear in stating that “There was a statistically significant reduction in nighttime flushing in the treatment group compared to the control group” and also in stating that “Further studies are warranted to examine the possible therapeutic effects of evening primrose oil for hot flashes”.
Meanwhile, the Annals of Internal Medicine review states that the Chenoy study cited by the author of the Alt Med Review actually found that:
Frequency of daytime hot flashes decreased in placebo but not evening primrose oil group; no difference between groups in frequency of nighttime hot flashes (decreased in both groups)
So while the frequency of daytime hot flashes were decreased in the placebo group only, the groups showed no difference in frequency of nightime flashes. In the discussion, evening primrose oil is listed (among other treatments) as being “ineffective for hot flashes”.
Now we have two seemingly contradictory statements; from the Alt Med Review “a statistically significant reduction in nighttime flushing in the treatment group compared to the control group” and from the Annals of Internal Medicine review “the groups showed no difference in frequency of nightime flashes“. It seems as if one of these statements must be wrong. But how to decide? This is where the original paper cited by both reviews comes in.
The authors of the original paper found that the placebo group, in contrast to the group treated with evening primrose oil, had a significant improvement “between the control cycle and last available treatment”. The authors then mention, as an aside, that there was “a reduction in the maximum number of night time flushes” in the treatment group – this specific finding was an exception and this was the context in which it was mentioned. They also state that “whether this is a true difference or chance improvement remains uncertain.” They go on to provide this comment in the discussion section of their paper:
The results from our pilot study show that gamolenic acid provided by evening primrose oil, although popularly believed to alleviate vasomotor symptoms of the menopause, offers no benefit over placebo.
A study that found a significant improvement in the placebo group and only mentioned (as an aside) the exception that there was “a reduction in the maximum number of night time flushes” in the treatment group is later represented by the author of a review in Alt Med Review as showing “a statistically significant reduction in nighttime flushing in the treatment group compared to the control group“. A paper that states quite clearly that “Given these results and the lack of a scientific reason for using gamolenic acid, the use of evening primrose oil in treating menopausal flushing cannot be supported” is cited in a review, seemingly as justification for the use of evening primrose oil.
The Alt Med Review article goes on to state that “botanicals, dietary supplements, and other CAM approaches have been shown to be effective in treating hot flashes in menopausal women” and that “The findings of the aforementioned research suggest there are effective alternative approaches to treating menopausal hot flashes” before advising that “an individualized approach should be applied” in the absence of definitive guidelines. While the author does not explicitly state that evening primrose oil should be used, there seems to be a positive slant toward evening primrose oil in the Alt Med Review article. It concludes: “With the information available to date, menopausal women can be encouraged to explore alternative approaches to alleviating hot flashes.”
I’m not sure that women should be encouraged to explore alternative approaches to alleviating hot flashes given the evidence put forward for evening primrose oil. I’m not sure that we should take seriously anything written in an article that seems to so misrepresent the research findings of a placebo-controlled trial. I am sure that we should question the worth of CAM journals such as the Alt Med Review as long as they are publishing articles that fail to accurately represent the papers they cite. Even if they do claim to be “a peer-reviewed journal dedicated to providing accurate, timely, and clinically relevant original articles, abstracts and literature reviews to the practicing preventive health-care professional.”
I have looked at a single citation in one alternative journal and compared the interpretation of the cited study to the interpretation given by a more conventional journal. It could be the case that the misinterpretation of the cited study that I have focused on is untypical of alternative journals in general. Future research could perhaps involve a more systematic review of citations in alternative journals.
I am far from the first commenter to pick up on problems with alternative journals. AP Gaylard highlighted some suggestions made by R Barker Bausell in his book “Snake Oil Science – The Truth About Complementary and Alternative Medicine“. One of these recommendations was to “Give more credence to trials published in well known medical journal and give no credence at all to those published in CAM journals.”
Dr Aust, meanwhile, has been quite outspoken on the trouble with CAM journals: here, and in a follow-up post. The follow-up post contains a summary of what Dr Aust perceives as being the modus operandi of the alternative journals.