Conventional And Alternative Journals

September 28, 2009 at 4:24 pm (Alternative Medicine, Nutritionism, Referenciness, Supplements) (, , , )

While there are conventional journals that will publish research on alternative medicine, some researchers into alternative medicine seem to prefer to publish their research in alternative journals. Some time ago, I spotted something rather interesting when reading up on food supplement research. It involved two reviews – one published in a conventional journal, and one in an alternative journal. Here are the PDFs: Annals Internal Med EPO review and AltMed EPO review.

Exploration

There is one paper that was cited by both reviews, but the authors of the two reviews seemed to come to different conclusions about what this paper meant. Here is a copy of the original PDF: EPO paper 1994. The author of the review published in Alternative Medicine Review stated that:

A single six-month, randomized, double-blind, placebo-controlled study examined EPO for menopausal hot flashes and night sweating in 56 women.[103] There was a statistically significant reduction in nighttime flushing in the treatment group compared to the control group, but not of daytime hot flashes. Further studies are warranted to examine the possible therapeutic effects of evening primrose oil for hot flashes.

The review is quite clear in stating that “There was a statistically significant reduction in nighttime flushing in the treatment group compared to the control group” and also in stating that “Further studies are warranted to examine the possible therapeutic effects of evening primrose oil for hot flashes”.

Meanwhile, the Annals of Internal Medicine review states that the Chenoy study cited by the author of the Alt Med Review actually found that:

Frequency of daytime hot flashes decreased in placebo but not evening primrose oil group; no difference between groups in frequency of nighttime hot flashes (decreased in both groups)

So while the frequency of daytime hot flashes were decreased in the placebo group only, the groups showed no difference in frequency of nightime flashes. In the discussion, evening primrose oil is listed (among other treatments) as being “ineffective for hot flashes”.

Now we have two seemingly contradictory statements; from the Alt Med Review “a statistically significant reduction in nighttime flushing in the treatment group compared to the control group” and from the Annals of Internal Medicine review “the groups showed no difference in frequency of nightime flashes“. It seems as if one of these statements must be wrong. But how to decide? This is where the original paper cited by both reviews comes in.

The authors of the original paper found that the placebo group, in contrast to the group treated with evening primrose oil, had a significant improvement “between the control cycle and last available treatment”. The authors then mention, as an aside, that there was “a reduction in the maximum number of night time flushes” in the treatment group – this specific finding was an exception and this was the context in which it was mentioned. They also state that “whether this is a true difference or chance improvement remains uncertain.” They go on to provide this comment in the discussion section of their paper:

The results from our pilot study show that gamolenic acid provided by evening primrose oil, although popularly believed to alleviate vasomotor symptoms of the menopause, offers no benefit over placebo.

A study that found a significant improvement in the placebo group and only mentioned (as an aside) the exception that there was “a reduction in the maximum number of night time flushes” in the treatment group is later represented by the author of a review in Alt Med Review as showing “a statistically significant reduction in nighttime flushing in the treatment group compared to the control group“. A paper that states quite clearly that “Given these results and the lack of a scientific reason for using gamolenic acid, the use of evening primrose oil in treating menopausal flushing cannot be supported” is cited in a review, seemingly as justification for the use of evening primrose oil.

The Alt Med Review article goes on to state that “botanicals, dietary supplements, and other CAM approaches have been shown to be effective in treating hot flashes in menopausal women” and that “The findings of the aforementioned research suggest there are effective alternative approaches to treating menopausal hot flashes” before advising that “an individualized approach should be applied” in the absence of definitive guidelines. While the author does not explicitly state that evening primrose oil should be used, there seems to be a positive slant toward evening primrose oil in the Alt Med Review article. It concludes: “With the information available to date, menopausal women can be encouraged to explore alternative approaches to alleviating hot flashes.”

I’m not sure that women should be encouraged to explore alternative approaches to alleviating hot flashes given the evidence put forward for evening primrose oil. I’m not sure that we should take seriously anything written in an article that seems to so misrepresent the research findings of a placebo-controlled trial. I am sure that we should question the worth of CAM journals such as the Alt Med Review as long as they are publishing articles that fail to accurately represent the papers they cite. Even if they do claim to be “a peer-reviewed journal dedicated to providing accurate, timely, and clinically relevant original articles, abstracts and literature reviews to the practicing preventive health-care professional.”

Limitations

I have looked at a single citation in one alternative journal and compared the interpretation of the cited study to the interpretation given by a more conventional journal. It could be the case that the misinterpretation of the cited study that I have focused on is untypical of alternative journals in general. Future research could perhaps involve a more systematic review of citations in alternative journals.

More

I am far from the first commenter to pick up on problems with alternative journals. AP Gaylard highlighted some suggestions made by R Barker Bausell in his book “Snake Oil Science – The Truth About Complementary and Alternative Medicine“. One of these recommendations was to “Give more credence to trials published in well known medical journal and give no credence at all to those published in CAM journals.”

Dr Aust, meanwhile, has been quite outspoken on the trouble with CAM journals: here, and in a follow-up post. The follow-up post contains a summary of what Dr Aust perceives as being the modus operandi of the alternative journals.

12 Comments

  1. Twitter Trackbacks for Conventional And Alternative Journals « Stuff And Nonsense [jdc325.wordpress.com] on Topsy.com said,

    […] Conventional And Alternative Journals « Stuff And Nonsense jdc325.wordpress.com/2009/09/28/conventional-and-alternative-journals – view page – cached While there are conventional journals that will publish research on alternative medicine, some researchers into alternative medicine seem to prefer to publish their research in alternative… (Read more)While there are conventional journals that will publish research on alternative medicine, some researchers into alternative medicine seem to prefer to publish their research in alternative journals. Some time ago, I spotted something rather interesting when reading up on food supplement research. It involved two reviews – one published in a conventional journal, and one in an alternative journal. Here are the PDFs: Annals Internal Med EPO review and AltMed EPO review. (Read less) — From the page […]

  2. apgaylard said,

    Nice spot. Another complimentary pair is provided by the homeopaths re-analysis of the legendary Shang et al: Lüdtke and Rutten and Rutten and Stolper. The latter get away with much more as they are published in Homeopathy, rather than a proper journal. (See here and hereblog – for an analysis of both papers.)

    The paper in the proper journal makes the obvious observation that if you do the meta-analysis differently you get different results; whilst the paper in the homeopath fanzine is able to claim post-hoc jiggery-pokery.

    Amother small example is Milgrom and Rey claiming that the latters 2003 findings were confirmed by van Wijk et al, when the paper says not:

    “We report here differences in thermoluminescence between C15 D2O and C15 LiCl, which correspond with the observations reported by Rey (2003). However, the difference from all of these recordings of these substances was not statistically significant.”

    After spending some time with the alt.med literature, I can see why Bausell suggests ignoring it: if the work is any good it’ll get published somewhere better.

  3. jdc325 said,

    Thank you for commenting. I’ve commented elsewhere that I’m wary of anything published in the journal Homeopathy (actually on a BHA discussion forum, where I quoted from Paul Wilson’s letter about Rutten and Stolper). I’m almost inclined to take the position that it is only worthwhile looking at papers from Alt Med journals in order to debunk them when they are being cited as support for the nonsense du jour. (But then I worry that I might miss something amusing – or something that is revealing.)

    (Edited for html. 10:25pm)

  4. draust said,

    Note also that the homeopath complaining loudly about “rotten sceptics!” in one homeo-journal will usually be on the editorial board of(or even chief editor) of another AltMed journal… which perhaps publishes a paper by the editor of the first journal complaining loudly about…

    …you get the idea.

    Question: who then are the editors likely to be approving, or the reviewers reviewing, these lengthy self-justificatory screeds? Anyone?

    Like they say, review by your peers. And how.

    I suppose the most charitable thing you can say about it is that it is rather like having a homeo-blogosphere, except that instead of blogs (like in scepticism) you have whole journals (thanks to the greed of the major publishing houses).

    Oh, and, since they are “journals”, there is usually limited or even no adverse / skeptical / questioning allowed – eCAM with its electronic comments thread being a notable exception.

    The other difference is that the standards of argument, scholarship and accuracy are typically higher in the sceptical blogosphere.

  5. draust said,

    Oops – forgot about WordPress link stripping my posts – one last try:

    http://ecam.oxfordjournals.org/cgi/eletters/4/1/7

  6. jdc325 said,

    Thanks for commenting Dr Aust.

    I’m inclined to agree that “the standards of argument, scholarship and accuracy are typically higher in the sceptical blogosphere”. I certainly wouldn’t claim that this were true of all sceptical bloggers, but if we take homeopathy as just one example I think one could make a strong argument for considering What The Hell Is This?, Hawk Handsaw and A Canna’ Change The Laws Of Physics as being more accurate and having higher standards of argument and scholarship than journals such as Homeopathy.

    I think that my blog is more a personal commentary on my experiences with advocates of alternative medicine and news coverage of health stories, whereas blogs like Holford Watch are more likely to critique an article in a scholarly manner.

    Hawk Handsaw
    A Canna’ Change The Laws Of Physics
    What The Hell Is This?

  7. draust said,

    Couldn’t agree more about the three blogs you just listed, jdc. They have submitted the claims of homeopaths in general, and Lionel Milgrom in particular, to a far, far more stringent and searching analysis that have the expert reviewers, editors and readers of the homeopathic journals… at least in my opinion. I think the efforts of all three are genuinely heroic, given that they all have full-time jobs and are doing this on the side.

    The depressing thing is that the Alt.Reality brigade will just go on puffing pompously about “published in a peer reviewed academic journal” etc etc… when really “hobby journal” would be a more accurate description. And far too many people in the academic medical establishment, in particular, will take these clowns at face value.

    *sigh*

    PS I think I am nearer to you in the “personal reaction blog” category. Personally I would blow a fuse if I tried to subject Lionel M’s meanderings to the kind of withering academic deconstruction Adrian Gaylard has produced. Actually I reckon, judging by his work on Milgrom, that AG would have been a much better academic than I am.

  8. The SkeptVet said,

    I spend a lot of time reviewing the literature regardingveterinary use of CAM, and I find the same phenomenon repeatedly. You can never trust a citation in support of an assertion without reading the cited paper itself to see if it says what the author claims it says. Another reason that just because it’s ina journal doesn’t eman it’s true. :-) Thanks for the post!(http://skeptvet.com/Blog/2009/07/you-cant-believe-everything-you-read-even-in-a-scientific-journal/)

  9. jdc325 said,

    Thanks for the link SkeptVet – your post makes for interesting reading.

  10. Neuroskeptic said,

    Re: “the standards of argument, scholarship and accuracy are typically higher in the sceptical blogosphere”, I think this is partly because a blog is a better medium for criticizing a paper than the “normal” academic channels.

    Say you read a paper and spot a massive gaping hole in the methods (which happens all the time, even in proper science, but especially in CAM). You could write a letter to the Editor of the journal who published it.

    It might get published, several months later. But in many cases the original authors will get an instant reply which gets published alongside it. In many cases this consists of obfuscation to muddy the waters and make the original gaping hole seem minor.

    And then you need to write another letter to point that out, if you can still be bothered. Etc.

    We’ve seen this very clearly in the Shang meta-analysis debate.

  11. draust said,

    Re: “the standards of argument, scholarship and accuracy are typically higher in the sceptical blogosphere”, I think this is partly because a blog is a better medium for criticizing a paper than the “normal” academic channels.

    This is partly why I am a fan of journal on-line comments threads, Neuroskeptic. Many people who read the paper will read the comments thread, and it is a much more immediate and visible way to respond than in a “Letter to the editor”, for the reasons you suggest.

    There is also, I think, a certain reluctance among many people in science to engage in public arguments, even involving their own work. People

    (i) don’t want it to be clear they are annoyed;
    (ii) hope it will all just go away;

    I say that partly because, in my editorial role, I have been contacted several times by people wanting to harrumph “I’m appalled you published….” “these people utterly misrepresented the literature (and didn’t cite us enough)” “this interpretation X published in your journal is wrong, wrong, wrong”.

    I have always offered people who say this a platform in print to say what they want to say.

    All, to a man or woman, have refused.

  12. Chiropractic For Autism « Stuff And Nonsense said,

    […] alternative journals and on one occasion found that a paper cited by the author of an article in an alternative journal was actually misrepresented by the […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: