Cultural Bias in Scientific Research

February 3, 2010 at 7:49 pm (Bad Science, Miscellaneous) (, )

This paper is a systematic review of controlled trials that asks the question “Do certain countries produce only positive results?” The authors conclude that:

Some countries publish unusually high proportions of positive results. Publication bias is a possible explanation. Researchers undertaking systematic reviews should consider carefully how to manage data from these countries.

The study looked at acupuncture trials originating in China, Japan, Hong Kong, and Taiwan – which were all positive – as well as trials published in Russia/USSR, of which 10 out of 11 were positive. Similar results were found in trials of interventions other than acupuncture.

Two days ago, I wrote a post about ‘boosting serotonin‘ and included some comment on St John’s Wort. This review of St John’s wort for major depression by Linde et al concluded that:

available evidence suggests that the hypericum extracts tested in the included trials a) are superior to placebo in patients with major depression; b) are similarly effective as standard antidepressants; c) and have fewer side effects than standard antidepressants. The association of country of origin and precision with effects sizes complicates the interpretation.

The note of caution regarding the country of origin relates to trials from German-speaking countries reporting findings more favourable to hypericum.

Neuroskeptic covered this review and pointed out that, although St John’s Wort was significantly more effective than placebo, this was only true when studies from German-speaking countries were examined.

Out of the 11 Germanic trials, 8 found that St John’s Wort was significantly better than placebo and the other 3 were all very close. None of the 8 non-Germanic trials found it to be effective and only one was close.

Neuroskeptic writes that “the obvious, cynical explanation is that there are lots of German trials finding that St John’s Wort didn’t work, but they haven’t been published because St John’s Wort is very popular in German-speaking countries”, but notes that the authors doubt that publication bias strongly influences their findings.

There is also speculation in the post by Neuroskeptic that St John’s Wort may actually work better in German trials because German investigators tend to recruit the kind of patients who respond well to St John’s Wort.

Perhaps the credibility of a therapy in a particular country actually influences trial results. If this is the case, then I should probably expect to see more positive results for acupuncture in China than in England, more positive trials of St John’s wort published in Germany than in England – and perhaps even a tendency for US studies of chiropractic to be more positive than those conducted in England.

It could be that the credibility of a therapy increases the possibility that researchers might be (consciously or subconsciously) biased in favour of that therapy. If they expect a therapy to work, their results may be more likely to show that it does.

There are numerous ways in which a study can be flawed. Properly randomised allocation of subjects, for example, is important partly because researchers may subconsciously put healthier subjects in the arm of the trial that features an intervention they believe is effective.

Depending on how well-blinded a trial is, it might even be the case that the subjects in German trials realise they are taking (for example) St John’s wort rather than placebo, and expect to improve.

I imagine that expectation may be an important factor in treating pain or depression. If subjects realise they are taking a medicine that is generally considered to be effective then their expectation of improvement could become a self-fulfilling prophecy.

Insufficient blinding, or unintended communication (of an expectation of efficacy) to the subject from researchers administering the treatment, may lead to an expectation of improvement and a subsenquent actual improvement.

This could perhaps partly explain the tendency for some countries to publish more positive results for specific therapies than others.

As I am English, perhaps I should avoid acupuncture and St John’s wort. Maybe I should be looking for “quintessentially English” remedies for any ailments I might suffer from?

Perhaps, then, the most appropriate treatment for me (for any minor, self-limiting conditions) would be advice to “keep a stiff upper lip”? This advice would probably be best delivered by a serious-looking person wearing a white coat.

10 Comments

  1. Teek said,

    “Maybe I should be looking for “quintessentially English” remedies for any ailments I might suffer from?”

    A strong cuppa and a slice of cake will cure what ails ya…

    the only possible way to get around this is to have a giant database, which registers every single clinical trial anywhere in the world, its intended outcome measures etc, and makes a note of those that don’t get published – as if that’ll ever happen…

  2. Brain Duck said,

    I would also wonder about what was in the German pills, cf others.

  3. jdc325 said,

    Yes, good point Duck. It could be that different trials used standardized extracts, while others used “whole herb” preparations. Also, there may have been different extracts used in different trials – with emphasis on different components of SJW (eg hypericin, hyperforin).

    Thanks for commenting.

  4. Johannesurt mot depresjon – for tyskere? « Skepsis blog said,

    […] positive funn i kinesisk akupunkturforskning (ca. 100%). Det kan også være andre årsaker, slik Stuff and Nonsense tar for seg: St John’s Wort may actually work better in German trials because German […]

  5. Neuroskeptic said,

    Good post.

    Re: the St John’s Wort meta-analysis, I just took a look at it again and I noticed something I missed the first time around –

    “Patients were recruited in private practices in all trials from German-language countries, in the trials from Sweden (Bjerkenstedt 2005) and Canada (van Gurp
    2002), and in one of the trials from the UK (Wheatley 1997). The
    second trial from the UK (Montgomery 2000) and the trial from
    France (Lecrubier 2002) were performed both in private practices
    and psychiatric outpatient departments. Three trials from the US
    (Shelton 2001; HDTSG 2002; Fava 2005) and the Brazilian trial
    (Moreno 2005) were performed in academic and/or community
    psychiatry research clinics. Two trials from the US and Denmark
    (Brenner 2000; Behnke 2002) did not report on the setting.”

    This could well be important – the kind of patients you get going to private psychiatrists are generally quite different to the kind you get in national health services, in the UK anyway, and I’d imagine it’s the same in Germany. So personally my interpretation is that SJW works for some people, not for others, and the kind of people it works for are the kind who go to German private psychiatrists.

  6. jdc325 said,

    “So personally my interpretation is that SJW works for some people, not for others, and the kind of people it works for are the kind who go to German private psychiatrists.”
    Ha, that’s quite a specific remedy then…

    Thanks for commenting Neuroskeptic.

  7. Holford, Burne and Serotonin Pills « Stuff And Nonsense said,

    […] Although there are some notes of caution sounded in the paper. For instance: “The association of country of origin and precision with effects sizes complicates the interpretation.” I wrote a blogpost about the association of country of origin here: Cultural Bias in Scientific Research. […]

  8. Homeopathic St John’s Wort « Stuff And Nonsense said,

    […] but most of it has been conducted in German-speaking countries. I wrote about this in my post on Cultural Bias in Scientific Research. The Cochrane review I link to in my post is worth […]

  9. The Year In Nonsense. And Stuff. « Stuff And Nonsense said,

    […] February began with a blogpost about various claims relating to “balancing neurotransmitters” and “boosting serotonin”. In writing this blogpost, I came across a review of St John’s wort for depression. The authors of the review noted that the interpretation of the evidence was complicated by the association of country of origin and precision with effects sizes. This led me to post a blog titled “cultural bias in scientific research“. […]

  10. An A-Z of Alternative Medicine « Stuff And Nonsense said,

    […] of country of origin and precision with effects sizes complicates the interpretation. See here for more on the association of country of origin with effects sizes. St John’s wort is one of […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: