This paper is a systematic review of controlled trials that asks the question “Do certain countries produce only positive results?” The authors conclude that:
Some countries publish unusually high proportions of positive results. Publication bias is a possible explanation. Researchers undertaking systematic reviews should consider carefully how to manage data from these countries.
The study looked at acupuncture trials originating in China, Japan, Hong Kong, and Taiwan – which were all positive – as well as trials published in Russia/USSR, of which 10 out of 11 were positive. Similar results were found in trials of interventions other than acupuncture.
available evidence suggests that the hypericum extracts tested in the included trials a) are superior to placebo in patients with major depression; b) are similarly effective as standard antidepressants; c) and have fewer side effects than standard antidepressants. The association of country of origin and precision with effects sizes complicates the interpretation.
The note of caution regarding the country of origin relates to trials from German-speaking countries reporting findings more favourable to hypericum.
Neuroskeptic covered this review and pointed out that, although St John’s Wort was significantly more effective than placebo, this was only true when studies from German-speaking countries were examined.
Out of the 11 Germanic trials, 8 found that St John’s Wort was significantly better than placebo and the other 3 were all very close. None of the 8 non-Germanic trials found it to be effective and only one was close.
Neuroskeptic writes that “the obvious, cynical explanation is that there are lots of German trials finding that St John’s Wort didn’t work, but they haven’t been published because St John’s Wort is very popular in German-speaking countries”, but notes that the authors doubt that publication bias strongly influences their findings.
There is also speculation in the post by Neuroskeptic that St John’s Wort may actually work better in German trials because German investigators tend to recruit the kind of patients who respond well to St John’s Wort.
Perhaps the credibility of a therapy in a particular country actually influences trial results. If this is the case, then I should probably expect to see more positive results for acupuncture in China than in England, more positive trials of St John’s wort published in Germany than in England – and perhaps even a tendency for US studies of chiropractic to be more positive than those conducted in England.
It could be that the credibility of a therapy increases the possibility that researchers might be (consciously or subconsciously) biased in favour of that therapy. If they expect a therapy to work, their results may be more likely to show that it does.
There are numerous ways in which a study can be flawed. Properly randomised allocation of subjects, for example, is important partly because researchers may subconsciously put healthier subjects in the arm of the trial that features an intervention they believe is effective.
Depending on how well-blinded a trial is, it might even be the case that the subjects in German trials realise they are taking (for example) St John’s wort rather than placebo, and expect to improve.
I imagine that expectation may be an important factor in treating pain or depression. If subjects realise they are taking a medicine that is generally considered to be effective then their expectation of improvement could become a self-fulfilling prophecy.
Insufficient blinding, or unintended communication (of an expectation of efficacy) to the subject from researchers administering the treatment, may lead to an expectation of improvement and a subsenquent actual improvement.
This could perhaps partly explain the tendency for some countries to publish more positive results for specific therapies than others.
As I am English, perhaps I should avoid acupuncture and St John’s wort. Maybe I should be looking for “quintessentially English” remedies for any ailments I might suffer from?
Perhaps, then, the most appropriate treatment for me (for any minor, self-limiting conditions) would be advice to “keep a stiff upper lip”? This advice would probably be best delivered by a serious-looking person wearing a white coat.