Anecdotal Evidence

July 24, 2012 at 6:15 pm (Anecdote, Evidence, Placebo) (, , , , , )

As is pointed out in the Wikipedia article on anecdotal evidence, such evidence is considered to be dubious support of a claim. Not only because it may be unrepresentative of a “typical” experience, but also because anecdotal evidence consists of casual observations rather than rigorous analysis. It is inevitable that casual observations will be subject to bias. Rigorous analysis is subject to bias too – but less so. Attempts are at least made to reduce the influence of biases.

Cognitive Biases

There is a useful list on Wikipedia of cognitive biases. Anecdotal evidence may be subject to, for example, confirmation bias – you might remember the times when a treatment seemingly worked and forget the times when it didn’t, or you might interpret new information as being favourable to your preconceptions when it actually isn’t. And that is just one of the 85 cognitive biases listed on that page. There are others that might be relevant too, for example illusory correlation – inaccurately perceiving a relationship between two unrelated events.

You might think that you aren’t prone to cognitive biases or are at least less prone than others. This is not an unusual viewpoint; mistaken, but not unusual. Lots of people think that others are more prone to cognitive bias than them. This is the cognitive bias of failing to compensate for one’s own cognitive biases – the bias blind spot.

We may selectively remember, misinterpret new information, or mistakenly assume that correlation equals causation. And it might be harder for us to spot when we do this than when others do it. It is easy to be mistaken, but not so easy to realise that we are.

A tale of two anecdotes

I tried two treatments, years apart, for similar problems (involving leg pain). They both ‘worked’.

Magnesium pills

Having had problems with leg pain that felt like it might be cramp, I tried some magnesium pills one evening and the next morning felt hardly any pain. This was a great improvement. Of course, I don’t know whether the improvement was due to the magnesium, due to the act of taking a pill (any pill), or whether I would have felt the improvement the next morning regardless… which is why this anecdote tells me nothing of value about whether magnesium pills can help with leg pain.

If the improvement that I felt could be due to the magnesium in the pill, to placebo effects, or to regression to the mean and I have made no effort to account for the placebo or regression to the mean possibilities I cannot rule them out and I cannot possibly know which of the three possible explanations might be the right one (or whether there is another explanation that I haven’t even considered). There may have been a correlation between taking the pills and perceiving an improvement, but I cannot know whether the pills caused the perceived improvement.

Ibuprofen gel

On another occasion when I had back pain I tried pain-killing gel. The gel was even more impressive than the magnesium pills: I bought a tube and it worked straight away… before I even used it. Which is ridiculous, of course.  A treatment cannot possibly work before you’ve applied it. In this case I can say for certain that the treatment itself was not responsible for the perceived improvement. If the perceived improvement had come shortly afterwards (as it did when I tried the magnesium pills), I might have assumed it was due to the treatment. I would have been mistaken.

Evidence-based medicine

It would be nice if we were able to tell what worked and what didn’t simply by trying it. Sadly, we can’t. Here, I wrote a rough guide to evidence-based medicine. EBM might not be perfect but carefully gathered evidence is probably a more reliable guide to what works than “try it and see”, at least when it comes to medical treatment.

Edit, 25th July 2012

As pointed out in the first comment below, anecdotes aren’t always ignorable. I can safely ignore my anecdotes about trying to treat leg pain, but if after trying a remedy my right arm turned green and swelled to twice its normal size you can bet I’d report it to my doctor and tell him which treatment(s) I’d tried. Such an anecdote wouldn’t prove that the treatment had caused the symptoms but in the absence of a more plausible explanation it would be reasonable to investigate further.

6 Comments

  1. softestpawn said,

    OK… yes… but…. as woodchopper pointed out many moons ago, anecdotes are still evidence. We should be as careful about dismissing it as about accepting rigorous research that has at least made an attempt to reduce the influence of bias.

    A single anecdote of a black swan, for example, is evidence that the theory that ‘all swans are white’ is false. Assessing the reliability of that anecdote against the existing theory should be done with care, as this is sort of a sliding scale of reliability (on probably several axiseses), not an either/or choice of reliable (research) and unreliable/ignorable (anecdote).

    Which you have said; it might be worth exploring what those scales look like – reputation of source (authority, and reputation to whom?), of institution (again), interpreting the reliability of the method (RCTs and p values when available, what about when they are not, etc) and the reporting of it (was it even carried out?) etc. It might throw up some reasons why reasonably sensible people practice homeopathy. Or it might not…

  2. jdc325 said,

    I didn’t intend to say that anecdotes should always be ignored, but I may have implied it. Post now amended with reference to your comment.

    Interesting comment re scales of reliability, by the way. Food for thought there…

  3. YaronD said,

    I noticed this a bit late, but anyway.

    softestpawn: No, a single anecdote of a black swan is absolutely NOT evidence that the claim that “all swans are white” is false.

    The *evidence* of a black swan will do that, sure, but an *anecdote* of seeing a black swan is not data that a black swan exists, is not evidence of a black swan, and so it doesn’t disprove anything that the actual *data* of a black swan would.

    Unless of course you expect that we have evidence that the claim “Elvis is dead” is false. And have evidence that the claim “Angels don’t wander among us” is false. And have evidence that the claim “There is no Loch Ness monster” is false. And have evidence that the claim “Aliens don’t kidnap people and anal probe them” is false.

    Because we do have multiple (better than just a single, right?) anecdotal reports of Elvis, of Angels, of Nessie, and of Alien abductions.

    An Anecdote isn’t evidence of anything other than having an anecdote. Anecdotes may give us a reason to suspect that something warrants checking. But that is about as far as they go. They are never evidence for anything else related to the anecdote.

  4. Sean said,

    Does the medical profession follow up on the swathes of anecdotal evidence that diet changes can help some diseases – I don’t see much evidence of them doing that as they still throw pills at you in the early stages of a disease. They are blind to the fact that there are ‘black swans’ everywhere!
    Young children being given ‘Methodrexate’ to treat onset of JRA (arthritis) – without any thought of advising a more natural approach – it makes me mad.
    There are also plenty of proper studies that are ignored!

  5. ChrisP said,

    Sean, modern medicine understands the role of diet and has been prominent in campaigns to improve diet. Sadly people are often their own worst enemies.

  6. I am NOT a Lab Rat! | ADD . . . and-so-much-more said,

    […] Anecdotal Evidence […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: