As is pointed out in the Wikipedia article on anecdotal evidence, such evidence is considered to be dubious support of a claim. Not only because it may be unrepresentative of a “typical” experience, but also because anecdotal evidence consists of casual observations rather than rigorous analysis. It is inevitable that casual observations will be subject to bias. Rigorous analysis is subject to bias too – but less so. Attempts are at least made to reduce the influence of biases.
There is a useful list on Wikipedia of cognitive biases. Anecdotal evidence may be subject to, for example, confirmation bias – you might remember the times when a treatment seemingly worked and forget the times when it didn’t, or you might interpret new information as being favourable to your preconceptions when it actually isn’t. And that is just one of the 85 cognitive biases listed on that page. There are others that might be relevant too, for example illusory correlation – inaccurately perceiving a relationship between two unrelated events.
You might think that you aren’t prone to cognitive biases or are at least less prone than others. This is not an unusual viewpoint; mistaken, but not unusual. Lots of people think that others are more prone to cognitive bias than them. This is the cognitive bias of failing to compensate for one’s own cognitive biases – the bias blind spot.
We may selectively remember, misinterpret new information, or mistakenly assume that correlation equals causation. And it might be harder for us to spot when we do this than when others do it. It is easy to be mistaken, but not so easy to realise that we are.
A tale of two anecdotes
I tried two treatments, years apart, for similar problems (involving leg pain). They both ‘worked’.
Having had problems with leg pain that felt like it might be cramp, I tried some magnesium pills one evening and the next morning felt hardly any pain. This was a great improvement. Of course, I don’t know whether the improvement was due to the magnesium, due to the act of taking a pill (any pill), or whether I would have felt the improvement the next morning regardless… which is why this anecdote tells me nothing of value about whether magnesium pills can help with leg pain.
If the improvement that I felt could be due to the magnesium in the pill, to placebo effects, or to regression to the mean and I have made no effort to account for the placebo or regression to the mean possibilities I cannot rule them out and I cannot possibly know which of the three possible explanations might be the right one (or whether there is another explanation that I haven’t even considered). There may have been a correlation between taking the pills and perceiving an improvement, but I cannot know whether the pills caused the perceived improvement.
On another occasion when I had back pain I tried pain-killing gel. The gel was even more impressive than the magnesium pills: I bought a tube and it worked straight away… before I even used it. Which is ridiculous, of course. A treatment cannot possibly work before you’ve applied it. In this case I can say for certain that the treatment itself was not responsible for the perceived improvement. If the perceived improvement had come shortly afterwards (as it did when I tried the magnesium pills), I might have assumed it was due to the treatment. I would have been mistaken.
It would be nice if we were able to tell what worked and what didn’t simply by trying it. Sadly, we can’t. Here, I wrote a rough guide to evidence-based medicine. EBM might not be perfect but carefully gathered evidence is probably a more reliable guide to what works than “try it and see”, at least when it comes to medical treatment.
Edit, 25th July 2012
As pointed out in the first comment below, anecdotes aren’t always ignorable. I can safely ignore my anecdotes about trying to treat leg pain, but if after trying a remedy my right arm turned green and swelled to twice its normal size you can bet I’d report it to my doctor and tell him which treatment(s) I’d tried. Such an anecdote wouldn’t prove that the treatment had caused the symptoms but in the absence of a more plausible explanation it would be reasonable to investigate further.