Experts versus Equations

October 19, 2008 at 4:15 pm (Miscellaneous) (, , , , )

Are expert predictions more accurate than predictions generated by computer analysis of data? I’ve been reading Ian Ayres’ book Super Crunchers, which looks at this question in a chapter titled “Experts versus Equations”.

Examples of experts versus equations are given throughout the book, including a regression equation that can predict the future price of wine better than world-famous wine experts, some nifty number crunching that can more accurately pick successful baseball players than can professional scouts and a guy named Chris Snijders taking on experienced buyers (“purchasing professionals”), which was published in Purchasing and Supply Management 191 (2003). Abstract; abstract of a follow-up study.

[Aside: The book also looks at testing social policy (I wrote about recidivism and sentencing recently) in the US and there is also work being done on testing social policy in the UK. There is a link at the bottom of this page to a paper about RCTs in social intervention in the UK and in 2007 Ben Goldacre wrote about a Professor and statistician named Sheila Bird who wished to do controlled trials on social policy.]

It is possible to test whether experts perform better than number crunching machines by looking at their comparative success in making predictions. Paul Meehl was one who looked at clinical versus statistical predictions and his early conclusion was that “the literature strongly favors statistical prediction”. William Grove considers that this conclusion “has stood up extremely well” in this article from the Journal of Clinical Psychology – Clinical Versus Statistical Prediction: The Contribution of Paul E. Meehl. [PDF]

I recall John Briffa asking me during a seemed to be an appeal to authority by a man who considers himself to be an expert on the basis of his own experiences with patients – he also asked if my interest in medicine was an essentially ‘academic’ pursuit, and is fond of writing about what he finds to work “in practice”. It is not uncommon in alternative medicine to claim that the statistics may well show one thing but I, the practitioner, find the situation to be quite different. Homeopathy being an obvious example. I’m always interested when people claim to know things ‘from experience’ or to know that something works ‘in practice’ as I wonder what makes this experience different from, and more reliable than, simple anecdote. Presumably what it means is that anecdata from an expert is more valuable than anecdata from non-experts. Is it more valuable than statistical evidence though?

Experts are not quite as expert as they believe themselves to be. Ayres relates a conversation with Theodore Ruger in Super Crunchers where Ruger is asked whether he could ‘beat the machine’ and he begins by saying “I should be able to”, before correcting himself. He had momentarily fallen into the same trap as other experts in overestimating his ability to make predictions in comparison to a statistical model. Ruger wrote an essay in Columbia Law Review that showed (in terms of a Supreme Courts decisions) that “The model predicted 75% of the court’s affirm/reverse decisions correctly, while the experts collectively got 59.1% right”. If a guy who knows that the model tends to perform better than the man (in terms of prediction) can fall into this trap then it is probably an easy trap to fall into and I shouldn’t be surprised when I see other experts do likewise.

Isabel, Grove on Meehl, In Praise of Clinical Judgement, RCTs of social interventions. [All PDFs]

Snapshot of Ruger’s Essay. [Google Scholar’s page].

4 Comments

  1. perceval said,

    i am reading the book, too, and it’s really a false dichotomy. for statistics as for computers, you have garbage in, garbage out. selecting the variables, taking the measurements, choosing the model, setting up the calculations all require background knowledge – and then there’s the danger of over interpreting the results

  2. jdc325 said,

    To be fair to Ayres, he does point out that, for starters: someone needs to come up with a hypothesis to test; there are areas where there is a lack of data and experts will be necessary as there will be no equation that will be useful; the best results in most cases will probably come from some kind of mixture of expertise and number crunching. If that doesn’t come across in my post it’s probably because it’s badly written. I was trying to drive home the point that experts aren’t as expert as they think – and we should beware the man who says “you can’t quantify what I do”.

  3. Robert Fischer said,

    Meehl thesis was that an algorithym or set of rules would be more reliable than an individual because the individual would be inconsistent in application of the methods, where as the algorithym would consistently come to the same conclusion. While it is ceratinly true that someone needs to crunch the numbers to come up with the rules, that is exactly Meehls point. It is better for the expert to use his judgment to come up with rules for the purposes of making the prediction than for the expert to make the prediction.

    Hillel Einhorn did some very intersting work with a group of pathologists. He asked them to review some slides and draw some conclusions about the slides based on 9 differnt factors. Einhorn used the ratings of the pathologists to create a prediction model that predicted the life expectancy of the patients with greater accuracy than the pathologists. It is all described in Faust’s book The Limits of Scientific Reasoning.

  4. jdc325 said,

    “It is better for the expert to use his judgment to come up with rules for the purposes of making the prediction than for the expert to make the prediction.”
    Well put Robert.

    Thanks for the notes on Hillel Einhorn too – I hadn’t heard about that work before.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: