Garbage in, garbage out


The other day, my friend told me that her math teacher’s favorite thing to say is: “If you take the average of a bunch of people who estimate the length of a cow, the outcome will be better than that of an expert.”  (This is false don’t believe this and walk away!)  I think that it’s implicit that the people can’t see the cow.  Being tired and considering that I have a 4 minute passing period at my high school, I believed it and moved on.  I mean after all, if you take the average of a million peoples’ guesses, the outcome should be more or less right, right?  Doesn’t the scientific method say that accuracy increases through repetition?

Wrong. (The statement about the cow, not the scientific method.)  As the title implies, the average of a bunch of garbage is still garbage.  What, do you expect all the guesses to be perfectly symmetric about the actual length of the cow?  I think my best resolution to this paradox (I think it’s a paradox maybe you aren’t so surprised) is that while accuracy does increase with more people, the accuracy increases more and more slowly and there’s a limit to how accurate you can get.  The problem is also pretty ill-defined, since after all, isn’t an expert defined to be someone who gets very very nearly the right answer?  You could call the first hit on Google for cow-length the expert answer, and maybe that helps convince you.  When you start asking questions like “how will the average of a bunch of amateur cow-guessers do” (pretend that they exist!) it gets fuzzy.

(ADDENDUM:  Another satisfying explanation is accuracy vs. precision.  People are easily swayed by many factors, and so while your results may be precise, they’re not accurate.  If I polled another million people in the same way, I would get a very close answer, but that doesn’t say anything about cows, it says something about people.  Thanks dad.)

By the way, the scientific method does work of course.  It’s not like you can ever say–well, let me just do this experiment really, really, really well and that’ll suffice.  No, you do your best in every experiment and taking the average definitely increases accuracy.

Ironically enough, this fact can be applied to the act of people guessing about this very problem.  If you ask a bunch of people what the answer is, they’ll probably say yes; especially if they’ve been swayed by public opinion.  If you ask an expert like Feynman, however, you can’t go wrong.  (By the way, Feynman’s example of this was when he was reviewing textbooks for the California Board of Education.  He found that people gave good reviews to a textbook that was blank, because people took the average of the reviews they saw and passed it on to other people.  A few good experts would be better in this scenario rather than polling tons of teachers and administrators. :D)

If you still don’t believe me (if there’s one thing to take away here it’s to always question a popular opinion) feel free to post your best argument for why it should be better in the comments section.  And even better, if you don’t believe me try doing the experiment yourself 😉

Advertisements

7 responses to this post.

  1. “The average of a bunch of garbage is still garbage.” Hehehe, indeed. Cool post!

    Reply

  2. Posted by hm29168 on January 25, 2011 at 7:20 pm

    The original saying was based on the weighing of an ox as witnessed by Francis Galton, an anecdote which was more recently published in the book: The Wisdom of Crowds.

    http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds#cite_note-0

    I haven’t read any of it and have only heard the story from outside sources, but it seems like a good read to check into and see if your theory holds up.

    Reply

  3. Thanks! Well the wikipedia article never claims that the crowd is better than an expert. I agree that the average is way better than the guess of a random person in the crowd! Also while I can’t mathematically disprove this without a better model, it’s pretty well-established to be true.

    Reply

  4. Ah, another issue is whether people are looking at the cow or not. I was assuming that people weren’t looking. In the wikipedia article people are looking at the ox and guessing its weight, which makes their guesses a lot more reasonable.
    So I maintain that if you ask the public something they have no idea about, no matter how many people you poll it wont be as accurate as someone who knows it pretty closely (in other words, it wont be that accurate). If people have some idea, things may change.

    Reply

  5. Posted by Kenny Uncle on February 4, 2011 at 11:44 pm

    Great post, Meena! This is an interesting hypothesis about averaging guesses by people about a subject on which they are not experts. Depending on the subject, people tend to guess “wrong” in predictable ways, especially with probability. For example, people tend to overestimate the risk of flying, and underestimate the risk of driving a car. I’m not sure if a larger sample size would correct for this. As far as guessing the weight or size of an ox, I wonder if it would make a difference if it was their ox, or their neighbor’s ox they were guessing about. That would be interesting to see. But now you have me thinking about this!

    Reply

    • Posted by Meena on February 6, 2011 at 1:11 pm

      Yah, trends are key to why this doesn’t work! That kind of error doesn’t decrease with more people. Also if it was someone’s own ox or their neighbor’s they might be more knowledgeable, but you’re right there isn’t a clear point where people are knowledgeable enough so that taking the average “works.” I would love to do a more detailed model of this where “experts” guess randomly within a close interval of the actual answer, and “amateurs” guess in a wider interval and see where the point is where the average of the amateurs win.

      Reply

  6. Posted by Tom Ato on March 20, 2012 at 10:43 pm

    Yeah. There was a contest at one of Hunter’s fairs a few years back to guess how many jelly beans were in a jar. There were already a few dozen votes, so I wrote them all down, went to the computer lab, and averaged them. I was sure I’d win. I didn’t win. 😥

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: