Overconfidence gets a bad rap among economists, and, it seems, for good reason. Overconfidence leads workers to overinvest in the companies they work for (putting both their employment and their retirement savings at risk if things go badly for their employer); it leads corporate leaders to propose and carry out foolish mergers based on hype and unrealistic expectations (remember AOL-TimeWarner?); it leads cocky individual investors with an addiction to CNBC to day trade themselves into the poorhouse.
But are there times when overconfidence can be a good thing – a seemingly unreasonable but in fact highly rational strategy that improves your odds for success?
That’s the thesis of a new paper from economist Peter Schanbacher of the University of Konstanz in Germany. Schanbacher starts with a fact that’s been well-documented: When asked empirical questions, people tend to overestimate their chances of getting the answer right. In a paper bristling with equations, Schanbacher shows that such overconfidence can actually be a rational choice – especially when people have no idea what the answer is. He argues that with “hard tasks the estimation risk is large. Being biased can reduce variance and improve the estimation.”
On his blog Stumbling and Mumbling, economist-turned-journalist Chris Dillow explains what Schanbacher is getting at in simpler terms.
Imagine a coin is biased to land on heads 80% of the time and tails 20%. Predict the next 10 tosses.
A good answer to this question would simply be 10 heads. This will be more likely to be right than any random-looking sequence of eight heads and two tails. But this answer is biased – it overpredicts heads.
In other words, overestimating the total number of heads that will come up – by guessing heads every time – will enable you to predict more accurately than if you guessed heads 80% of the time. As Dillow goes on to note:
A similar problem faces the ignorant individual when asked about his chances of solving a problem correctly. The same lack of expertise that makes someone ignorant can also mean there is large variance around his estimate of his competence. One rational solution to this, says, Schanbacher, is for him to give a biased answer – to over-estimate his competence – just as a biased answer to our coin-toss question is reasonable.
In the real world, there’s considerable evidence that in certain situations overconfidence can be a very good thing indeed. Overconfident lotharios who ask out every woman they see, and who aren’t fazed when they get turned down, are over time likely to do better than wallflowers who never ask anyone out.
Similarly, it takes a certain kind of delusional confidence for entrepreneurs to push their visions and stick with their companies when everything looks bleakest. Entrepreneurs may wildly overestimate their chances of success, but only by convincing themselves they’re going to succeed are they able to motivate themselves to work hard enough to give their business a real chance of success.
(MORE: Home Test for HIV)
This sort of confidence is contagious: The charismatic Steve Jobs’ was said to generate a “reality distortion field” that enabled him to infect all those around him with his own brand of overconfidence. If you’re reading this on an iPad, you owe it to Jobs’ faith in his own visions.
This isn’t a new notion. Back in 1896, in a famous paper called “The Will to Believe,” philosopher William James argued that irrational faith was often necessary to give people the confidence they needed to succeed. When you don’t know the answer, but need to have an answer, you need to make a leap of faith – and leap as if your life depended on it. (Sometimes, it does.) James illustrated this logic with a somewhat melodramatic quotation from English judge and man of letters Fitzjames Stephen:
We stand on a mountain pass in the midst of whirling snow and blinding mist through which we get glimpses now and then of paths which may be deceptive. If we stand still we shall be frozen to death. If we take the wrong road we shall be dashed to pieces. We do not certainly know whether there is any right one.
So what did James suggest we should do? Pick a path, and start marching, as if the path you’ve chosen is the only one that will save you.
I’m pretty confident that’s good advice.