Illusions

I am still reading Daniel Kahneman's Thinking, Fast and Slow, and I came across an interesting quote

People can maintain an unshakable faith in any proposition, however absurd, when they are sustained by a community of like-minded believers. 

He used the example of stock traders, who think they are successful in picking stocks even though evidence points otherwise. Two out of three mutual funds underperform the overall market in any given year. However stock traders still believe in their approach.

I thought of the example of project management. I went to my first PMI conference in 1999. I was surrounded by like-minded believers in the waterfall methodology for project management, even though evidence such as the Chaos Report showed that our approach was not effective.

Will we see the same effect with agile approaches? Will we fool ourselves into believing we are successful even if the evidence suggests otherwise? Maybe not.

Philip Tatlock did research for his 2005 book Expert Political Judgment: How Good Is It? How Can We Know? Tatlock interview 284 people covering 80,000 predictions and even though these people were "experts" their results were worse then if these predictions were simply assigned probabilities. To compound the effect, the experts were very reluctant to admit that they were even wrong.

Kahneman sums it up pretty nicely. He says that the "errors of prediction are inevitable because the world is unpredictable." (p. 220) He goes on to say that short-term trends can be predicted but not longer term ones; though even the dividing line between the two is unpredictable. The implication to planning is clear; keep the planning horizon short and don't fool yourself into believing you are better than you really are.