Except in the most clearly defined environments, the bottom line is that estimates depend upon people. The majority of estimates are produced not by trained estimators but by the doers – the people who will be doing the work and should have a good idea of how long it will take and how much it will cost.
These people tend to fall into one of four categories:
The mere fact that so many tasks end up taking longer than was first thought indicates that many people are compulsive optimists, especially technical people who have an enthusiasm for the work and a rosy view of their own abilities.
Equally problematic to your estimates are the born cynics. Maybe they have been made to suffer for optimistic estimates in the past; maybe they have been in the same job too long or are just defensive about their own abilities.
Far and away the most dangerous animal on your project is the inconsistent estimator. This is probably someone who sees the whole project management process as bureaucracy that just gets in the way of doing the real job. When you ask their opinion on a task they are likely to come with up any old number just to make you go away and leave them alone.
The consistently accurate estimator is a very rare beast indeed, some would say mythical. If you find one, chain them to the desk so they can't get away.
This may sound flippant, but the problem is a very real one. There is no substitute for experience but following these ‘Golden Rules’ may help you along the way:
Break down work to a point where the estimator and/or an available expert can visualise the work being done
Estimate for the person who will actually be doing the work, not the estimator or the expert
Obtain alternative estimates, using the same method or alternative methods
Feedback alternative estimates to sources, seeking consensus
Issue estimates to operators ensuring that the working method is the one used in the estimate
Watch for changes in method which may affect estimates
Obtain immediate, equivalent feedback on performance versus estimates
Adjust estimates in light of feedback
Review at end of project for possible adjustments to estimating methods used