This story starts from a time not so long ago. I was meant to build a data warehouse, but this time with a new technology and from unfamiliar data sources. I split the project to bite size tasks and gave an estimate for each task. I then summed up the individual task estimates and announced my total estimate. Fast forward two months and it was clear than my estimate was shit.

It’s been bugging me ever since…

Where Did I Go Wrong?

I spent many sleepless nights wondering all the mistakes I did with the estimate. My mistakes were:

1. Not making a clear separation between definition and execution phases
2. Giving absolute estimates without limitations instead of estimate ranges
3. Not giving a probability indicating how sure I was with my estimates
4. Allowing the client to affect my estimates

After some reflection, here’s a better approach for work estimates.

1. Separate Definition from Execution

Working with a familiar client, it’s easy to resort to little shortcuts like skipping proper definition phase and just rushing into doing the work. It’s also not easy to make the mental distinction between continuous work and a new project for same client.

Basically any new scope that takes several days should be a new project.

First thing to do is to use a project plan template and the process for better definitions. It’s not possible to give accurate estimates without detailed requirements.

When I gave the shitty estimate, I only had a shallow knowledge of the requirements. I should have actually looked at the data to see how complex it was.

There are two big questions marks in business intelligence projects that can cause significant delays.

1. Data complexity
2. Data availability

Simple data can be modeled in a day, complex data might take 50 days of work. Same goes with API’s. Sometimes they are documented well and easy to use, sometimes a lot of reverse engineering is required.

Only way to know, is to dig into the data source.

Good understanding results to better work estimates, but it takes a bit more time up front.

2. Absolute Estimates vs. Ranges

Rarely I have seen estimates that were too large. I recommend making an estimate and then setting a range for the estimate by adding 20 - 30 % on top of the original one. Ranges also highlight the fact that we are talking about estimates instead of fixed values.

Not everyone will agree with me on this one. Another alternative would be to only show the max value to the client and create a positive surprise by beating the estimate.

I like realism and I don’t like surprises. You might be different.

3. Probability for The Estimate to Be True

When there’s an estimate, there needs to be a probability for that estimate. Think about the weather forecast, it’s completely different to forecast considerable amount of rain with 5 % probability or 80 % probability. In terms of project work most people are usually satisfied with just the amount of work.

How often have you received a probability with work amount estimates?

Whenever there’s a forecast, there needs to be a probability to set realistic expectations.

Edit: After publishing this story I got some feedback on probabilities. It’s a lot better to clearly state the terms and conditions behind the estimate. If things don’t go as planned, it’s a lot easier to say that this is because term XYZ was not true instead of using the probability as basis or justification.

4. Deny Customer Influence to Estimates

Most of Power BI or Data Platform projects are sold with time and material pricing simply meaning with an hourly rate.

I have couple times met customers that start negotiating the work time estimates. Sometimes I have even reduced my estimates due to this. Unfortunately, these behaviors are just stupidity. The hours will be billed regardless of the estimate, so it’s best to have realistic expectations.

• If you have a client trying to negotiate the estimates down, don’t let that happen
• If you are client trying to negotiate estimates down, just don’t. Focus your negotiations on the price

The better the estimates, the happier everyone is.

End

Without a crystal ball, all estimates are hard. Especially if you are trying to create something for the first time. There always room for improvement.

If it’s not possible to get enough details beforehand, it’s always good to state that the estimates might change during the definition phase.