Estimating software

I touch the third rail of technology delivery

Photo by Aaron Lefler on Unsplash

Today, there was supposed to be an app update article for paid subscribers. But I’ve been thinking a lot about the process of software estimation lately and felt this article was more interesting to write on this fine Wednesday. We will be back to our regularly scheduled content next week.

---

We have a problem—a major problem.

All around the world, interested parties are inquiring about when they can have some technology thing that they want. And all around the world, technologists are gripping pens in a blind rage that someone would dare ask that question.

Estimating software can be defined by two truths:

1. It is hard.

2. No one is good at it.

And yet, estimates are given out every single day as if we all know what we are doing.

So, what are we to do?

Acknowledging the difference between an estimate and a commitment

The first step to building a better estimation process is to be better at distinguishing between “an estimate” and “a commitment.” It should be obvious that asking for an estimate means you are getting an estimate, but my theory is that most of the people asking are only using the word “estimate” because that’s the word they’ve heard their peers use when talking to technologists about projects. What they really want to know is when they can pencil in delivery.

They want a commitment.

Maybe it sounds less confrontational to ask for an estimate. Maybe they are unsure about the work and want to leave the door open to change their minds on scope. Maybe, again, that is just how they think you ask for these things.

So the improvement here is to be clear about what an estimate is and to commit to something far smaller: the date when the estimate will be delivered along with a confidence interval based on the known scope today.

Research on project estimation produced what is known as the “cone of uncertainty” which demonstrates the wide range that programs can have in size (up or down) at various milestones within the program. When we have a request for an estimate for work, we are typically on the far left of the cone (called “Feasibility” here). This is the realm of “semi-educated guessing” and any estimate given should take this into account.

Aroonvatanaporn, Pongtip & Sinthop, Chatchai & Boehm, Barry. (2010). Reducing estimation uncertainty with continuous assessment: tracking the "cone of uncertainty". 337-340. 10.1145/1858996.1859065.

Instead of “We are estimating a 3-month build” for the feature, the estimate should follow this pattern:

Given the initial understanding of the problem that you’re trying to solve, I am X% confident that delivery will be between Y and Z. We can commit to a revised estimate with a higher degree of confidence in N days|weeks|etc.

Notice how this changes the calculus of the problem? Both parties are getting what they want. The estimator can deliver a range that accounts for some level of uncertainty. The requestor gets a commitment in the form of a target date that is much closer to now than the completed work would be. Both get the feeling of having a conversation where participants in the conversation are speaking the same language.

Are we estimating effort time or calendar time?

Similar to our estimate vs commitment problem, we also need to be clear about effort and calendars. I’ve seen plenty of estimates that call for “X development weeks.” The requestor hears “Whatever today is, add X weeks and the work is delivered. Hooray!”

The estimator, though, probably meant this as “assuming I have access to the environments, have been allow-listed to call the endpoints I need, have access to the data I need for testing, it will be 4 weeks of development time. This doesn’t include any change control processes I have to navigate once development is complete.”

The problem is not with the assumptions, it’s that they weren’t stated in the estimate.

Unpacking “effort” further, does “1 week to develop” mean “40 hours of development spread across 2 weeks because I’m in so many meetings that I can’t do more than 20 hours of work a week?”

Fair question, isn’t it?

I think the requestor is in the right here to get an all-in estimate. Assumptions must be included, but we cannot expect someone operating outside of our technology bubble to understand all of the extra stuff that goes into the delivery of a request.

Do I estimate based on my worst-performing engineer? Or an average engineer? Or who I think will be assigned to the work?

Amazon had an interesting section within the engineering performance guide that called out the “SDE 1 Coding Bar.” This was an assumption that development skills were pegged to what an “SDE 1” could do. Meaning there was a base-level understanding of code structure, algorithms, development philosophies, data structures, etc. across all engineers in the organization. Differentiation would come down to the application of skills, not the skills themselves.

I found that section very interesting because it helped me understand Amazon's philosophy on fungibility amongst software engineers. I think it could also be used as part of the estimation process for Amazon’s delivery by asking the question “How long would this take an SDE 1 to complete?”

The challenge I have with this approach is that it doesn’t then account for what happens when an SDE 2 or an SDE 3 takes on the work. Should I give an estimate based on an SDE 1 when I know I have an SDE 2 picking up the work? What if your organization doesn’t define the “SDE 1 Coding Bar” and there is significant variability across the engineers on the teams?

My take: Use the average engineer time to deliver on the team doing the work if the request is going to be delivered by multiple engineers. If it is going to be assigned to an individual, estimate the work based on the individual that might be assigned to the request.

My estimate was way off and we are never making the date our partners are tracking toward

A good Cabernet Sauvignon can be aged for 10-30 years from its vintage. It will smooth out and develop deeper, more complete flavors while reinforcing the concept of “aging like fine wine.”

Bad news, on the other hand, is like being attacked by a hungry grizzly bear. It is not great and only gets worse the longer it lasts.

If you estimate software, you will be wrong a bunch of times. The best thing that you can do is build trust with your partners before you need it and avoid letting the bad news linger. Bring an honest assessment of the issues being faced and a plan to get back to green.

Some people will not react well. That’s just life. Nothing you do will prevent a negative reaction should the receiving party want to give one. Focus on doing your best and being upfront about the situation.

Okay, how do I actually estimate this thing?

I’d love to quip “It depends” and close out the article, but I can’t bring myself to do that.

Estimation is based on experience with similar work. If you are adding a new set of UIs to an existing mobile app, how long did it take to deliver screens of similar complexity previously? If you are building a new API from scratch, ask around your organization about other APIs built from scratch.

Don’t start from zero every time.

There is no magic formula in this process. You will need to exercise judgment and you will need to validate your assumptions over time. By continuously learning and refining the estimates you will move toward a higher degree of certainty on the target date.

If you are completely lost, look for a senior engineer in your organization and buy them a coffee.

I hope this guide helps a bit. If there are questions that you feel went unanswered, please drop them in the comment section.