Why Program Increment Estimates Miss: Hidden Variables for Planning

Bill Nichols

What is a credible range of dates for when the work in a product increment might finish? Software development work includes many sources of uncertainty, including the size of the work product, developer work effort, complexity, and waits that are sometimes predictable. We found that story point counts often don’t add information that is useful for projecting durations to completion, and wanted to understand whether the problem was in estimation or execution. To investigate, we compared data to determine the correlation between estimates of stories, the actual story direct time, and the calendar durations for story completion, in the hope that this would help to better predict credible completion ranges.

https://softwareexcellencealliance.org/why-program-increment-estimates-miss-hidden-variables-for-planning/

David Tuma

In real life, wait time also occurs when someone has multiple tasks on their plate and they're multi tasking. They could be working on Work Item A, and need to put it on hold while they switch to Work Item B. That could happen multiple times per day. Does your definition of wait time include or exclude that concept?

Bill Nichols

David Tuma Yes, that would be captured in the way we defined wait times. They are broadly defined as any gaps between the work sessions for a particular issue.

For automated data gathering, the practical implication is that you would need some mechanism for determining what task is actually being worked on. We'd really like to find cues for this inside the data. For example, we might examine the Git repository or other tools to see what is actually being changed, and what other tasks might be waiting.

Jeff Schwalb

Would wait time include stepping away from focused work on a task to talk to peers, managers, or stakeholders about various project issues?

Bill Nichols

Jeff Schwalb Yes, that is correct - coordination is important and valuable, but it is still something that pulls us from direct task work. Most people significantly underestimate how large a factor it is.

In regards to our source data, we have to trust that the teams were recording interruptions when they weren't specifically working on a task in their plan. But as we've seen, the TSP data appears to be surprisingly good. Tapajit independently showed that about 95% of the tasks had essentially accurate time data. The 5% noise level was consistent with my earlier findings based on Benford distributions.

Attendee

Brian Gallagher: When using TSP, it seems you measure wait time as part of the process. For teams that aren't using TSP (like typical government DevSecOps), what can they do to better capture that?

Bill Nichols

Attendee And that's the real money question.

Let's start with the first part. TSP didn't specifically set out to measure wait time. Instead, it encouraged engineers to capture stopwatch time spent on each of their work items. Wait times can be extracted from this by comparing timestamps on consecutive work sessions.

So for modern teams, how would we get something that approximates the wait time? And how much of that wait time do we need to capture? These are open questions that we need to investigate.

One thing we could possibly do is look at other tools in the chain and try to triangulate, to determine which work items are on hold. It might be a piece of code waiting for inspections to come in, or it might be a piece of code that a developer put down while he or she worked on something else.

One possibility would be some kind of lightweight toggle for an individual to indicate whether an item was on hold or in progress. Another possibility might be retrospectively looking at git logs to determine that a piece of work was sitting idle for a certain amount of time. These are problems that we haven't solved yet; but the data indicates this could help us close the information loop.

David Tuma

Bill Nichols A little nugget that might help in that research: Dr. Brad Hodgins, in his dissertation, found no statistical difference in the success of teams that recorded time using a stopwatch, and teams that backfilled time at the end of the day. So it might make a huge difference if your data stream allows for human-in-the-loop feedback at least once a day.

Bill Nichols

David Tuma Thank you - that tracks with what we're seeing. We might infer that as long as data is collected with some underlying consistency, then you have a proxy you can correct with some sort of calibration. It's the intermittent noise that causes the big issues, because they stem from a bunch of different special causes.

That is one of the problems we've seen in using flow metrics for prediction. The moment you start getting variations - like changes in the amount of work in progress - the data gets noisy. Then you lose the ability to make strong projections on important questions, like which stories are going to get done.

Attendee

Yoshihiro Akiyama: What does your analysis suggest for the improvement of estimation and scheduling for TSP teams? And how do the sources of noise you found impact the likelihood of success?

Bill Nichols

Attendee That could be an interesting study: to do a sensitivity analysis on the data outliers. That would help you measure the degree to which outliers are limiting your ability to control your program. Do they affect things like the credible range? Or will they significantly change your 90% confidence level? In principle, that's something we should be able to do if we understand the underlying distributions.

One thing that does stand out is the TSP teams did a surprisingly good job with the estimates. The unfortunately honest truth is that we don't really know how well agile teams are estimating - because the data we have on individual stories is incomplete. The noise is too high in the relationship between story points, calendar duration, and actual effort. These poor correlations make them a poor measure even of throughput, which is nothing more than an average. Without estimation feedback, you cannot get any distributional insight.

Velocity and cycle time do have their uses. But in this research we've quantitatively demonstrated the limits of their ability to assist with predictive questions. And found inspiration on new avenues to study to close that gap.