Effectiveness of a product stream
I’ve pulled together some data for the first year of a product stream we created and plotted it as charts for throughput, rework and effectiveness.
The first chart shows the weekly rework. I’ve talked about rework previously so I won’t cover it again here. The blue line indicates the remaining technical debt and the blue bars the repaid technical debt. The pink line indicates the remaining defects and the pink bar the fixed defects. Week by week, it can be seen that defects were fixed as soon as they were discovered to reduce the remaining defect count to zero. Also the technical team continuously repaid technical debt to keep the remaining amount of rework small.
The second chart shows the amount of throughput every week.
The peak at week ending 18/12, without any throughput during the 8 preceding weeks, demonstrates a flush of inventory amounting to 140 odd stories and corresponds to the alpha release of the product. In the rework chart, a small increase in fixed defects can be observed during the same week. Inventory again builds up for two weeks, as improvements are made to the automated deployment system, until the next peak at week ending 15/01. At which point 42 completed stories are flushed. Releases then occurred every week and while some variation is observed the throughput remained stable.
To improve the performance of the product for the beta release during week ending 26/02, 7 technical debt cards were completed. As the system experienced more rigorous use by editorial users, 8 defects were fixed plus a further 6 the following week due to increased traffic. The official launch was completed in the week ending 18/03. During that week some data inconsistencies were encountered following migration from the old content management system, which resulted in 6 defects. In response to traffic loads the load balancers were tuned with 5 technical debt cards. This effort continued the following week with a further 8 technical debt cards and 5 fixed defects as traffic increased to approximately 180 million page impressions and 3.7 million unique users per month with an average page weight of 2Mbytes. Further peaks in technical debt of 8, 7 and 10 can be seen during the weeks ending 06/05, 17/06 and 24/06, respectively. This work concentrated on the expansion of the product with reconfiguration of the production environment to support additional channels.
In the Lean manufacturing world there’s a measurement called First-Time-Through (FTT), which monitors whether a cell is making products right the first time. It’s a measure of the effectiveness of the cell’s standardized work and shows the percentage of product made without any need for rework or scrap.
FTT = ( Total units processed – Rejects or Reworks ) / Total units processed
If the standardized work is adhered to, the product will be made right first time and FTT will be 100%. However, flawed materials, faulty components and operator error all contribute to rework and scrap.
I was interested to read about FTT because I’ve been thinking for a while now about the effectiveness of software teams. I’ve long considered an effective team as one that is able to sustain throughput (i.e. the number of cards released to production that deliver value) while fixing defects immediately and repaying technical debt to keep the amount of rework small.
I consider technical debt and defects to be rework, and technical debt to be a natural byproduct of software development. It stems from earlier decisions, based on what we knew at the time, and requires attention later when the system has outgrown the outcomes of those decisions. It is necessary rework that keeps the emerging design relevant and the software healthy and habitable, reducing risks and medium to long-term costs. Defects are basically mistakes. They happen. How we create software determines whether we have a small and manageable amount of rework or a crippling amount of rework. If we’re responsible, skilled and bake quality into code we can minimize rework to technical debt and occasional defects. If we’re irresponsible and cut corners, or we’re rubbish and write crap code, then rework can become so large that the only viable option is to cancel or start again.
Technical debt requires careful management and continuous investment while defects should be fixed as soon as they are found. A proportion of a team’s capacity is therefore always expended doing an amount of rework. That’s a good thing providing:
- the completed rework is small compared to the throughput so that capacity mostly focuses on value demand, and
- the completed rework is enough to keep the remaining rework small compared to the throughput, thus minimizing further failure demand.
(Throughput excludes repaid technical debt and fixed defects that went live).
On a weekly basis then, the throughput in relation to the remaining technical debt and defects might be a useful measure of a team’s effectiveness.
The effectiveness of the product stream is defined as:
Effectiveness = ( Throughput – Rework ) / Throughput
Throughput = number of cards released to production (excluding completed rework)
Rework = the number of technical debt and defect cards in inventory and work-in-process
The final chart shows the weekly effectiveness of the product stream.
The lows at weeks ending 29/01, 25/03 and 01/04 can be attributed to marked dips in throughput. At 29/01, 12 cards were queued as inventory whereas a small increase in the amount of remaining rework was present at 25/03 and 01/04. Clearly the product stream is most effective when the completed rework was small compared to the throughput and was enough to keep the remaining rework small compared to the throughput.