Whidbey Update

ASP.NET 2.0 and Visual Web Developer are coming along well, and my team is cranking towards our Beta2 release. 

My test team is in the midst of completing a full test pass right now in preparation for our zero bug bounce push for Beta2.  A full test pass is where we run and analyze every single automated test we have currently built for the product, and then also manually run every test we plan to automate before the product ships.  My team currently has 102,000 test cases which test 505,000 different test scenarios – so our full test passes tend to be fairly involved things. 

My dev team is cranking on keeping up with the incoming bug flow and driving towards what we call “ZBB” or Zero Bug Bounce – which is a point in time when we have no bugs older than 48 hours in a given milestone.  This is a project management trick we use on projects at Microsoft to focus and push teams to sustain a high rate of bug fixing over a period of time. 

My devs have been doing an amazing job the last few weeks handling the incoming flow and cranking out fixes.  This past week the test team opened 514 bugs on ASP.NET and Visual Web Developer Whidbey – which is abnormally high because of the full test pass and some automated FXCop runs that tend to generate lots of little bugs.  The dev team really responded, though, and resolved 648 bugs (130 a day which is pretty stunning) – providing us a net gain of 134 less bugs than where we were a week ago.

Our goal is to not push any bugs beyond the Beta2 milestone – which means that our goal for this ZBB push is to hit a true zero count.  We have three weeks to go…

After the ZBB date we’ll absorb the “bounce” of the ZBB push – which is basically when bugs end up being re-opened because of regressions caused by thousands of fixes being made in only a few weeks of time.  My dev team maintains a pretty tight check-in process to try and minimize these regressions (for managed code my team’s regression rate is around 4% -- meaning approximately 1 in 20 fixes introduces a new bug in the product, which is actually considered pretty good). 

All code checkins must always be peer-reviewed prior to checking in (this is even true when the most senior developer checks in).  Code changes must then run through a few hours of checkin suites that provide base-level unit test coverage of the product prior to checkin.  For ASP.NET Whidbey, these checkin suites currently yield a block-level code coverage of about 64% (meaning 64% of all code blocks are exercised and run during the tests).  We then run more exhaustive nightly tests over the product to catch issues in the latest builds of the product. 

There is a balance/trade-off between running more or less tests during checkin – the more tests the more coverage, but the more tests the longer it takes to checkin.  If we ran our entire automated test bed it would take about 12 hours for each code change prior to checkin – which is why we’ve trimmed the checkin suites to be more of a key subset that provides broad scenario coverage (and then rely on nightly and full automation runs to be comprehensive).  We then continually add/remove checkin suites during the product cycle in reaction to regressions that get missed.  

After we absorb the ZBB bounce and drive back down to a steady low bug count, we’ll start our Whidbey security push – which is where we’ll do an in-depth multi-week security analysis of the code in the product.  This is a pretty involved process which I’ll try to blog about in more depth in the future as we go through it.  Needless to say the goal at the end is to feel confident that the product is solid and ready to be attacked by thousands of hackers out there while running on the Internet.

After we complete the security push we’ll go through what we call “tell” and “ask” mode for Beta2.  This is another project management trick that we use to slow the rate of code churn in the product and force teams to be deliberate about what bugs are fixed.  Basically during these modes the “triage bar” that a bug has to hit in order to be fixed starts going up – meaning trivial or corner case bugs stop getting fixed and the focus slowly moves to only really bad “show stopper” bugs. 

During tell mode, teams within our division are still given discretion to fix any bugs they want – they just need to be prepared to present and explain why they choose the ones they did to the central division ship room.  This ends up ensuring a common bar across the division, and slows the rate of fixes and slowly brings up build quality.  You might naturally wonder how not fixing bugs could possibly bring up build quality – since this obviously seems counter intuitive.  Basically the answer lies in the regression % I talked about earlier for checkins – even with a low regression number you end up introducing new bugs in the product (and when you have a division of over 1000 developers even a low percentage regression rate can mean lots of bugs introduced a week).  By slowing the rate of checkins you slow the number of regressions – and if you focus the attention on bad bugs and add additional review process to make sure these fixes don’t introduce regressions, the quality will go up significantly.

During ask mode, teams within our division then need to ask permission of our central ship room committee before making a checkin – which adds additional brakes to slow the checkin rate.  In addition, all bugs in ask mode must go through a full nightly automation run and buddy testing (which takes at least 12 hours) to further guard against introducing problems.  Ask mode will also be the time when we’ll drive our stress passing numbers up to super-high levels, and we’ll use the low rate of checkins to find and fix pesky hard to find stress failures. 

Beta2 will provide the first “go-live” license we’ll ship with Whidbey – meaning we’ll allow any customer to build and run production applications on it.  We’ll have a big caveat warning which is that it is still not a finished product, but we won’t ship it until we feel confident that the quality level meets production deployment levels.  We’ll then deploy it on ~20 microsoft.com and MSN sites to enable us to closely monitor how it is doing, and find any corner cases that we might have missed prior to the final RTM version and ensure we have an awesome quality release.

Needless to say, I’m really looking forward to seeing what people build on top of it....

October 30th Update: For more details on how we are testing Whidbey, check out my new post here: http://weblogs.asp.net/scottgu/archive/2004/10/28/249458.aspx