Hitting Code Complete

This weekend has been a pretty busy one for my team (ASP.NET and the Visual Studio web tools) as we drive towards hitting code complete (CC) for our current V2 feature milestone.  Our target date for hitting CC was this past Friday (the 14th) -- which really means late Sunday night.  About half of my team has been here at somepoint this weekend finishing up -- we look to be on track for hitting our goal.

Clarrification: Note that the Code Complete we hit this past weekend wasn't for the entire product -- just the particular coding milestone we are currently in (see more details below).

One of the frequent questions people ask me is what the methodology is at Microsoft for developing software.  The process varies by team and division, but are usually organized around several fixed feature development milestones (typically 5-7 milestones for a V1 product, 2-4 milestones for a product upgrade).

Feature milestones are usally split up into 3 components: planning, coding and stabilization.  During the planning phase the PM team drives finalizing the milestone feature list, and complete detailed design specs for each feature.  During the coding phase (usually 8 weeks) the development team implements the features and complete checkin suites to provide basic verification coverage of them.  During the stabilization phase (usually 6-10 weeks) the QA team works to implement automation tests that provide detailed coverage of the milestone features -- and the dev team fixes the corresponding bugs that are found.

Our goal when exiting a milestone is to leave it with a stable product that contains a compelling feature set, and that has enough automation tests written against it to ensure that we can maintain "self-host" quality coverage of the features on a nightly basis going forward. 

The automation of these tests is critical.  Because we typically localize products into 8-32 languages (VS.NET is localized into 8 languages, ASP.NET and the .NET Framework into all 30+ Windows languages) we can't rely on manual testing.  It is simply not possible to manually run tens of thousands of tests 32 different times (one for each language) every day.  Instead our QA team builds all tests to plug into an automation framework (affectionately named "MadDog").  The automation framework is then integrated into our test lab -- and allows a tester to kick off a test run in a matter of minutes. 

A tester -- sitting in their office -- can configure the operating system and hardware they want to use (for example: Windows Server 2000, Advanced Server Edition on a 2 processor hardware configuration with a German language build) as well as the test suite they want to run (for example: all session state tests).  The automation framework with then find a hardware machine that meets the specification selected, reformat and re-image the operating system for the choice selected, and then execute the test suite run.  All pass/fail results can then be viewed by the tester on their own machine with their office (fairly fancy -- we are quire proud of this testing framework).  On my team the test organization typically runs 20% of their automated tests every day (varying the system configuration).  We then do a full automation run of the remaining test cases every 2 weeks.

After we finish our feature milestones, we'll typically have a long stabilization period (8-14 extra weeks) and then begin our formal beta process.  The goal with the betas are two fold -- 1) help find bugs, and 2) start getting people excited about the release. 

Most products at Microsoft these days have either 2 or 3 widespread beta releases.  As we get close to the end, we'll also usually cut a few release candidate (RC) releases that we hand out to a small list of our most active beta customers to sign-off that all their bugs are fixed and that the release is ready.  Once we feel all the release criteria are met, we'll declare the product RTM (released to manufacturing) -- after which the product will be formally launched and released to customers.

And of course after a product is released, the whole process repeats itself again (2-4 new feature milestones, 2-3 betas, and RTM again)....