Fear and Loathing - Archives - 2007

The World Doesn't Need Yet Another .NET Testing Framework

Saturday, September 22, 2007

1 Comment

I have a lot of respect for Jim Newkirk but I was reading Roy Osheroves blog entry about the new XUnit.NET framework that has just been released and I'm scratching my head wondering why. While it has been a few years since the original NUnit framework was released, .NET has evolved and continues to do so, Jim (and others) felt it was time to clean the slate. What bothers me are some of the same things mentioned in Roy's blog.

I do agree with changes they've done on some parts. For example, I'm now in the camp that [SetUp] and [TearDown] are evil and need to be abolished. Easy enough. I tell people not to use them and if I really had to I could create a custom FxCop rule or something to prevent check-ins with them. However in XUnit.NET they just don't exist. The problem I have with XUnit.NET is that other things have changed, and quite drastically (after downloading and playing with it for a couple of hours). [ExpectedException] is gone in favor of the more JUnit like Assert.Throws() approach. This is fine, but it means more code changes to tests that already work.

On the positive side, there is extensibility but I have to ask. Why oh why can't we all get along? NUnit is great but I've recently (the last couple of months) shifted over to MbUnit which seems to have a little more life and there are some nice features I'm getting productivity out of. Why couldn't say the 3 cheeses (Andrew, James, and Charlie) get together to say "Let's build a unified framework". On unit testing framework for .NET to rule them all. True, it might drag along the Microsoft approach to backward compatibility, but when we're out there trying to get people to do Agile and write good code with unit tests, I think it's a good thing rather than having to tell them "Oh yeah, and you have to rewrite X% of your already-working-and-nothing-wrong-with-them tests". Sigh.

Roy mentioned a few other things that irk me. No Assert.Fail. Well, the only reason I used it was to call it at the end of a test that had an [ExpectedException] tag so that it would fail if it fell through to there (I know, it would probably fail on the missing exception so it might be moot). Still, dropping this completely isn't cool. The bigger thorn in my side is Assert.Equals being deprecated in favor of Assert.Equal<TypeToAssertAgainst>. That's just plain mean and goes back to rewriting X% of your working tests. That would be like changing some HTML tag just because some cool new syntax is available. I see no redeeming value in doing this based on looking at the code for both. The TheoryAttribute is nice, but I already have that (and more) with MbUnit's [RowTest] and extended attributes. Frankly, I see XUnit.NET as a framework that a) removes some features which some people use b) forces you to rewrite perfectly working tests and c) adds very little that you can't get elsewhere already.

Like Roy, I'm not about to jump on this bandwagon and I don't suggest you do either. If you're looking to start writing new tests with it, I guess it's okay (other than the things mentioned above) but that's for you to decide. If you're looking to replace NUnit or MbUnit with XUnit.NET forget it. It's not worth it as a lot of your tests might break or worse case, you'll have to actually just replace code via rename because of the syntax. For me it took me about 10 minutes to move 2,200 unit tests from NUnit to MbUnit in a project. I would suspect looking over the list of changes in XUnit.NET that it would be a significant change to move those tests to XUnit.NET. Not worth it in my books and a new unit testing framework today, unless it's really fixing a real problem we have, doesn't make a heck of a lot of sense.

I go back to why can't we all get together and build a unified testing framework for .NET and be done with it. MbUnit (and NUnit for that matter) could have been giving some attention and perhaps the extensibility model could have been applied here. I'm not saying that we need one and only one unit testing framework, but we certainly don't need a dozen. CsUnit seems to have gone by the wayside in favor of NUnit and MbUnit. VSTSUnit (or whatever it's called, if it has a name) only exists because it's there and people don't know any different. I'm all for competition and adding something new to the mix but only if there's value-add and we're not just telling people that the new Red is Blue. Now it's getting silly with all the choices because there's little bang for your buck with some of them (especially XUnit.NET). I personally feel like I'm getting shafted like I do at the video store and my DVD collection. DVD comes out in it's normal form. Pay. DVD comes out in Directors Cut. Pay. DVD comes out in Extended Directors Cut, but this time it contains a special version with extra commentary you can't get anywhere, but they decide to drop some of the additional features that came out in the Directors cut so if you want both features you have to buy both copies (plus the original because it contains some special interview that got discarded along the way). Too many choices, too confusing to end developers to try to choose and frankly, why should they? Slam them all together and call it a day.