On the Future of Software Development #4 – The need for specialization

In my previous postings I talked about two perceptions: 1. I don´t see any real signs that make our trade of software development singular and hard to compare to other industries. 2. The rate of change in our industry has increased very much in the past 10 to 20 years and makes it impossible to keep up our attitude of allround programmers.

So what do I make of these perceptions?

Conclusion: Software Development needs much more specialization

My conclusion is, software developers need to change to be able to cope with the raging techology tsunami.

The past 20 years we´ve always waited for technologies to become better to solve the software crisis. The next compiler would be faster, the next UI tool would be more intuitive, the next server generation raise us to a new level of abstraction.

But what have we got? More languages, more database access APIs, more servers, more complex IDEs, more non-functional aspects to look after. Has any programmer been fired because of more productive tools? I haven´t heard of any. (If they get fired, then it´s because of some merger/acquisition or customers are not paying anymore etc. It´s because of economic reasons.)

So all the changes in tools so far have not ended the software crisis. There is more software with higher quality than in the past to produce, than todays programmers can deliver.

For me this means, we should start to change another parameter in our system. We went from Client/Server to N-Tier, we went from Waterfall to Agile, we´ve got OOA, patterns, Intellisense. Everything is new and better (and still ever changing) – except for the people.

The people haven´t changed much in the past 20 years. They are still living in this world of “I once learned the trade and now I´m able to solve any problem.” They still see themselves as allrounders. And the process of software production follows this attitude in many projects.

My believe is, there lies a lot of potential for improvement.

And since our trade is not singular, we should look left and right how others have evolved over time.

An obvious trait of everything we buy is, it is the product of a process in which many highly specialized individuals/companies cooperate. Look at your car, TV, cell phone, house, even the next hamburger you eat. They are all the result of a long chain of producers.

A high division of labor and a shallow production depth are common to all production processes behind our everyday things.

And look at the education market: There are dozens of professions you can learn to be involved with the product “car”. Or have you ever sat through the credits of a major movie? Did you count the number of different jobs/professions taking part in its production? There are hundreds of people involved and all are highly trained (or at least experienced) professionals. (Otherwise they would not be on the payroll of a major Hollywood production.)

Now think about an average software project. How many professions do you see at work there? Is it 2, 3, or 5? Let´s see… I count “programmer”, and … Well, I´m not talking about role, I´m talking about professions, i.e. a trade you can actually learn, where there exists a curriculum, a common agreement on what you should now as an xyz expert.

Even though on an average (!) software project you will probably find at least “project leader”, “architect”, “programmer”, “tester”, “deployer”, “documenter” as roles – they are most often staffed with people who just learned the trade of a software developer. A typical project leader is just a very senior programmer, the same goes for an architect. A tester sometimes is a programmer who´s turn it is to test. Working on deployment sometimes is viewed as punishment. And let´s not talk about documentation…

Most of those roles are filled by people who may be switched to other roles at any time. So it´s perfectly understandable, that they entertain an allrounder-attitude.

Back in the good old times, when programmers truly were artists, deployment was just copying a couple of files, documentation was for whimps, testing was not needed (or was done by the customer), and architecture was emergent. Well, at least we all remember it to have been like that, don´t we? :-)

However nowadays, all of these roles require very different skills. There are many tools for testers, deployment is not easy at all (despite XCOPY deployment and ClickOnce for the .NET Framework), and architects probably have the most difficult job of them all. (They need to be the true generalists with a very good knowledge of a lot of technological options, plus they need people skills.)

In addition to the many tools and material options (see the box “What I mean by tools and materials” in posting #2 of this series) for those roles, there is a vast number of challenges for th unsuspicious programmer role. The term programmer does not distinguish between a guy working on a GUI or on domain logic or database access or security or internationalization or application distribution etc.

Microsoft now has started to provide tools in different flavors for different roles (see the box on the right of the Visual Studio Team Systems homepage; it lists architects, developers, testers, and project managers), but nobody seems to see a need for defining more subroles within the programmer role. This smells of underspecification.

Since long programmers used to specialize in a vertical industry. A game developer rarely will switch to embedded systems for industrial plant automation; a compiler specialist does not want to become a guy on a business software team; a domain expert in the insurance industry working on enterprise applications will not switch to the telecommunication industry.

But a horizontal specialization is comparatively rare among programmers.

To be successful in the future, though, I think it is inevitable.

I don´t know exactly where the boundaries will lie between specialized areas of expertise, but a rough sketch would probably separate architecture/design, frontend, domain logic, data access, security, deployment, and testing from oneanother. (Maybe it´s worthwhile to specialize even more and separate Smart Client frontends from WebForms from mobile form factors.)

Where the knowledge horizon lies, will also be determined by how many tools/materials an area like frontend or data access contains. At least to be equally knowledgeable in WinForms/GDI+/WebForms etc. and ADO.NET/SQLXML/T-SQL etc. I think is impossible.

But why specialize? Because only by specialization quality of products can be attained.

The quality of a software product is determined not by delivering a functional complete program. Fulfilling functional requirements is the premise. We don´t need to talk about that, don´t we?

Rather software quality is determined by non-functional aspects like performance, scaleability, flexibility, correctness, robustness, maintainability, speed of development etc.

Of course, almost any programmer can bang out some code that somehow fulfills the functional requirements of a project. But did he use the technologies to their best? Did he even choose the most adequate technologies available?

As I understand software development, one must have considerable experience with a technology to put it to good use in a mission critical project. Also, to select between several technologies one needs to have experience with all of them, lest the decision is uninformed.

Now, to gain experience or even considerable experience it takes time. However, the time in a programmer´s professonal life is pretty much constant.

What happens, when the number of technologies relevant for a programmer increases? The time he can devote to each to gain (considerable) experience diminishes. That´s a very simple and obvious calculation. This becomes worse, if the complexity of the technologies also rises. And it becomes even worse, if the number of non-functional aspects to heed increases too.

I doubt the nowadays common allround programmer can cope with this situation and wholeheartedly claim, he can still produce high quality software. If this is possible, I´m a whimp and should think about switching to some other profession.

This conclusion might make feel many uneasy. I can understand that. Whenever a deep believe is shattered or a habit needs to be changed, (emotional) resistance springs up. But feeling the status-quo should not change is not really a proof for anything.

I emphasize with the many allround programmers and small shops who think, they depend on them being allrounders. But I nevertheless can´t imagine how they will be able to claim before their customers, they know it all and have chosen the best tools/materials while at the same time keep up with all the different non-functional requirements (the customer not even thought about, but nevertheless expects to be considered).

To go even further, I need to say, specialization will entail many other changes. A very obvious one I´ll mention right away: Once programmers start to specialize there will be not enough work for everyone on every project. A small shop with three allround developers not continuously will have enough work for maybe a security specialist or a deployment expert. A large software department in a bank on the other hand will be able to keep the guy who specializes on Web frontends, because there´s always some project going on, which needs his expertise.

So what I see as an inevitable implication of a trend/need towards specialization is an increase in outsourcing. Get an expert on your project when you need him! This there will be a larger market for freelance programmers in the future. But of course, specialists are free to flock together and found a company that in itself specializes on something. Their foci can be homogenious or heterogenious; both could make sense. What will not make sense, though, is to succumb to the feeling, you´d be better off as an allrounder.

Instead of keeping broad knowledge, start to focus.

Instead of being a loner, start to network.

Allround skills in the future of software development will manifest itself not in individuals anymore, but in teams.

When something starts to overwhelm you, the natural reaction is to focus. When you first drive a car, a tunnel view is dangerous. When you need to cope with an ever changing and broadening environment of technologies, a tunnel view is the basis for your survival.

Think of it: We are programmers and not gatherers anymore. We are already highly specialized. Why stop at the current level?

So far, the customers of software products have not (often) asked, how software is produced and who produces the software. But that will change! The demand for higher quality software will drive to the software manufacturers to lay open, how (!) they are ensuring quality. And I don´t think any customer will be satisfied with an answer like “Hey, we´re all allrounders here at Xyz Software Inc. Any developer can jump in for the other any time. We don´t believe in specialists.” But that´s the situation today in many, many software developing departments/shops.

But would you, when buying your next car, go for the car manufacturer who´d answer you in that way? “Hey, our engineers are not specialized! We are proud our motor engineer can jump in for a car body engineer any time!”

Cathedrals were build by a large number of different professions. Aircrafts are build by a large number of professions. So are TV sets. Just software should be singular in that a single profession – software developer – builds the actual product? That cannot be true.

But of course, specialization leads to even more implications. Once developers start to focus their skills, a lot of thinks need to change. Some of those implications will be the topic of future installments of this series.

[See here for an overview of all postings of the series on the future of software development.]