The Shift
Lots of people reading Jeremy's post about programming manifesto. Well written I must say. I love reading the reaction of other’s to such a posts… it’s all shock in awe. Cheers. Amen Brother!! New thinking. Wow thinking as a matter of fact.
Unfortunately, these same topics have been discussed about for well over 15 years and I'm still saddened at how developers these days are just figuring it out. I guess many haven't been living this passion as long as I have. Almost a decade ago I read Rapid Development by Steve McConnell. It was the first book that made me think "hell ya!" I carried it around with me everywhere. I told my friends about it. I told my mom about it. I told my colleagues bout it. I was “one with the book” I assure you. All you need to do is read the books list of references to see the massive list of books that have been written on the same topic dating back yet another 15 years. Hell, the entire Agile movement was based on most of the same principles.
I chuckle humbly to myself (who am I kidding.. humble.. heh) as I remember yelling out my office once "Don't use the F#$%N wizards!!!" and "We're not writing a computer science project here people" or “Simple code is better than elegant code” I also remember ranting about "think as an object like a service.. stateless with a strong interface" long ago as well - trying to promote stateless objects with defined and open interfaces. Oh well. No one listened then either <grin>.
Mike, one of the new enlightened ones (Mike has come a long way into this mindset in the past year), states that a change is coming regarding the way we develop software. I can't agree more, however, I'm not sure when this will happen simply due of the slow rate of mindset change and, of course, the tools used to promote these changes.
One interesting statement from the manifesto was “Software Engineering over Computer Science”. Let's talk about that for a second. What is Software Engineering? Well, Wikipedia states:
Software Engineering (SE) is the design, development, and documentation of software by applying technologies and practices from computer science, project management, engineering, application domains, interface design, digital asset management and other fields.
The term software engineering was popularized after 1968[1], during the 1968 NATO Software Engineering Conference (held in Garmisch, Germany) by its chairman F.L. Bauer, and has been in widespread use since.
(funny how it says applying technologies from computer science.. )
Is there really an engineering practice for software? Nope. Not yet. Any thoughts as to why? Wikipedia states that:
Engineering is the application of scientific and mathematical principles to develop economical solutions to technical problems, creating products, facilities, and structures that are useful to people.[1][2] One who practices engineering is called an engineer, and those licensed to do so have formal designations such as Professional Engineer. Engineers use imagination, judgment, and reasoning to apply science, technology, mathematics, and practical experience. The result is the design, production, and operation of useful objects or processes. The broad discipline of engineering encompasses a range of specialized subdisciplines that focus on the issues associated with developing a specific kind of product, or using a specific type of technology.
Guess what - Software Engineering is not even close to this yet. It's not nearly as structured. It's not nearly as scientific - especially when it comes to the development of business solutions We don't require licenses to practice software engineering. Apparently, you need to go to school to become an engineer - a school that teaches specific disciplines of engineering. And damn it.. Professional Engineers have those damn cool rings – it’s like the Freemasonry secret handshake or something. Last time I checked you don't need to obtain a special degree for the "Software Engineering" discipline - you can have any degree or education you want and still be a programmer and call yourself a software engineer and your trade software engineering. Tomorrow ,I will become a Doctor because I know CPR. There are still fights between people with different degrees on who are better programmers.. computer science grads, community college grads, or self taught. Shouldn't we have some standards about who can call themselves software engineers or software architects?
One of the big movements I've seen lately is the "Pattern Movement." Old news. Boring. Been there - done that. It truly shocks me that this is still "new" to many developers. Ask 100 developers to list 10 software design patterns and to explain their use.. what do you think you will get? Software patterns have been around for decades and are even studied in Computer Science. Wait a second.. I thought the manifest said Software Engineering (which doesn't exist according to MY definition – likely not yours if you are a software engineer) over Computer Science.. anyway, moving on.
There is a shift coming and it has to do with the split between building software for business and building software that can be used to build software for business. Now - I'm a computer science guy - so I might get flamed for being all.. "computer science".. but in computer science we studied and predicted this shift a very long time ago. We knew there needed to be higher abstraction required. We knew this was the natural trend. It started with binary coding - we moved to assembly language. We then moved to more abstract languages. This change took a while because there were those in the industry who claimed that you could never get better assembly code than hand written assembly code - compiler optimization - um, computer science - solved that and gave us the birth of the mighty modern computer language. There has been some amazing languages since then. SmallTalk. Lisp. Prolog. Cobol, Pascal. PL/1, APL, C, C++, C# ( the LIST goes on). I find it hysterical so very few people in this industry even know who Noam Chomsky is (outside his political exploits). A pioneer in linguistics providing the cornerstone, and if you look closely, a roadmap for computer languages in any form (eg, domain specific languages). Come on people – do some reading as well as coding.
If we try to categorize software languages, we also run into some interesting results. According to this post, software engineering is about application programming where we solve a domain specific problem within the abstraction of a language. Presto - that's it.
If I were a betting man, and I am since I own a few companies (that or I'm a stupid man - or both) I'm going to guess that there will be a huge separation between "engineers" and "installers" of software in much the same way we see a gap between engineers and trades people. Today, this gap in the software industry isn't as apparent at the individual level as it is at the commercial level - companies who write custom business software - companies who write tools to write custom business software and companies who write business software. Ten years ago there wasn't a big gap in the skills needed to write tools or business solutions. It's my bet that we're going to see this gap widen as businesses are demanding better tools to meet business needs faster. I believe that the next paradigm shift will be equivalent to the time we moved from assembly languages to more abstract languages. I see the rebirth of the next generation languages and grammars to help us get there. Many scoff this idea in much the same way they scoffed moving to more abstract languages in the first place. "Generated code is bloat ware" - "Generated code can never perform as well as hand written code" - these are the exact same statements made 30+ years ago - but today we'd rarely think of every writing the 1's and 0's ourselves. Compiler optimization is simply magic. Take a look at SQL Server 2005 today with its query optimization features - simply magic. You'd simply never write a compiler optimizer as part of a business solution. You would write that as part of a platform building solution to make it easier for business solutions to be built and run. The gap widens. Rarely do we have to tune our own indexes these days.
Taking this to a simpler paradigm, there may be a number of people who code religiously in notepad, even hand writing the code that manages their GUI. Bravo - you're all wonderful coders. The great many business developers out there use a GUI design tool to generate the code. Not everyone has always been happy with auto generated gui's (take ASP and ASP 1.1 for example) however, its’ seriously getting much better to the point where we rarely, if ever, hand code our own gui's - especially if you are writing windows forms based applications. This will continue as we get into the world of XAML and better designers for the user experience. There is only one reason for these tools - to increase the gap. To allow business developers to develop amazing business applications faster and with less complexity (although I argue that in sometimes we're seeing higher complexity solutions emerge instead – that’s what happens when you give really smart people really good tools).
Fourth Generation Programming languages are not new. They tried to come alive in the late 60's and early 70's would you believe. They failed at that time and were discarded.. given a bad rap and locked up in jail.. all subscribers to the theory labeled heretics and imbeciles. You even mention the word 4GL in some circles and you will be labeled a dreamer and chastised for all eternity.
Then there came 5GL languages... in the 90's considered the wave of the future. Clearly the complexity of solving such a solution was overwhelming - and still is. Unfortunately, from my perspective - I don't think computer science tried hard enough. The 5GL languages I read about took a big bang approach - a Problem->Implementation style. The all or nothing approach simply can’t work. If we are truly going to get to the 5GL mindset we need much more granular and abstract intermediate representations to govern the complexity and allow for real life applicability and extensibility.
Where does this leave us? Stuck it seems. We seem to have generations of programmers solving the same problems over and over again claiming the same small victories over and over again (patterns, code generators, different ways of abstracting and getting the same results, new forms of code interop). Truly, I think the answer to the question does lie in computer science - not software engineering.
By the way, I don't disagree with any of the manifesto's mentioned in this post. I applaud them. I’m just very cynical as this all seems to be a repeating pattern to me. It also saddens me that we can’t seem to break out of it even though we have evolved sooo much technically since the 70’s. I believe we need to work better in the 3GL world we live in. I believe in Agile methods and agile designs. I believe in developer empowerment. I believe in writing code for maintenance instead of elegance. I buy into it all. Most of us build business solutions - wild and crazy apps that take data from a user, save it in a database, and report on it (amazing - I know!!!). The problem doesn't seem to change - and neither does the continual debate about how these types of solutions should be created. In the end, the majority of software engineers simply hand write the code - sometimes relying on mini-factories (gui code gen tools, DDL gen tools, DML gen tools, BO skeleton gen tools, etc) - but these partial software factories solve hamster coding problems at best. However, we need to work much harder if we want to reap the rewards. We are close to the same type of change the manufacturing world realized from principles based on the theory of constraints, Lean Production, Just in Time Inventory, Six Sigma, and Chaos and Uncertainty theory. Those who embrace these style of changes will be the companies that lead the next generation of software.
Based on my previous statement, you can probably guess that I believe that before any paradigm shift in technology is discovered, we need to change the process by which we develop it first. We’ve proven time and time again that technology isn’t making us better at what we do. The massive changes in manufacturing that has altered our lives so significantly started with process changes, not technology changes (thank you Deming). Let’s revolutionize and adopt these changes first and only then can we be better prepare to change our design factory.