To an external observer it would appear that the consensus about the way software should be developed changes dramatically every second or third year, more frequently than the whims of fashion. Trends seem to come and go with no rhyme or reason, and it seems that the label you adopt is more important than the results that you get or the things that you actually do.

Are we working in engineering or in a fashion industry?

Have you ever taken the time to investigate a new method or practice only to find that it is just the re-branding and re-gurgitation of ideas that you have seen many times before?

Have you ever got frustrated that every new idea about software development that comes along seems to be at the expense and in aggressive competition with everything that has gone before?

Does it seem to you that following that latest software development fashion has become more important than producing great software?

In their hurry to become fashionable people seem to throw away the good with the bad. Instead of learning from their own experience and building on top of the good things that they do, they heedlessly throw everything away and start with something they believe is fundamentally new. It is as though they have no solid knowledge to stand upon. Thus they swing easily to a new trend without being able to preserve what they have learnt and experienced.

Big companies around the world carelessly discard expensive process and tool investments, almost before they have even tried them. Every project adopts a new method which new team members must first learn and master before they can begin to work.  Every time someone changes their job they have to learn a new approach before they can get on with the real task at hand.  This is not effective; we cannot learn from experience as we are forever starting over.

Nothing new ever really becomes established. Everything is marketed to the trend setters and early adopters, those who most desire to be seen as different and fashionable. People who have moved on even before the new ideas have truly established themselves or shown their real worth. We cannot get anything to stick.

In reality very little seems to change.  As in the fashion world, there is much ado about next to nothing. In something as trivial as clothing this may be acceptable, but with the size of our investment in software this is wasteful, expensive and absurd.

We are all deadly tired of all the buzz that we suffer from.

What about agile?

The latest fashion to sweep the industry is “being agile”.  Now let’s be quite clear here, the “agile” movement has made a very positive contribution to the software industry. It has reminded us that people matter first and foremost when developing software.  This is not really new but it is important, and it did seem to have been neglected by the previous, more technically oriented, fashions such as object orientation, and programming in java. By presenting a set of values the agile manifesto created something robust and resilient that can withstand the waves of change that will be bought on by the next fashion.

It’s a shame that the same cannot be said for the many agile methods that have promoted themselves as supporting the agile philosophy. For a movement that values people over process and tools it has certainly given us a lot of “new” processes and tools. The majority of these have been effective in promoting agile values by putting the team back at the heart of what is done to develop software but in bringing these things back into focus, much has been lost or obscured as new terms are introduced for old things, creating the illusion of something completely new.

The end result of this constant re-packaging and re-branding of old ideas is a constant churn in the way that software development teams work. Rather than re-focusing of peoples’ efforts away from wasted work onto the production of quality software all that seems to happen is an arbitrary renaming of the things they do and produce.

Even with something as correct and beneficial as the agile philosophy it can start to get lost in the churn and hype. Already we are starting to see a backlash against agile and our fear is that the benefits will be lost as early adopters move onto the next bandwagon and the late majority re-assert their right not to change to something that has obviously gone out of fashion.

Your current way-of-working – It’s a soup of ideas with no clear ingredients

Regardless of whether you are using a commercially developed or bespoke software development process, regardless of how it is documented or communicated, regardless of whether you use the latest buzzwords or the language of twenty years ago I am sure that it is really just a combination of bits and pieces found in other published processes with a little bit of local knowledge related to your specific domain or business.

Each new method may be presented as completely new, but in reality they are all (even mine) just re-combinations of old ideas with the occasional new in-sight thrown in for good measure.

This results in a lot of wasted effort as old truths are rediscovered but cloaked in apparently new clothing. New clothing that can prove hard to resist. Younger and less experienced coworkers promote new trends, following new gurus, supported by the hype of a media always hungry for “news”.  Managers who have lost touch with actual development find themselves in a hopeless situation: resist the newest fashion and they brand themselves as out of touch. Pilot projects are run to force proof of merit for the new approach, but motivated developers can make anything work on a small scale.  As a result the new approach overtakes the old, and all that was working with the old approach is thrown out along with the parts that were not.  Only too late do they discover that the new approach itself has parts that don’t work along with the parts that do. It is not survival of the fittest but survival of the most fashionable.

If we can separate the ingredients from the soup we can empower people to build the way-of-working that they need and to be able to adapt the method to cope with a changing industry and incorporate new ideas that complement and build on their own hard won experience.

Addressing the challenge

We need to stop forever chasing after fads and easy answers that forever disappoint without discouraging innovation and new ideas.  We need to stop constantly re-packaging and re-branding old ideas just for the sake of it. We need to help people understand how to build great software.  But how?  This is a problem I have thought about for at the least ten years and of course I have a concrete idea on how we can get there.

At the root of our problems is a deep misunderstanding of the nature of software development.  Researchers have tried to attack this problem with new theories like formalism to prove correctness of programs, or through formal languages which never have been adopted outside the academic world.  Industry efforts have spent years standardizing swollen meta-models that defy easy understanding.

There are many ways for people to learn software engineering but the only way to get good at it is through practice and experience.  Universities, technical institutes and commercial training courses teach us a particular way of working, such as UP, XP, Scrum (or whatever happens to be popular at the time) but they give us very little practical experience. When we later meet the challenges of the real world we find that the way-of-working has changed again and we have no solid knowledge to stand upon.  Thus we swing easily to a new trend without being able to preserve what we have learnt and experienced.  This is not effective; we cannot learn from experience as we are forever starting over.   Thus we need a theory.

In myopinion, this theory is right in front of us – we just need to grab it. We can start with all these methods, processes, and practices out there and find the “truth” of software engineering.

The solution looks somewhat like this:

1.   Find the Kernel – the Mother of all Methods

All methods have a lot in common.  After all, they are all used to develop software, and they all acknowledge that there are certain things that we always need to do when we develop software.  For instance, we always write code, we always test it (in one way or the other), we always think about requirements (documented or undocumented), we always have a backlog (explicit or implicit), and we always have a plan on a paper or in our heads. To borrow an overused metaphor, we want to find the DNA for software development.

We need to find the “essential core” or “kernel” of software development, that which cannot be made simpler.

By studying about 50 methods, including XP and Scrum, my team has identified a kernel with some 20+ elements, things we always do or always produce. On the surface, there may appear to be large differences in these studied methods and the ways we work with them. As an example, you can capture requirements with features or with use cases.  But there is a common basis for the many methods, which we capture in our kernel elements.

2.  Understand the nature of methods or processes

An important insight is that every method is a soup of practices, whether or not the practices are explicitly called-out.  For example, RUP consists of many integrated practices, while Scrum essentially consists of only one practice, project management, and a few agile work patterns.

3.  Describe each interesting method using the kernel.

With the kernel in place all methods can be described in a uniform way, as specializations or extensions to this kernel.  We can harvest the implicit practices from all widely used and proven methods or processes, such as architecture, components, iterations, etc.  Some practices will overlap, e.g. risk driven iterations from RUP and backlog driven sprints from Scrum.  Some practices will complement one another, e.g. use cases and project management.

The kernel clears away the cosmetic differences between methods, such as the same thing being called by different names in different methods.  For example RUP talks about iterations whereas Scrum talks about sprints, but they really mean pretty much the same thing. By doing this we can clearly see the real differences between different methods.

This is not just a theory but something that my team has already started to work on. Today around 15 such practices have been developed using our initial, prototype kernel. Since the kernel is agnostic in relation to any specific practice, we can simply find out what is the actual difference between different practices, not just on the surface but in depth. This decreases the element of religion in which every method is embedded.

It would be excellent if our technical institutes or universities would educate students in the basics of software engineering, followed up by training the students in a set of good practices using that base. Education will become more logical since it focuses on individual ideas instead of the particular soup of ideas that forms every method, process, or methodology. I believe students will love it. There is also space for a lot of relevant research here.

Remember Kurt Lewin’s words: “there is nothing as practical as a good theory.” A good theory makes it easy to learn and develop your knowledge without going overly religious.  That would be smart.

The first step is to build on the experience of our initial kernel and build a new kernel. It is clear that we in my team made a few mistakes during our initial work but the end result is that we know that this is not an abstract theory but a pragmatic approach that is proven to work. Now we need to build an improved kernel together. One that is more accessible, more intuitive, and more generally applicable. To do this we need your help. To find out more about how you can get involved see further down in this paper.

Realizing the Benefits

This isn’t just something that affects methodologists, process geeks and academics; it is something that will benefit everybody involved in software development.

What is in it for the industry?

Most large companies have their own home-grown method or process, one which is a soup of standard ideas complemented by some of their own more business specific ideas.  Usually these processes are described in a thick book or a website, and lots of money has been invested to document them. Sometimes people are trained on the process, sometimes they are just pointed toward it.

In reality the process is often ignored; the only parts that are actually used are the parts that form the “oral tradition” of the organization.  This is explained by the re-discovered law of nature: people don’t read process books.  New ideas are introduced into the organization, and the old processes fall out of fashion and their books become shelf-ware.

In these large companies there may even be more than one process.  For example the big system integrators may have ten to twenty different processes.  Sometimes they are quite similar, but the similarities are hidden behind all the cosmetic differences.

If your company was to adopt the practice idea, you wouldn’t need to throw out your entire way of working because something new and sexy is becoming popular.  Instead you would improve your existing way of working one practice at a time.  You could even adopt practices used by other companies without throwing out those existing practices that seem to be working well.

To get started you would look at your existing way of working as a set of practices. Then you would look for your pain points and complement your existing way of working by removing any practices that are not working and replacing them with practices that address the weaknesses.  This is easy to do once you have understood the kernel and its usage.

In a large organization with many different ways of working, you can use this approach to successively improve each way of working without forcing everyone to use the same method or process.

This approach will make it easier to adopt new practices without having to change the other practices you have.  Imagine that you have already introduced the kernel and described your practices a couple of years ago.  Then you would be able to introduce Scrum easily by replacing your existing practice for project management by Scrum without having to make any major modifications to your other practices.  Looking into the future, Scrum will most likely be replaced by a new practice, and you will be able to do that easily, too.

What’s in it for the academic and educational community?

Most university professors have made an academic career and have never really had a chance to practice software development on a large scale.  Still they have to teach software engineering which of course is not easy or stimulating.  They just do it because it is in the curriculum and not because they have something to contribute.  They have no theory to teach, just a set of ideas or one specific approach.  When challenged, one professor, who is successful in computer science and who teaches software engineering, said: “Amazingly, the students like to bathe in the mess we ship to them”.  I realize this was not serious, but for sure this teacher was not proud of what he was doing.

A theory will fundamentally change this situation.  Students will learn the basics of software.  They will get a language to communicate about software process, practices, patterns, etc.  I can imagine that they will get a language with a grammar that represents the kernel and language constructs to describe the practices being constituents of the process.  The language needs to be executable so that the practices can come alive.  By this I mean that practices are not just specifications but also executables.  When running a project, the practices will start to run and instances of activities, work products, roles with competencies will be created and populated by real things.  Aspects seem to fit well to model practices.  There are very interesting semantic rules that need to be identified and defined.  There is whole new world that opens up to the students to understand the fundamentals of software engineering.  Not to mention the whole new world for practically and theoretically interested researchers.

What’s in it for the methodologists?

Thinking back on my own career from 1987, I was asked by many people to write a book on the methodology I created.  At that time Objectory had a number of new ideas such as use cases, use case driven development which is a kind of test-driven design, collaborations, sequence diagrams, components and component based development.  Most of the rest was not special.  Implementation, unit testing, system test, performance test, configurations, planning was quite conventional.  Of course, I had experience from the whole lifecycle, but I was not a world-class expert on everything.  However, to write a book I had to cover the whole lifecycle, even if much of it was not my expertise.

With the new theory we are looking for, there will be no need to describe anything else than what is new.  You won’t need to write a book to publish your new ideas and put them in the context of all the other things a software development team needs to do.  You will just describe your new practice or your new pattern and it will be available to the whole world the next day.  Anybody in the whole world with great ideas can contribute and become successful.

What’s in it for the teams developing the software?

Finally we will all be able to escape from endless churn caused by slavishly following fashions and become proper software engineering teams. Teams that build and extend their knowledge by practicing good software development based on a solid foundation, one that is not constantly changing and forcing you to learn the same things over and over again. One that lets you demonstrates your expertise by the results you’ve achieved rather than the courses you’ve attended. One that lets you bring in new ideas and new team mates easily and seamlessly without performance dips or wasted effort.

Final Words

Our understanding of software engineering lacks a basic theory.  As a result, we keep reinventing old approaches with slightly different words, obscuring the real innovations and making it harder to use the good parts of the new while discarding the bad parts of the old.  The theory will help us to substantially improve our education in software engineering.  It will help us to be less naïve in how we react to new ideas popping up around us.  Finally, it will also help us to adopt new ideas faster than we can today.

At its core the theory relies on practices.  Now there are good and bad practice concepts.  The idea of practices has been around for fifty years, but these practices are loosely defined.  We need a practice concept that is more precise.

1. Our practices need to be designed for the developers and not as we described process historically, for the process engineers.
2. We need to be able to keep the practices separate, so we can compose them and in that way create our own way of working (our process).
3. Practices need to be accessible and extensible – providing lightweight practical advice that is easily put into practice rather than comprehensive academic texts. Practices should focus on the essentials – the minimal path rather than the maximal path.
4. Finally, we need practices that can come alive and be executable. Practices must be more than descriptions.

The real beneficiary of this theory will be the software industry, as has already been proven in many companies.  We will be able to easily educate our people, get them up to speed, improve the way we work with our products, reengineer (a stronger word than refactoring) our products systematically, and continually improve the way we work.

The result will be better software, significantly faster and at dramatically reduced costs. As mentioned above this is something we need to work on together. To find out more about how you can get involved visit www.ivarblog.com and give your feedback to my blog “In need of a theory for software engineering” or e-mail us at a_theory@ivarjacobson.com.