Friday, October 06, 2006

Does agile process need architecture upfront?

September turned out was almost the busiest month of the whole 2006 so far. It was the end of my last contract, and as usual the deadline of some critical milestones of my current project, plus I just moved to my new apartment last week and still no internet yet, I am writing this post in a Starbucks. Now put that all together with a couple of camping and cottage trips (because of the tight schedule I did not get to take vacation in the whole summer, this is some kind of compensation for that before temperature drops below zero again) thats the reason why not a single post in September from me.

Now since the craziness is slowly cooling down a bit, I think its time to attempt to talk about something that has been repeatedly popping up and becoming the center of many discussions I had in the last two months and so. There are many faces of this topic, such as “Do we need a architect?” or “Do we need to design this now?” or even the “How come component A looks just like component B, and by the way what are they for?”, but it all boils down to one fundamental question “Does agile process need an architecture upfront”. Before I could think through this question, based on my agile nature my first instinct was to answer “of course not, that's what agile is all about – no big and expensive architecture upfront and by relying on many best practices such as TDD(Test Driven Design/Development), Refactory, and Paired Programming to guarantee the architectural integrity and taking an evolving approach to eventually arrive at the best architecture”. Before I can even feel proud and good about my hardcore Agileness, many discussions and debates I had in last two months led me to re-examine the reality; although there is no official number to show the project success rate after applying Agile approach but many study suggested the failure rate is probably around 30% or even higher, so why that many projects are still failing. You can argue that “People do not know how to apply Agile approach” or even somewhat degrading excuse like “They don't know what they are doing”, you can see just how many of this kind of excuses flying around everywhere now-a-days, however if we take a more constructive approach then you will quickly realize that there must be something we as the Agile community is not doing right, and one of these things that seems to confuse a hell out of most of the people is about the necessity of architecture in Agile approach. People I met are either completely lost on this if they just moved from a traditional model to an Agile method, or following certain practices but without any rationale, or even creating architecture for Agile project but just too ashamed to tell any of his/her friend that almost felt like they are some kind of traitors of Agile thinking. After a couple of days of meditation in the middle of nowhere in the Killarny provincial park, I am writing this post and hopefully can give some answers to all those confusion, pride, guilt, and even betrayal :-)

Before I can just answer “yes” or “no” to this question, I recalled something I learned when I was 10 years old “Other than Black and White, there is also something in this world called Gray”, but to make the answer not complete useless we need to find a precise definition for the gray, something more than “The answer is yes and no”. Now lets take a look at what is Software Architecture? In the traditional sense Software Architecture is defined as

The software architecture of a system comprises its software components, their external properties, and their relationships with one another. The term also refers to documentation of a system's software architecture. Documenting software architecture facilitates communication between stakeholders, documents early decisions about high-level design, and allows reuse of design components and patterns between projects. (by Bass, Len, Paul Clements, and Rick Kazman in Software Architecture In Practice (p. 23-28) and quoted by Wikipedia)

This definition is just abstract and almost useless like other textbook definitions such as the IEEE definition, and pointed out loud and clear by Ralph Jahnson in extreme programming mail-list in 2003 that:

There is no highest level concept of a system. Customers have a different concept than developers. Customers do not care at all about the structure of significant components. So, perhaps an architecture is the highest level concept that developers have of a system in its environment.

According to Johnson in his post, architecture is just a social contract among the developers, regardless it is on the paper or just in their mind, and most importantly it has almost nothing to do with other major stakeholders since they don't really care (As a matter of fact in the reality some stakeholders don't even necessarily want the project to succeed.). So far our question is almost answered, the answer is “Software architecture is whatever your developers agree upon, a merely enough consensus among them before they can move on to the construction stage”, in other words even in Agile you still need some Architecture upfront but whether its zero documentation or cutting down a forest its all developers' call, moreover in Agile you should never document architecture for users or any other stakeholders. Now some of you might wonder again this answer sounds like a good idea and very agile but it does not seems very helpful or providing any guidance, so how much really do I have to create for the architecture before I can proceed without too much worry? To answer this in short, you should create just enough architecture based on your project size and complexity (This is another reason why my favorite methodology is RUP, because it scales as long as you capture the spirit, see Per Runeson and Peter Greberg's Extreme Programming and Rational Unified Process – Contrasts or Synonyms? for more information on this topic), but if you are just like most of the people I talked to still have no concrete ideas about just how much is enough, I want to recommend a very interesting research paper I read recently to help you understand what is considered and proven to be important in Agile process. Dr. Diane Kelly published her research paper “A Study of Design Characteristics in Evolving Software Using Stability as a Criterion” in IEEE Transactions on Software Engineering Vol. 32, NO. 5 May 2006. The original idea of this paper has nothing to do with Agile methodology but it was the target project she picked for conducting this research captured my attention as soon as I realize this is a perfect study that provides us some very rare insight into what really matters when it comes to software architecture by closely examine a large, complex, and actively maintained scientific software system and its 30 years of active history. The target project used in her study is a system that:

Simulates thermal hydraulic subsystems in Canadian designed nuclear generating station. This software was first released in 1975 and is still actively maintained and in use. Awards were given to its developers from peers in the Canadian nuclear industry. That, its long life, and its use for safety-related, operational, and design issues in the international nuclear community point to the success of this software.

Clearly this is a very successful software system, an ideal role model for almost all the software systems out there we are trying to build, but I can just imagine that 9 out of the 10 companies I have worked for if they were about to build a system like this, no one in the planning meeting would even brave enough to just mention the word Agile or XP, maybe... just maybe someone will whisper the name of Agile Unified Process (If you are interested in how Agile performs in large scale project here is another fascinating article Agile Software Testing in a Large-Scale Project published on IEEE Software July/August 2006 issue). OK now just lets take a look at what they did 30 years ago:

Developers of scientific software are generally domain experts and not software engineers. Applications require deep domain knowledge and have extensive mathematical content.....Software engineering practices are generally not used...However, from the author's observations, there was a common culture of disciplined software practices in all the groups....Often little or no documentation is produced.

Doesn't that sound familiar to you? And remember this was done 30 years ago when systemic software design, architecture, and even the development life cycle theory is still in its cradle. With all these characteristics we observed from the system and now we also know how it was constructed, its time to show you the best part of this study that is to reveal the secret of the major design related factors that contributed to the success of this software system. The author listed the following major contributing factors:

  • The use of common blocks was the establishment of a common vocabulary made up of the variables names associated with the common blocks...Some of the research described in this paper reveals that the common blocks provided a kind of road map into the software code and a stable basis for long-term maintainability...this finding suggests that the developers relay on the common block structure more than the procedural portion of the code (My interpretation: Domain Object Model, Responsibility Model, and Key Building Blocks)

  • Rather than attempt to foresee and design for predicted changes, create a design that allows the software to evolve in response to unforeseeable changes. This may point, for example, to the architectural style typical of scientific software being appropriate and supportive of the changes that must be absorbed over the long term by this type of software.(My interpretation: Simplicity, Just Good Enough Design, Just In Time Implementation, and Refactory)

  • The software in this study has a high percentage of variable whose semantics and names relate to entities in the application and user domains. (My interpretation: Proper Object-Oriented Design and Programming - even on non-object oriented platform)

  • First, initial attention to naming convention is key. Second, maintaining those convention both over time and throughout all parts of the software provides a form of stable documentation. (Coding/Naming Convention and Self Descriptive Coding Practice)

Hopefully these factors I mentioned above can serve you as a starting point for your own Agile Architecture, and as well hoping after reading this post you can be as excited as I am right now in this ever intriguing but promising software engineering world, like Herb Shulter said in the September issue of 2006 Dr. Dobbs “It is a wonderful time to be a software engineer” !