From six months release cadence to continuous deployment

Since it has been a while, almost ten years, since I blogged about what I'm doing at work I thought a bit of a summary would be appropriated. I don't intend to talk you through every task I've done since no one would be interested in that, but I thought perhaps it might be interesting to some of you to read about me joining the Swedish Pensions Agency (Pensionsmyndigheten, in Swedish) eight years ago and how we collectively since then managed to move from an enforced three, but in reality six, months release cadence for all systems to a continuous deployment scheme where individual teams and the part of the business they serve is in charge of how and when new and changed features are brought to production.

Disclaimer: This is my story, my perception of how it was, what happened and what it has become. Many of my current and former colleagues who are/were involved might have a slightly, or totally, different view. I'm writing this with a hope it will inspire others to discuss and question matters of organisation and software development processes and how they affect the daily joy and development efficiency of an IT-organisation.

How it was 8 years ago

In 2014 I decided I was done with consulting, at least for now, and wanted to try out being an employee of the organisation where I spend my working days. I found a position as systems architect/developer within the development side of the IT-department at the Swedish Pensions Agency. The ad spoke about an agile and forward looking agency basing their development on Java and open-source tools I liked. At that time we were about 80 developers and testers maintaining and developing systems where some had been around a good ten years already, integrating with even older systems with their roots in the 80's or 90's.

We were all divided into three, about the same sized, silos without much direct communication, each managing a related set of systems. Communication and coordination was mainly handled through people being included in smaller or larger projects. Releases to production were mandated to happen every third month (for all systems synchronized), with a good set of manual end-to-end-testing leading up to it. Any release between those where to only include really critical bug fixes. And yes, we did a fair amount of those.

In my silo, "case management", we were further divided into people (most of us) working in the ever (it seemed) ongoing "program for case management" with a six month release cadence of new functionality, and the maintenance team (the rest of us). In the maintenance team we got a load of new functionality (and code) to maintain every six months and most of what we did was fixing bugs in someone else's code and then spend a disproportional amount of time merging the fix into branches representing the release due in a month's time, the one four months away and sometimes also the one due seven months into the future since development had already begun there.

We used daily meetings (standing!), but no one would call that "agile" today, and I don't think no one really did back then either. The "agile" in the ad I think was a more correct description of the state of mind or spirit within the organisation which came as a very positive surprise to me. There were already a great number of small experiments (or "pilots" as they were called) running in the organisation and whenever a new idea was presented by someone the answer was almost always to try it out as a "pilot". Moving to use Jira for internal organisation of work was one of those "pilots" that came to stuck.

What happened

In beginning of 2015 we were all pretty fed up with the current situation. One of our neighboring silos, "external web", started up an official pilot to try out agile delivery with self organized, cross-functional teams and a six weeks planning horizon. At the same time they had started moving towards a brand new architecture with microservices and continuous deployment pipelines. Despite maintaining a large monolithic system with about half a million lines of code and no ambitions, nor opportunities to change the architecture, we still wanted to find another way of working also in the "case management" silo.

With support of first line management we organized a first workshop aiming to define what "awesome" would look like with respect to development organisation and delivery process within our silo. The result described cross-functional teams, each focused on one or two subdomains, responsible for both developing new features and maintenance tasks, working in a continuous delivery manner where changes were deployed to production weekly or more frequently when need arise. Many would say going to production weekly isn't continuous delivery. I think the important thing is to be able to do it more frequently, but actually doing it is a business decision and they thought, and still do, that once a week is great for the internal user base of case management clerks.

In cooperation with first line management we formed a small team to meet once or twice a week to discuss, develop and coordinate our abilities to meet this description of "awesome case management". We defined roles and teams. We engaged with parties outside our silo to find out boundaries for our new processes and to create ways to cooperate. We enhanced our technical abilities.

In order to do deployments to production every week, instead of just four times a year, we had to do those deployments during office hours, which meant they had to be zero downtime from a business perspective. We also had to make the whole process, from build and test to creation of release notes, automated. Luckily the system already had a heavy set of automated unit and system level tests which made us confident that if all those passed we were good to go to production. With a small team of dedicated engineers, helped by the fact that the whole agency at the same time moved sources to Git/Bitbucket, we built Jenkins pipelines, integrated with Jira and Bitbucket, to manage building artifacts, running tests and creation of release tickets, including information on all changes going into each release.

After working out the organisational, process related and technical issues with everyone involved we turned our silo into the new team structure and ways of working in march 2016. After a bit over a year of discussions and preparations we stopped moving people to work and stared moving work to people (teams). And we didn't look back. Of course it wasn't perfect, but with continuous improvement processes on several levels we were able to improve as needed. The area where we had the most problems was in prioritization and scheduling of larger activities, all coming from differently sized projects still run in parallel across the agency. This was a bit outside our reach, since it crossed several silos. We needed something more.

Luckily, with two silos having turned their organisation and way of working around in similar ways there were a pretty good case for the agency to continue forward and find new, more agile, ways for the whole development organisation. SAFe is a framework for prioritizing and coordinating work across many teams in an agile and lean way and since our new practices heavily resembled team and delivery practices in SAFe it came natural for the agency to decide to embark on that route. Of course there where a lot of education and discussions to be done, especially with upper management, but since the director general of the agency was a firm proponent decisions were made in due time and a first SAFe Agile Release Train (ART) was started up as a pilot late 2019.

What it has become

After about a year of trying out SAFe practices and organisation structures, as well as changing to full time remote work during the pandemic, the organisation decided we had gathered enough experience to have the whole development organisation go full-on SAFe. About one and a half year later we are running five ARTs and a few supporting groups for tooling and very specialized services. Each ART is organized around parts of the agency's rather broad business domain and employ both business and IT-specialists in their teams and management groups. SAFe's lean portfolio management helps us organize and prioritize all the large or cross-ART initiatives and the time of projects fighting over individuals for work on full- or part-time is forever over.

With the pandemic no longer directing how society works we have been able to employ both on-site PI-planning events and frequent "teamdays" at the office. All our gained experiences over the past two years have proven to us that SAFe practices works both in an on-site and in a fully remote setting. Our ARTs and teams now operates with a mix of working remote and being at the office. This gives everyone a great deal of personal freedom and also lays the foundation for team being able to function despite being split over offices in different parts of Sweden.

Also the infrastructure side has continued to develop. With a modern cloud platform and CI/CD-tooling, including a formalized microservice archetype, it's now possible to spin up new services in a few days and have them deployed to production whenever the business sees fit, several times a day if needed.

Conclusion

It has been a really interesting, sometimes frustrating, but overall joyful journey for eight years. We have moved from projects throwing software "over the wall" to teams taking full responsibility for their systems and services all the way to production, from endless discussions over priorities and attempts to maximize utilization to pull-based planning and focus on throughput, from excessive merges and an in-reality six months release cadence to releasing new features as they are ready and we still want more. We are each day looking for ways to make our services better and our organisation a bit more efficient and capable. If you stay tuned chances are you may hear of some of that on this blog. 

If you happen to be living in Sweden (and talking Swedish) and fancy come working with us, checkout our current job openings. Over the years our IT-organisation has grown, and it continues to grow. Our society are in the middle of the digital transformation and IT is at the heart of our agency's business. I think we will see many interesting and challenging opportunities going forward.

Ideas and books forming me as a Software Engineer

This September it was 15 years since I joined university to study Software Engineering, the official name was “Informatics”, but I think software engineering (or authoring and gardening as argued in another post) is better describing what I was interested in and, luckily, what I'm currently doing. Since then I've got some 10+ years of experience from working in the industry and a whole 15 years of continuous learning. As an exercise for myself I've set out to go through the major ideas I've picked up and used along with the people and books that collaboratively have formed me to the software engineer I am today. Hopefully you find this post at least a bit interesting and inspiring as well.


Object Orientation (OO)
In 1996, and the following years, OO was the new hot thing in software design, at least at my university. I learned to model domains (even though I didn't know that word by then) in objects using the OMT (Object Modeling Technique) notation by James Rumbaugh (later on one of the main contributers to UML). This way of abstracting the real world into objects with attributes, methods and relations very much appealed to me as it offered the tools to unambiguously document the analysis result by drawing a diagram. I did also believe my teachers when they talked about OO as the enabler of universal libraries of small reusable objects. Now I, and I'm sure they too, know this is never going to happen, but the OO-paradigm still brings structure for reuse within a system and hence is a great tool for honoring the DRY-principle. 
Ever since those first courses I started to think of OO as the way to design and build software systems. Later on I've come to see OO as the way of designing and building transaction based information systems where manipulation of state is the primary focus. Other types of systems might be better suited to use some other paradigm, but this type of systems for administering information is what I've been working with almost the entire time since I graduated. In university I learned the theory of OO, but we didn't actually turn any of our OO-designs into working software. We sure learned programming, but then it was to solve more traditional algorithmic problems. When leaving university I was all eager to learn how to do OO in real world projects. And boy, was I disappointed!
For several years I was employed in projects where I learned lots of things about programming and projects in general but nowhere was OO used to anything but data diagrams (now drawn in UML notation which I learned through Martin Fowler's “UML Distilled”), i.e. objects with attributes and relations but no behavior, that was later turned into database tables and data structures. All behavior was programmed into transaction scripts, either directly in the GUI components or as free standing functions or stored procedures. I was desperately looking for a real world example of true OO implementation because on my own I couldn't really figure out what it would look like in order to work properly. The first piece of the solution came when I got to read Craig Larman's “Applying UML and Patterns” . In this book he describes all aspects of OOA/OOD, how the software can be structured with boundary classes, controllers and entities, and shows by example code how it all comes together. Still, this isn't the approach intuitively encouraged by the structure in popular frameworks such as Java EE and Spring Framework. Out-of-the-box they rather suggests a static service structure operating on DTO:s, and that is, I think, the main reason why most systems are built with data and behavior separated and not using the true OO-paradigm.


Domain-Driven Design (DDD)
In a Jfokus 2008 tutorial I was first introduced to Domain-Driven Design. The three hour tutorial only scratched the surface of DDD but it was enough for me to understand that this was the description of how the full OO-paradigm is used to build real world systems. Soon after, I got to read Eric Evans' book “Domain-Driven Design - Tackling complexity at the heart of software”, which I have posted about previously, and since then I firmly believe that DDD is the approach to use for designing and implementing complex software systems.
DDD is not a method, nor is it simply a technique or an architectural pattern. I would rather call it an approach to software development, including analysis, design and implementation.
Perhaps what most of my colleagues first think of when DDD is mentioned is the design sketches I use to draw with the business domain at the center and technical integration packages all around. The domain being built on the conceptual building blocks, entities, value objects, repositories and services, as introduced by Eric Evans. But even though that part of the approach might be the easiest to pick up, and also important in building working software with DDD, I think what makes the biggest difference to me is the ubiquitous language. This practice of building a model with a shared language based on, but with more precise definitions than, the language spoken by the domain experts is the basis of my philosophy to always build the software so that structure and logic follows the business domain. I think that is the only way to guarantee a flexible design such that a small change to the business is always a small change to the software, never a big one. My experience is that the second you deviate from that principle, most often due to time pressure or pure laziness, you are asking for trouble further down the road. It might take a year or so, but it will come back and bite you!


Test Driven Development (TDD)
If DDD provides the approach on analysis, design and implementation, TDD is clearly what integrates that approach with quality awareness and assurance. TDD is my way to ensure a testable design and a correct implementation that is easy and secure to improve further. In a previous post, "Are you testinfected?", I told the story of how I got introduced to TDD, how I was skeptical at first and how I later on proved the usefulness to myself. Now I rarely, and definitely reluctantly, writes any code without first writing a failing test.


A Clean Coder producing Clean Code
Ever since the start of my career I’ve wanted to produce code that I’m proud of. However, the benchmarks for that assessment have changed over the years. Now I consider well tested and easy to read code being the standard to achieve. Since any piece of code is read much more often than it is written I think readability is prime quality measurement for code. Lately I’ve been reading both Clean Code and The Clean Coder by Robert C Martin. Both are excellent books. The first one giving good advice on how to make the code readable, the second really demanding reflection on what it means to be a professional software developer.


Agile with Scrum and Kanban
Another interest of mine is software development processes and techniques. At first I learnt how to deliver in a world of waterfalls, later I got to experience RUP. Despite all bad things said about RUP, I’ve always found the core messages on iterative, risk driven development to be right and frankly quite agile. However, agile approaches such as Scrum, Kanban and developmenttechniques gathered in XP take this a long way further. The major inspiration here has been Henrik Kniberg, first in his Jfocus tutorial in 2008 and through his two books on Scrum and Kanban.
Currently I’m working in a world of mixing and matching. I’m picking the best parts out of all those above to create a process and way of working that supports being as agile and lean as possible in an otherwise RUP-ified world. I think, at the current state it is somewhat of coming success.


“All problems are not worth solving”
That was one of the advice I got from an experienced colleague of mine many years ago when I was aspiring to go from being a sheer hacker to become a CapgeminiCertified Software Engineer. He probably gave me a bunch of other advice too, but this is the one I still remember. And I also think, for me personally, it might be the most valuable advice I’ve ever got. Since problem solving is my trade I happily go of designing solutions, then it is good to remember thinking for a minute or two about whether the problem really deserves being solved or if it is something we can as happily live with. At the bottom of this is that all resources are limited and needs to be applied where they make the best return on investment. On the other hand, we need to be careful not to spend too much time on deciding if a problem is worth solving, i.e. it is not always worth solving the prioritization problem if the solution offered is cheap enough.


The future
This pretty much sums up where I am today, but of course I will continue to evolve and in the near future I see the following ideas as the most interesting.
- Strategic design – How DDD concepts such as Bounded Context is applied at a larger scale to make strategic decisions on how to factor architectures and apply different design and development strategies. In most discussions “architecture” seems to mean frameworks, databases, application servers and technical layering. I think architectures would be much more interesting and usable if they concerned business domains.
- CQRS – I have only touched the surface of this approach to structure systems with separated command and query sides and event stores and messaging instead of relational database schemas and object/relational mappers. However, I think is an interesting approach and I definitely plan to dig in deeper.
- DCI  – Even though DDD has come to be my preferred approach I must admit the structure of the code base with behavioral code localized in every domain object has its draw backs. With requirements based on use cases or user stories, it is sometimes hard to answer the not so infrequent question “how is this UC implemented?”. DCI (Data, Context, Interaction) solves this problem through keeping the behavioral code in role objects implemented within a context representing a single UC. The code in the role objects are then injected into the data objects representing the domain. To me, it sounds like an interesting add-on to the ideas of DDD and in the future I like to continue exploring the benefits of this approach and the demands it poses on the implementation language.
- Functional programming – In order to build simpler programs that are safe to parallelize on multiple cores functional programming has gained new interest in the last years. Up until now I’ve lived only in the world of imperative programming and grasping the concepts and underlying mechanics of functional programming will pose a challenge. A challenge I’m happy to take on.
- Scala – Of all the new languages having emerged on the JVM I think Scala is the most interesting. It combines functional programming with additional powers in OO-programming without losing the benefits of static typing. All of which I think are interesting qualities. I think Scala might be the enabling programming language for many of the ideas I listed here for future exploration.

Value Producers and Supporters

In the lean and agile world anything that produces value for the client is a good thing, everything else should be carefully scrutinized before carried out. This doesn’t mean you shouldn’t produce any documentation, it just means you should produce only the documentation that someone is likely to read. It doesn't mean you shouldn't produce any automated tests, it just means you should make sure you produce them in such a way you optimize value during the development period and for future maintenance. To do this right you need to understand which part of the project organization is producing direct value for the client and which part is supporting the Value Producers.

In many projects you could find three groups of Value Producers: programmers who produce code that builds the product, testers who make sure the product does what it is supposed to do and technical writers who produce the user documentation. Everyone else, requirements analysts, architects, designers, etc, are usually in a supportive role. The Supporters does not themselves produce anything of direct value to the client, however that doesn’t mean that their work is not important. It is. As long as they make it easier for the Value Producers to do their work, now and in the future.

That said one of the challenges in running an efficient agile project is to understand who is supporting who and how to make it as easy as possible to produce maximized value for the client. In order to do that you first need to understand what deliverables (both artifacts and activities) that produce value to the client. In some projects the client might get great value from the structuring and questioning on  business processes done by the requirements analysts in order to define the system requirements. In other projects the client is perfectly clear on their business and the requirements specs are only needed to formally document the systems functionality and drive testing. Next you remove any activity or artifact that does not  add value to the client or is invaluable in order to support efficient value production and make sure everyone inside and outside of the project understands their role and for whom they are supposed to produce value, in other words: if  they are a Value Producer or a Supporter.

Define and apply “The boy scout rule”

“The boy scout rule”, as defined by Robert C. Martin, tells us to make small improvements to existing code as we pass by implementing new features. It doesn't mean we need to make the whole class/file perfect, it just means we should make some small improvement in addition to the new feature, which of course should have a clean implementation. This is a good principle to include in the culture of any project. But how to do make it happen in a team with many developers, each bringing their own experiences and standards? In my project we are currently making a serious effort. Here are the actions we are taking. Please leave a comment with your own experiences/ideas.

First, talk about it. Introduce the name “The boy scout rule” - it is a quite funny name, and easy to remember. Load it further with the reference to real world scouts not caring who left the garbage found at the camping site, but always making sure the site is cleaned and left a bit better than found. Transfer the values into software development as a way to continuously refactor an existing code base while at the same time implementing new features.

Second, define, in concrete terms, what “The boy scout rule” means in the project. In our project we have made the following definition, with reference to our design principles:

• Increase readability
  • Choose intention revealing names from the business domain for classes/methods/attributes/parameters/variables.
  • Extract methods to enforce the principle of Command Query Separation (CQS) and intention revealing method names.
• Enhance the structure
  • Move methods to the“right” location to enforce the Single Responsibility Principle (SRP) for classes.
• Make it easier to do changes without introducing bugs:
  • Increase readability (as above).
  • Enhance the structure (as above).
  • Use domain objects for method parameters (e.g. use a ZipCode object instead of a simple String).

Third, lead by example. Start with the the informal leaders (often the most experienced developers) and have them spread the practice through interaction and pairing in daily work. Personally, when pairing with colleagues I try to explicitly point out what part of the work we are doing is implementing the new feature and what part is applying “The boy scout rule”. By doing so I hope to spread both the knowledge and practice to apply the rule in daily work.

No man is an island

What’s the most common reason for failure in a software project, or perhaps in any project? You will get many different answers depending on who you ask. The designers might say “The requirements were not clear enough and the programmers didn't follow the architecture”. Testers on the other hand might say “The programmers never delivered what we were supposed to get when we were supposed to get it” and programmers tend to say “We got lots of ‘bug-reports’ that where really outside the scope”. The list can be long, but at the end of the day it all comes down to lack of communication.

Let’s face it, developing software is a complex and creative handiwork. That’s why we try to stay away from the over-planned, defined, way of waterfall and turn to iteration based approaches like RUP or Scrum. True iterations give us great feedback to base the next, (hopefully) better, version on. However, working with iterations is not enough. The saying “No man is an island” origins back to about year 1600 and is as true as ever when it comes to software projects. For many years we have set up the projects so that specialist testers form a test team, specialist designers form a design team, specialist programmers form implementation teams, and so on. This for sure brings advantages in having specialists learning from each other becoming even more specialised. But it also introduces a great deal of hand-overs between the teams, each missing some vital information. I believe this way of organising the development project is one of the key reasons why projects fail.

In the agile community we talk about “Cross-functional teams“, meaning that people with different specializations work together to achieve a common goal. In software development that means that a programmer sits next to a designer and a tester, working together on transforming a requirement into a piece of well designed, well tested code ready for production. The agile practices are all about maximizing communication, e.g. by locating people in the same room with access to broadband communication aids such as whiteboards.

If you are currently in a project where the test team and development team sit at different floors and the primary tool for communication is release notes, then you have a superb opportunity to increase the likeliness of project success just by re-locating so that developers and tester form collaborative teams working together on both design, test design, implementation and test execution.

Automated functional tests - an organisational change

Introducing automated systems (functional) testing to a project used to run manual functional and regression tests at the end of every development cycle is not a technical issue – well, it is to some extent, but there are some good solutions out there – it is a fundamental change to the work your testers perform.

In the manual approach a typical set up of roles dictate that test designers create written test specifications in some tool or in ordinary documents. The specifications contain everything the tester needs to execute the test, including detailed step-by-step instructions.  The tests are then manually executed over and over again making sure to catch any regression in the system functionality.

When moving to automated testing the previously text based test specifications are transformed into an executable format. Perhaps the behaviour driven approach is used with some short textual description that underneath is connected to code exercising the system, preferably through the GUI if such exists. Since execution of the tests is automated the testers previously doing the manual test execution are no longer needed. Instead test implementers, with the ability to build the code that executes the tests, are working alongside the test designers. Tests are executed by the build system, perhaps nightly or even more frequent making the feedback loop faster.

This is a big change - and yes, it is a good one - but it is important to understand the impact to the organization of your test/project team and the new skills required. It is also important to understand that the automated tests are now a fundamental artifact for your system, just like the written test specifications used to be, and it is to be owned by the test lead. Just because it is automated and uses code to run, it is not to be handed over to the developers.

The main pillars of Scrum according to me

In this post I'll share with you what I think are the most valuable pieces of Scrum, pieces I'm never willing to compromise over.

Iterative/incremental

Scrum is truly iterative mandating to produce a fully functional increment of the final product, including at least some business functionality, in each sprint. In addition each sprint ends with a sprint review and retrospective making the feedback loop explicit.

Transparency 

In Scrum everything is visible. Any misconception in the organisation, design or infrastructure is pulled into the light. Honesty is just the first name!

Continuous improvement

Fast feedback loops, in larger and larger circles, and transparency are a solid ground for continuous improvement both to the product under development and to the process itself. Always strive to do things faster and with higher quality than the last time around.

Self-managing teams 

Scrum is based on the belief that a team of professionals is most effective if left alone to organise, solve its problems and work towards the sprint goal instead of having management telling them what to do and how to do it. Managers are to take the role of facilitators/sponsors making sure the team has everything it needs to produce as much value as possible.

Definition of done 
When is a task done? In order to truly complete a piece of work and deliver a high quality increment it is absolutely vital to have a solid definition of done. Any work that should have been done to get the increment ready for production will not go away if left undone. Instead it will accumulate over the following iterations and add up exponentially to a large, undefined, piece of work requiring a stabilisation phase of unknown length to meet quality criteria.

These are my fundamental values which I try to stick to. What do you think? Do you have other/additional most pressures parts?

Borders make communication break down. Don’t let them pile up!

Communication involves transferring information from the sender to the receiver. In human communication the sender and receiver are each in separate personal contexts based on their personal history, interests and the reason for which they engage in the act of communication. The data being transferred (i.e. the words spoken or written, pictures drawn, the body language being used, etc) are formed based on the information being processed and filtered through the sender's context and then re-interpreted to information through the context of the receiver. Successful communication relies on the information not being to much obscured through this process.

This said, all human communication occurs between contexts, the recipe for successful communication is that the differences between the contexts are as small as possible. So what is it that makes these context differ? What is it that makes communication hard? I like to use the term "borders". In professional communication, e.g within a project or within a department or a company, many different types of borders can be identified. If you were to communicate without crossing any of those boarders the communication wouldn't be very interesting or valuable, so that is not a success factor. But when laying out your organisation you should take care not to place different borders at the same place since borders tend to reinforce each other and pile up to something higher than the sum of its parts, i.e. making important communication unnecessary hard.

Let's have a look at some commonly found borders in professional life to make the idea clear.

Borders between professions
It is easy for a developer to talk to a fellow developer and for a tester to talk to another tester since they share the same professional context, same professional interests, similar experiences, etc. It is much harder for a developer to talk to a tester or a requirements analyst since that communication is crossing the border of professions.

Borders between area of responsibility
Area of responsibility could be the project you are currently working in or the sub-system (within a larger project) that you are assigned to, but it could also be the requirements for a project as opposed to the code base of the same project. People with different areas of responsibilities tends to have different focuses and priorities and hence communicates out of different contexts.

Borders between geographic locations
Many software projects of today are distributed over several different locations. It might be different countries, different cities, different buildings or as little as different floors or even different rooms. Never the less we, being humans, quickly develops a common context with those people sitting in the same room as opposed to those at another location, hence communication over geographical boarders is harder. Having to consider not being able to speak face to face with people at other locations but being bound to telephones, e-mail, IM, etc, doesn't get communication any better
.
Borders between chains of command
People separated into teams with different managers also tend to have a harder time communicating than those being assigned to the same team. In some cases management policies might not allow any communication except through the chain of command and that is obviously a problem. But even when direct communication is allowed it suffices from the different context resulting from different priorities in the two teams.

Borders between cultures
The clash of cultures is pretty common in more or less any software project these days. It might be due to organisational mergers, consultants being brought in to work alongside employees or part of the project being outsourced to another company or another country. The more cultural differences the higher the communication border.

So, what to do with all this? It is something to think carefully about when setting up your project organisation. E.g. consider a pretty common organisational pattern: You start with placing, analysts, architects, developers, testers and CM:s each in their separate team with a separate manager only reporting to you, being the top line project manager. Then you decide that development should be outsourced to another company placed in another country (typically with lower cost per developer) and you provide them with no other means of communication than e-mail, IM and phones. Said and done, after some time you start to wonder why there is so much frustration within the other teams, everyone saying it is impossible to work with the development team. Yeah, not so hard to understand, right? You have managed to put the development team in a context where all borders, profession, area of responsibility, location, chain of command and culture, are placed at the exact some place, i.e. right between the development team and the rest of the project. Given this you should be happy if any communication at all is taking place.

Of course, it isn't possible to have the whole project in the exact same context, since it wouldn't be very practical to have only developers or only testers making up the whole project. So, some cross-boarder communication is necessary and that is just find. Usually it works well if you take care to set up your organisation so that borders do not coincide. E.g., this is one of the reasons why cross-functional teams are good. The team members have to cross the professional boarders and perhaps also cultural once to communicate but they are all within the same area of responsibility and (hopefully) in the same location. If you need to have an off-site development team (perhaps with another cultural background) it is a very good idea to ease the communication by having at least one developer on-site handling the communication with the development team over the geographical boarder.

The bottom line advice: Watch out for context borders and don't let them pile up to destroy communication within your project!

Iterations based on feedback deepens your understanding

A few weeks ago I came across a challenge in the form of a competition from one of the JFokus exhibitors. It seemed fun and small enough for a spare-time project so I set out to solve it.

For those of you not reading Swedish I'll give you a translation of the problem statement:

"What should be done is to reduce a string as much as possible. You do this by repeatedly removing sub-strings. Not any sub-string, but only from a given set.

The string to reduce is:

VOCDIITEIOCRUDOIANTOCSLOIOCVESTAIOCVOLIOCENTSU

To reduce it you are allowed to remove occurrences of the following words:

TDD, DDD, DI, DO, OO, UI, ANT, CV, IOC, LOC, SU, VO

E.g. some repeated removals could be:

1) ...BIDDDOCDDD...  (remove DDD)
2) ...BIDDDOC...     (remove DDD again) 
3) ...BIOC...        (remove IOC) 
4) ...B...           and so on... 

What does the shortest string you can produce in this way look like?

To be a correct solution the program should be able to produce the shortest string given these in data and also for other start and reduction strings. In other words, it should be a general solution for this type of problem."

In an initial analysis session I managed to solve the given string by hand and thereby deduced an algorithm in three steps:

• First split the start string based on "unreducible" chars, i.e. chars in the start string not part of any of the reduction strings.
• Reduce each of the reducible sub-strings by recursive removal of reduction strings.
• Concatenate the results into the shortest possible string.

Due to a limited (and unknown) amount of time to spend on the problem I decided to split the work into two iterations each resulting in a delivered solution to maximize my chances of being able to post at least some solution to the competition.

Iteration One
In the first iteration I decided to work on the second step of my deduced algorithm, i.e reduce the string by recursive removal of reduction strings, since it could stand as a solution on its own. Yes, this is definitely a brute-force, not so elegant, solution but never the less it solves the initially stated problem. I made the recursion happen inside a loop over all reduction strings to explore the shortest rest of applying the reduction strings in different order. From a performance point-of-view this pretty soon get out of control when the length of the start string and number of reduction strings grow, but for a iteration-one-solution I think it was OK.
Before packaging up my application I worked it over to ensure easily readable and neatly factored code guided by unit tests.

Iteration Two
Soon I got feedback that my solution did not solve all invariants of in-data used to evaluate submissions, i.e. it wasn't a good enough general solver. I supposed this was due to the severe performance problems kicking in for a bit longer start strings and higher number of reduction strings.
I quickly wrote some additional end-to-end unit tests with tougher in-data and got to work on adding implementation for the first and third steps of my initial algorithm. The well tested and factored code made it easy to add the new functionality. In addition I added caching to ensure that no sub-string is ever evaluated more than once. Running my end-to-end tests through the profiler tool provided evidence for a strong performance boost; the brute-force solution of iteration one did almost 140 000 recursions over the initially stated in-data to find the shortest string, now I was down to 92 recursions!
Architecturally-wise my solution wasn't fancy or innovative, I rather stayed with the OO-best-practices I normally use. The solution was implemented in Java with a simple interface describing the API for the logic.

public interface StringReducer {
    String reduce(String startString);
}

I now had three implementations of this interface (UnReducibleCharBasedStringReducer, RecursionBasedStringReducer and CachingStringReducer) that could be linked together to implement the complete algorithm.

Iteration Three
Again I got feedback that my solution did not solve all invariants of in-data used to evaluate submissions. At first I was a bit surprised since I really thought the solution was complete, but the feedback made me think harder (and open up another iteration) and soon I realized that my algorithm was lacking an important piece: For the general case, there is no guarantee that there will be a single string that is the shortest. It might very well be that the shortest result includes two or more strings with equal length. Consider for example the start string "ABC" and reduction strings "AB" and "BC". Reduction using "AB" gives the rest "C" while reduction using "BC" gives the rest "A", i.e. two different strings with equal length depending on the order in which the reduction strings are applied. So far my solution had randomly returned the latter of these results, omitting the first. Now I changed the StringReducer interface to include the notion of a Rest wrapping one or more strings with equal length. This required a pretty significant re-write of the implementations but the tests already in place guided the work and secured a high quality result.

In the process of re-write I also found another situation that could yield multiple results. Consider the start string "IOIOI" and the reduction string "IOI". This set-up could return two alternative results, "IO" and "OI", depending on which of the two occurrences of the reduction string that is removed first. Adding this to the RecursionBasedStringReducer was as easy as adding a new test and adding another loop in the recursive function.
With these modifications I submitted my solution for the third time, a couple of days before the dead-line of the competition.

Iteration Four
This time the feedback said that my submission was accepted and I waited for the jury to make their decision on the perceived best solution. I didn't win - but I followed the heated discussions taking off right after the jury had announced their decision. In this discussions several suggestions for hard-to-handle in-data were put forward. For each I added new tests to my solution and found that it could handle most of them. The exception was a rather long start string (not possible to split) with associated rather many reduction strings that could be used to reduce the entire start string. I'm pretty confident my solution eventually would have come to the right conclusion but the large number of reduction strings made the number of needed recursions huge. But of course the solution is to pay attention to the first reduction path returning an empty string. After that there is no need to continue searching since no string could possibly be shorter. Again adding a test and injecting a simple condition in the right placeRecursionBasedStringReducer was enough to solve the problem.
In doing so I also found a case of "false reduction strings", i.e. reduction strings containing characters not found in the start string. These strings will never be able to reduce the start string and hence can be removed right away instead of causing extra reduction paths to be evaluated. This finding caused another small update to the implementation and that's the current state of my StringReducer.

Conclusion
My take-away from solving this challenge is this real life example showing that working iteratively based on feedback helps you to arrive at a deeper understanding of the problem domain and a step-wise better solution, something we should always strive for in any project regardless of scope and domain.
In addition it was great fun!

We are all Gardeners and Authors

In the previous book on Software Engineering that I read the work of designing and developing software systems was characterised as gardening. The authors talked about the ever ongoing effort of sustaining and improving the internal quality of the code in terms of nurturing, guiding and pruning. I kind of liked that picture of the system under development as something fragile and tender that we should lovingly care about to make sure it will grow into something really valuable for the business we set out to serve.

In terms of software development this work of a gardener's can be translated into finding the proper input of both domain knowledge and reusable pieces of existing code (nurturing), applying a sound architecture to shape the system form (guiding) and constantly refactor to improve existing code and get rid of what is not serving its purpose any more (pruning). At our disposal we have tools like Domain Driven Design (DDD), Test Driven Development (TDD), different levels of software patterns and a powerful IDE. Growing a system is also a great metaphor in these times of agility. Doing the agile development right we will continuously add small pieces of new functionality to a working system always being careful not to wreck the tender "plant" we are growing. In other words - we should all be Gardeners.

We should also be Authors. In the current book I'm reading, "Clean Code" by Robert C Martin, "Uncle Bob" argues that good source code should be just like prose. It should be easy and enjoyable to read as well as informative and clear about its intentions. After all the code we write will be read by ourselves and others many, many, many times in the future. Writing  code that can be correctly and efficiently executed by a computer is not that hard. In most cases the compiler and run-time optimizer will take care and by adding some tests to the mix we should not get that many surprises. The really hard part is to write code that can correctly and efficiently be read, understood and changed by humans. So, next time you sit down to do some programming, try to think of yourself not as someone talking to the computer but as an Author talking to your fellow and future developers with the source code as your means of communication.

True iterations deliver value

Everyone wants to be iterative and incremental these days. Iterations are indeed nothing new; they have been around inside and outside of RUP and other methodologies for a long time and they show up again in SCRUM (called Sprints) and XP. Many projects claims to be iterative, but are they really?

In my view you are truly iterative only if you for each iteration go through the whole cycle of analysis, design, implementation, integration, test and deploy to really produce a high quality increment of the final system. Many projects I’ve seen claims to be iterative but tends to focus only on analysis and design in the first iteration, hence implicitly converting the iterations to phases in a good old fashioned waterfall.

The only way you get to benefit the true value of iterative development is to run through the whole cycle and collect feedback from everyone involved. Then you know how to do even better in your next iteration. As a Software Engineer we should really push for true iterations in our projects since that is the only way we are sure to be involved and able to give and receive early feedback.

Software Engineering Essential Reading 1 - Growing Object-Oriented Software, Guided by Tests

Under the title "Software Engineering Essential Reading" I plan to share short reviews of books and articles I find most useful and an essential read for every software engineer.

First up is an (in my opinion) outstanding book on Test Driven Development (TDD) written by Steve Freeman and Nat Pryce. The title is "Growing Object-Oriented Software, Guided by Tests" and, as the title suggests, this book is more about how to successfully build changeable and maintainable software using TDD than a technical book on a specific testing framework. Sure, through the book they are using jUnit 4.x and jMock2, but the book is not a go-through of framework features, rather it uses lots of example code to show how they "grow" (just like a gardener does with flowers) software. These frameworks just happens to be their favorite tools.

The book roughly consists of three parts:

1. First the authors go over their motivation for TDD and the principles and techniques they apply to both testing and OO-design and -implementation. Part of this is really heuristics to follow for a clean, readable and maintainable design of production code which also is greatly discusses in other books, but I like the fact that it is included here because it brings the whole picture of responsible software development rather than just explaining TDD ripped out of its context.
   
2. In the second part they go through a long (about 150 pages) example of "growing" an application using the principles an techniques from part 1. The example is very well done and easy to follow. In the beginning I felt it was very valuable, but at the end I thought it was running a bit long. But just as I was about to skip some pages the book went on into part three.
   
3. In the last part they wrap up and enforces their key ideas by going over their principles in more depth backed-up with the examples from part two. In many passages this part is so dense and right-to-the-spot that you have to pause a minute or so after each paragraph just to think over the truths that just were brought to you.

All in all, this is a truly great book and I would say it goes straight to the top of all SE-related books I've ever read. I strongly recommend anyone engaged in software development to get a copy.

Are you test infected?

That was the words a Finnish colleague used when first introducing me to unit testing using jUnit. It was back in 2002 when I had the pleasure of working with Capgemini Finland in a Helsinki-based project for about 18 months. The thought of writing test code to test my production code seemed good in theory since it would inevitably help me to find some bugs. But, wouldn’t it be at least twice as much code to write? And wouldn’t writing twice the amount of code make the whole project take twice the time to complete? I wasn’t in any way convinced that the practise of writing unit tests would make me a better developer.

I might be skeptical but I try hard not to be ignorant to new ideas. I tried writing a few tests and saw them pass without modifying my production code at all. I didn’t find any bugs, so what was the benefit of this? It didn’t mean I didn’t have any bugs in my code, it was just that since my design was not made with unit tests in mind I found it to hard to write any tests that really drove out the bugs. Instead we found them in systems testing and, like we were used to, had ourselves some late nights in-front of the debugger.

Since then there has been a good deal of writing and discussions on Test Driven Development (TDD) and I’ve been experimenting with both jUnit and TDD. Over the years I have come to solve more and more complex problems in my java designs and for some years I’ve been very reluctant to write or change any code without writing a test first to prove to myself what has to be done and that my code meets the requirement without introducing new bugs. Thanks to the excellent jUnit support in NetBeans, my IDE of choice, it isn’t a huge effort to write some extra code just to be sure the production code really works.

Now I’m happy about being test infected since I have proved to myself over and over again that I save time by writing tests as I design and code. In my experience it doesn’t take more than a few hours of coding, even on a very small project, to have a test fail unexpectedly and as soon as it does it means you have found a bug that you at some late point might have had to spend hours to debug.

So, if you are not yet test infected make sure to take a look at the xUnit framework for your language of choice. And if you are, please share your TDD experiences by commenting on this post.

No contract can compensate for lack of trust

Many buyers of IT development projects seem to think that all problems and unpleasant surprises can be avoided by buying a fixed-price, fixed-time, fixed-functional contract. My experience, however, is that if you haven't built enough trust between buyer and supplier you will at some point, often late in the project, end up in endless discussions on what was really part of the scope and how to interpret the requirements written in the contract. This is just natural since both parts, lacking trust for each other, are focusing on maximizing their own profit from the business relationship.

In the agile world one giant step forward has been taken - to embrace change. This results in fixed-price, fixed-time, fixed-functional-effort contracts where decisions on functionality can be deferred and revisited in each iteration. This gives the buyer the benefit of actually receiving what they need instead of what they thought they would need in the future by the time of contract signing.

However, even in the agile approach you may end up in long discussions on how to define the effort of any given feature to be implemented - if you are in a non-trust-relationship that is. I think that relationships between organisations aren't any different from relationships between people. It is just not possible to base a lifetime of living together with another person on contracts. It has to be based on deep and honest trust that both parties honour.

So, before you decide to by an IT development project from a vendor, decide on whether the vendor is to be trusted. Also try to find out if the vendor has trust in you as a buyer. If so, go forward, form a common goal and sign the contracts you think are necessary for guidance. But do not try to build a relationship on contracts if you are lacking the trust.

Use SCRUM to increase flexibility, efficiency and transparency in the delivery process

In this post I’ll give a short overview on why I think using SCRUM is a good way to increase flexibility, efficiency and transparency in the delivery process.

Flexibility
In SCRUM we work with short, iterative, releases (sprints) where the customer (product owner), at the beginning of the sprint, is allowed to decide what is most important right now and should be included in the next release. Hence "time-to-market" for a typical feature can be reduced to only the length of a sprint, typically 2 to 4 weeks.

Efficiency
SCRUM emphasises the delivery team as responsible for the delivery they have committed to within the sprint. The team is also empowered to make decisions on how to organise the work to achieve maximum efficiency now and in future sprints.

Communication on priorities, requirements and possible designs is short-circuited between the team and the product owner. Hence, additional management and detailed planning is eliminated and feedback cycles are sped up. Everyone involved is contributing directly to the delivery of high-quality code, following the lean principle of "eliminate waste".

Transparency
SCRUM introduces complete transparency to the delivery process. Expectations for the next sprint is set and agreed between the product owner and the team in the "sprint planning meeting" at the beginning of the sprint. Current status is tracked every day in "daily scrums" where all team members discuss what has been done and what should be done next. Anyone is invited to listen in on "daily scrums".

At the end of the sprint a "sprint demo" is organised where the team demos the features implemented in this sprint. Everyone is invited to the "sprint demo" to see and comment on what has been done and what needs to be done next.

The SCRUM team
A typical SCRUM team consists of 5-8 people including every competency needed to produce a production ready release in every sprint. Competency in the team includes:

- Analysis
- Design
- Implementation
- Test
- Configuration Management
- Scrum Master (responsible for the delivery process)