From six months release cadence to continuous deployment

Since it has been a while, almost ten years, since I blogged about what I'm doing at work I thought a bit of a summary would be appropriated. I don't intend to talk you through every task I've done since no one would be interested in that, but I thought perhaps it might be interesting to some of you to read about me joining the Swedish Pensions Agency (Pensionsmyndigheten, in Swedish) eight years ago and how we collectively since then managed to move from an enforced three, but in reality six, months release cadence for all systems to a continuous deployment scheme where individual teams and the part of the business they serve is in charge of how and when new and changed features are brought to production.

Disclaimer: This is my story, my perception of how it was, what happened and what it has become. Many of my current and former colleagues who are/were involved might have a slightly, or totally, different view. I'm writing this with a hope it will inspire others to discuss and question matters of organisation and software development processes and how they affect the daily joy and development efficiency of an IT-organisation.

How it was 8 years ago

In 2014 I decided I was done with consulting, at least for now, and wanted to try out being an employee of the organisation where I spend my working days. I found a position as systems architect/developer within the development side of the IT-department at the Swedish Pensions Agency. The ad spoke about an agile and forward looking agency basing their development on Java and open-source tools I liked. At that time we were about 80 developers and testers maintaining and developing systems where some had been around a good ten years already, integrating with even older systems with their roots in the 80's or 90's.

We were all divided into three, about the same sized, silos without much direct communication, each managing a related set of systems. Communication and coordination was mainly handled through people being included in smaller or larger projects. Releases to production were mandated to happen every third month (for all systems synchronized), with a good set of manual end-to-end-testing leading up to it. Any release between those where to only include really critical bug fixes. And yes, we did a fair amount of those.

In my silo, "case management", we were further divided into people (most of us) working in the ever (it seemed) ongoing "program for case management" with a six month release cadence of new functionality, and the maintenance team (the rest of us). In the maintenance team we got a load of new functionality (and code) to maintain every six months and most of what we did was fixing bugs in someone else's code and then spend a disproportional amount of time merging the fix into branches representing the release due in a month's time, the one four months away and sometimes also the one due seven months into the future since development had already begun there.

We used daily meetings (standing!), but no one would call that "agile" today, and I don't think no one really did back then either. The "agile" in the ad I think was a more correct description of the state of mind or spirit within the organisation which came as a very positive surprise to me. There were already a great number of small experiments (or "pilots" as they were called) running in the organisation and whenever a new idea was presented by someone the answer was almost always to try it out as a "pilot". Moving to use Jira for internal organisation of work was one of those "pilots" that came to stuck.

What happened

In beginning of 2015 we were all pretty fed up with the current situation. One of our neighboring silos, "external web", started up an official pilot to try out agile delivery with self organized, cross-functional teams and a six weeks planning horizon. At the same time they had started moving towards a brand new architecture with microservices and continuous deployment pipelines. Despite maintaining a large monolithic system with about half a million lines of code and no ambitions, nor opportunities to change the architecture, we still wanted to find another way of working also in the "case management" silo.

With support of first line management we organized a first workshop aiming to define what "awesome" would look like with respect to development organisation and delivery process within our silo. The result described cross-functional teams, each focused on one or two subdomains, responsible for both developing new features and maintenance tasks, working in a continuous delivery manner where changes were deployed to production weekly or more frequently when need arise. Many would say going to production weekly isn't continuous delivery. I think the important thing is to be able to do it more frequently, but actually doing it is a business decision and they thought, and still do, that once a week is great for the internal user base of case management clerks.

In cooperation with first line management we formed a small team to meet once or twice a week to discuss, develop and coordinate our abilities to meet this description of "awesome case management". We defined roles and teams. We engaged with parties outside our silo to find out boundaries for our new processes and to create ways to cooperate. We enhanced our technical abilities.

In order to do deployments to production every week, instead of just four times a year, we had to do those deployments during office hours, which meant they had to be zero downtime from a business perspective. We also had to make the whole process, from build and test to creation of release notes, automated. Luckily the system already had a heavy set of automated unit and system level tests which made us confident that if all those passed we were good to go to production. With a small team of dedicated engineers, helped by the fact that the whole agency at the same time moved sources to Git/Bitbucket, we built Jenkins pipelines, integrated with Jira and Bitbucket, to manage building artifacts, running tests and creation of release tickets, including information on all changes going into each release.

After working out the organisational, process related and technical issues with everyone involved we turned our silo into the new team structure and ways of working in march 2016. After a bit over a year of discussions and preparations we stopped moving people to work and stared moving work to people (teams). And we didn't look back. Of course it wasn't perfect, but with continuous improvement processes on several levels we were able to improve as needed. The area where we had the most problems was in prioritization and scheduling of larger activities, all coming from differently sized projects still run in parallel across the agency. This was a bit outside our reach, since it crossed several silos. We needed something more.

Luckily, with two silos having turned their organisation and way of working around in similar ways there were a pretty good case for the agency to continue forward and find new, more agile, ways for the whole development organisation. SAFe is a framework for prioritizing and coordinating work across many teams in an agile and lean way and since our new practices heavily resembled team and delivery practices in SAFe it came natural for the agency to decide to embark on that route. Of course there where a lot of education and discussions to be done, especially with upper management, but since the director general of the agency was a firm proponent decisions were made in due time and a first SAFe Agile Release Train (ART) was started up as a pilot late 2019.

What it has become

After about a year of trying out SAFe practices and organisation structures, as well as changing to full time remote work during the pandemic, the organisation decided we had gathered enough experience to have the whole development organisation go full-on SAFe. About one and a half year later we are running five ARTs and a few supporting groups for tooling and very specialized services. Each ART is organized around parts of the agency's rather broad business domain and employ both business and IT-specialists in their teams and management groups. SAFe's lean portfolio management helps us organize and prioritize all the large or cross-ART initiatives and the time of projects fighting over individuals for work on full- or part-time is forever over.

With the pandemic no longer directing how society works we have been able to employ both on-site PI-planning events and frequent "teamdays" at the office. All our gained experiences over the past two years have proven to us that SAFe practices works both in an on-site and in a fully remote setting. Our ARTs and teams now operates with a mix of working remote and being at the office. This gives everyone a great deal of personal freedom and also lays the foundation for team being able to function despite being split over offices in different parts of Sweden.

Also the infrastructure side has continued to develop. With a modern cloud platform and CI/CD-tooling, including a formalized microservice archetype, it's now possible to spin up new services in a few days and have them deployed to production whenever the business sees fit, several times a day if needed.

Conclusion

It has been a really interesting, sometimes frustrating, but overall joyful journey for eight years. We have moved from projects throwing software "over the wall" to teams taking full responsibility for their systems and services all the way to production, from endless discussions over priorities and attempts to maximize utilization to pull-based planning and focus on throughput, from excessive merges and an in-reality six months release cadence to releasing new features as they are ready and we still want more. We are each day looking for ways to make our services better and our organisation a bit more efficient and capable. If you stay tuned chances are you may hear of some of that on this blog. 

If you happen to be living in Sweden (and talking Swedish) and fancy come working with us, checkout our current job openings. Over the years our IT-organisation has grown, and it continues to grow. Our society are in the middle of the digital transformation and IT is at the heart of our agency's business. I think we will see many interesting and challenging opportunities going forward.

What about PMs in Scrum-based development?

In scrum terminology there is no such thing as a project manager (PM). However, in addition to the product owner (PO) and the scrum master (SM) Henrik Kniberg in his excelent "from the trenches"-book talks about a "sponsor". Someone that provides for the teams long-term staffing and work environment as well as comitting to remove impediments outside the team. He describes himself, in his position as development line manager, as having this role.

In my experience many large organisations are totally project organised bringing together people from different parts of the organisation as well as external consultants to form a team. In such organisations the traditional PMs, if accepting to have an empowered team, is often ideal for this sponsor role.
That brings me to propose the following division of responsibility:

Product Owner: Responsible for what the team is to build. Merges requirements from different stakeholders and manages priorities.

Scrum Masters: Guides the team and PO in following the agreed process. Makes sure the surroundings understand how the team works and do not interfere. The SM role is usually not full time and is therefore filled by a member of the team.

Project Manager: Responsible for long-term staffing and external contacts in order to provide the team with the best possible environment to meet the needs of the organisation.

Value Producers and Supporters

In the lean and agile world anything that produces value for the client is a good thing, everything else should be carefully scrutinized before carried out. This doesn’t mean you shouldn’t produce any documentation, it just means you should produce only the documentation that someone is likely to read. It doesn't mean you shouldn't produce any automated tests, it just means you should make sure you produce them in such a way you optimize value during the development period and for future maintenance. To do this right you need to understand which part of the project organization is producing direct value for the client and which part is supporting the Value Producers.

In many projects you could find three groups of Value Producers: programmers who produce code that builds the product, testers who make sure the product does what it is supposed to do and technical writers who produce the user documentation. Everyone else, requirements analysts, architects, designers, etc, are usually in a supportive role. The Supporters does not themselves produce anything of direct value to the client, however that doesn’t mean that their work is not important. It is. As long as they make it easier for the Value Producers to do their work, now and in the future.

That said one of the challenges in running an efficient agile project is to understand who is supporting who and how to make it as easy as possible to produce maximized value for the client. In order to do that you first need to understand what deliverables (both artifacts and activities) that produce value to the client. In some projects the client might get great value from the structuring and questioning on  business processes done by the requirements analysts in order to define the system requirements. In other projects the client is perfectly clear on their business and the requirements specs are only needed to formally document the systems functionality and drive testing. Next you remove any activity or artifact that does not  add value to the client or is invaluable in order to support efficient value production and make sure everyone inside and outside of the project understands their role and for whom they are supposed to produce value, in other words: if  they are a Value Producer or a Supporter.

No man is an island

What’s the most common reason for failure in a software project, or perhaps in any project? You will get many different answers depending on who you ask. The designers might say “The requirements were not clear enough and the programmers didn't follow the architecture”. Testers on the other hand might say “The programmers never delivered what we were supposed to get when we were supposed to get it” and programmers tend to say “We got lots of ‘bug-reports’ that where really outside the scope”. The list can be long, but at the end of the day it all comes down to lack of communication.

Let’s face it, developing software is a complex and creative handiwork. That’s why we try to stay away from the over-planned, defined, way of waterfall and turn to iteration based approaches like RUP or Scrum. True iterations give us great feedback to base the next, (hopefully) better, version on. However, working with iterations is not enough. The saying “No man is an island” origins back to about year 1600 and is as true as ever when it comes to software projects. For many years we have set up the projects so that specialist testers form a test team, specialist designers form a design team, specialist programmers form implementation teams, and so on. This for sure brings advantages in having specialists learning from each other becoming even more specialised. But it also introduces a great deal of hand-overs between the teams, each missing some vital information. I believe this way of organising the development project is one of the key reasons why projects fail.

In the agile community we talk about “Cross-functional teams“, meaning that people with different specializations work together to achieve a common goal. In software development that means that a programmer sits next to a designer and a tester, working together on transforming a requirement into a piece of well designed, well tested code ready for production. The agile practices are all about maximizing communication, e.g. by locating people in the same room with access to broadband communication aids such as whiteboards.

If you are currently in a project where the test team and development team sit at different floors and the primary tool for communication is release notes, then you have a superb opportunity to increase the likeliness of project success just by re-locating so that developers and tester form collaborative teams working together on both design, test design, implementation and test execution.

Automated functional tests - an organisational change

Introducing automated systems (functional) testing to a project used to run manual functional and regression tests at the end of every development cycle is not a technical issue – well, it is to some extent, but there are some good solutions out there – it is a fundamental change to the work your testers perform.

In the manual approach a typical set up of roles dictate that test designers create written test specifications in some tool or in ordinary documents. The specifications contain everything the tester needs to execute the test, including detailed step-by-step instructions.  The tests are then manually executed over and over again making sure to catch any regression in the system functionality.

When moving to automated testing the previously text based test specifications are transformed into an executable format. Perhaps the behaviour driven approach is used with some short textual description that underneath is connected to code exercising the system, preferably through the GUI if such exists. Since execution of the tests is automated the testers previously doing the manual test execution are no longer needed. Instead test implementers, with the ability to build the code that executes the tests, are working alongside the test designers. Tests are executed by the build system, perhaps nightly or even more frequent making the feedback loop faster.

This is a big change - and yes, it is a good one - but it is important to understand the impact to the organization of your test/project team and the new skills required. It is also important to understand that the automated tests are now a fundamental artifact for your system, just like the written test specifications used to be, and it is to be owned by the test lead. Just because it is automated and uses code to run, it is not to be handed over to the developers.

Borders make communication break down. Don’t let them pile up!

Communication involves transferring information from the sender to the receiver. In human communication the sender and receiver are each in separate personal contexts based on their personal history, interests and the reason for which they engage in the act of communication. The data being transferred (i.e. the words spoken or written, pictures drawn, the body language being used, etc) are formed based on the information being processed and filtered through the sender's context and then re-interpreted to information through the context of the receiver. Successful communication relies on the information not being to much obscured through this process.

This said, all human communication occurs between contexts, the recipe for successful communication is that the differences between the contexts are as small as possible. So what is it that makes these context differ? What is it that makes communication hard? I like to use the term "borders". In professional communication, e.g within a project or within a department or a company, many different types of borders can be identified. If you were to communicate without crossing any of those boarders the communication wouldn't be very interesting or valuable, so that is not a success factor. But when laying out your organisation you should take care not to place different borders at the same place since borders tend to reinforce each other and pile up to something higher than the sum of its parts, i.e. making important communication unnecessary hard.

Let's have a look at some commonly found borders in professional life to make the idea clear.

Borders between professions
It is easy for a developer to talk to a fellow developer and for a tester to talk to another tester since they share the same professional context, same professional interests, similar experiences, etc. It is much harder for a developer to talk to a tester or a requirements analyst since that communication is crossing the border of professions.

Borders between area of responsibility
Area of responsibility could be the project you are currently working in or the sub-system (within a larger project) that you are assigned to, but it could also be the requirements for a project as opposed to the code base of the same project. People with different areas of responsibilities tends to have different focuses and priorities and hence communicates out of different contexts.

Borders between geographic locations
Many software projects of today are distributed over several different locations. It might be different countries, different cities, different buildings or as little as different floors or even different rooms. Never the less we, being humans, quickly develops a common context with those people sitting in the same room as opposed to those at another location, hence communication over geographical boarders is harder. Having to consider not being able to speak face to face with people at other locations but being bound to telephones, e-mail, IM, etc, doesn't get communication any better
.
Borders between chains of command
People separated into teams with different managers also tend to have a harder time communicating than those being assigned to the same team. In some cases management policies might not allow any communication except through the chain of command and that is obviously a problem. But even when direct communication is allowed it suffices from the different context resulting from different priorities in the two teams.

Borders between cultures
The clash of cultures is pretty common in more or less any software project these days. It might be due to organisational mergers, consultants being brought in to work alongside employees or part of the project being outsourced to another company or another country. The more cultural differences the higher the communication border.

So, what to do with all this? It is something to think carefully about when setting up your project organisation. E.g. consider a pretty common organisational pattern: You start with placing, analysts, architects, developers, testers and CM:s each in their separate team with a separate manager only reporting to you, being the top line project manager. Then you decide that development should be outsourced to another company placed in another country (typically with lower cost per developer) and you provide them with no other means of communication than e-mail, IM and phones. Said and done, after some time you start to wonder why there is so much frustration within the other teams, everyone saying it is impossible to work with the development team. Yeah, not so hard to understand, right? You have managed to put the development team in a context where all borders, profession, area of responsibility, location, chain of command and culture, are placed at the exact some place, i.e. right between the development team and the rest of the project. Given this you should be happy if any communication at all is taking place.

Of course, it isn't possible to have the whole project in the exact same context, since it wouldn't be very practical to have only developers or only testers making up the whole project. So, some cross-boarder communication is necessary and that is just find. Usually it works well if you take care to set up your organisation so that borders do not coincide. E.g., this is one of the reasons why cross-functional teams are good. The team members have to cross the professional boarders and perhaps also cultural once to communicate but they are all within the same area of responsibility and (hopefully) in the same location. If you need to have an off-site development team (perhaps with another cultural background) it is a very good idea to ease the communication by having at least one developer on-site handling the communication with the development team over the geographical boarder.

The bottom line advice: Watch out for context borders and don't let them pile up to destroy communication within your project!