DDD on top of Services, DDD inside Services - A Domain Driven approach to SOA

This post is about Domain Driven Design (DDD) and Service Oriented Architecture (SOA) and how the former can be used to build services for the latter. But it starts of in a few observations about current state of the union.

The "DDD needs a database" assumption

I've come to find it being a pretty common understanding that software systems using DDD building blocks have to be backed by a database, and, even more specific, having repository implementations using an ORM framework. This leads to the assumption that domain entities, once added to the repository, are attached to an EntityManager that is responsible for "auto-magically" saving changed data as transactions are completed. I find these assumptions bringing to many constraints on a domain driven system architecture without any need. In fact, I can't find anything in the DDD approach, as put forward by Eric Evans, that supports this understanding. Instead I see the domain model as a piece of code completely free of any explicit or implicit dependencies on the surroundings. For a couple of years I've been working using this approach and in this post I will describe the architecture of such a system where repository implementations rely on services published by other systems as well as storing part of the data in a dedicated database.

SOA as spaghetti on top of CRUD

I've also found Service Oriented Architectures (SOA) to be implemented as mostly CRUD operations for fine grained data objects, or services with, even really complex, business logic firmly placed in Transaction Scripts (TS). TS might be good enough for simple things, but quite quickly the code turns into spaghetti as complexity rises. Complex business logic is where DDD really shines, but given the assumptions discussed above, many people don't see it as an option when there is no underlying database, but a set of CRUD like services.

And there we have another problem with many service implementations. When just using simple DTOs as parameters and/or return values and presenting the client with a CRUD-like service API, what support are we then giving the client-side developer? A bunch of getters and setters that could be set in any combination! How to know what (from a business point-of-view) makes up a valid object? And how to know what are valid modifications? Such APIs force the client-side developer to have intimate knowledge about the server-side implementation, and changes to the API won't show up as compiler errors. When APIs that I depend on change, I like the compiler to be able to notify me.

I think we can do much better and offer the client-side developer much better support by extending the service API with some DDD building blocks like Entities and Value Objects. In this post I will explain how we did that, in effect implementing a published domain.

Separation of concerns - The independent domain model

One of the principles of DDD that I think are most important to apply is "Separation of concerns". It is also part of Robert C Martin's SOLID principles by the name "Single responsibility principle". In short, every piece of code should deal with one problem only and have only one reason to change. It is also an important part of writing clean code since it increases clarity if a piece of code only does one thing. In the DDD perspective I apply this principle by separating code that models the business domain in order to solve business problems from technical code that glues the application together or handles communication with the outside world, such as databases or services exposed by other systems. This leads to a system with a traditional layered architecture with a slight twist: the domain model is kept in the center with no outgoing dependencies whatsoever. The domain model is concerned only with solving business problems, while the surrounding integration layer (or "anti-corruption layer" in Evans' terms) is concerned only with technical issues. The domain model is built using the DDD building blocks (entities, value objects, repositories, services and factories), with repositories (and sometimes also services and factories) represented as interfaces used by the domain, but implemented in the integration layer. Thereby creating the dependency from the infrastructure layer to the domain, and that's the twist! This is perhaps an unusual architectural style, but it is not new. Similar models have previously been described as “Hexagonal Architecture” and “Onion Architecture” among others. Robert C Martin has a nice summary of the history of this architecturalstyle on his blog.

This architectural style brings the following benefits:

- Easy to read business logic, since it is not mixed with code for interaction with databases, messaging services or other technical concerns. This caters for fewer mistakes as the software evolves.

- Easy to deploy. The domain code can be deployed in different runtime environments without changes, since no dependencies exist. At one point we made great use of this as we were moving to a new deployment platform while at the same time developing new functionality for the business.

- Easy to test business logic. All business logic can be tested by automation without any need for a runtime container. Doesn't really have to argue why that is good...

Repository on top of services

Since the domain is only concerned with business logic and the business doesn't care how entities are stored, only the fact that they can be stored and at what step in the business process that happens, all repositories in the domain are represented as interfaces. It is now up to the integration layer to provide implementations for those repositories.

Regardless whether entities are to be stored in a database, in files, or by calling a remote service (In my case it was services exposed via a customer specific API on top of stateless EJB, i.e. basically RMI, but it could be web services or any other protocol as well), or a combination, a repository needs access to the internal state of the entity in order to get (for storing) and set (for re-constructing) values. Typically most of that state isn't public in the domain model, so the repository implementation needs to gain access. Depending on security settings, reflection might be an alternative. In most cases it is not. Instead we made the members of the entity protected and created a specific sub-class of the entity in the integration layer that we used both for reading when state was not public, and for re-constructing entities on read requests.

With the problem of gaining access to internal state solved it is pretty easy to write a repository implementation that reads and writes most of the data over one or several  remote service calls and keeps the rest in a few database tables in a dedicated database. This split storage model is often a result from the fact that existing external services might not fully support the needs of the domain model. Another reason might be performance. In some cases we had to cache carefully selected data pieces in our own database to ensure timely retrieval. The beauty of this approach is that we can do all those tricks needed in the repository implementation without having any part of it leak into the domain model code; the separation of concern is total.

Published domain and services with business meaning

As mentioned above, CRUD-oriented services that let a client store and retrieve DTOs with getters and setter for every attribute push a great burden onto the developer of the client code. If using DDD to implement the service internally, we could do better by offering that domain knowledge to the client, packaged as a published domain and services that carries business meaning. A CRUD-service will never have a place outside the integration layer of the client, while the signature of a service well crafted in business terms might be a good candidate for a service API used directly in the domain model of the client. Of course in the shape of an interface with a small implementation in the integration layer to carry out the technical lifting of making a remote call. But the point is, the service signature can be used verbatim and the integration layer can be kept thin since the client domain doesn’t have to re-define what the service means in business terms; hard earned domain knowledge is reused.

The same goes for the business objects. If we publish those parts of the domain that can be used outside our service implementation, we offer the client-side developer to directly benefit from the domain knowledge we have gathered in building our service. But what parts can be published? Well, if you think about it, the part of a good API that is most useful is the limitations it imposes, i.e. the help you get to avoid doing stupid things. By publishing a model of domain objects as plain Java objects, with constructors and accessors ("getters", but in business terms and not necessarily following the Java Beans convention), we help the client-side developer to avoid constructing invalid objects and accidentally changing state that, from a business perspective, should be immutable. With only this much the  client-side developer gets much better support than with only the raw data format offered by our stateless services. In addition it might also be appropriate to offer a thin layer on top of the service calls that that exposes the services in terms of these domain objects and takes care of transforming them to/from the raw format used to go over the wire.

Having a published domain gives the client-side developer the choice to either just model transformations of it into  a more suitable model for the client context or, if contexts are closely related, decide to take on a conformist approach and extend the domain objects with additional functionality. I've done both in different contexts and it is so much better than having to experiment with DTOs to find out what are valid combinations of attribute values.

Conclusion

DDD is suitable for implementing domains on top of external services. It is also suitable for the implementation of such services, and if we carefully select parts of the domain model to publish we offer great help to the client-side developer.

Ideas and books forming me as a Software Engineer

This September it was 15 years since I joined university to study Software Engineering, the official name was “Informatics”, but I think software engineering (or authoring and gardening as argued in another post) is better describing what I was interested in and, luckily, what I'm currently doing. Since then I've got some 10+ years of experience from working in the industry and a whole 15 years of continuous learning. As an exercise for myself I've set out to go through the major ideas I've picked up and used along with the people and books that collaboratively have formed me to the software engineer I am today. Hopefully you find this post at least a bit interesting and inspiring as well.


Object Orientation (OO)
In 1996, and the following years, OO was the new hot thing in software design, at least at my university. I learned to model domains (even though I didn't know that word by then) in objects using the OMT (Object Modeling Technique) notation by James Rumbaugh (later on one of the main contributers to UML). This way of abstracting the real world into objects with attributes, methods and relations very much appealed to me as it offered the tools to unambiguously document the analysis result by drawing a diagram. I did also believe my teachers when they talked about OO as the enabler of universal libraries of small reusable objects. Now I, and I'm sure they too, know this is never going to happen, but the OO-paradigm still brings structure for reuse within a system and hence is a great tool for honoring the DRY-principle. 
Ever since those first courses I started to think of OO as the way to design and build software systems. Later on I've come to see OO as the way of designing and building transaction based information systems where manipulation of state is the primary focus. Other types of systems might be better suited to use some other paradigm, but this type of systems for administering information is what I've been working with almost the entire time since I graduated. In university I learned the theory of OO, but we didn't actually turn any of our OO-designs into working software. We sure learned programming, but then it was to solve more traditional algorithmic problems. When leaving university I was all eager to learn how to do OO in real world projects. And boy, was I disappointed!
For several years I was employed in projects where I learned lots of things about programming and projects in general but nowhere was OO used to anything but data diagrams (now drawn in UML notation which I learned through Martin Fowler's “UML Distilled”), i.e. objects with attributes and relations but no behavior, that was later turned into database tables and data structures. All behavior was programmed into transaction scripts, either directly in the GUI components or as free standing functions or stored procedures. I was desperately looking for a real world example of true OO implementation because on my own I couldn't really figure out what it would look like in order to work properly. The first piece of the solution came when I got to read Craig Larman's “Applying UML and Patterns” . In this book he describes all aspects of OOA/OOD, how the software can be structured with boundary classes, controllers and entities, and shows by example code how it all comes together. Still, this isn't the approach intuitively encouraged by the structure in popular frameworks such as Java EE and Spring Framework. Out-of-the-box they rather suggests a static service structure operating on DTO:s, and that is, I think, the main reason why most systems are built with data and behavior separated and not using the true OO-paradigm.


Domain-Driven Design (DDD)
In a Jfokus 2008 tutorial I was first introduced to Domain-Driven Design. The three hour tutorial only scratched the surface of DDD but it was enough for me to understand that this was the description of how the full OO-paradigm is used to build real world systems. Soon after, I got to read Eric Evans' book “Domain-Driven Design - Tackling complexity at the heart of software”, which I have posted about previously, and since then I firmly believe that DDD is the approach to use for designing and implementing complex software systems.
DDD is not a method, nor is it simply a technique or an architectural pattern. I would rather call it an approach to software development, including analysis, design and implementation.
Perhaps what most of my colleagues first think of when DDD is mentioned is the design sketches I use to draw with the business domain at the center and technical integration packages all around. The domain being built on the conceptual building blocks, entities, value objects, repositories and services, as introduced by Eric Evans. But even though that part of the approach might be the easiest to pick up, and also important in building working software with DDD, I think what makes the biggest difference to me is the ubiquitous language. This practice of building a model with a shared language based on, but with more precise definitions than, the language spoken by the domain experts is the basis of my philosophy to always build the software so that structure and logic follows the business domain. I think that is the only way to guarantee a flexible design such that a small change to the business is always a small change to the software, never a big one. My experience is that the second you deviate from that principle, most often due to time pressure or pure laziness, you are asking for trouble further down the road. It might take a year or so, but it will come back and bite you!


Test Driven Development (TDD)
If DDD provides the approach on analysis, design and implementation, TDD is clearly what integrates that approach with quality awareness and assurance. TDD is my way to ensure a testable design and a correct implementation that is easy and secure to improve further. In a previous post, "Are you testinfected?", I told the story of how I got introduced to TDD, how I was skeptical at first and how I later on proved the usefulness to myself. Now I rarely, and definitely reluctantly, writes any code without first writing a failing test.


A Clean Coder producing Clean Code
Ever since the start of my career I’ve wanted to produce code that I’m proud of. However, the benchmarks for that assessment have changed over the years. Now I consider well tested and easy to read code being the standard to achieve. Since any piece of code is read much more often than it is written I think readability is prime quality measurement for code. Lately I’ve been reading both Clean Code and The Clean Coder by Robert C Martin. Both are excellent books. The first one giving good advice on how to make the code readable, the second really demanding reflection on what it means to be a professional software developer.


Agile with Scrum and Kanban
Another interest of mine is software development processes and techniques. At first I learnt how to deliver in a world of waterfalls, later I got to experience RUP. Despite all bad things said about RUP, I’ve always found the core messages on iterative, risk driven development to be right and frankly quite agile. However, agile approaches such as Scrum, Kanban and developmenttechniques gathered in XP take this a long way further. The major inspiration here has been Henrik Kniberg, first in his Jfocus tutorial in 2008 and through his two books on Scrum and Kanban.
Currently I’m working in a world of mixing and matching. I’m picking the best parts out of all those above to create a process and way of working that supports being as agile and lean as possible in an otherwise RUP-ified world. I think, at the current state it is somewhat of coming success.


“All problems are not worth solving”
That was one of the advice I got from an experienced colleague of mine many years ago when I was aspiring to go from being a sheer hacker to become a CapgeminiCertified Software Engineer. He probably gave me a bunch of other advice too, but this is the one I still remember. And I also think, for me personally, it might be the most valuable advice I’ve ever got. Since problem solving is my trade I happily go of designing solutions, then it is good to remember thinking for a minute or two about whether the problem really deserves being solved or if it is something we can as happily live with. At the bottom of this is that all resources are limited and needs to be applied where they make the best return on investment. On the other hand, we need to be careful not to spend too much time on deciding if a problem is worth solving, i.e. it is not always worth solving the prioritization problem if the solution offered is cheap enough.


The future
This pretty much sums up where I am today, but of course I will continue to evolve and in the near future I see the following ideas as the most interesting.
- Strategic design – How DDD concepts such as Bounded Context is applied at a larger scale to make strategic decisions on how to factor architectures and apply different design and development strategies. In most discussions “architecture” seems to mean frameworks, databases, application servers and technical layering. I think architectures would be much more interesting and usable if they concerned business domains.
- CQRS – I have only touched the surface of this approach to structure systems with separated command and query sides and event stores and messaging instead of relational database schemas and object/relational mappers. However, I think is an interesting approach and I definitely plan to dig in deeper.
- DCI  – Even though DDD has come to be my preferred approach I must admit the structure of the code base with behavioral code localized in every domain object has its draw backs. With requirements based on use cases or user stories, it is sometimes hard to answer the not so infrequent question “how is this UC implemented?”. DCI (Data, Context, Interaction) solves this problem through keeping the behavioral code in role objects implemented within a context representing a single UC. The code in the role objects are then injected into the data objects representing the domain. To me, it sounds like an interesting add-on to the ideas of DDD and in the future I like to continue exploring the benefits of this approach and the demands it poses on the implementation language.
- Functional programming – In order to build simpler programs that are safe to parallelize on multiple cores functional programming has gained new interest in the last years. Up until now I’ve lived only in the world of imperative programming and grasping the concepts and underlying mechanics of functional programming will pose a challenge. A challenge I’m happy to take on.
- Scala – Of all the new languages having emerged on the JVM I think Scala is the most interesting. It combines functional programming with additional powers in OO-programming without losing the benefits of static typing. All of which I think are interesting qualities. I think Scala might be the enabling programming language for many of the ideas I listed here for future exploration.

Is Domain Driven Design always applicable?

Disclamer: This blog post discusses use of DDD in information centric systems, which in my experience is most of the systems out there. It does probably not apply to technical software such as compilers and operating systems.

In my opinion the answer to the question in the title depends on how you define DDD. Do you do DDD if you do not use all technical building blocks? I think so. Is DDD more than the tachnical building blocks? I definately think so.

Personally, I would opt for using DDD in almost any situation but those that I know for sure will not evolve for more than a day or so and where I'm dead sure of the requirements. In any other situation I would start building the Ubiquitous language together with the domain experts to get a common vocabulary. This doesn't sound so much, but it is huge to avoiding expensive misunderstandings.

For the systems architecture I would always deploy the strategy to keep business logic and infrastructure code separated. It makes wonders for readability and testability. My rule of thumb is that business logic should be easily testable using unit tests (and end-to-end tests independent of the run-time environment) and infrastructure code should be so simple you generally need no other testing but a few basic integration tests.

For design my strategy is to keep the code close to the business in terminology, structure and logic. This is important to make sure that the impact of a change is about equal in size in both business and software. Here the ubiquitous language is an absolute need.

Then we come to the technical building blocks and now it is time for some choices. In general I think you could talk about two types of systems, or parts of systems; those mostly concerned with changes in object state and those mostly concerned with processing data streaming through the system.

In the first case entities, aggregates and repositories are a natural fit, in the second I think transaction scripts (in DDD context called domain services, since they do only concern domain logic, no infrastructure code, as per discussed above) are a nice fit. When the most important feature is to crunch some data, perhaps modify it and then route it further to some recepient (like another system or some persistent store) I think it is the "processing pipeline", i.e the stateless service code, that should be emphazised. So in those cases the internals of the data isn't very interesting and might be better left in some simple DTO format.

Whether going with entities or servicies I think the practice of using typed value objects for input parameters give a lot in terms of clarity and IDE-support when writing and reading the code. From a technical perspective, this is the easiest DDD building block to use and I think it has great advantages even if it is the only part of DDD that you apply to the system.

In conclusion, for the types of information systems I've been developing over the years DDD is the way to do it, however that doesn't necessarily mean everything should be entities or value objects. DDD is so much more.

Software Engineering Essential Reading 2 - Domain Driven Design

In 2004 Eric Evans published his book “Domain Driven Design – Tackling complexity at the heart of software”. It's quite a heavy bit of literature, but every page is filled with insightful discussions of the benefits and practices on designing object oriented software with a model true to the business domain, and thereby worth every minute spent reading and reflecting. Eric has solid experience from developing software in support of complex business domains and he openly shares and discusses both success and failures.
The book discusses domain driven design (DDD) on several different levels:

• The importance to use the same domain model in all aspects of a project – in writing, in the spoken language and in the source code.
• The stereotyped building blocks used in modeling, design and implementation of domain centric software systems. Entity, Value Object, Repository, Service, Aggregate and a few more.
• How to use Strategic design and concepts such as Bounded Context to define current and future logical systems or enterprise wide architectures without having to unify the whole enterprise in a single domain model.

The book is the definitive starting point in learning and applying DDD, one of the really important design paradigms now and in the future. This book, together with the active community, both locally and on the Internet, has helped me in successfully designing and deliver complex software under complex technological and organizational circumstances. This is definitely one of the books I'm happy to come back to now and then and I think every professional software designer should have a copy. Before I first read it I was wondering why no one out in the “real” world used OO-design even remotely similar to what I learned and practiced in university. In his book Eric put this straight and he made me instantly feel “at home” with a true OO-design, carefully tuned to model the businesses, as the way to go about software design. Thanks Eric for sharing!

Define and apply “The boy scout rule”

“The boy scout rule”, as defined by Robert C. Martin, tells us to make small improvements to existing code as we pass by implementing new features. It doesn't mean we need to make the whole class/file perfect, it just means we should make some small improvement in addition to the new feature, which of course should have a clean implementation. This is a good principle to include in the culture of any project. But how to do make it happen in a team with many developers, each bringing their own experiences and standards? In my project we are currently making a serious effort. Here are the actions we are taking. Please leave a comment with your own experiences/ideas.

First, talk about it. Introduce the name “The boy scout rule” - it is a quite funny name, and easy to remember. Load it further with the reference to real world scouts not caring who left the garbage found at the camping site, but always making sure the site is cleaned and left a bit better than found. Transfer the values into software development as a way to continuously refactor an existing code base while at the same time implementing new features.

Second, define, in concrete terms, what “The boy scout rule” means in the project. In our project we have made the following definition, with reference to our design principles:

• Increase readability
  • Choose intention revealing names from the business domain for classes/methods/attributes/parameters/variables.
  • Extract methods to enforce the principle of Command Query Separation (CQS) and intention revealing method names.
• Enhance the structure
  • Move methods to the“right” location to enforce the Single Responsibility Principle (SRP) for classes.
• Make it easier to do changes without introducing bugs:
  • Increase readability (as above).
  • Enhance the structure (as above).
  • Use domain objects for method parameters (e.g. use a ZipCode object instead of a simple String).

Third, lead by example. Start with the the informal leaders (often the most experienced developers) and have them spread the practice through interaction and pairing in daily work. Personally, when pairing with colleagues I try to explicitly point out what part of the work we are doing is implementing the new feature and what part is applying “The boy scout rule”. By doing so I hope to spread both the knowledge and practice to apply the rule in daily work.

Proposed design principles for internal design

In my current project we are to implement additional functionality in an existing code base. The system is SOA-based, composed of many sub systems, each implementing a set of services, as detailed by the architectural guidelines given to the project. In practice this means that each sub system hosts a set of domain logic behind a façade of Java EE Stateless Session Beans or Message Driven Beans. As the service façade approach is described in detail in the guidelines, any guidelines of the internal design is equally absent. Therefore I’ve, with great inspiration from the DDD community, put together this list of design principles for proposed use within the project, but I think they could be equally applied to internal design of most administrative IT-systems. What do you think? Should something be added or subtracted?

General design principles

• Adhere to the Single Responsibility Principle (SRP), i.e. every module (class/method) has one responsibility, not several.
• Choose names from the business domain for classes, methods, attributes, parameters and variables.
• Use domain objects as parameters instead of primitives like string or int.
• Design the system such that domain classes can be unit tested individually.
• Each (part of a) sub system should use one design metaphor, either 
• domain centered - business logic placed in domain objects, preferable when changes to state is most important -  or
• service centered – business logic placed in stateless services that operate on data in DTO:s, preferable when the data flow is most important. 

From this follows

• Keep business logic and technical implementation details separate, i.e. do not combine these two types of complexity.
• Adhere to Command Query Separation (CQS), i.e. each method should be either a command (having a side-effect and possibly returning a result) or a query (returning a result without any side-effect).

On a more detailed level

• As a tool for communication use the following stereotypes for objects implementing the business domain logic:
• Value Objects – immutable, created when needed, not persisted.
• Entities – mutable (as per allowed by business rules), loaded/persisted through Repositories.
• Repository – interface describing load/persist of Entities.
• Factory – interface describing creation of Entities and Value Objects when external dependencies (to Repositories, Factories or Services) must be fulfilled.
• Service – stateless class housing business logic that operate on data in DTO:s
• Dependencies are preferably handled through dependency injection to facilitate loose coupling between objects.

Application of principles
These principles define a solid approach to design of new (sub) systems. When used in work with an existing code base it can be used for continuously applying small improvements in accordance with “the boy scout rule” and as vision to compare with current state in order to form a road map for larger refactoring efforts.