A reference architecture (Part 2)

…continued from part 1.

In the previous part of this series I explained the general architectural guidelines. Of  course it is very important to have a consistent organization of all the solutions and projects in an organization, so that’s what I will discuss in this post.

Solution structure

The general pre-defined structure of solution and projects looks like this:

Solution structure

As you can see this structure reflects the architectural layers as described in previous part of this series:

• The Manager solution folder contains all managers
• The Agent solution folder contains all agents
• The Utility solution folder contains all utilities
• The BusinessEntity solution folder contains all business entities
• The Client solution folder contains all client applications

Manager solution folder in detail

Each manager has the same structure. As an example, this is how the structure of the Stock manager looks like:

Manager structure

Again, this reflects the architectural design as described earlier: the Stock manager is located in the solution folder Manager\Stock. This folder has the following projects:

• Schepers.Manager.Stock.ServiceInterface: service contracts, message contracts and data contracts of the Stock manager
• Schepers.Manager.Stock.ServiceAdapter: service adapter of the Stock manager
• Schepers.Manager.Stock.Business: business logic of the Stock manager

Agent and utility solution folder in detail

The agent and utility structure is identical to the manager structure, except that we also have ResourceAccess projects. As an example, have a look at the structure of the Email utility:

Agent/Utility structure

the Email utility is located in the solution folder Utility\Email, and has the following projects:

• Schepers.Utility.Email.ServiceInterface: contains service contracts, message contracts and data contracts of the Email utility
• Schepers.Utility.Email.ServiceAdapter: service adapter of the Email utility
• Schepers.Utility.Email.Business: business logic of the Email utility
• Schepers.Utility.Email.ResourceAccess.Repository: repository of the Email utility
• Schepers.Utility.Email.ResourceManager.NetMail: resource manager of the Email utility

BusinessEntity folder in detail

The project containing all business entities looks like:

Business entities

In more complicated environments you can use solution folders to divide the various business entities in categories.

Client folder in detail

The client folder contains all client projects:

Client projects

Other folders

If needed, you can add other solution folders, for example a Test folder that contains all unit tests for your projects.

Solution separation

Solutions are all put in a separate folder, and we define two types of Visual Studio solutions: entity and master solutions.

Entity solutions

The idea of an entity solution is that it should contain only those projects relevant for the solution, and relevant to one deployable entity (that may consist of a number of assemblies). So entity solutions never mix separate deployable projects, for example:

Solutions
\Schepers.Utility.Email.ResourceAccess-VS2008.sln (contains only resource access projects)
\Schepers.Manager.Stock.Business-VS-2008.sln (contains only business project)

Obviously, this approach has a number of advantages:

• You see only what is relevant for the project
• You only build the smallest relevant deployable entity
• Various team can work on separate solutions without having to worry about conflicts

Master solutions

A master solution contains all projects that are involved in a business application, so it moves beyond the boundaries of the entity solution because it contains parts that are going to be deployed separately. A master solution is typically used for testing purposes, that is to say: to test the entire business solution. Example:

Solutions
\Schepers.Master-VS2008.sln (contains all projects for this customer)
\Schepers.Manager.Stock-VS-2008.sln (contains all projects related to the Stock manager)

Guidelines

Folder and file names

Inside visual studio, folder names are short: Utility, Agent, Client,… but project names contain the full namespace: Schepers.Manager.Stock.ServiceInterface, Schepers.Utility.Email.ResourceAccess.ResourceManager.netMail,…

Naming conventions

This one is easy: we use the naming conventions as defined at http://msdn.microsoft.com/en-us/library/xzf533w0(VS.71).aspx.

Contract namespaces

It’s important to have naming conventions for the contract namespaces and that we apply them in a consistent way.

Starting points

These are some starting points concerning naming guidelines:

• a contract namespace should make clear to what domain data and message contracts belong to. It does not care about the solution structure (the .NET namespace) that is being used.
• a contract namespace includes a version indication
• a contract namespace is in lower case
• a contract namespace does not care about it’s an enumeration or other specific type
• a contract namespace does not contain the name of the property itself

Conventions

Convention for data contracts: http://schemas.schepers.com/servicetype.servicename/subspecification/yyyy/mm/dd

Convention for message contracts: http://schemas.schepers.com/servicetype.servicename/request-response-name/yyyy/mm/dd

where

• servicetype is the type of service (agent, manager, utility, cross,…) without ‘service’ (mandatory)
• servicename is the name of the service (storage, user,…) (mandatory)
• subspecification is a sub-category when needed (render, view, abstract,…) (optional)
• yyyy is the year in four digits (2009, 2010,…) (mandatory)
• mm is the month in two digits (04, 11,…) (mandatory)
• dd is the day in two digits (06, 24,…) (mandatory)
• request-response-name is the name of the request or response it belongs too (addbookmarkrequest, addbookmarkresponse,…) (mandatory)

For versioning we choose a date indication yyyy/mm/dd instead of version number, because it’s a W3C standard and has more meaning.

Example

The contract namespaces for the EmailService are named as follows:

• data contracts
  • EmailAddress: http://schemas.schepers.com/utility.email/2009/07/02
  • EmailAttachment: http://schemas.schepers.com/utility.email/2009/07/02
  • EmailPriority: http://schemas.schepers.com/utility.email/2009/07/02
• message contracts
  • SendEmailRequest: http://schemas.schepers.com/utility.email/sendemailrequest/2009/07/02

Advantages and disadvantages

Although the amount of layers may seem overwhelming at first, evolving to a service oriented design has undoubtedly advantages for IT as well as business.

Advantages for IT

Loose coupling

Services are loosely coupled, meaning that they work entirely independent from each other; so changing one service does not affect another service in anyway. Because of the layered approach, this is also true for each other component in the architecture. Examples:

• If a resource manager changes or is replaced, it has no impact on the existing assets and consumers won’t even notice.
• A real implementation can easily be replaced by a mock implementation

Single-Responsibility

The Single-Responsibility principle means that a class should have only one responsibility, and has therefore only one reason to change. Examples:

• The resource manager’s task is to implement resource access to a specific resource and using a specific technology (for example, using LINQ). It’s about data logic, no business or other logic.
• A repository is an abstraction of the data functionality needed for a service. It contains only the abstraction for that particular service, and only uses the resource manager(s) needed for its scope.
• An agent is fine-grained; it offers functionality within its limited scope only.

Interface Segregation Principle

According to the Interface Segregation Principle, many client specific interfaces are better than one general purpose interface, so we design our services in such a way that they expose tailored business functionality to clients and only what they really need. Because of this, services are more robust and better maintainable, due to fewer dependencies. Of course, a service should describe its capabilities and characteristics in a clear and stateless (no shared context) way without hidden assumptions. If this is done well, redundancy is low and by combining and recombining services project delivery will speed up – with lowered costs as a logical consequence. It promotes reuse on business and technological level: use what’s already there and extend what you already have. It also results in a very scalable solution: it is easy to scale out your services by deploying them into a load-balanced server farm, using software load balancers as well as hardware solutions.

Same mindset

If everyone has the same mindset, it’s easier to work together. Same approach, same methodology, same vision makes it possible to better understand what everyone is doing and how things work. There is a one-time investment of learning this – but because the entire team uses it, the inter-group collaboration becomes a lot better because we are already speaking the same architectural language.

Advantages for business

Think business, not technology

The design forces you to think in terms of business processes and service interactions, instead of monolithic applications. Therefore business has a clear view on what capabilities are available without having to worry about implementation details or technology choices. Example: business should not care how data is stored! Business changes can be implemented faster, because business handling is done in 1 place only by the development team in collaboration with business team. Moreover, business and IT speak the same business language – not technology!

Eliminate redundancy

Services can be leveraged cross multiple business projects – reducing redundancy. Reuse and (re)combining services allows faster project delivery and lower costs. Use what’s already there and extend what you already have (reuse on business level).

Disadvantages

The downside is that this flexibility comes at a price: the initial investment of setting up the design and basic services is higher, and there’s a lot of boilerplate code needed to achieve the layered structure (which can be auto generated for the most part). You also need a clear view on what you are trying to achieve; and last but not least: real sponsorship from the business to be able to succeed.

To be continued…

A reference architecture (Part 1)

Introduction

IT and business needs have changed over the years, demanding system design to follow. In today’s tough economical conditions, there is more than ever the need to control costs and as such it is indispensable that an enterprise has to be able to respond very quickly to change with minimal effort. Traditionally projects start in the IT department, mostly without taking into account business requirements, but this just doesn’t work anymore. Instead, projects should be driven by business needs, and evidently the underlying IT solution has to support this.

I have built a systems architectural design template, adhering to proven SOA guidelines, to tackle this challenge.  Of course, every environment is different – and so is every design – but the idea is to have a template that uses best practises and that is extensible to your own needs. The result is a multi-layered architecture based on resources like Juval Lowy’s master class, the Web Service Software Factory guidelines, feedback from other architects and experiences with developers using this architecture.

Architectural guidelines

To begin, the template below visually represents the multi-layered systems architecture I have developed:

Description of the layers

The client layer represents all components that consume the underlying managers. It’s more than just a presentation layer, because other than graphical user interface clients this layer also contains non-visible clients that can trigger managers (for example, an email or an incoming file on an FTP server). These clients can perform non-business related validation on user input, but they don’t contain other business or data access logic.  A client can never talk directly with agents or utilities, it only talks to managers.

The manager layer contains the top-level coarse-grained systems, and uses the agents/utilities to delegate work:

• Processes:
  • Expose business processes
  • Typically long running (workflow) and invoked using a one-way or notification (reservation) pattern
  • Encapsulates rules particular to the business process
  • Includes transaction control using compensation
  • One process may act as a controller for another and invoke it
  • Can be stateful
  • Invoke underlying agents/utilities
• Activities:
  • Expose business activities that encapsulate calls to multiple agent and/or utilities to perform a single activity
  • Each operation scoped as a logical unit of work
  • Typically immediate request-response pattern
  • Includes rules that are common to the organization

The agent and utility layer contains the low-level granular and fine-grained components. Agents provide business-centric functional context, while utilities are more related to infrastructure functionality like logging and emailing.

An agent:

• Exposes business entity data
• Typically immediate request-response pattern
• Encapsulates data consistency rules and CRUD that are common to the entire organization
• May invoke utilities
• Each operation scoped as a logical unit of work

There are different types of agents, for example:

• Proxy agent: referred to as wrapper, broker, or gateway, they act as a bridge between other members of the service-oriented system and incompatible subsystems
• Device agent: a specialized type of proxy that stands in front of a hardware device.
• Entity agent: exposes a single uncluttered concept that represents a business entity. They are independent from the business processes.

An utility :

• Exposes shared functionality that is not specific to a business process or entity (not domain specific)
• Typically immediate request-response
• Does not invoke other agents

The business entities layer models the real-world, global domain-specific objects, like Customers and Orders.

The resource layer contains resources, which can be a database, external service or any other kind of resource.

Important to know is that managers are service-enabled, and that agents and utilities can be service enabled but don’t have to.

Communication flow

Communication between clients, managers, agents, resource access layer and resource layer is only allowed from top to bottom. These are the rules:

• A client can consume the services from manager layer, but it cannot access the agent or resource access layer directly.
• A component from the manager layer can consume the agent and utility layer, but it cannot access the resource access layer directly.
• The enterprise business entities are used by manager, agent, utility and resource access layer, and so they are never exposed to the outside world.

The layers in detail

The manager layer has the same structure of the agent/utility layer, except that it does not have a resource access layer. So, the manager structure is designed according to the following structure:

Manager layer

And the agent and utility layer is designed according to the following structure:

Agent/Utility layer

Note that for the manager the service interface layer is mandatory: a manager is always exposed as a service to the clients. But for an agent or utility, the service interface layer is optional. This means that:

• The service business layer of a manager can use the service interface layer of an agent
• The service business layer of a manager can use the service business layer of an agent directly (and later, if the agent is service enabled, talk to the service interface layer instead)

Service interface layer

The service contract contains the interface for the service. It defines a logically and semantically related set of operations grouped on an interface and is about how a service behaves.

The data contract contains all data contracts that the service uses. A data contract is a formal agreement between a service and a client that abstractly describes the data to be exchanged. It precisely defines, for each parameter or return type, what data is serialized (turned into XML) to be exchanged. An important characteristic is that a data contract can be reused within the service. To communicate, the client and the service do not have to share the same types, only the same data contracts.

The message contract contains all message contracts needed by the service. A message contract is a wrapper around the data contracts and is mostly used when you need explicit control over the format of the SOAP (Simple Object Access Protocol) message, but in the architecture we always use them for the following reasons:

• clarity and uniformity: use the same approach in all services
• explicit implementation of the Request/Response pattern

An important characteristic is that a message contract is designed not to be reused.

The fault contract contains all fault contracts. A fault contract provides a way to describe faults in a contract and mark certain types of exceptions as appropriate for transmission to the client, shielding underlying system exceptions. As a rule we use fault contracts for errors that are not part of the normal program flow, for example:

• void SendEmail(SendEmailRequest) throws fault exception when subject or other fields are not filled in
• GetCustomersResponse GetCustomers() throws fault exception when underlying data source is unavailable

The service adapter contains the actual implementation of the service interface. It consists of the adapter, which invokes business actions after it uses entity translators to translate between message types or data types and types required by the business actions.

Service business layer

The business layer takes care of enterprise business entity validation and other custom business logic.

Resource access layer

The resource access layer contains the repositories and resource managers. Repositories are abstractions that interact with the resource managers and provide business interfaces to the domain model: so they are coarse-grained and closer to the domain. A repository can work with an underlying resource manager or it can act as a repository to mock data (a convenient way to do unit testing). Using dependency injection techniques, the correct repositories can be invoked at runtime by specifying them in a configuration file. A resource manager is an implementer towards a specific resource, so they are fine-grained and closer to the database. They typically implement the basic CRUD (create, read, update and delete) and other data operations, and use entity translators to do the mapping between data entities and enterprise business entities.

Rules:

• An agent can use its repository (only one!) from the resource access layer, and can consume utilities.
• A utility can only use its own repository (only one!) from the resource access layer.
• A repository will mostly use only one resource manager, but in some cases it can use one or more resource managers as long as they belong to the same business context and business entities (to implement master-data relationships). Example: an OrderRepository can use a resource manager that gets partial order information from one database and another resource manager that gets another part of order information from an xml file, because the same business entity is being constructed out of it. An OrderRepository cannot use a resource manager to get for example customer information; because this does not belong to the OrderRepository’s core tasks (in this case a manager service has to be created to aggregate order and customer information).

Data flow

The following diagram illustrates the request data flow:

Data flow

Continued in part 2.