A reference architecture (part 2)

…continued from part 1

Separating the commands from the queries

Another key principle I really like is CQS (Command Query Separation). It took me a while to see its power, but since that moment I have found that it works quite well and actually simplifies a lot, although it requires a mind shift and needs you to think a bit different. You also have to take it into account up front the design process. Basically it means that you separate the commands from the queries, as they are completely different in nature. Executing commands involves domain logic and requires the fact that you should express the business intend of your system (which goes hand in hand with a task oriented user interface); while the query side just needs to retrieve data in a specific format.

Let’s take a simple example to describe the difference.

Imagine a web application that shows personal information about a user and offers the possibility to change it. Displaying personal information of a user is straightforward: you want to retrieve the data that is shown on the page, nothing more, and nothing less; and there’s no need to execute domain logic because the query does not have any side effects: it’s just getting the data in a specific format. Changing data however is more complex – and you probably don’t want to have an operation like ModifyPersonalInformation, because this is merely a CRUD operation that doesn’t express any business intend. If we want to change an email address for example, we don’t want to pass al the personal information; but only the new email address and probably the password to be able to validate the request. So we better create a ChangeEmailAddress command, that contains only the needed information so that the command can be executed correctly. By doing this, we also very explicitly make clear that this will change email address which involves some domain validation and logic, which will cause side effects.

As a result, the query side can be kept simple, as its only purpose is to somehow get the data from one or more data sources, and maybe transform and aggregate it into a format usable by the consumer. The command side however can have a fully-fledged domain model with complex logic, validation, transaction management, workflow and everything needed to process commands.

Both query and command side can operate on the same data source, or they could each have their own data source: in this case the query side would work with a data source optimized for reading data (which can be a relational database with dedicated views or maybe a NOSQL system), while the command side would work with a data source optimized for writing data (like a relational database). Note that if you use separate data sources, you also introduce the additional need of keeping the read data source side in sync with the write data source; so it’s an important design decision that causes additional overhead.

Not so strict CQS

As discussed above, the command side is responsible for processing commands requested by a consumer. A command is purely a data container that holds enough information to be able to execute the command. The actual execution is handled by its designated command hander, which contains all the logic needed for the actual execution of the command.

If you look at CQS in a strict way, a command should have only the responsibility of executing the command, and not return data. Following this pattern has fundamental consequences, and because I’m not a purist, for now I’m going to allow our commands to return simple data (like an id or a status), and they can also throw faults in case something goes wrong. Sure, this violates the CQS principle, but the main goal now is to have the commands use the underlying domain model while the queries use the underlying database directly.

Next time I will start implementing a concrete example that demonstrates the architecture and principles.

A reference architecture (part 1)

Introduction

Software architecture is part of my job. And in my experience I can honestly say that if you would ask several people to design a system, they would all come up with something different. As such, it’s often very subjective. Of course there are objective arguments for choosing this or that, but quite often they come with a flavor of personal taste.

Anyway, my point is that in the course of previous years the ideas I had about architecting systems have evolved. As a matter a fact, if I look back at some of the designs I did, I would probably do it completely different nowadays. Because with experience comes maturity, and of course, also technology evolves. I’m confident that the ideas I have now will change again in the future, and in fact this is exactly what makes my job fun to do. Just always remember that there is no such thing as a perfect architectural design that is a solution for every problem.

The goal of this series is to show you an example how you could design a system. It’s kind of a reference architecture that I like to use (I have used it – a number of times in middle-sized projects, and I’m still quite happy about it), but it’s up to you to decide if you find some ideas to be usable in your specific environment. In no way I claim that these ideas are the one and only truth, instead, I would like to take this opportunity to learn from your comments and maybe change ideas based on feedback and new insights. So, this will always be a work in progress.

In this series…

Currently I posted 12 parts in this series:

Part 1: introduction, layers, onion architecture
Part 2: separating the commands from the queries, not so strict CQS
Part 3: a simple scenario, visual studio solution and projects, the domain layer
Part 4: the application layer
Part 5: the infrastructure layer (data access)
Part 6: the service layer
Part 7: the infrastructure layer (tests)
Part 8: the query side
Part 9: throwing business faults
Part 10: validation
Part 11: logging
Part 12: conclusion, where is the source code?

Layers

According to me, the design of a system should be as simple as possible (but not too simple), yet adaptable and extendable when needed. It means: don’t create anything that you might need later, only what you really need now, but be sure everything is in place so that you can relatively easy create it later if you need it (scale out). In other words, don’t make everything hopelessly complex just because you can and it looks nice, but choose wisely because there’s a valid reason for it.

In order to build a scalable and flexible system, it’s a good idea to group components into logical layers, define the relation between them as well as their dependencies. In general, we can define five layers.

Presentation layer

The presentation layer consists of everything that is visible to the consumer of the system, in other words, everything that has a UI. This is a thin layer, so it delegates all work to the service layer, and is only dependent on that layer. Typically it uses service operations that have been designed in the most optimal way to be used by the UI.

Service layer

The service layer holds everything that is needed to expose functionality to other systems. Typically exists of service contracts that define the interface of the exposed service operations, data transfer objects that define the requests and responses for those operations and service adapters that glue the layer to the application layer. It takes care of serialization of data, defining communication protocols and other configuration needed for interaction with other systems. This layer has dependencies on the application and domain layers.

Application layer

The application layer is a thin layer responsible for application workflow. It orchestrates the work by using the domain layer and calls to external components, and defines transaction scopes in which they should be executed. This is also a good place to implement security and high-level logging. The logic in this layer can be exposed by the service layer in an asynchronous or synchronous way, or can be logic that is triggered by events. It has only a dependency on the domain layer.

Domain layer

This is the business heart that models the system and where the domain logic is implemented. Typically it contains domain entities, domain services and repository interfaces, all used by the application layer to execute domain logic. It has no dependencies to other layers.

Infrastructure layer

This layer contains everything that is related to infrastructure, so this is the layer that is the most coupled to technological decisions: data access logic needed to talk to databases (concrete implementations of the repository interfaces defined in the domain layer), external libraries and providers, logging frameworks, file system logic and so on. This is the outer layer which is dependent on all the other layers.

Onion architecture

To define relations between the layers, you probably know the traditional layered architecture that exists of UI on top, then business and then infrastructure layer, and where each layer talks to the layer directly under it. However, the problem is that in this representation the business layer is coupled to infrastructural concerns.

So instead of the traditional layered approach, I’m a big fan of the onion architecture to define dependencies between the layers of the system and to determine the coupling between them. Representing layers as an onion means that every layer can only depend on layers more central: coupling is done towards the center:

This architecture is an exact match to visualize the dependencies of the various layers that we defined; and goes hand in hand with inversion of control principle: implementation details are injected by the outer layers at runtime; the inner layers just define and use interfaces. As an example, the domain layer defines the repository interfaces it needs in order to be able to perform the work; but the concrete implementation is coded and injected by the infrastructure layer.

Next time I will continue on this, and talk a little about one of the key principles: separating queries and commands.

Avoiding the use of static variables when using log4net

The typical usage for log4net is to define static variable as follows:

public class FileRegistratorJob : IJob
{
    private static readonly ILog _log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);    

    public FileRegistratorJob()
    {
    }

    public void Execute(IJobExecutionContext context)
    {
        try
        {
            // do something here
            _logger.Debug("Job completed!");

        }
        catch (Exception ex)
        {
            _logger.Error("An error occured", ex);
        }
    }
}

The use of this static variable _log bothers me, as I tend to avoid using static variables whenever I can. I want something that works nicely with my inversion of control container (StructureMap), and so I want my logging mechanism to be injected instead of hardcoded like this.

The solution is to first create a logging interface that contains all the logging operations:

public interface ILogger

   void Debug(string message);
   void Debug(string message, Exception ex);        
   void Info(string message);
   void Info(string message, Exception ex);
   void Warn(string message);
   void Warn(string message, Exception ex);
   void Error(string message);
   void Error(string message, Exception ex);
   void Fatal(string message);
   void Fatal(string message, Exception ex);
}

 

Then we create the actual implementation for it that uses log4net to do the logging:

public class Log4NetLogger : ILogger
{
    private readonly ILog _log;

    /// <summary>
    /// Constructor.
    /// </summary>
    /// <param name="type"></param>
    public Log4NetLogger(Type type)
    {
        _log = LogManager.GetLogger(type);
    }

    public void Debug(string message)
    {
        _log.Debug(message);
    }

    public void Debug(string message, Exception ex)
    {
        _log.Debug(message, ex);
    }

    public void Info(string message)
    {
        _log.Info(message);
    }

    public void Info(string message, Exception ex)
    {
        _log.Info(message, ex);
    }

    public void Warn(string message)
    {
        _log.Warn(message);
    }

    public void Warn(string message, Exception ex)
    {
        _log.Warn(message, ex);
    }

    public void Error(string message)
    {
        _log.Error(message);
    }

    public void Error(string message, Exception ex)
    {
        _log.Error(message, ex);
    }

    public void Fatal(string message)
    {
        _log.Fatal(message);
    }

    public void Fatal(string message, Exception ex)
    {
        _log.Fatal(message, ex);
    }
}

All it takes to actually use this is to set it up in a StructureMap registry:

public class StructureMapRegistry : Registry
{
    public StructureMapRegistry()
    {
        // Logging
        For<ILogger>().AlwaysUnique().Use(s => s.ParentType == null ? new Log4NetLogger(s.BuildStack.Current.ConcreteType) : new Log4NetLogger(s.ParentType));
        var applicationPath = Path.GetDirectoryName(Assembly.GetAssembly(GetType()).Location);
        var configFile = new FileInfo(Path.Combine(applicationPath, "app.config"));
        XmlConfigurator.ConfigureAndWatch(configFile);
        
        // Other registrations here        
    }
}

And then you could constructor inject it in your classes as follows:

public class FileRegistratorJob : IJob
{
    private ILogger _logger;

    public FileRegistratorJob(ILogger logger)
    {
        _logger = logger;
    }

    public void Execute(IJobExecutionContext context)
    {
        try
        {
            // 
            _logger.Debug("Job completed!");

        }
        catch (Exception ex)
        {
            _logger.Error("An error occured", ex);
        }
    }
}

 

I much cleaner solution because we removed the coupling to log4net.

Parsing large XML files

Today I had to parse large xml files and in various ways: getting the value of an attribute, counting elements, and getting all elements and their contents.

There are various ways of parsing xml files: using DataSet, XmlSerializer, XPathDocument, XmlDocument, XDocument and XmlTextReader. Now because my xml files could be large, it was obvious that I did not want to load them completely in memory before parsing them.

Obviously the choice of method would then be XmlTextReader, because this allows you to advance to the next nodes and attributes without having to load the file completely.

For simplicity, assume that my xml file looks like this:

<?xml version="1.0" encoding="utf-8" ?> 
<personImport batchType="PmH">
  <persons>
    <person>
      <senderData>
        <name>Lincoln</name>
    <fileCreationDate>25/09/2012</fileCreationDate>
      </senderData>
      <personData>
        <lastName>Steward</lastName>
        <firstName>Michael</firstName>
      </personData>
    </person>
    <person>
      <senderData>
        <name>Mercator</name>
    <fileCreationDate>25/09/2012</fileCreationDate>
      </senderData>
      <personData>
        <lastName>Miles</lastName>
    <firstName>David</firstName>
      </personData>
    </person>
  </persons>
</personImport>

The first thing I needed to do was to retrieve the batchType value from the root element. So I created an ImportFileTypeResolver to do the work:

public class ImportFileTypeResolver : IImportFileTypeResolver
{
    public ImportFileType Resolve(string filePath)
    {
        using (var reader = new XmlTextReader(filePath))
        {
            reader.ReadToFollowing("personImport");                
            switch (reader.GetAttribute("batchType"))
            {
                case "PmH": return ImportFileType.Person;
                default: return ImportFileType.Unknown;
            }
        }
    }        
} 

 

As you see there are a number of convenience methods that make it very easy to advance to the next element/attribute, using ReadToFollowing and MoveToAttribute.

 

The next thing was to return the number of person elements, so I created a PersonImportFileJobCountResolver:

 

public class PersonImportFileJobCountResolver : IImportFileJobCountResolver
{
    public int GetNumberOfJobs(string filePath)
    {
        using (var reader = new XmlTextReader(filePath))
        {
            var nodeCount = 0;
            while (reader.ReadToFollowing("person")) nodeCount++;                
            return nodeCount;
        }            
    }        
}

 

Again, all I needed to do was use the ReadToFollowing operation until the file has been read completely and for each iteration increase the counter.

 

The third thing I wanted to do was to retrieve all person elements as Person objects. To do that I created a PersonImportBatchFileExtractor as follows:

 

public interface IImportBatchFileExtractor<TEntity> : IDisposable
{
    TEntity ExtractNext();
}

 

public class PersonImportBatchFileExtractor : IImportBatchFileExtractor<PersonImportBatchFileExtractor.Person>
{
    private XmlReader _xmlReader;

    public Person ExtractFirst(string personImportBatchFilePath)
    {
        _xmlReader = new XmlTextReader(personImportBatchFilePath);
        return ExtractNext();
    }

    public Person ExtractFirst(XDocument data)
    {
        var stream = new MemoryStream();
        data.Save(stream);
        stream.Position = 0;
        _xmlReader = XmlReader.Create(stream);
        return ExtractNext();
    }

    public Person ExtractNext()
    {
        if (_xmlReader == null) throw new ApplicationException("Call ExtractFirst before calling ExtractNext");

        _xmlReader.ReadToFollowing("person");
        if (_xmlReader.NodeType == XmlNodeType.None) return null;

        var xDocument = XDocument.Parse(_xmlReader.ReadOuterXml());
        if (xDocument.Root == null) throw new ApplicationException("Something went wrong during parsing a person");

        var name = xDocument.Root.XPathSelectElement("senderData/name").Value;
        var fileCreationDate = xDocument.Root.XPathSelectElement("senderData/fileCreationDate").Value;
        var lastName = xDocument.Root.XPathSelectElement("personData/lastName").Value;
        var firstName = xDocument.Root.XPathSelectElement("personData/firstName").Value;

        return new Person(xDocument, new SenderData(name, DateTime.Parse(fileCreationDate)), new PersonData(lastName, firstName));
    }

    public void Dispose()
    {
        if (_xmlReader!=null) _xmlReader.Close();
    }

    public class Person
    {
        public Person(XDocument xmlRepresentation, SenderData senderData, PersonData personData)
        {
            XmlRepresentation = xmlRepresentation;
            SenderData = senderData;
            PersonData = personData;
        }
        public XDocument XmlRepresentation { get; set; }
        public SenderData SenderData { get; set; }
        public PersonData PersonData { get; set; }
    }

    public class SenderData
    {
        public SenderData(string name, DateTime fileCreationDate)
        {
            Name = name;
            FileCreationDate = fileCreationDate;
        }
        public string Name { get; set; }
        public DateTime FileCreationDate { get; set; }
    }

    public class PersonData
    {
        public PersonData(string firstName, string lastName)
        {
            FirstName = firstName;
            LastName = lastName;
        }
        public string FirstName { get; set; }
        public string LastName { get; set; }
    }
}

 

This simply looks for the next person element, creates an XDocument based on it and then uses xpath to find the elements within that person to build a Person object and return it.

 

Usage is like:

 

using (var personImportBatchFileExtractor 
        = new PersonImportBatchFileExtractor())
{
    var person = personImportBatchFileExtractor.ExtractFirst
                 (@"C:\temp\batchimport\pmh.xml");                
    while (person != null)
    {
        person = personImportBatchFileExtractor.ExtractNext();
    }
};

 

Communicating through sockets

A Socket is an end-point of a bidirectional communication link between two applications running on the same network. So basically it means you can use it to allow applications to communicate with each other. You have a server, that listens for incoming connections, and clients that make a connection to the server. Then, data can be transferred between the clients and the server.

First create a console application and call it SocketServer. The main looks like:

class Program
{
    private static System.Net.Sockets.Socket _socket;
    private static AsyncCallback _dataTransferCallBack;
    private const int MaxLengthOfPendingConnectionsQueue = 10;        
    private const short SocketPort = 7761;


    static void Main(string[] args)
    {            
        _socket = new System.Net.Sockets.Socket(AddressFamily.InterNetwork, 
                      SocketType.Stream, ProtocolType.Tcp);
        var endPoint = new IPEndPoint(IPAddress.Any, SocketPort);
        _socket.Bind(endPoint);
        _socket.Listen(MaxLengthOfPendingConnectionsQueue);
        // Begins an asynchronous operation to accept an incoming 
        // connection attempt.
        _socket.BeginAccept(OnConnectRequest, null);
        Console.WriteLine("Server started, listening...");
        Console.ReadLine();
        // Clean up socket.
        _socket.Close();
        GC.Collect();
        GC.WaitForPendingFinalizers();    
    }
}

The socket is created, and bound to an endpoint at a specific port number. Then the BeginAccept operation makes the application wait for an incoming connection request (asynchronously). If a connection request comes in, then the OnConnectRequest operation is executed:

 

public static void OnConnectRequest(IAsyncResult asyn)
{
    try
    {
        // EndAccept completes a call to BeginAccept. 
        // It returns a new Socket that can be used to send data 
        // to and receive data from the remote host.                
        var dataTransferSocket = _socket.EndAccept(asyn);
        WaitForData(dataTransferSocket);
        _socket.BeginAccept(OnConnectRequest, null);
    }
    catch (ObjectDisposedException)
    {
        Console.WriteLine("OnConnectRequest: Socket has been closed.");
    }
    catch (SocketException ex)
    {
        Console.WriteLine(
           string.Format("Something fishy happened: {0}", ex.Message));
        throw;
    }
}

In this operation we first call the WaitForData operation, and then call the BeginAccept operation again to continue listening to incoming connection requests of other clients. The WaitForData starts the actual receive of data: if the connected client sends data, the OnDataReceived callback operation is executed:

public static void WaitForData(System.Net.Sockets.Socket dataTransferSocket)
{
    try
    {
        if (_dataTransferCallBack == null) 
               { _dataTransferCallBack = OnDataReceived; }
        var socketPacket = new SocketPacket(dataTransferSocket);
        // Now start listening for data.
        dataTransferSocket.BeginReceive(socketPacket.DataBuffer, 0, 
                      socketPacket.DataBuffer.Length, SocketFlags.None, 
                     _dataTransferCallBack, socketPacket);
    }
    catch (SocketException ex)
    {
        Console.WriteLine(
             string.Format("Something fishy happened: {0}", ex.Message));
        throw;
    }
}

 

Where SocketPacket is like:

 

public class SocketPacket
{
    public System.Net.Sockets.Socket Socket { get; set; }
    public byte[] DataBuffer { get; set; }

    public SocketPacket(System.Net.Sockets.Socket socket)
    {
        Socket = socket;
        DataBuffer = new byte[1024];
    }                                   
}

In the OnDataReceived operation we check if there is data, if there is, we keep calling the WaitForData again to get the next data. If there is no data anymore, we stop the transfer and close the socket:

 

public static void OnDataReceived(IAsyncResult asyn)
{
    try
    {
        var socketPacket = (SocketPacket)asyn.AsyncState;
        int numberOfBytesReceived = socketPacket.Socket.EndReceive(asyn);

        if (numberOfBytesReceived <= 0)
        {
            Console.WriteLine("Client {0}, disconnected", 
                         socketPacket.Socket.RemoteEndPoint);
            socketPacket.Socket.Close();
            return;
        }

        var receivedData = Encoding.ASCII.GetString(socketPacket.DataBuffer, 
                    0, numberOfBytesReceived);
        Console.WriteLine("{0}", receivedData);
        WaitForData(socketPacket.Socket);
    }
    catch (ObjectDisposedException)
    {
        Console.WriteLine("OnConnectRequest: Socket has been closed.");
        throw;
    }
    catch (SocketException ex)
    {
        Console.WriteLine(
                 string.Format("Something fishy happened: {0}", ex.Message));
        throw;
    }
}

Now what’s left is to write the client side code to connect to the server and send messages. Create a new console application and call it SocketClient. For the client, I’m going to write a SocketPublisher that contains all the logic to connect and send the messages, which is used by the client. This is the code:

 

public interface ISocketPublisher
{
    void SendMessage(string message);
}

public class SocketPublisher : ISocketPublisher, IDisposable
{
    private System.Net.Sockets.Socket _socket;
    private readonly string _remoteAddress;
    private readonly short _remotePortNumber;
   
    public SocketPublisher(string remoteAddress, 
                                   short remotePortNumber)
    {
        _remoteAddress = remoteAddress;
        _remotePortNumber = remotePortNumber;
        Connect();
    }

    public void SendMessage(string message)
    {
        var messageBytes = 
               System.Text.Encoding.ASCII.GetBytes(message);
        _socket.Send(messageBytes);
    }

    public void Dispose()
    {
        Disconnect();
    }

    private void Disconnect()
    {
        if (_socket != null)
        {
            _socket.Shutdown(SocketShutdown.Both);
            _socket.Close();
        }
    }

    private void Connect()
    {
        if ((_socket == null) || (!_socket.Connected))
        {
            var remoteIpAddress = 
                     System.Net.IPAddress.Parse(_remoteAddress);
            var remoteEndPoint = new System.Net.IPEndPoint(
                                 remoteIpAddress, _remotePortNumber);
            _socket = new System.Net.Sockets.Socket(
                        AddressFamily.InterNetwork, 
                        SocketType.Stream, ProtocolType.Tcp);
            _socket.Connect(remoteEndPoint);
        }
    }
}

And in the Main you can then send a message as follows:

 

class Program
{
    private const string SocketAddress = "127.0.0.1";
    private const short SocketPort = 7761;

    static void Main(string[] args)
    {
        using (var socketPublisher = 
                 new SocketPublisher(SocketAddress, SocketPort))
        {
            socketPublisher.SendMessage("Hello world!");
        }
    }
}

Now start the server, then start the client, and you should see the message coming through at the server:

The server can also send messages back to the clients; if you want to do that you need to keep track of connected clients sockets and then you can send messages back.