Endo is the new fluent API

I tweeted this over the weekend ..

a good title for a possible blog post .. endo is the new fluent API ..

— Debasish Ghosh (@debasishg) June 1, 2013

My last two blog posts have been about endomorphisms and how it combines with the other functional structures to help you write expressive and composable code. In A DSL with an Endo - monoids for free, endos play with Writer monad and implement a DSL for a sequence of activities through monoidal composition. And in An exercise in Refactoring - Playing around with Monoids and Endomorphisms, I discuss a refactoring exercise that exploits the monoid of an endo to make composition easier. Endomorphisms help you lift your computation into a data type that gives you an instance of a monoid. And the mappend operation of the monoid is the function composition. Hence once you have the Endo for your type defined, you get a nice declarative syntax for the operations that you want to compose, resulting in a fluent API. Just a quick recap .. endomorphisms are functions that map a type on to itself and offer composition over monoids. Given an endomorphism we can define an implicit monoid instance ..

implicit def endoInstance[A]: Monoid[Endo[A]] = new Monoid[Endo[A]] {
  def append(f1: Endo[A], f2: => Endo[A]) = f1 compose f2
  def zero = Endo.idEndo
}

I am not going into the details of this, which I discussed at length in my earlier posts. In this article I will sum up with yet another use case for making fluent APIs using the monoid instance of an Endo. Consider an example from the domain of securities trading, where a security trade goes through a sequence of transformations in its lifecycle through the trading process .. Here's a typical Trade model (very very trivialified for demonstration) ..

sealed trait Instrument
case class Security(isin: String, name: String) extends Instrument

case class Trade(refNo: String, tradeDate: Date, valueDate: Option[Date] = None, 
  ins: Instrument, principal: BigDecimal, net: Option[BigDecimal] = None, 
  status: TradeStatus = CREATED)

Modeling a typical lifecycle of a trade is complex. But for illustration, let's consider these simple ones which need to executed on a trade in sequence ..

1. Validate the trade
2. Assign value date to the trade, which will ideally be the settlement date
3. Enrich the trade with tax/fees and net trade value
4. Journalize the trade in books

Each of the functions take a Trade and return a copy of the Trade with some attributes modified. A naive way of doing that will be as follows ..

def validate(t: Trade): Trade = //..

def addValueDate(t: Trade): Trade = //..

def enrich(t: Trade): Trade = //..

def journalize(t: Trade): Trade = //..

and invoke these methods in sequence while modeling the lifecycle. Instead we try to make it more composable and lift the function Trade => Trade within the Endo ..

type TradeLifecycle = Endo[Trade]

and here's the implementation ..

// validate the trade: business logic elided
def validate: TradeLifecycle = 
  ((t: Trade) => t.copy(status = VALIDATED)).endo

// add value date to the trade (for settlement)
def addValueDate: TradeLifecycle = 
  ((t: Trade) => t.copy(valueDate = Some(t.tradeDate), status = VALUE_DATE_ADDED)).endo

// enrich the trade: add taxes and compute net value: business logic elided
def enrich: TradeLifecycle = 
  ((t: Trade) => t.copy(net = Some(t.principal + 100), status = ENRICHED)).endo

// journalize the trade into book: business logic elided
def journalize: TradeLifecycle = 
  ((t: Trade) => t.copy(status = FINALIZED)).endo

Now endo has an instance of Monoid defined by scalaz and the mappend of Endo is function composition .. Hence here's our lifecycle model using the holy monoid of endo ..

def doTrade(t: Trade) =
  (journalize |+| enrich |+| addValueDate |+| validate).apply(t)

It's almost the specification that we listed above in numbered bullets. Note the inside out sequence that's required for the composition to take place in proper order.

Why not plain old composition ?

A valid question. The reason - abstraction. Abstracting the composition within types helps you compose the result with other types, as we saw in my earlier blog posts. In one of them we built larger abstractions using the Writer monad with Endo and in the other we used the mzero of the monoid as a fallback during composition thereby avoiding any special case branch statements.

One size doesn't fit all ..

The endo and its monoid compose beautifully and gives us a domain friendly syntax that expresses the business functionality ina nice succinct way. But it's not a pattern which you can apply everywhere where you need to compose a bunch of domain behaviors. Like every idiom, it has its shortcomings and you need different sets of solutions in your repertoire. For example the above solution doesn't handle any of the domain exceptions - what if the validation fails ? With the above strategy the only way you can handle this situation is to throw exceptions from validate function. But exceptions are side-effects and in functional programming there are more cleaner ways to tame the evil. And for that you need different patterns in practice. More on that in subsequent posts ..

A DSL with an Endo - monoids for free

When we design a domain model, one of the issues that we care about is abstraction of implementation from the user level API. Besides making the published contract simple, this also decouples the implementation and allows post facto optimization to be done without any impact on the user level API.

Consider a class like the following ..

// a sample task in a project
case class Task(name: String) 

// a project with a list of tasks & dependencies amongst the
// various tasks
case class Project(name: String, 
                   startDate: java.util.Date, 
                   endDate: Option[java.util.Date] = None, 
                   tasks: List[Task] = List(), 
                   deps: List[(Task, Task)] = List())

We can always use the algebraic data type definition above to add tasks and dependencies to a project. Besides being cumbersome as a user level API, it also is a way to program too close to the implementation. The user is coupled to the fact that we use a List to store tasks, making it difficult to use any alternate implementation in the future. We can offer a Builder like OO interface with fluent APIs, but that also adds to the verbosity of implementation, makes builders mutable and is generally more difficult to compose with other generic functional abstractions.

Ideally we should be having a DSL that lets users create projects and add tasks and dependencies to them.

In this post I will discuss a few functional abstractions that will stay behind from the user APIs, and yet provide the compositional power to wire up the DSL. This is a post inspired by this post which discusses a similar DSL design using Endo and Writers in Haskell.

Let's address the issues one by one. We need to accumulate tasks that belong to the project. So we need an abstraction that helps in this accumulation e.g. concatenation in a list, or in a set or in a Map .. etc. One abstraction that comes to mind is a Monoid that gives us an associative binary operation between two objects of a type that form a monoid.

trait Monoid[T] {
  def append(m1: T, m2: T): T
  def zero: T
}

A List is a monoid with concatenation as the append. But since we don't want to expose the concrete data structure to the client API, we can talk in terms of monoids.

The other data structure that we need is some form of an abstraction that will offer us the writing operation into the monoid. A Writer monad is an example of this. In fact the combination of a Writer and a Monoid is potent enough to have such a DSL in the making. Tony Morris used this combo to implement a logging functionality ..

for {
  a <- k withvaluelog ("starting with " + _)
  b <- (a + 7) withlog "adding 7"
  c <- (b * 3).nolog
  d <- c.toString.reverse.toInt withvaluelog ("switcheroo with " + _)
  e <- (d % 2 == 0) withlog "is even?"
} yield e

We could use this same technique here. But we have a problem - Project is not a monoid and we don't have a definition of zero for a Project that we can use to make it a Monoid. Is there something that would help us get a monoid from Project i.e. allow us to use Project in a monoid ?

Enter Endo .. an endomorphism which is simply a function that takes an argument of type T and returns the same type. In Scala, we can state this as ..

sealed trait Endo[A] {
  // The captured function
  def run: A => A
  //..
}

Scalaz defines Endo[A] and provides a lot of helper functions and syntactic sugars to use endomorphisms. Among its other properties, Endo[A] provides a natural monoid and allows us to use A in a Monoid. In other words, endomorphisms of A form a monoid under composition. In our case we can define an Endo[Project] as a function that takes a Project and returns a Project. We can then use it with a Writer (as above) and implement the accumulation of tasks within a Project.

Exercise: Implement Tony Morris' logger without side-effects using an Endo.

Here's how we would like to accumulate tasks in our DSL ..

for {
  _ <- task("Study Customer Requirements")
  _ <- task("Analyze Use Cases")
  a <- task("Develop code")
} yield a

Let's define a function that adds a Task to a Project ..

// add task to a project
val withTask = (t: Task, p: Project) => p.copy(tasks = t :: p.tasks)

and use this function to define the DSL API task which makes an Endo[Project] and passes it as a Monoid to the Writer monad. In the following snippet, (p: Project) => withTask(t, p) is a mapping from Project => Project, which gets converted to an Endo and then passed to the Writer monad for adding to the task list of the Project.

def task(n: String): Writer[Endo[Project], Task] = {
  val t = Task(n)
  for {
    _ <- tell(((p: Project) => withTask(t, p)).endo)
  } yield t
}

The DSL snippet above is a monad comprehension. Let's add some more syntax to the DSL by defining dependencies of a Project. That's also a mapping from one Project state to another and can be realized using a similar function like withTask ..

// add project dependency
val withDependency = (t: Task, on: Task, p: Project) => 
  p.copy(deps = (t, on) :: p.deps)

.. and define a function dependsOn to our DSL that allows the user to add the explicit dependencies amongst tasks. But this time instead of making it a standalone function we will make it a method of the class Task. This is only for getting some free syntactic sugar in the DSL. Here's the modified Task ADT ..

case class Task(name: String) {
  def dependsOn(on: Task): Writer[Endo[Project], Task] = {
    for {
      _ <- tell(((p: Project) => withDependency(this, on, p)).endo)
    } yield this
  }
}

Finally we define the last API of our DSL that glues together the building of the Project and the addition of tasks and dependencies without directly coupling the user to some of the underlying implementation artifacts.

def project(name: String, startDate: Date)(e: Writer[Endo[Project], Task]) = {
  val p = Project(name, startDate)
  e.run._1(p)
}

And we can finally create a Project along with tasks and dependencies using our DSL ..

project("xenos", now) {
  for {
    a <- task("study customer requirements")
    b <- task("analyze usecases")
    _ <- b dependsOn a
    c <- task("design & code")
    _ <- c dependsOn b
    d <- c dependsOn a
  } yield d
}

In case you are interested I have the whole working example in my github repo.

Composing Heterogeneous DSLs in Scala

When we use DSLs to model business rules, we tend to use quite a few of them together. We may use a DSL for computing date/time, another one for manipulating money with currency, and a few others for implementing the actual rules of the domain. And not all of them will be developed by us - third party DSLs play an equally important role out here.

But when we use a bunch of heterogeneous DSLs, our application needs to be flexible enough to adapt to the independent evolution paths of each of them. We need to be able to embed entire DSLs within our application structure and yet not be coupled to the implementation of any of them.

In this blog post, I will demonstrate how we can organize heterogeneous DSLs hierarchically and achieve the above goals of keeping individual implementations decoupled from each other. This is also called representation independence and is yet another strategy of programming to an interface. I will use Scala as the implementation language and use the power of Scala's type system to compose these DSLs together.

The most interesting part of this implementation will be to ensure that the same representation of the composed DSL is used even when you extend your own DSL.

We are talking about accounts and how to compute the balance that a particular account holds. We have a method balanceOf that returns an abstraction named Balance. No idea about the implementation details of Balance though ..

trait Account {
  val bal: Balances
  import bal._

  def balanceOf: Balance
}


Account is part of our own DSL. Note the abstract val Balances - it's a DSL which we embed within our own abstraction that helps you work with Balance. You can make a Balance abstraction, manipulate balances, change currencies etc. In short it's a utility general purpose DSL that we can frequently plug in to our application structure. Here we stack it hierarchically within our own DSL.

Here's how Balances look ..

trait Balances {
  type Balance

  def balance(amount: Int, currency: String): Balance
  def amount(b: Balance): Int
  def currency(b: Balance): String
  def convertTo(b: Balance, currency: String): Balance
}


Note Balance is abstracted within Balances. And we have a host of methods for manipulating a Balance.

Let's now have a look at a sample implementation of Balances, which also concretizes a Balance implementation ..

class BalancesImpl extends Balances {
  case class BalanceImpl(amount: Int, currency: String)
  type Balance = BalanceImpl

  def balance(amount: Int, currency: String): Balance = {
    BalanceImpl(amount, currency)
  }

  def amount(b: Balance) = b.amount
  def currency(b: Balance) = b.currency

  def convertTo(b: Balance, toCurrency: String): Balance = {
    BalanceImpl(b.amount * 2, toCurrency)
  }
}


Note Balances is a complete DSL totally decoupled from our own DSL that has the Account abstraction.

Now on to some concrete Account implementations ..

trait BankAccount extends Account {
  // concrete implementation of Balances
  val bal = new BalancesImpl

  // object import syntax
  import bal._

  // dummy implementation but uses the Balances DSL
  override def balanceOf = balance(10000, "USD")
}
object BankAccount extends BankAccount


It's a bank account that uses a concrete implementation of the Balances DSL. Note the balanceOf method uses the balance() method of the Balances DSL accessible through the object import syntax.

Now the fun part. My BankAccount uses one specific implementation of Balances. I would like to add a few decorators to my account, which will have richer versions of the API implementation. How do I ensure that all decorators that I may define for BankAccount also get to use the same DSL implementation for Balances?

Here's how .. Not only do we compose decorator and decoratee abstractions hierarchically, we use Scala's singleton type to ensure that the same representation of the Balances DSL gets to flow from the decoratee to the decorator.

// decorator
trait InterestBearing extends Account {
  // decoratee
  val semantics: Account

  // singleton type pulls the same representation from up
  val bal: semantics.bal.type

  // object import ensures that any method of
  // Balances that we use below comes from the same singleton type
  import bal._

  def interest: Int = 100 // dummy implementation

  override def balanceOf = {
    val b = semantics.balanceOf
    balance(amount(b) + interest, currency(b))
  }
}


So we have done a hierarchical composition of two heterogeneous DSLs and ensured that a single representation of one DSL is used uniformly within the other even in the face of extensions and decorations. The process has been made easy by the power of Scala's static type system.

Now we can have a concrete instance of an interest bearing bank account as a single Scala module ..

object InterestBearingBankAccount extends InterestBearing {
  val semantics = BankAccount
  val bal: semantics.bal.type = semantics.bal
}



Combinators as the sublanguage of DSL syntax

In my presentation on DSLs in PhillyETE I had talked about organizing DSL syntax around a sub language of combinators. The underlying implementation model needs to be segregated from the engine that renders the syntax of your DSL. Doing this you can decouple your semantic model and reuse it in the context of other DSLs or even in some entirely different context. I called this the model-view architecture of DSLs. In this post I would like to extend my thoughts along similar lines and claim that publishing combinators that wire your model elements is a very effective way towards compositionality of your DSL syntax.

Functions compose naturally - hence if your DSL publishes combinators then your client can compose them to form larger structures. But for this composition to be effective, you need to have the proper types for each of them - yes, we are talking about type-safe composition.

In Scala I tend to use the curried syntax a lot when I design combinators. Let’s have a look at an example .. we assume that we have a domain model designed using some paradigm - it can be OO, it can be functional or maybe a mix of the two. When we implement a business rule, we use these model elements, wire them up and present a proper syntax to the user. As an example here's a domain rule for enrichment of a securities trade as it goes through its processing pipeline ..

1. get the tax/fee ids for a trade
2. calculate tax/fee values
3. enrich trade with net cash amount

Here are a few snippets of combinators that help users implement the rule ..

// get the list of tax/fees applicable for this trade
// depends on the market
val forTrade: Trade => Option[List[TaxFeeId]] = {trade =>
  taxFeeForMarket.get(trade.market) <+> taxFeeForMarket.get(Other)
}

// all tax/fees for a specific trade
val taxFeeCalculate: Trade => List[TaxFeeId] => List[(TaxFeeId, BigDecimal)] = {t => tids =>
  tids zip (tids ? valueAs(t))
}

val enrichTradeWith: Trade => List[(TaxFeeId, BigDecimal)] => BigDecimal = {trade => taxes =>
  taxes.foldLeft(principal(trade))((a, b) => a + b._2)
}


Note each of these combinators return a function instead of a concrete business abstraction. Of course combinators are supposed to return a function only. But my point is that with such a strategy it becomes easier to build larger abstractions out of these smaller ones. And these combinators will be domain facing - each of them implements a snippet of a business rule. Hence take special care to name them appropriately so that you have that ubiquitous language going within your DSL syntax. Here’s how we combine the above to form a larger business rule that implements the “enrichment of trade” functionality ..

// enrichment of trade
// implementation follows problem domain model
val enrich = for {
  // get the tax/fee ids for a trade
  taxFeeIds      <- forTrade 

  // calculate tax fee values
  taxFeeValues   <- taxFeeCalculate 

  // enrich trade with net amount
  netAmount      <- enrichTradeWith 
}
yield((taxFeeIds map taxFeeValues) map netAmount)

// enriching a trade
trade map enrich should equal(…)


.. very concise, very succinct and brings out the domain semantics very explicitly. This composition was possible only because we used the curried syntax in designing our combinators. Haskell has curry-by-default - no wonder it revels in designing of embedded DSLs.

So, is combinators as the building block of DSL syntax a good idea to follow ?

I have been using this strategy for some time now and it has been working out quite nicely for me. It’s not essential that your underlying domain model has to be purely functional. I have been using libraries like scalaz in scala that offers a good hierarchy of typeclasses which help a lot in designing generic combinators that work across an unrelated family of abstractions. In my PhillyETE talk I discussed how functors allows you to map over entities across diverse abstractions like List, Option and Functions. And once you make your combinators generic, they tend to be adopted within the DSL as part of the ubiquitous language.

Why I made DSLs In Action polyglotic

Since I started writing DSLs In Action (buy here)^*, a lot of my friends and readers asked me about my decision to make the book polyglotic. Indeed right from the word go, I had decided to treat the topic of DSL based design without a significant bias towards any specific language. Even today after the book has been published, many readers come up to me and ask the same question. I thought I would clarify my points on this subject in this blog post.

A DSL is a vehicle to speak the language of the domain on top of an implementation of a clean domain model. Whatever be the implementation language of the DSL, you need to make it speak the ubiquitous language of the domain. And by language I mean the syntax and the semantics that the experts of the particular domain are habituated to use.

A DSL is a facade of linguistic abstraction on top of your domain's semantic model. As a DSL designer it's your responsibility to make it as expressive to your users as possible. It starts with the mapping of the problem domain to the solution domain artifacts, converging on a common set of vocabulary for all the stakeholders of the implementation and finally getting into the nuts and bolts of how to implement the model and the language.

It's a known fact that there's NO programming language that can express ALL forms of abstraction in the most expressive way. So as a language designer you always have the flexibility to choose the implementation language based on your solution domain model. You make this choice as a compromise of all the forces that come up in any software development project. You have the timeline, the resident expertise within your team and other social factors to consider before you converge to the final set of languages. In short there's always a choice of the language(s) that you make just like any other software development effort.

Being Idiomatic

Same problem domains can be modeled in solution domain using radically different forms of abstraction. It depends on the language that you use and the power of abstraction that it offers. The same set of domain rules may be implemented using the type system of a statically typed language. While you need to use the power of meta-programming to implement the same concepts idiomatically in a dynamically typed language. Even within dynamic languages, idioms vary a lot. Clojure or the Lisp family offers compile time meta-programming in the form of macros to offer your users the specific custom syntactic structures that they need for the domain. While Ruby and Groovy do the same with runtime meta-programming.

Here's an example code snippet from my book of an internal DSL being used to register a security trade and computes its net cash value using the principal amount and the associated tax/fee components. It's implemented in Ruby using all of the Ruby idioms.

str = <<END_OF_STRING
new_trade 'T-12435' for account 'acc-123' to buy 100 shares of 'IBM',
                    at UnitPrice = 100
END_OF_STRING

TradeDSL.trade str do |t|
  CashValueCalculator.new(t).with TaxFee, BrokerCommission do |cv|
    t.cash_value = cv.value 
    t.principal = cv.p
    t.tax = cv.t
    t.commission = cv.c
  end
end


The above DSL has a different geometry than what you would get with the same domain concepts implemented using Clojure .. Have a look at the following snippet of a similar use case implemented in Clojure and executed from the Clojure REPL:

user> (def request {:ref-no "r-123", :account "a-123", :instrument "i-123", 
         :unit-price 20, :quantity 100})
#'user/request

user> (trade request)
{:ref-no "r-123", :account "a-123", :instrument "i-123", :principal 2000, :tax-fees {}}

user> (with-tax-fee trade
        (with-values :tax 12)
        (with-values :commission 23))
#'user/trade

user> (trade request)
{:ref-no "r-123", :account "a-123", :instrument "i-123", :principal 2000, 
  :tax-fees {:commission 460, :tax 240}}

user> (with-tax-fee trade
        (with-values :vat 12))
#'user/trade

user> (trade request)
{:ref-no "r-123", :account "a-123", :instrument "i-123", :principal 2000, 
  :tax-fees {:vat 240, :commission 460, :tax 240}}

user> (net-value (trade request))
2940


The above DSL is implemented using syntactic macros of Clojure for custom syntax building and standard functional programming idioms that the language supports.

In summary, we need to learn multiple languages in order to implement domain models idiomatically in each of them. DSLs In Action discusses all these ideas and contains lots and lots of implementations of real world use cases using Java, Scala, Clojure, Ruby and Groovy.

Hope this clears my intent of a polyglotic treatment of the subject in the book.


^* Affiliate Link

DSLs In Action is out!

It all started with a mail from Christina of Manning expressing their interest in publishing a book on building parsers in Scala. She mentioned that they have come across a blog post of mine titled External DSLs made easy with Scala Parser Combinators and Marjan Bace (Publisher) would like to talk to me on this possible venture. I still remember that hour long discussion with Marjan that ultimately led to the thoughts of coming out with a book on DSL design.

The ebook of DSLs In Action (buy here)^* has been published, the print version is due Dec 24, just in time for Christmas. The final book is sized at 377 pages with 9 chapters and 7 appendices. Before publication the book was reviewed at various levels by an illustrious panel of reviewers. I received lots of feedbacks and suggestions at Manning online Author's Forum. The current incarnation of the book is the result of assimilating all suggestions, feedbacks and critiques aided by the constructive editing process of the Manning crew. Jonas Boner has been kind enough to write the foreword of the book. The result is now a reality for all of you to read, enjoy and review.

Some one asked to me pen down my thoughts on DSLs In Action in one page. Not the literal summary of what it contains, but the impressions of some of the thought snippets that it can potentially induce into the minds of its readers. This post is an attempt towards that.

DSL and polyglotism

A DSL is targeted for a specific domain. Your implementation should be expressive enough to model the ubiquitous language that your domain users use everyday. Hence the practice od DSL driven development has an aura of polyglotism in it. Assuming you have the relevant expertise in your team, target the implementation language that's best suited for developing the syntax of the domain. DSLs In Action considers case studies in Ruby, Groovy, Clojure and Scala and focuses its readers to the implementation patterns of DSLs in each of them.

A DSL as a collaboration medium

The one most standout dimension that DSL driven development adds to a team is an increased emphasis on collaboration. The main value proposition that a DSL brings to table is a better collaboration model between the developers and the domain users. It's no secret that most of today's failed projects attribute their debacle to the huge gap in the developers' understanding of the domain. This mainly results from the lack of communication between the developers and the domain users and also between the different groups of developers. When you have code snippets like the following as part of your domain model ..

import TaxFeeImplicits._
override def calculatedAs(trade: Trade): PartialFunction[TaxFee, BigDecimal] = {
  case TradeTax   => 5.  percent_of trade.principal
  case Commission => 20. percent_of trade.principal
  case Surcharge  => 7.  percent_of trade.principal
  case VAT        => 7.  percent_of trade.principal
}


you can collaborate very well with your domain experts to verify the correctness of implementation of the business rule for tax calculation on a security trade. DSLs In Action dispels the myth that a DSL will make your domain analysts programmers - it will not. A DSL best serves as the collaboration bridge to discuss business rules' implementation with your analysts.

Importance of the semantic model

A DSL is a thin layer of linguistic abstraction atop a semantic model. I have blogged on this topic some time ago. The semantic model is the set of abstractions on which you grow your DSL syntax. The semantic model can potentially be reused for purposes other than developing your DSL layer - hence it needs to be as loosely coupled as possible with the syntax that your DSL publishes. In DSLs In Action I discuss the concept of a DSL Facade that helps you decouple basic abstractions of the domain model from the language itself.

DSL design is also abstraction design

Throughout DSLs In Action I have highlighted many of the good qualities that a well-designed abstraction needs to have. In fact Appendix A is totally dedicated to discussing the qualities of good abstractions. In almost every chapter I talk about how these qualities (minimalism, distillation, composability and extensibility) apply in every step of your DSL design. In real world when you implement DSLs, you will see that entire DSLs also need to be composed as abstractions. Use a proper language that supports good models of abstraction composition

External and Internal is only one dimension of classifying DSLs

Though broadly I talk about external and internal DSLs, actually there are lots of different implementation patterns even within them. Internal DSLs can be embedded where you emded the domain language within the type system of the host language. Or they can generative where you allow the language runtime to generate code for you, like you do in Ruby or Groovy using dynamic meta-programming. With the Lisp family you can create your own language using macros. Some time back I blogged about all these varying implementation patterns with internal and external DSLs.

Hope you all like DSLs In Action. If you have reached this far in the post, there are chapters 1 and 4 free on Manning site taht would help you get a sneak peak of some of the stuff that I talked about .. Happy Holidays!

^* Affiliate Link

Composable domain models using scalaz

I have been having some solid fun working through scalaz - it's possibly as close you can get to Haskell with a postfunctional language like Scala, which also supports object oriented paradigms. One of the ways I do learn languages is by developing domain models using the idioms that the language offers and try to make the model as expressive as possible. I pick up domains on which I have worked before - so I have an idea of how much I can gain in epressivity using the new language compared to implementations in older languages.

Securities trading is a domain on which I have been working since the last 10 years. I have implemented domain models of securities trading back office systems in Java and Scala. It's time to add scalaz to the mix and see how much more functional my model turns out to be. I have created a playground for this - tryscalaz is a repository on my github that hosts some of my experiments with scalaz. I have started building a domain model for trading systems. It's far from being a realistic one for production use - its main purpose is to make myself more conversant with scalaz.

Scalaz is a wonderful experiment - it's definitely what functional Scala programs should look like. It has a small but wonderful community - Jason (@retronym) and Runar (@runarorama) always help me proactively both on the mailing list and on Twitter.

I am not going into every detail of how my trade domain model shapes up with Scalaz. I implemented a similar domain model in Haskell very recently and documented it here, here and here on my blog. If nothing else, it will help you compare the various aspects of both the implementations.

In this post let me go through some of the features of Scalaz that I found wonderfully expressive to model your domain constraints. You can get a lot out of using Scala only. But with Scalaz, you can take your composition at a much higher level through the various combinators that it offers as part of implementing typeclasses for functors, applicatives, monads and arrows. I haven't yet explored all of these abstractions - yet many of those are already very useful in making your domain models concise, yet expressive.

Here's some example of composition using the higher order functions that Scalaz offers ..



Note how we can compose the functions much like the Haskell way that I described in the earlier posts. In the above composition, I used map, which we can do in Scala for lists or options which explicitly support a map operation that maps a function over the collection. With scalaz we can use mapping of a function over any A of kind *->* for which there exists a Functor[A]. Scala supports higher kinds and scalaz uses it to make map available more generally than what you get in the Scala standard library.

Now let's infuse some more Scalaz magic into it. Frequently we need to do the same operations on a list of trades, which means that instead of just a map, we need to lift the functions through another level of indirection. Much like ..



Note how the functions forTrade, taxFees etc. get lifted into the List of Options.

Another nice feature that becomes extremely useful with scalaz in a domain model is the use of non-breaking error handling. This is made elegant by designing the Validation[] abstraction as an applicative functor. You can design your validation functions of the domain model as returning an instance of Validation[]. They can then be wired together in a variety of ways to implement accumulation of all failures before reporting to the user .. Here's a simple example from the Trade domain model ..



Validation[] in scalaz works much like Either[], but has a more expressive interface that specifies the success and error types explicitly ..



You can use Validation[] in comprehensions or as an applicative functor and wire up your domain validation logic in a completely functional way. Here's how our above validations work on the REPL ..



When we have invalid trade arguments all validation errors are accumulated and then reported to the user. If all arguments are valid, then we have a valid Trade instance as success. Cool stuff .. a recipe that I would like to have as part of my domain modeling everytime I start a new project ..

Updates on DSLs In Action - Into Copy Editing

I have completed writing DSLs In Action. As we speak, the book has moved from the development editor to the copy editor. I will be starting the process of copy editing along with the team of helpful copy editors of Manning.

The Table of Contents has been finalized. Have a look at the details and send me your feedbacks regarding the contents of the book.

DSLs In Action is a book for the practitioner. It contains real world experience of writing DSLs in a multitude of JVM languages. As the table of contents show, I have used Java, Groovy, Ruby, Scala and Clojure to demonstrate their power in DSL design and implementation. I have also focused on the integration aspects between these languages, which is fashionably known today by the name of polyglot programming.

All examples in the book are from the real world domain of securities trading and brokerage systems. I have intentionally chosen a specific domain to demonstrate the progression of DSL implementation from small trivial examples to serious complex and non-trivial ones. This also goes to bust a common myth that DSLs are applicable only for toy examples.

Another recurring theme throughout the book has been a strong focus on abstraction design. Designing good DSLs in an exercise in making well-designed abstractions. A DSL is really a thin linguistic abstraction on top of the semantic model of the domain. If the underlying model is expressive enough and publishes well behaved abstractions, then designing a user friendly syntax on top of it becomes easy. The book discusses lots of tools and techniques that will help you think in terms of designing expressive DSLs.

The book is replete with code written in multiple languages. You can get it all by cloning my github repo which contains maven based instructions to try most of them yourself.

And finally, thanks to all the reviewers for the great feedback received so far. They have contributed a lot towards improvement of the book, all remaining mistakes are mine.

DSL Interoperability and Language Cacophony

Many times I hear people say that DSL based development often leads to a situation where you have to manage a mix of code written in various languages, which are barely interoperable and often need to be integrated using glues like ScriptEngine that makes your life more difficult than easier. Well, if you are in this situation, you are in the world of language cacophony. You have somehow managed to pitchfork yourself into this situation through immature decisions of either selecting an improper language of implementation for your DSL or the means of integrating them with your core application.

Internal DSLs are nothing but a disciplined way of designing APis which speak the Ubiquitous Language of the domain you are modeling. Hence when you design an internal DSL, stick to the idioms and best practices that your host language offers. For domain rules that you think could be better modeled using a more expressive language, select one that has *good* interoperability with your host language. If you cannot find one, work within the limitations of your host language It's better to be slightly limiting in syntax and flexibility with your DSL than trying to force integration with a language that does not interoperate seamlessly with your host language. Otherwise you will end up having problems not only developing the interfaces between the DSL and the host language, you will have other endless problems of interoperability in the face of exceptions as well.

Don't let the language cacophony invade the horizons of your DSL implementation!

Consider the following example where we have a DSL in Groovy that models an Order that a client places to a securities broker firm for trading of stocks or bonds. The DSL is used to populate orders into the database when a client calls up the broker and asks for buy/sell transactions.



I am not going into the details of implementation of this DSL. But consider that the above script on execution returns an instance of Order, which is an abstraction that you developed in Groovy. Now you have your main application written in Java where you have the complete domain model of the order processing component. Your core application Order abstraction may be different from what you have in the DSL. Your DSL only constructs the abstraction that you need to populate the order details that the user receives from their clients.

It's extremely important that the Order abstraction that you pass from executing the Groovy script be available within your Java code. You can use the following mechanism to execute your Groovy code ..



This runs the script but does not have any integration with your Java application. Another option may be using the Java 6 ScriptEngine for talking back to your Java application. Something like the following snippet which you can have wiothin your main application to execute the Groovy script using ScriptEngine ..



Here you have some form of integration, but it's not ideal. You get back some objects into your Java code, can iterate over the collection .. still the objects that you get back from Groovy are opaque at best. Also since the script executes in the sandbox of the ScriptEngine, in case of any exception, the line numbers mentioned in the stack trace will not match the line number of the source file. This can lead to difficulty in debugging exceptions thrown from the DSL script.

Groovy has excellent interoperability with Java even at the scripting level. When integrating DSLs with your main application always prefer the language specific integration features over any generic script engine based support. Have a look at the following that does the same job of executing the Groovy script from within a Java application. But this time we use GroovyClassLoader, a ClassLoader that extends URLClassLoader and can load Groovy classes from within your Java application ..



You will have to make your DSL script return a Closure that then gets called within the Java application. Note that within Java we can now get complete handle over the Order classes which we defined in Groovy.

This is an example to demonstrate the usage of proper integration techniques while designing your DSL. In my upcoming book DSLs In Action I discuss many other techniques of integration for DSLs developed on the JVM. The above example is also an adaptation from what I discuss in the book in the context of integrating Groovy DSLs with Java application.

Thanks to Guillame Laforge and John Wilson for many suggestions on improving the Groovy DSL and it's interoperability with Java.

Thrush in Clojure

Quite some time back I wrote a blog post on the Thrush Combinator implementation in Scala. Just for those who missed reading it, Thrush is one of the permuting combinators from Raymond Smullyan's To Mock a Mockingbird. The book is a fascinating read where the author teaches combinatorial logic using examples of birds from an enchanted forest. In case you've not yet read it, please do yourself a favor and get it from your favorite book store.

A Thrush is defined by the following condition: Txy = yx. Thrush reverses the order of evaluation. In our context, it's not really an essential programming tool. But if you're someone who takes special care to make your code readable to the human interface, the technique sometimes comes in quite handy.

Recently I came across Thrush in Clojure. You don't have to implement anything - it's there for you in the Clojure library implemented as a macro ..

Conside this simple example of bank accounts where we represent an account as a Map in Clojure ..

(def acc-1 {:no 101 :name "debasish" :type 'savings :balance 100})
(def acc-2 {:no 102 :name "john p." :type 'checking :balance 200})


We have a list of accounts and we need to find all savings accounts and compute the sum of their current balances .. well not too difficult in Clojure ..

(defn savings-balance
  [& accounts]
  (apply +
    (map :balance
      (filter #(= (:type %) 'savings) 
        (seq accounts)))))


To a programmer familiar with the concepts of functional programming, it's quite clear what the above function does. Let's read it out for all of us .. From a list of accounts, filter the ones with type as savings, get their balances and report the sum of them. That was easy .. but did you notice that we read it inside out from the implementation, which btw is a 4 level nested function ?

Enter Thrush ..

Being a permuting combinator, Thrush enables us to position the functions outside in, in the exact sequence that the human mind interprets the problem. In our Scala version we had to implement something custom .. with Clojure, it comes with the standard library .. have a look ..

(defn savings-balance
  [& accounts]
  (->> (seq accounts)
       (filter #(= (:type %) 'savings))
       (map :balance)
       (apply +)))


->> is implemented as a macro in Clojure that does the following :

1. threads the first form (seq accounts) as the last argument of the second form (the filter), which makes (seq accounts) the last argument of filter
2. then makes the entire filter form, including the newly added argument the last argument of the map

.. and so on for all the forms present in the argument list. Effectively the resulting form that we see during runtime is our previous version using nested functions. The Thrush combinator only dresses it up nicely for the human eyes synchronizing the thought process with the visual implementation of the logic. And all this at no extra runtime cost! Macros FTW :)

->> has a related cousin ->, which is same as ->>, but only threads the forms as the second argument of the next form instead of the last. These macros implement Thrush in Clojure. Idiomatic Clojure code is concise and readable and using a proper ubiquitous language of the domain, makes a very good DSL. Think about using Thrush when you feel that reordering the forms will make your API look more intuitive to the user.

Thrush also helps you implement the Decorator pattern in a very cool and concise way. In my upcoming book, DSLs In Action I discuss these techniques in the context of designing DSLs in Clojure.

DSL : Grow your syntax on top of a clean semantic model

A DSL primarily has two components - a semantic model that abstracts the underlying domain and a linguistic abstraction on top that speaks the dialect of the user. The semantic model is the model of the domain where you can apply all the principles of DDD that Eric Evans espouses. And the linguistic abstraction is a thin veneer on top of the underlying model. The more well abstracted your model is, easier will be the construction of the layer on top of it. Here's a general architecture of a DSL engineering stack :-

It's interesting to observe that the two components of the stack evolve somewhat orthogonally.

The Semantic Model evolves Bottom Up

The semantic model usually evolves in a bottom up fashion - larger abstractions are formed from smaller abstractions using principles of composition. It can be through composition of traits or objects or it can be through composition of functions as well. How beautiful your compositions can be depends a lot on the language you use. But it's important that the semantic model also speaks the language of the domain.

Here's an example code snippet from my upcoming book DSLs In Action that models the business rules for a trading DSL. When you do a trade on a stock exchange you get charged a list of tax and fee components depending on the market where you execute the trade. The following snippet models a business rule using Scala that finds out the list of applicable tax/fee heads for a trade ..

class TaxFeeRulesImpl extends TaxFeeRules {
  override def forTrade(trade: Trade): List[TaxFee] = {
    (forHKG orElse 
       forSGP orElse 
         forAll)(trade.market)
  }
   
  val forHKG: PartialFunction[Market, List[TaxFee]] = { 
    case HKG => 
      // in real life these can come from a database
      List(TradeTax, Commission, Surcharge)
  }
    
  val forSGP: PartialFunction[Market, List[TaxFee]] = {
    case SGP => 
      List(TradeTax, Commission, Surcharge, VAT)
  }
    
  val forAll: PartialFunction[Market, List[TaxFee]] = {
    case _ => List(TradeTax, Commission)
  }

  //..
}


The method forTrade clearly expresses the business rule, which reads almost as expressive as the English version ..

"Get the Hong Kong specific list for trades executed on the Hong Kong market OR Get the Singapore specific list for trades executed on the Singapore market OR Get the most generic list valid for all other markets"

Note how Scala PartialFunction s can be chained together to give the above model an expressive yet succinct syntax.

The Language Interface evolves Top Down

Here you start with the domain user. What dialect does he use on the trading desk ? And then you try to build an interpreter around that which uses the services that the semantic model publishes. I call this thin layer of abstraction a DSL Facade that sits between your DSL script and the underlying domain model and acts as the glue.

It also depends a lot on the host language as to how you would like to implement the facade. With a language like Lisp, macros can come in very handy in designing an interpreter layer for the facade. And with macros you do bottom up programming, bending the host language to speak your dialect.

When you are developing an external DSL, the EBNF rules that you specify act as the DSL Facade for growing your syntax. Within the rules you can use foreign code embedding to interact with your semantic model.

In summary, when you design a DSL, the semantic model is as important as the dialect that it speaks. Having a well designed semantic model is an exercise in designing well-engineered abstractions. And as I mention in my book, the four qualities of good abstractions are minimalism, distillation, extensibility and composability.