DHL delivered a box of these at my doorstep today

Happy New Year! Have a blast .. See ya all in 2011 ..

A case study of cleaner composition of APIs with the Reader monad

In my earlier post on composable domain models, I wrote about the following DSL that captures the enrichment of a security trade by computing the applicable tax/fees and then the net cash value of the trade. It uses chained composition of scalaz functors .. In this post we are going to improve upon the compositionality, introduce a new computation structure and make our APIs leaner with respect to type signatures ..

scala> trd1 ° forTrade ° taxFees ° enrichWith ° netAmount  
res0: Option[scala.math.BigDecimal] = Some(3307.5000)


Here are the building blocks for the above .. the individual functions and the type definitions for each of them ..

forTrade: Trade => (Trade, Option[List[TaxFeeId]])
taxFees: (Trade, Option[List[TaxFeeId]]) => (Trade, List[(TaxFeeId, BigDecimal)])
enrichWith: (Trade, List[(TaxFeeId, BigDecimal)]) => RichTrade
netAmount: RichTrade => Option[BigDecimal]


and here's the chaining in action with wiring made explicit ..

Note how we explicitly wire the types up so as to make the entire computation composable. Composability is a worthwhile quality to have for your abstractions. However in order for your functions to compose, the types for input and output for each of them must match. In the above example, we need to have forTrade spit out a Trade object along with the list of tax/fee id, in order for it to compose with taxFees.

For an API to be usable, the secret sauce is to make it lean. Never impose any additional burden on to your API's interface that smells of incidental complexity to the user. This is exactly what we are doing in the above composition. Note we are carrying around the Trade argument pipelining it through each of the above functions. In our use case the Trade is a read-only state and needs to be shared amongst all functions to read the information from the object.

Enter the Reader Monad

Refactor the above into the Reader monad. A Reader is meant to be used as an environment (it's also known as the Environment monad) for all the participating components of the computation. What we need to do for this is to set up a monadic structure for our computation. Here are the modified function signatures .. I have changed some of the names for better adaptability with the domain, but you get the idea ..

val forTrade: Trade => Option[List[TaxFeeId]] = //..
val taxFeeCalculate: Trade => List[TaxFeeId] => List[(TaxFeeId, BigDecimal)] = //..
val enrichTradeWith: Trade => List[(TaxFeeId, BigDecimal)] => BigDecimal = //..


Every function takes the Trade but we no longer have to do an explicit chaining by emitting the Trade also as an output. This is where a monad shines. A monad gives you a shared interface to many libraries where you don't need to implement sequencing explicitly within your DSELs.

And here's our DSEL that runs through the sequence of enriching a trade while using the passed in trade as an environment .. (thanks @runarorama for the help with the Reader in scalaz)

val enrich = for {
  taxFeeIds      <- forTrade        // get the tax/fee ids for a trade
  taxFeeValues   <- taxFeeCalculate // calculate tax fee values
  netAmount      <- enrichTradeWith // enrich trade with net amount
}
yield((taxFeeIds ° taxFeeValues) ° netAmount)


This is a comprehension in Scala which is like the do notation of Haskell. Desugar it as an exercise and explore how flatMap does the sequencing.

Here's what the type of enrich looks like ..

scala> enrich
res1: (net.debasishg.domain.trade.dsl.TradeModel.Trade) => Option[BigDecimal] = <function1>


enrich is monadic in nature and follows the usual structure of a monad that sequences its operations through bind to give it an imperative look and feel. If any of the above sub-computations fail, the whole computation fails. But show it to a person who knows the domain of security trading - the steps in enrich nicely models the ubiquitous language.

I have the entire DSL in my github repo. You can get the use of enrich here in the test case ..

DSLs In Action is out!

It all started with a mail from Christina of Manning expressing their interest in publishing a book on building parsers in Scala. She mentioned that they have come across a blog post of mine titled External DSLs made easy with Scala Parser Combinators and Marjan Bace (Publisher) would like to talk to me on this possible venture. I still remember that hour long discussion with Marjan that ultimately led to the thoughts of coming out with a book on DSL design.

The ebook of DSLs In Action (buy here)^* has been published, the print version is due Dec 24, just in time for Christmas. The final book is sized at 377 pages with 9 chapters and 7 appendices. Before publication the book was reviewed at various levels by an illustrious panel of reviewers. I received lots of feedbacks and suggestions at Manning online Author's Forum. The current incarnation of the book is the result of assimilating all suggestions, feedbacks and critiques aided by the constructive editing process of the Manning crew. Jonas Boner has been kind enough to write the foreword of the book. The result is now a reality for all of you to read, enjoy and review.

Some one asked to me pen down my thoughts on DSLs In Action in one page. Not the literal summary of what it contains, but the impressions of some of the thought snippets that it can potentially induce into the minds of its readers. This post is an attempt towards that.

DSL and polyglotism

A DSL is targeted for a specific domain. Your implementation should be expressive enough to model the ubiquitous language that your domain users use everyday. Hence the practice od DSL driven development has an aura of polyglotism in it. Assuming you have the relevant expertise in your team, target the implementation language that's best suited for developing the syntax of the domain. DSLs In Action considers case studies in Ruby, Groovy, Clojure and Scala and focuses its readers to the implementation patterns of DSLs in each of them.

A DSL as a collaboration medium

The one most standout dimension that DSL driven development adds to a team is an increased emphasis on collaboration. The main value proposition that a DSL brings to table is a better collaboration model between the developers and the domain users. It's no secret that most of today's failed projects attribute their debacle to the huge gap in the developers' understanding of the domain. This mainly results from the lack of communication between the developers and the domain users and also between the different groups of developers. When you have code snippets like the following as part of your domain model ..

import TaxFeeImplicits._
override def calculatedAs(trade: Trade): PartialFunction[TaxFee, BigDecimal] = {
  case TradeTax   => 5.  percent_of trade.principal
  case Commission => 20. percent_of trade.principal
  case Surcharge  => 7.  percent_of trade.principal
  case VAT        => 7.  percent_of trade.principal
}


you can collaborate very well with your domain experts to verify the correctness of implementation of the business rule for tax calculation on a security trade. DSLs In Action dispels the myth that a DSL will make your domain analysts programmers - it will not. A DSL best serves as the collaboration bridge to discuss business rules' implementation with your analysts.

Importance of the semantic model

A DSL is a thin layer of linguistic abstraction atop a semantic model. I have blogged on this topic some time ago. The semantic model is the set of abstractions on which you grow your DSL syntax. The semantic model can potentially be reused for purposes other than developing your DSL layer - hence it needs to be as loosely coupled as possible with the syntax that your DSL publishes. In DSLs In Action I discuss the concept of a DSL Facade that helps you decouple basic abstractions of the domain model from the language itself.

DSL design is also abstraction design

Throughout DSLs In Action I have highlighted many of the good qualities that a well-designed abstraction needs to have. In fact Appendix A is totally dedicated to discussing the qualities of good abstractions. In almost every chapter I talk about how these qualities (minimalism, distillation, composability and extensibility) apply in every step of your DSL design. In real world when you implement DSLs, you will see that entire DSLs also need to be composed as abstractions. Use a proper language that supports good models of abstraction composition

External and Internal is only one dimension of classifying DSLs

Though broadly I talk about external and internal DSLs, actually there are lots of different implementation patterns even within them. Internal DSLs can be embedded where you emded the domain language within the type system of the host language. Or they can generative where you allow the language runtime to generate code for you, like you do in Ruby or Groovy using dynamic meta-programming. With the Lisp family you can create your own language using macros. Some time back I blogged about all these varying implementation patterns with internal and external DSLs.

Hope you all like DSLs In Action. If you have reached this far in the post, there are chapters 1 and 4 free on Manning site taht would help you get a sneak peak of some of the stuff that I talked about .. Happy Holidays!

^* Affiliate Link

Monads, Applicative Functors and sequencing of effects

Monads and applicative functors are both used to model computations - yet it's interesting to note the subtle differences in the way they handle sequencing of effects. Both of them support an applicative style of effectful programming that lets you write code in pointfree style (in Haskell) making your code look so expressive.

Applicative functors are more general than monads and hence have a broader area of application in computation. Haskell does not define monads to be a derivative of applicative functors, possibly for some historical reasons. Scalaz does it and does it correctly and conveniently for you to reduce the number of abstractions that you need to have.

In Haskell we have sequence and sequenceA implemented for monads and applicatives separately ..

-- in Control.Monad
sequence  :: Monad m => [m a] -> m [a]

-- in Data.Traversable
class (Functor t, Foldable t) => Traversable t where
  sequenceA :: Applicative f => t (f a) -> f (t a)


In scalaz we have the following ..

// defined as pimped types on type constructors
def sequence[N[_], B](implicit a: A <:< N[B], t: Traverse[M], n: Applicative[N]): N[M[B]] =
    traverse((z: A) => (z: N[B]))


sequence is defined on applicatives that works for monadic structures as well ..

Monads are more restrictive than applicatives. But there are quite a few use cases where you need to have a monad and NOT an applicative. One such use case is when you would like to change the sequence of an effectful computation depending on a previous outcome.

A use case for monads

Have a look at the function ifM (monadic if) defined in scalaz ..

// executes the true branch of ifM
scala> true.some ifM(none[Int], 4.some)
res8: Option[Int] = None


Here's how ifM is defined ..

def ifM[B](t: => M[B], f: => M[B])(implicit a: Monad[M], b: A <:< Boolean): M[B] = 
  >>= ((x: A) => if (x) t else f)


It uses the monadic bind that can influence a subsequent computation depending on the outcome of a previous computation.

(>>=) :: m a -> (a -> m b) -> m b


Now consider the same computation implemented using an applicative ..

scala> val ifA = (b: Boolean) => (t: Int) => (f: Int) => (if (b) t else f)            
ifA: (Boolean) => (Int) => (Int) => Int = <function1>

// good!
scala> none <*> (some(12) <*> (some(false) map ifA)) 
res41: Option[Int] = None

// bad!
scala> none <*> (some(12) <*> (some(true) map ifA)) 
res42: Option[Int] = None


<*> just sequences the effects through all computation structures - hence we get the last effect as the return value which is the one for the else branch. Have a look at the last snippet where even though the condition is true, we have the else branch returned.

(<*>) :: f(a -> b) -> f a -> f b


So <*> cannot change the structure of the computation which remains fixed - it just sequences the effects through the chain.

Monads are the correct way to model your effectful computation when you would like to control the structure of computation depending on the context.

A use case for applicatives

scalaz implements Validation as an applicative functor. This is because here we need to accumulate all the effects that the validation can produce. As I noted in my last post on composing abstractions in scalaz, the following snippet will accumulate all validation errors before bailing out .. quite unlike a monadic API ..

def makeTrade(account: Account, instrument: Instrument, refNo: String, market: Market, 
  unitPrice: BigDecimal, quantity: BigDecimal) =
  (validUnitPrice(unitPrice).liftFailNel |@| 
    validQuantity(quantity).liftFailNel) { (u, q) => Trade(account, instrument, refNo, market, u, q) }


Let's look into it in a bit more detail ..

sealed trait Validation[+E, +A] {
  //..
}

final case class Success[E, A](a: A) extends Validation[E, A]
final case class Failure[E, A](e: E) extends Validation[E, A]


Note in case of success only the actual computation value gets propagated, as in the following ..

trait Validations {
  //..
  def success[E, A](a: A): Validation[E, A] = Success(a)
  def failure[E, A](e: E): Validation[E, A] = Failure(e)
  //..
}


With a monadic bind, the computation will be aborted on the first error as we don't have the computation value to be passed to the second argument of >>=. Applicatives allow us to sequence through the computation chain nicely and accumulate all the effects. Hence we can get effects like ..

// failure case
scala> makeTrade("a-123", "google", "ref-12", Singapore, -10, 600)
res2: scalaz.Validation[scalaz.NonEmptyList[String],net.debasishg.domain.trade.Trades.Trade] = 
  Failure(NonEmptyList(price must be > 0, qty must be <= 500))


One other interesting abstraction to manipulate computation structures is an Arrow, which makes an interesting comparison with Applicative Functors. But that's for some other day, some other post ..

Composable domain models using scalaz

I have been having some solid fun working through scalaz - it's possibly as close you can get to Haskell with a postfunctional language like Scala, which also supports object oriented paradigms. One of the ways I do learn languages is by developing domain models using the idioms that the language offers and try to make the model as expressive as possible. I pick up domains on which I have worked before - so I have an idea of how much I can gain in epressivity using the new language compared to implementations in older languages.

Securities trading is a domain on which I have been working since the last 10 years. I have implemented domain models of securities trading back office systems in Java and Scala. It's time to add scalaz to the mix and see how much more functional my model turns out to be. I have created a playground for this - tryscalaz is a repository on my github that hosts some of my experiments with scalaz. I have started building a domain model for trading systems. It's far from being a realistic one for production use - its main purpose is to make myself more conversant with scalaz.

Scalaz is a wonderful experiment - it's definitely what functional Scala programs should look like. It has a small but wonderful community - Jason (@retronym) and Runar (@runarorama) always help me proactively both on the mailing list and on Twitter.

I am not going into every detail of how my trade domain model shapes up with Scalaz. I implemented a similar domain model in Haskell very recently and documented it here, here and here on my blog. If nothing else, it will help you compare the various aspects of both the implementations.

In this post let me go through some of the features of Scalaz that I found wonderfully expressive to model your domain constraints. You can get a lot out of using Scala only. But with Scalaz, you can take your composition at a much higher level through the various combinators that it offers as part of implementing typeclasses for functors, applicatives, monads and arrows. I haven't yet explored all of these abstractions - yet many of those are already very useful in making your domain models concise, yet expressive.

Here's some example of composition using the higher order functions that Scalaz offers ..



Note how we can compose the functions much like the Haskell way that I described in the earlier posts. In the above composition, I used map, which we can do in Scala for lists or options which explicitly support a map operation that maps a function over the collection. With scalaz we can use mapping of a function over any A of kind *->* for which there exists a Functor[A]. Scala supports higher kinds and scalaz uses it to make map available more generally than what you get in the Scala standard library.

Now let's infuse some more Scalaz magic into it. Frequently we need to do the same operations on a list of trades, which means that instead of just a map, we need to lift the functions through another level of indirection. Much like ..



Note how the functions forTrade, taxFees etc. get lifted into the List of Options.

Another nice feature that becomes extremely useful with scalaz in a domain model is the use of non-breaking error handling. This is made elegant by designing the Validation[] abstraction as an applicative functor. You can design your validation functions of the domain model as returning an instance of Validation[]. They can then be wired together in a variety of ways to implement accumulation of all failures before reporting to the user .. Here's a simple example from the Trade domain model ..



Validation[] in scalaz works much like Either[], but has a more expressive interface that specifies the success and error types explicitly ..



You can use Validation[] in comprehensions or as an applicative functor and wire up your domain validation logic in a completely functional way. Here's how our above validations work on the REPL ..



When we have invalid trade arguments all validation errors are accumulated and then reported to the user. If all arguments are valid, then we have a valid Trade instance as success. Cool stuff .. a recipe that I would like to have as part of my domain modeling everytime I start a new project ..