DSL Interoperability and Language Cacophony

Many times I hear people say that DSL based development often leads to a situation where you have to manage a mix of code written in various languages, which are barely interoperable and often need to be integrated using glues like ScriptEngine that makes your life more difficult than easier. Well, if you are in this situation, you are in the world of language cacophony. You have somehow managed to pitchfork yourself into this situation through immature decisions of either selecting an improper language of implementation for your DSL or the means of integrating them with your core application.

Internal DSLs are nothing but a disciplined way of designing APis which speak the Ubiquitous Language of the domain you are modeling. Hence when you design an internal DSL, stick to the idioms and best practices that your host language offers. For domain rules that you think could be better modeled using a more expressive language, select one that has *good* interoperability with your host language. If you cannot find one, work within the limitations of your host language It's better to be slightly limiting in syntax and flexibility with your DSL than trying to force integration with a language that does not interoperate seamlessly with your host language. Otherwise you will end up having problems not only developing the interfaces between the DSL and the host language, you will have other endless problems of interoperability in the face of exceptions as well.

Don't let the language cacophony invade the horizons of your DSL implementation!

Consider the following example where we have a DSL in Groovy that models an Order that a client places to a securities broker firm for trading of stocks or bonds. The DSL is used to populate orders into the database when a client calls up the broker and asks for buy/sell transactions.



I am not going into the details of implementation of this DSL. But consider that the above script on execution returns an instance of Order, which is an abstraction that you developed in Groovy. Now you have your main application written in Java where you have the complete domain model of the order processing component. Your core application Order abstraction may be different from what you have in the DSL. Your DSL only constructs the abstraction that you need to populate the order details that the user receives from their clients.

It's extremely important that the Order abstraction that you pass from executing the Groovy script be available within your Java code. You can use the following mechanism to execute your Groovy code ..



This runs the script but does not have any integration with your Java application. Another option may be using the Java 6 ScriptEngine for talking back to your Java application. Something like the following snippet which you can have wiothin your main application to execute the Groovy script using ScriptEngine ..



Here you have some form of integration, but it's not ideal. You get back some objects into your Java code, can iterate over the collection .. still the objects that you get back from Groovy are opaque at best. Also since the script executes in the sandbox of the ScriptEngine, in case of any exception, the line numbers mentioned in the stack trace will not match the line number of the source file. This can lead to difficulty in debugging exceptions thrown from the DSL script.

Groovy has excellent interoperability with Java even at the scripting level. When integrating DSLs with your main application always prefer the language specific integration features over any generic script engine based support. Have a look at the following that does the same job of executing the Groovy script from within a Java application. But this time we use GroovyClassLoader, a ClassLoader that extends URLClassLoader and can load Groovy classes from within your Java application ..



You will have to make your DSL script return a Closure that then gets called within the Java application. Note that within Java we can now get complete handle over the Order classes which we defined in Groovy.

This is an example to demonstrate the usage of proper integration techniques while designing your DSL. In my upcoming book DSLs In Action I discuss many other techniques of integration for DSLs developed on the JVM. The above example is also an adaptation from what I discuss in the book in the context of integrating Groovy DSLs with Java application.

Thanks to Guillame Laforge and John Wilson for many suggestions on improving the Groovy DSL and it's interoperability with Java.

PubSub with Redis and Akka Actors

Redis (the version on the trunk) offers publish/subscribe based messaging. This is quite a big feature compared to the typical data structure oriented services that it had been offering so far. This also opens up lots of possibilities to use Redis as a messaging engine of a different kind. The sender and the receiver of the messages are absolutely decoupled from each other in the sense that senders do not send messages to specific receivers. Publishers publish messages on specific channels. Subscribers who subscribe to those channels get them and can take specific actions on them. As Salvatore notes in his weekly updates on Redis, this specific feature has evolved from lots of user requests who had been asking for a general notification mechanism to trap changes in the key space. Redis already offers BLPOP (Blocking list pop operation) for similar use cases. But still it's not sufficient to satisfy the needs of a general notification scheme. Salvatore explains it in more details in his blog post.

I have been working on a Scala client, which I forked from Alejandro Crosa's repository. I implemented pubsub very recently and also have integrated it with Akka actors. The full implementation of the pubsub client in Scala is in my github repository. And if you like to play around with the Akka actor based implementation, have a look at the Akka repository.

You define your publishers and subscribers as actors and exchange messages over channels. You can define your own callbacks as to what you would like to do when you receive a particular message. Let's have a look at a sample implementation at the client level .. I will assume that you want to implement your own pub/sub application on top of the Akka actor based pubsub substrate that uses the redis service underneath.

Implementing the publisher interface is easy .. here is how you can bootstrap your own publishing service ..



The publish method just sends a Publish message to the Publisher. Publisher is an actor defined in Akka as follows:



The subscriber implementation is a bit more complex since you need to register your callback as to what you would like to do when you receive a specific type of message. This depends on your use case and it's your responsibility to provide a proper callback function downstream.

Here is a sample implementation for the subscriber. We need two methods to subscribe and unsubscribe from channels. Remember in Redis the subscriber cannot publish - hence our Sub cannot do a Pub.



I have not yet specified the implementation of the callback. How should it look like ?

The callback will be invoked when the subscriber receives a specific type of message. According to Redis specification, the types of messages which a subscriber can receive are:

a. subscribe
b. unsubscribe
c. message

Refer to the Redis documentation for details of these message formats. In our case, we model them as case classes as part of the core Redis client implementation ..



Our callback needs to take appropriate custom action on receipt of any of these types of messages. The following can be one such implementation .. It is customized for a specific application which treats various formats of messages and gives appropriate application dependent semantics ..



Note in the above implementation we specialize some of the messages to give additional semantics. e.g. if I receive a message as "+t", I will interpret it as subscribing to the channel "t". Similarly "exit" will unsubscribe me from all channels.

How to run this application ?

I will assume that you have the Akka master with you. Also you need to have a version of Redis running that implements pubsub. You can start the subscription service using the above implementation and then use any other Redis client to publish messages. Here's a sample recipe for a run ..

Prerequisite: Need Redis Server running (the version that supports pubsub)



For running this sample application :-

Starting the Subscription service



Starting a Publishing service



Another publishing client using redis-cli



Have fun with the message formats



The full implementation of the above is there as a sample project in Akka master. And in case you are not using Akka, I also have a version of the above implemented using Scala actors in the scala-redis distribution.

Have fun!

Thrush in Clojure

Quite some time back I wrote a blog post on the Thrush Combinator implementation in Scala. Just for those who missed reading it, Thrush is one of the permuting combinators from Raymond Smullyan's To Mock a Mockingbird. The book is a fascinating read where the author teaches combinatorial logic using examples of birds from an enchanted forest. In case you've not yet read it, please do yourself a favor and get it from your favorite book store.

A Thrush is defined by the following condition: Txy = yx. Thrush reverses the order of evaluation. In our context, it's not really an essential programming tool. But if you're someone who takes special care to make your code readable to the human interface, the technique sometimes comes in quite handy.

Recently I came across Thrush in Clojure. You don't have to implement anything - it's there for you in the Clojure library implemented as a macro ..

Conside this simple example of bank accounts where we represent an account as a Map in Clojure ..

(def acc-1 {:no 101 :name "debasish" :type 'savings :balance 100})
(def acc-2 {:no 102 :name "john p." :type 'checking :balance 200})


We have a list of accounts and we need to find all savings accounts and compute the sum of their current balances .. well not too difficult in Clojure ..

(defn savings-balance
  [& accounts]
  (apply +
    (map :balance
      (filter #(= (:type %) 'savings) 
        (seq accounts)))))


To a programmer familiar with the concepts of functional programming, it's quite clear what the above function does. Let's read it out for all of us .. From a list of accounts, filter the ones with type as savings, get their balances and report the sum of them. That was easy .. but did you notice that we read it inside out from the implementation, which btw is a 4 level nested function ?

Enter Thrush ..

Being a permuting combinator, Thrush enables us to position the functions outside in, in the exact sequence that the human mind interprets the problem. In our Scala version we had to implement something custom .. with Clojure, it comes with the standard library .. have a look ..

(defn savings-balance
  [& accounts]
  (->> (seq accounts)
       (filter #(= (:type %) 'savings))
       (map :balance)
       (apply +)))


->> is implemented as a macro in Clojure that does the following :

1. threads the first form (seq accounts) as the last argument of the second form (the filter), which makes (seq accounts) the last argument of filter
2. then makes the entire filter form, including the newly added argument the last argument of the map

.. and so on for all the forms present in the argument list. Effectively the resulting form that we see during runtime is our previous version using nested functions. The Thrush combinator only dresses it up nicely for the human eyes synchronizing the thought process with the visual implementation of the logic. And all this at no extra runtime cost! Macros FTW :)

->> has a related cousin ->, which is same as ->>, but only threads the forms as the second argument of the next form instead of the last. These macros implement Thrush in Clojure. Idiomatic Clojure code is concise and readable and using a proper ubiquitous language of the domain, makes a very good DSL. Think about using Thrush when you feel that reordering the forms will make your API look more intuitive to the user.

Thrush also helps you implement the Decorator pattern in a very cool and concise way. In my upcoming book, DSLs In Action I discuss these techniques in the context of designing DSLs in Clojure.