Why I made DSLs In Action polyglotic

Since I started writing DSLs In Action (buy here)^*, a lot of my friends and readers asked me about my decision to make the book polyglotic. Indeed right from the word go, I had decided to treat the topic of DSL based design without a significant bias towards any specific language. Even today after the book has been published, many readers come up to me and ask the same question. I thought I would clarify my points on this subject in this blog post.

A DSL is a vehicle to speak the language of the domain on top of an implementation of a clean domain model. Whatever be the implementation language of the DSL, you need to make it speak the ubiquitous language of the domain. And by language I mean the syntax and the semantics that the experts of the particular domain are habituated to use.

A DSL is a facade of linguistic abstraction on top of your domain's semantic model. As a DSL designer it's your responsibility to make it as expressive to your users as possible. It starts with the mapping of the problem domain to the solution domain artifacts, converging on a common set of vocabulary for all the stakeholders of the implementation and finally getting into the nuts and bolts of how to implement the model and the language.

It's a known fact that there's NO programming language that can express ALL forms of abstraction in the most expressive way. So as a language designer you always have the flexibility to choose the implementation language based on your solution domain model. You make this choice as a compromise of all the forces that come up in any software development project. You have the timeline, the resident expertise within your team and other social factors to consider before you converge to the final set of languages. In short there's always a choice of the language(s) that you make just like any other software development effort.

Being Idiomatic

Same problem domains can be modeled in solution domain using radically different forms of abstraction. It depends on the language that you use and the power of abstraction that it offers. The same set of domain rules may be implemented using the type system of a statically typed language. While you need to use the power of meta-programming to implement the same concepts idiomatically in a dynamically typed language. Even within dynamic languages, idioms vary a lot. Clojure or the Lisp family offers compile time meta-programming in the form of macros to offer your users the specific custom syntactic structures that they need for the domain. While Ruby and Groovy do the same with runtime meta-programming.

Here's an example code snippet from my book of an internal DSL being used to register a security trade and computes its net cash value using the principal amount and the associated tax/fee components. It's implemented in Ruby using all of the Ruby idioms.

str = <<END_OF_STRING
new_trade 'T-12435' for account 'acc-123' to buy 100 shares of 'IBM',
                    at UnitPrice = 100
END_OF_STRING

TradeDSL.trade str do |t|
  CashValueCalculator.new(t).with TaxFee, BrokerCommission do |cv|
    t.cash_value = cv.value 
    t.principal = cv.p
    t.tax = cv.t
    t.commission = cv.c
  end
end


The above DSL has a different geometry than what you would get with the same domain concepts implemented using Clojure .. Have a look at the following snippet of a similar use case implemented in Clojure and executed from the Clojure REPL:

user> (def request {:ref-no "r-123", :account "a-123", :instrument "i-123", 
         :unit-price 20, :quantity 100})
#'user/request

user> (trade request)
{:ref-no "r-123", :account "a-123", :instrument "i-123", :principal 2000, :tax-fees {}}

user> (with-tax-fee trade
        (with-values :tax 12)
        (with-values :commission 23))
#'user/trade

user> (trade request)
{:ref-no "r-123", :account "a-123", :instrument "i-123", :principal 2000, 
  :tax-fees {:commission 460, :tax 240}}

user> (with-tax-fee trade
        (with-values :vat 12))
#'user/trade

user> (trade request)
{:ref-no "r-123", :account "a-123", :instrument "i-123", :principal 2000, 
  :tax-fees {:vat 240, :commission 460, :tax 240}}

user> (net-value (trade request))
2940


The above DSL is implemented using syntactic macros of Clojure for custom syntax building and standard functional programming idioms that the language supports.

In summary, we need to learn multiple languages in order to implement domain models idiomatically in each of them. DSLs In Action discusses all these ideas and contains lots and lots of implementations of real world use cases using Java, Scala, Clojure, Ruby and Groovy.

Hope this clears my intent of a polyglotic treatment of the subject in the book.


^* Affiliate Link

Syntactic Sugars - What makes them sweet ?

Stephen Colebourne talks about implementing for-each in Java maps. He has proposed changes to be made to javac and queued up his request for approval by appropriate authorities. It is good to see Java community leads taking some serious steps towards syntactic sugars in the language. I am always for intention revealing syntactic sugars - after all .. Programs must be written for people to read, and only incidentally for machines to execute ?

Syntactic sugars, when properly designed, reduce the semantic distance between the problem domain and the solution domain. Syntactic sugars do not add new features or capabilities to an existing language. Still we value them mainly for social reasons - they can make your abstractions much more explicit, thereby making your intentions much more direct. And syntactic sugars often lead to concise and succinct code much pleasing to your eyes.

Java is not a language that boasts of concise syntax. Yet the smart for loop introduced in Java 5 reduces a lot of accidental complexity and makes the programmer intention much more explicit ..

for(String name : names) {
  // process name
}


is much more succinct than

for(Iterator<String> it = names.iterator(); it.hasNext(); ) {
  String name = it.next();
  // process name
}


judging from the fact that the latter snippet has its intentions buried down into verbosity of structures not directly related to the intention of the programmer.

names foreach println


is better, though not Java.

Many of the languages being used today offer lots of syntactic sugars abstracting rich capabilities of the underlying language. Take, for example, the Groovy Builder syntax, which exploits the mechanics of meta-programming, closures and named arguments to implement elegant, concise, intuitive APIs. Java developers use binding frameworks to manipulate XML and bind them to the model or the relational database schema. Not that there is anything wrong with it. But the developer has to go through all the hoops of mapping the object structure to the XML schema and use an external framework like JAXB to come up with a much longer version of the same solution than using Groovy MarkupBuilders.

Syntactic sugars are nothing new in the landscape of programming languages. It all started (possibly) with Lisp, offering macros as the means to design syntactic abstractions. To get a little sugar to the language offered syntax, you need not have to wait till the next official release. In Lisp, the syntax of the program is a direct representation of the AST, and with macros you can manipulate the parse tree directly. Languages like Lisp are known to offer syntax extensibility and allows developers to implement his own syntactic sugar.

Ruby offers runtime meta-programming, another technique to add your own syntactic sugars. Ruby does not have a macro system where you can play around with the abstract syntax tree, though we have had a ruby parser released by Ryan Davis that has been written entirely in Ruby. The standard meta object protocol offered by Ruby allows developer control over the language semantics (not the syntax) and has the capability to generate classes and methods dynamically at runtime. Meta-programming, method_missing, open classes, optional parentheses are some of the features that make Ruby a great language to build syntax abstractions for runtime processing.

A language built on the philosophy of bottom up programming offers extensible syntax (be it through the syntactic abstractions of Lisp or the semantic customizations of Ruby), on which syntactic sugars can be constructed by developers. Java believes in democratization of all syntax offered by the language, and it may take quite a few years to officialize the little sugar that you have been yarning for. Remember the explosion in the number of blog posts in celebration of the for-each loop when it came out with Java 5. In other languages, people build new syntax by the day and evolve new vocabulary within the same language that maps into the domain that they model. If you miss those features which you enjoyed in your earlier language, just build it over the new language. And it does not necessarily have to be the process of hooking onto the compilation cycle or plugging in customized modules into your language parsers.

Many of today's languages offer strong enough capabilities to build structures that look like syntax extensions. Scala is an example that makes the cut in this category. The advanced type system of Scala enables developers write control structures within the syntax of the language that looks like syntactic abstractions. Max likes deterministic finalization in C# and its idiomatic usage with "using" keyword. He has implemented the same syntax in Scala using closures, view bounds and implicit conversions. Besides eliminating lots of boilerplates, his extension looks charmingly useful for the domain he is using it.

Syntax extensibility is a necessity if you want your language to support evolution of DSLs. Extensible syntax scales much better than the framework based approach so popularized by Java. When you add a new syntactic structure in your language, it meshes so nicely with the rest of the language constructs that you never feel that it has been externally bolted on to you. Although in reality it is nothing more than syntactic sugars being presented in a form that makes more sense when coding for the particular problem in hand. When we talk about a language, we think in terms of parsers - this is no different when we think about DSLs. Implementing an external DSL is hard, considering the enormous complexity that parser generators make you go through. Scala offers monadic parser combinators where you can directly map your EBNF syntactic structures into implementations. And all this is done through syntactic sugars on top of closures and higher order functions that the language offers.

Higher Order Functions - The Secret Sauce ?

There has been lots of debates on whether object-oriented interfaces scale better than syntax extension capabilities in a language design. While OO certainly has its place in modularizing components and abstracting away relationships between them, there are situations when objects force us fit the round peg in a square hole. How many times have you cursed Java for forcing you define an unnecessary interface just to apply a function over a set of abstractions defining a specific set of contracts ? You can do the same in Scala using structural typing (aka anonymous types) and higher order functions. Higher order functions seem to be the secret sauce for offering syntax extensibility in programming languages.

Open Classes in Ruby - Too easy to be misused ?

I read this ..

Open classes makes invention possible in real time. No committee. No waiting for the next language revision. If you think of it, you can try it. Does Symbol#to_proc make it worthwhile? How about andand? Why don’t you tell me? I was serious when I asked you what you think of it.

and soon after .. this ..

I am all with Raganwald that "dangerous features enable bottom-up language evolution". My main concern, however, is that, I have seen too much of Ruby code where people tend to open up classes without even considering other design choices. On many of these occasions did I feel that mixins could have been a more pragmatic alternative. Maybe, one reason could be that, open classes in Ruby are too easy to implement. And developers give in to the temptation of using this "dangerous feature", without giving the adequate amount of thought that the design decision deserves. Scala's lexically scoped alternative is less sexy, but looks more pragmatic.

Why I like Scala's Lexically Scoped Open Classes

Ruby allows you to open up any class definition and plug in your methods into the guts of the existing class. John Carter opens up Integer and adds factorial ..

class Integer
  def factorial
    return 1 if self <= 1
    self * (self-1).factorial
  end
end


This allows him to write

puts 10.factorial


While this can lead to designing nice looking expressions of "pleasing symmetry", that read consistently from left-to-right (5.succ.factorial.succ.odd), it looks scary to me. Not the openness of the class, but the global context in which it is open.

Matz, replying to a thread, on the difference between ruby and python, mentions ..

"open class" is so strong (often too strong), we can break things easily. In other word, Ruby trust you to give you sharp knives, where Python don't. From the Python point of view, it's wrong, I guess.

In a way, I think the implementation of globally open class in Ruby brings up similar problems as the obliviousness property of aspects. You really never know what has been added to a class and whether the new implementation that you are plugging in, breaks apart any of the methods added by another developer. And it is only because all extensions in Ruby classes take place on the global namespace without any context whatsoever. Check out the discussion between Reginald Braithwaite and Avi Bryant in the comments section of the above Raganwald post.

Scalability problems in large projects ?

Scala has put in better thoughts in designing lexically scoped open classes. They call it "implicits" and allow extension of existing classes through a lexically scoped implicit conversion. Taking the example from Martin Odersky's blog, in order to add the append() method to Scala's built-in Array, define a class as ..

class RichArray[T](value: Array[T]) {
    def append(other: Array[T]): Array[T] = {
        val result = new Array[T](value.length + other.length)
        Array.copy(value, 0, result, 0, value.length)
        Array.copy(other, 0, result, value.length, other.length)
        result
    }
}


and add an implicit conversion from plain arrays to rich arrays:

implicit def enrichArray[T](xs: Array[T]) = new RichArray[T]


Now we can apply the append() method to plain arrays as well. It's not as organic as Ruby's open classes, but gives you a much better control towards evolution of APIs as your application codebase scales up. Let's see how ..

I love to program in languages that offer extensibility of abstractions - no, not the inheritance way .. public inheritance is possibly the second most tightest coupling between abstractions (guess what the first one is .. correct! .. the friend class in C++ ..). Scala offers extensibility even to classes written in Java, and with a rich repertoire of features. It almost obviates the necessity of plugging in your favorite Dependency Injection framework. And implicits offer features to add extensions to existing class structures.

Multidimensionally Open Classes ..

The core APIs that any class exposes abstract its prime responsibility, which are invariant across the entire lifecycle of the domain. This is the minimalist view of designing a class (when in doubt, leave it out !). Whether the large Array class of Ruby is the ideal approach of designing an abstraction, has been beaten to death. But definitely it is the result of a mindset that offers open classes - if you do not put assoc in Array, someone else will.

When we consider extensibility of abstractions, typically we want to extend an abstraction in multiple dimensions. A core domain object often needs to be extended with behaviors that help developers design smart APIs. But, more often than not, these behaviors make sense in a specific context of the application and may seem irrelevant or redundant in other contexts. In John Carter's earlier example, extending Integer with factorial may make sense when you are designing smart APIs for mathematical calculations. But why should an Integer class generally be bothered about computing it's own factorial ? This way, adding all possible math calculations in the global class object, can only lead to a big big bloat violating all good principles of OO design. The extension has to be in a specific context of the class and should be invisible to any other context, for which it seems like a noise.

Have a look at how Scala handles this wreck effect through a simple example ..

I have a class Person, which is a domain object in the core package ..

package org.dg.biz.core;

class Person(val lastName: String,val firstName: String,val age: Int) {
    //..
    //.. details

    override def toString(): String = {
        lastName + " " + firstName + " " + age
    }
}


While working with my UI classes, I would like to have an API person.toLabel() which will display a JLabel Swing component for rendering a person's details on a frame. And while working on the messaging part of the application, I would like to have a person.toXML(), which will help me generate an XML message out of the Person object. But obviously I would not like to have any javax.swing imports in my messaging component.

I would like to extend the same core abstraction (Person), but along two mutually orthogonal dimensions in a share nothing mode. The messaging component should not be able to invoke person.toLabel(), even though it is the same core abstraction that it is extending. The extensions have to honor the context in which they are being used. Then only can we ensure proper separation of concerns and the right modularity in designing application component boundaries.

In Ruby, class objects are global variables, and indiscriminate extension methods plugged into existing classes can lead to reduced maintenability and long term reliability of class structures. Not with Scala *implicits* though ..

Within the ui package, I define an enhancer class RichPerson, which provides all extensions to my Person class *only* for the UI context ..

package org.dg.biz.ui;

import org.dg.biz.core._
import javax.swing._

class RichPerson(person: Person) {
    def toLabel(): JLabel = {
        new JLabel(person.toString)
    }
    //.. other UI context extensions
}


and define a mixin that provides me the implicit conversion ..

package org.dg.biz.ui;

import org.dg.biz.core._

trait UIFramework {
    implicit def enrichPerson(person: Person): RichPerson = {
        new RichPerson(person)
    }
}


When I write my UI components, the extension designed for the UI context kicks in and serves me the UI-only view of the extended core abstraction. I can use the nice person.xxx() syntax on the core abstraction itself, the compiler does the magic underneath through invocation of the conversion function ..

package org.dg.biz.ui;

import org.dg.biz.core._
import java.awt.event.WindowAdapter
import javax.swing._

object Main extends UIFramework {

    def main(args: Array[String]) = {
        val p = new Person("ghosh", "debasish", 35)
        val frame = new JFrame()
        frame.addWindowListener(new WindowAdapter(){})
        frame.getContentPane().add(p.toLabel) // nice person.toLabel() usage
        frame.pack()
        frame.setVisible(true)
    }
}


Trying to use person.toXML() will result in a syntax error ! In fact the IDE can only display the available extensions as part of auto-completion features.

Similarly for the messaging component, I define the following extensions in the messaging package ..

package org.dg.biz.msg;

import org.dg.biz.core._

class RichPerson(person: Person) {
    def toXML(): scala.xml.Elem = {
        <person>
            <name>
                <lastname>{person.lastName}</lastname>
                <firstname>{person.firstName}</firstname>
            </name>
            <age>{person.age}</age>
        </person>
    }
}


and the mixin ..

package org.dg.biz.msg;

import org.dg.biz.core._;

trait MessageFramework {

    implicit def enrichPerson(person: Person): RichPerson = {
        new RichPerson(person)
    }
}


and my messaging application code ..

package org.dg.biz.msg;

import org.dg.biz.core._

object Main extends MessageFramework {

    def save(person: scala.xml.Elem) = {
        scala.xml.XML.saveFull("person.xml", person, "UTF8",
                true, null)
    }

    def main(args: Array[String]) = {
        val p = new Person("ghosh", "debasish", 41)
        save(p.toXML)
    }
}


and with Controlled Visibility ..

We extend our Person class in different dimensions for two orthogonal concerns and both of the extensions are mutually exclusive of each other. We cannot access the UI extension from within the messaging part of the application and vice versa. The visibility of the *implicit* conversion functions, along with Scala mixin techniques, ensure that the UI component *only* gets access to the RichPerson class meant for UI extension. Hence the global namespace does not get polluted and the class structure does not get bloated. Yet we have the power of open classes.

Language Explorations on the JVM - An Application Developer's perspective

Sometime ago I had reported on our first experience of using Rhino scripting in a Java EE application for a large client. It was exactly what Ola Bini suggests in his post on language explorations. Some of the modules of the application needed the dynamism, were required to be hot swappable and customizable by the domain users. And the compilation cycle was getting in the way in trying to meet up these requirements through the standard server side language. We went for Rhino scripting for all such controllers using the new scripting engine available for executing Javascript within the JVM.

Since that application has been successfully deployed, we have been fiddling around with some more options towards polyglotism. This post is a brief summary of some of the languages / language bridges we explored in the process. All of what we did so far has been on the JVM as the underlying Polyglot platform - we have not yet explored anything on the .NET world.

Web controllers are areas which may need lots of dynamic nature, since they deal with user interactions, page flows, stateful storage across requests and many other control flow structures for realizing one complex use case. Spring Web Flow provides one viable option for modeling this. Another option from the scripting world is Rhino in Spring, which integrates Mozilla Rhino JavaScript interpreter with Spring Framework. The value add is to offer to the user the flexibility of a dynamic language to model the dynamic parts of the application on the Java platform, while integrating with the dependency injection principles of the Spring framework. Spring also offers nice support of plugging in managed script based controllers in multiple languages - this will surely provide more options towards evolution of polyglot programming in today's applications.

Another area where we explored the possible usage of an expressive language is the configuration of an application. Applications today mostly use XML based configurations, which feels too noisy for human consumption. SISC offers a lightweight Scheme scripting engine atop the JVM and comes bundled with a small footprint of around 230 KB. I had blogged before on using Scheme as an executable XML :

In SISC bridging is accomplished by a Java API for executing Scheme code and evaluating Scheme expressions, and a module that provides Scheme-level access to Java objects and implementation of Java interfaces in Scheme.

Talking about what Ola Bini calls the "stable layer", I fully agree that static type safety helps here, since the entire application infrastructure will be built upon this layer. Till today Java is my #1 choice as the language and Spring is my only choice as the framework for this layer. I have talked on this a number of times before, but I guess it is worth repeating that I love the non-intrusiveness of Spring as far as declarative programming on the JVM is concerned. As it stands now, I will not forego Spring if I am developing on the JVM platform.

It will be really interesting to see how Scala shapes up its future as a potential candidate for this layer. Scala is a feature rich language with an advanced type system, nice syntax, less verbosity and more elegance than Java. Where Scala lacks are tooling, documentation and industry patronage, all of which can improve with more and more users joining the community.

In the domain layer, most applications rely on pure Java to model business rules. As Ola has mentioned, this layer is a strong candidate for DSL based implementation. Irrespective of what language(s) you use to implement your DSL, the business application rules should always be based on the DSL only. My feeling is that in today's scenario, Java is not really an ideal language to design a DSL. Hence we tend to find almost all applications implementing the domain layer at lower levels of abstraction. This makes the domain layer of today more verbose and less maintainable.

Powerful and expressive languages with conciseness of syntax are better fit for designing DSLs. While JRuby and Scala make suitable candidates for designing DSLs for the domain layer, I think the static typing of Scala makes it a better fit here. I may be biased, but when I am thinking of reusable API design to be used by big teams, somehow static typing (possibly done better than Java) makes me more comfortable. However, considering the state of enterprise software development today, there is a significant entry barrier for average programmers to both Scala and JRuby. Idiomatic Scala or Ruby is primarily based on functional paradigms, something which is still not intuitive to a Java programmer today. With most of today's new generation languages embracing FP, this may be the single most deciding factor that will determine the amount of acceptability that polyglot programming will find within the behemoth called enterprise software development. But there is no doubt that a well designed DSL using languages like Scala or JRuby will find tomorrow's domain model at a much higher level of abstraction than what it is today.

Party time Over for Rails .. It's time to Deliver !

People have increasingly started talking about Erlang and putting the concurrency-idol on the hot seat for an upcoming session of dissecting its anatomy. Possibly the fan fare of Rails has started taking a backseat. Is this a sign of maturity for Ruby on Rails ? Or a case of disillusionment ?

As Brian points out :

In terms of the Gartner Hype Cycle, Ruby and Rails are nearing the end of the "Peak of Inflated Expectations" and are moving into the "Trough of Disillusionment".

It's indeed true that Rails have enjoyed the limelight of being the darling of the community and a promise (or panacea) to put an end to the verbosity and *inelegance* of Java based Web development. It has been quite some time though, yet we hear people like Obie utter the following in the context of choosing Java over Rails in developing Big Enterprise Applications :

Like Gavin pointed out, you can't do an app with thousands of model classes in Rails. Okay, but I'm really not trying to say that you should try. Right now, I personally have no interest in working on those kinds of monstrosities, but to each his own. If that big enterprise application exposed web interfaces, I might be inclined to use JRuby on Rails to do those..

It's time to deliver for Rails ! How long will Rails shrug away from the enterprise applications branding them as monstrous ? It's time to take Rails out of the playful scaffolding model that delivers basic CRUD applications at lightening speed. To deliver complex enterprise applications, you need a performant application server with production quality deployment support, a real fast virtual machine and solid domain modeling capabilities. We all know the ActiveRecord model is never going to scale up for large complex domain models. The Rails community, till now, do not acknowledge how important it is to have a separate domain model, decoupled (or loosely coupled) with the persistence layer. And ActiveRecord will never take you towards that end. How much strong metaprogramming capabilities you may use to offer fancy dynamic finders in Rails, ultimately you need to have a strong decoupled domain model to deliver big enterprise applications. Till then, I guess, JRuby will be your best bet and Rails will remain just another Java library sucked up within a Java application server.

Why Thinking in Ruby Matters

I was going through a great article by Jim Weirich where he discusses why it is important to think in Ruby, when you are programming in Ruby. He goes on to describe an implementation of the Adapter design pattern in Ruby and compares the implementation with a similar (but much scaled down) version in Java. Ruby gives you the power to express more in less words - conciseness of syntax, along with great metaprogramming abilities leads to the expressivity that Ruby offers.

Consider the Strategy design pattern, which allows encapsulation of an algorithm and allows users to vary them independent of the context. There is nothing object-oriented about a Strategy design pattern. It so happens that the GOF book describes it in the context of an object-oriented language, C++. Stated succinctly, the design pattern espouses yet another best practice of software design, separation of concerns, by decoupling the implementation of an algorithm from the context.

In Java, the Strategy design pattern is implemented through the composition of a polymorphic hierarchy. The strategy is composed into the context through an interface, which can support multiple implementations transparently. Here is an example from a Payroll application, where the algorithm for calculation of a salary is encapsulated into a strategy :

class SalaryComputation {
  // injected through dependency injection
  private SalaryComputationStrategy strategy;
  //..
  //..

  public BigDecimal compute(..) {
    //..
    //..
    return strategy.compute(..);
  }
  //..
}

Here SalaryComputationStrategy is an interface which can have multiple implementations. Typically in a Java project, the implementations are injected transparently through a DI container like Spring or Guice.

interface SalaryComputationStrategy {
  BigDecimal compute(..);
}

class DefaultSalaryComputationStrategy
  implements SalaryComputationStrategy {
    //.. implementation
}

class SpecialSalaryComputationStrategy
  implements SalaryComputationStrategy {
    //.. implementation
}

One of the immediate consequences of the above implementation in Java is the number of classes and the inheritance hierarchies involved in the implementation. Looks like an accidental complexity in implementation, but this is quite a normal way of dealing with a nail when all you have is a hammer.

How do you encapsulate an algorithm and ensure flexible swapping of implementations in a language like Ruby that offers powerful syntax extensibility ? Remember, Ruby is also object oriented, and the above Java implementation can be translated with almost no effort using equivalent Ruby syntaxes. And we have a Strategy implementation in Ruby. Can we do better ?

Abstraction Abstraction!

Abstraction is the key - when you program in a language, always choose the best form of abstraction that suits the problem you are modeling. In a strategy design pattern, all you are encapsulating is an algorithm, which is, by nature a functional artifact. The very fact that Java or C++ does not support higher order functions (anonymous inner classes and function pointers are for the destitutes) had forced us to use encapsulating objects as holders of algorithms. And that led to the plethora of class hierarchies in the Java implementation. Ruby supports higher order functions in the form of blocks and coroutine support in the form of yields - keep reading and hope for a better level of abstraction support.

class SalaryComputation
  def compute
    ## fetch basic, allowances
    @basic = ..
    @allowance = ..

    ## fixed logic goes here

    ## coroutine call for the specific algorithm
    yield(@basic, @allowance)
  end
  ##
end

and the algorithm can be injected inline by the client through a Ruby block :

SalaryComputation.new.compute {
  |b, a|
  tax = (b + a) * 0.2
  b + a - tax
}

and we do not have to define a single extra class for the strategy. The strategy is nicely embedded inline at the caller's site. Another client asking to use a different algorithm can supply a different block at her call site.

What if I want to use the same default algorithm for different clients, yet keeping the flexibility of plugging in multiple implementations ? DRY it up within a Ruby Module and use the power of Ruby mixins :

module DefaultStrategy
  def do_compute(basic, allowances)
    tax = basic * 0.3
    basic + allowances - tax
  end
end

class SalaryComputation
  include DefaultStrategy ## mixin

  def initialize basic
    @basic = basic
    @allowances = basic * 0.5
  end

  def compute
    do_compute(@basic, @hra)
  end
end

And for some places where I would like to inject a special strategy, define another Module for the same and extend the SalaryComputation class during runtime :

module SpecialStrategy
  def do_compute(basic, allowances)
    tax = basic * 0.2
    basic + allowances - tax
  end
end

s = SalaryComputation.new(100)
s.extend SpecialStrategy
puts s.compute

Handling Fine Grained Variations

Suppose a part of the algorithm is fixed, while the computation of the tax only varies. The typical way to handle this in Java will be through the Template Method pattern. When designing the same in Ruby, the mechanism melds nicely into the language syntax :

def compute
  @basic + @allowances - yield(@basic)
end

and the identity of the pattern disappears within the language. Ruby supports fine grained variations within algorithms through powerful language syntax, which can only be done using extra classes in Java.

And finally ..

Keep an eye on the Ruby open classes feature. You can rip open any class and make changes either at the class level or at a single instance level. This feature offers the most coarse grained way to implement strategies in Ruby.

## open up an existing class
class SalaryComputation

  ## alias for old method
  ## you may need it also
  alias :compute_old :compute

  def compute
    ## new strategy
  end
end

So, how many ways does Ruby offer to implement your Strategy Design Pattern ?

Design Tip: Localize your Object Creation Logic

Always try to localize the logic of object creation. This has been one of the big lessons that the Factory design pattern teaches us. This is also a teaching of the Separation of Concerns, which every well-designed software should honor. Don't let your object creation logic get spitted into the processing code. Always program by the interface returned from the factory contract - this way the processing logic remains independent of individual concrete subclasses. All this we knew from the GOF patterns, right ?

Factory with a Strategy

Another benefit of localizing your object creation code is to have the flexibility of plugging in various creation strategies transparently without impacting your clients. In a real life Java application, we were having some performance problems, which could be traced down to creation of a large number of some specific objects in memory. While the objects were not too heavy, the sheer numbers were bringing down the performance curve. Luckily we had the creation logic behind a factory abstraction in Java.

public class BorderObjectFactory {

  //..
  public BorderObject createBorderObject(int type,
    double thickness, Color color, BorderCategory category) {
    //..
  }
  //..
}

And all clients were using the factory for creating BorderObject instances. After a careful review of the creation pattern for BorderObjects, we concluded that we need to implement instance sharing through the Flyweight pattern. This will imply a change in the creation logic, which would have impacted all clients had they not been behind the factory firewall. Here goes the flyweight ..

public class BorderObjectFactory {

  //.. needs to be a singleton

  // pool of flyweights
  private Map<String, BorderObject> pool =
    new HashMap<String, BorderObject>();

  // don't instantiate me directly
  private BorderObjectFactory() {}

  public BorderObject createBorderObject(int type,
    double thickness, Color color, BorderCategory category) {

    //..
    // make the hash key
    String key = new StringBuilder()
        .append(type)
        .append(thickness)
        .append(color.toString())
        .append(category.toString()).toString();

    BorderObject bo = pool.get(key);
    // if it finds from pool, return it
    if (bo != null) {
      return bo;
    }

    // first time entry so create
    bo = new BorderObject(type, thickness, color, category);
    // cache for later use
    pool.put(key, bo);
    return bo;
  }
}

In Ruby, Factories are smell

Ruby has open classes - every class is an object, which can be changed during runtime. If you want to change the creation logic of an object, just open it up and plug in the new logic at runtime. Here is the similar looking Border class in Ruby ..

class Border

  attr_reader :width, :height

  def initialize(wid, ht)
    @width, @height = wid, ht
  end
  # other logic

end

In case you want to change the creation logic through an implementation of the Flyweight based pooling, just define a separate Flyweight module :

module Flyweight
  def Flyweight.included(klass)
    klass.instance_eval %{
      @_pool = {}

      alias :orig_new :new

      def new(*key)
        p "invoking new on " + self.to_s + " args = " + key.to_s
        (@_pool ||= {})[key] ||= orig_new(*key)
      end
    }
  end
end

and mix it in the Border class ..

class Border
  include Flyweight

  # rest similar
  #

end

and clients enjoy the benefit of sharing instances transparently without any change of code. And the best part is they continue to use the same Border.new(..) call to create new Border objects.

b1 = Border.new(2, 10)
b2 = Border.new(4, 20)
b3 = Border.new(4, 20) ## instance sharing with b2
b4 = Border.new(2, 10) ## instance sharing with b1

Depending on the language in which you are designing, the factory implementation may vary in shape and size. In Ruby you don't need to have separate factory abstractions, the factory design pattern is melded into the power of the language. But keeping the object creation logic localized always gives you the extra yard with respect to your design scalability.

Rails Applications and Law of Demeter

Jay Fields talks about Law of Demeter violations in Rails programs and suggests a fix using the Ruby module Forwardable, which allows you implement delegation without too much littering in your codebase. Along the same line Jeffrey Allan Hardy introduces the class method delegate in Ruby on Rails to automate delegation tasks. Both of these techniques allow syntax extensibility to keep your code clean from the manual delegate methods.

My question is .. does this really fix the violation of the Law of Demeter in Rails applications ?

The basic philosophy behind the concepts of Adaptive Programming or the Law of Demeter is to provide a better separation between the behavior and object structure in OO programs. A typical Rails application is tied strongly to the underlying ActiveRecord model. The domain model for a Rails application follows the same structure as the database model - hence the navigation of the object graph is a replica of the underlying relational table structure. To put it bluntly, there is no domain object model, which abstracts the behavior of the system. The constructs like Forwardable or class method delegate offer syntactic sugars that may help in code refactoring, but do not add to the reusability of the model. In fact the principle of least knowledge that LoD preaches is violated the moment you make your behavioral model knowledgable about the underlying persistence structure.

I am not an expert in Ruby or Rails - I would like to know what others feel about this ..

Thinking Differently with Design Patterns, Java and Accidental Complexity

Bob Lee has posted a very useful tip on performant singletons. Have a look if you are writing concurrent Java applications for the enterprise. From plain old synchronization to DCL to IODH - I guess this pattern implementation has come a full circle in Java. If you are into Java, follow this idiom .. possibly this is the best you can get for a fast, thread-safe, lazily loaded singleton with JLS guarantee.

This post is not about singletons, although we start with the code for implementing the same pattern based on post Java 5 specifications :

public class Singleton {
  static class SingletonHolder {
    static Singleton instance = new Singleton();
  }

  public static Singleton getInstance() {
    return SingletonHolder.instance;
  }
}

Everytime you need a singleton in your application, make use of the above idiom. Unless you are coding a trivial application, very soon you will feel the spiralling cost of the growing number of classes. This is a basic problem of many of the Java idioms and design patterns when we try to force functional programming paradigms through nested classes or anonymous inner classes. This is, what many refer to as accidental complexity in modeling, which, very often, tends to overshadow the domain complexity, thereby resulting in lots of glue codes.

I am not in the league to snub Java. Myself, I am a Java programmer and have been doing OO with Java and C++ for the last 10 years. The GOF design patterns book has been my bible and all my thoughts have, so far, been soaked in the practices and principles that the book professes. With Ruby and Lisp, I have started to think about programming a bit differently. And as Alan Perlis has epigrammed, "A language that doesn’t affect the way you think about programming, is not worth knowing".

Singleton Pattern Elsewhere

require 'singleton'
class Foo
  include Singleton
end

That's it ! Ruby's powerful mixin functionality automatically makes our class a singleton - the new method is rendered private and we get an instance method for getting the object.

Another new generation language Scala offers the object keyword for implementing singletons.

object SensorReader extends SubjectObserver {
  // ..
}

In this example the declaration for SensorReader creates a singleton class that can have a single instance.

Design Patterns in Java and Accidental Complexity

As a language, Java does not offer powerful functional abstractions that Ruby, Scala or Lisp provides. Hence many design patterns which look invisible in these languages stand out as elaborate design constructs in Java. These add to the *accidental complexity* in a Java application codebase and often turns out more difficult to manage than the *actual complexity*, which is the complexity of the domain that you are trying to model. Technologies like aspects and metadata based annotations are attempts to improve the abstraction level of the Java programming language. Unfortunately these can never give the programmer that seamlessness in extending the syntax of the core language. The programmer will never be able to program bottom up or carve out a DSL as elegant as Rails using Java. Norvig has an excellent presentation on how dynamic languages make many of the GOF patterns invisible within them. The presentation illustrates how macros can make the implementation of Interpreter design pattern easier, method combinations can make Observers seamless and multi-methods can ease the implementation of Builder design pattern. Mark Dominus has posted a very thought provoking essay that concludes that patterns are signs of weakness in programming languages. What he means is that, languages where we need to write repetitive code to implement solutions to recurring problems lack in the abstraction power. The very fact that we have to repeat the code for implementing the Strategy design pattern for every instance of application of the pattern in Java, implies that the language lacks the extensibility to imbibe the design construct within itself. And by doing so, the implementation inherits lots of *accidental complexity* or yellow marker as part of the codebase. OTOH, in a typical functional implementation, the strategy is a simple variable whose value is a function, and with first class functions, the pattern is invisible.

The singleton pattern implementation in Java, despite providing a performant solution, also contributes to this accidental complexity of the codebase. This is more true for many other pattern implementations in Java or C++.

Thinking Differently

I cannot imagine myself writing about lack of abstractions in OO languages had I not been exposed to Ruby, Scala or Lisp. I realize the truth in the 19th epigram of Alan Perlis - these languages have really affected my thinking on programming at large. Now I can appreciate why Steve Yeggey thinks design patterns [in Java] as mostly pounding star shaped pegs into square holes.

Syntax Extensibility, Ruby Metaprogramming and Lisp Macros

Over the last few days I have been feeling a bit Lispy. It's not that I have been immersed in Lisp programming, I still do Java for my day job and enjoy the process of staring at reams of standard object oriented api calls and the big gigantic frameworks that provide the glue code for the enterprise software. Java is still my favorite programming language, I still enjoy writing Java and have been recently working on bigger commitments to write more Java with more Spring and more Hibernate.

The only difference is that I have started reading Paul Graham's On Lisp again !

I am convinced that I will not be programming production level business applications in Lisp in the near foreseeable future. But reading Lisp makes me think differently, the moment I start writing event listeners in Java Swing, I start missing true lexical closures, I look forward to higher level functions in the language. Boilerplates irritate me much more and make me imagine how I could have modeled it better using Scheme macros. True, I have been using the best IDE and leave it to its code generation engine to generate all boilerplates, I have also put forth a bit of an MDA within my development environment that generates much of the codes from the model. I am a big fan of AOP and have been using aspects for quite some time to modularize my designs and generate write-behind logic through the magic of weaving bytecodes.

The difference, once again, is that, I have been exposed to the best code generator of all times, the one with simple uniform syntax having access to the whole language parser, that gets the piece of source code in a single uniform data structure and knows how to munch out the desired transformation in a fail-safe manner day in and day out - the Lisp macro.

Abstractions - Object Orientation versus Syntax Construction

For someone obsessed with OO paradigm, thriving on the backbones of objects, virtual functions and polymorphism, I have learnt to model abstractions in terms of objects and classes (the kingdom of nouns). I define classes on top of the Java language infrastructure, add data members as attributes, add behavior to the abstractions through methods defined within the classes that operate on the attributes and whenever need be, I invoke the methods on an instantiated class object. This is the way I have, so far, learnt to add abstraction to an application layer. Abstraction, as they say, is an artifact of the solution domain, which should ultimately bring you closer to the problem domain. We have :

Machine Language -> High Level language -> Abstractions in the Solution Domain -> Problem Domain

In case of object oriented languages like Java, the size of the language is monstrous, add to that at least a couple of gigantic frameworks, and abstractions are clear guests on top of the language layer. Lisp, in its original incarnation, was conceived as a language with very little syntax. It was designed as a programmable programming language, and developing abstractions in Lisp, not only enriches the third block above, but a significant part of the second block as well. I now get what Paul Graham has been talking about programming-bottom-up, the extensible language, build-the-language-up-toward-your-program.

Take this example :

I want to implement dolist(), which effects an operation on each member of a list. With a Lisp implementation, we can have a natural extension of the language through a macro

dolist (x '(1 2 3)) (print x) (if (evenp x) (return)))

and the moment we define the macro, it blends into the language syntax like a charm. This is abstraction through syntax construction.

And, the Java counterpart will be something like :

// ..
Collection<..> list = ... ;
CollectionUtils.dolist(list,
    new Predicate() {
      public boolean evaluate() {
        // ..
      }
    });
// ..


which provides an object oriented abstraction of the same functionality. This solution provides the necessary abstraction, but is definitely not as seamless an extension of the language as its Lisp counterpart.

Extending Extensibility with Metaprogramming

Metaprogramming is the art of writing programs which write programs. Languages which offer syntax extensibility provide the normal paths to metaprogramming. And Java is a complete zero in this regard. C offers more trouble to programmers through its whacky macros, while C++'s template metaprogramming facilities are no less hazardous than pure black magic.

Ruby offers excellent metaprogramming facilities through its eval() family of methods, the here-docs, open classes, blocks and procedures. Ruby is a language with very clean syntax, having the natural elegance of Lisp and extremely powerful metaprogramming facilities. Ruby metaprogramming capabilities have given a new dimension to the concept of api design in applications. Have a look at this example from a sample Rails application :

class Product < ActiveRecord::Base
  validates_presence_of :title, :description, :image_url
  validates_format_of :image_url,
    :with => %r{^http:.+\.(gif|jpg|png)$}i,
    :message => "must be a URL for a GIF, JPG, or PNG image"
end

class LineItem < ActiveRecord::Base
  belongs_to :product
end


It's really cool DSL made possible through syntax extension capabilities offered by Ruby. It's not much of OO that Rails exploits to offer great api s, instead it's the ability of Ruby to define new syntactic constructs through first class symbols that add to the joy of programming.

How will the above LineItem definition look in Lisp's database bindings ? Let's take this hypothetical model :

(defmodel <line_item> ()
(belongs_to <product>))


The difference with the above Rails definition is the use of macros in the Lisp version as opposed to class functions in Rails. In the Rails definition, belongs_to is a class function, which when called defines a bunch of member functions in the class LineItem. Note that this is a commonly used idiom in Ruby metaprogramming where we can define methods in the derived class right from the base class. But the main point here is that in the Lisp version, the macros are replaced in the macro expansion phase before the program runs and hence provides an obvious improvement in performance compared to its Rails counterpart.

Another great Lispy plus ..

Have a look at the following metaprogramming snippet in Ruby, incarnated using class_eval for generating the accessors in a sample bean class :

def self.property(*properties)
  properties.each do |prop|
    class_eval <<-EOS
      def #{prop} ()
        @#{prop}
      end
      def #{prop}= (val)
        @#{prop} = val
      end
    EOS
  end
end


Here the code which the metaprogram generates is embedded within Ruby here-docs as a string - eval ing on a string is not the recommended best practice in the Ruby world. These stringy codes are not treated as first class citizens, in the sense that IDEs do not respect them as code snippets and neither do the debuggers. This has been described in his usual style and detail by Steve Yeggey in this phenomenal blog post. Using define_method will make it IDE friendlier, but at the expense of readability and speed. The whacky class_eval runs much faster than the define_method version. A rough benchmark indicated that the class_eval version ran twice as fast on Ruby 1.8.5 than the one using define_method.

def self.property(*properties)
  properties.each do |prop|
    define_method(prop) {
      instance_variable_get("@#{prop}")
    }

    define_method("#{prop}=") do |value|
      instance_variable_set("@#{prop}", value)
    end
  end
end


Anyway, all these are examples of dynamic metaprogramming in Ruby since everything gets done at runtime. This is a big difference with Lisp, where the code templates are not typeless strings - they are treated as valid Lisp data structures, which the macro processor can process like normal Lisp code, since macros, in Lisp operates on the parse tree of the program. Thus code templates in Lisp are IDE friendly, debugger friendly and real first class code snippets. Many people have expressed their wish to have Lisp macros in Ruby - Ola Bini has some proposals on that as well. Whatever little I have been through Lisp, Lisp macros are really cool and a definite step forward towards providing succinct extensibility to the language through user defined syntactic control structures.

OO Abstractions or Syntax Extensions ?

Coming from an OO soaked background, I can only think in terms of OO abstractions. Ruby is, possibly the first language that has pointed me to situations when syntax extensions scale better than OO abstractions - Rails is a live killer example of this paradigm. And finally when I tried to explore the roots, the Lisp macros have really floored me with their succinctness and power. I do not have the courage to say that functional abstractions of Lisp and Ruby are more powerful than OO abstractions. Steve Yeggey has put it so subtly the natural inhibition of OO programmers towards extended syntactic constructs :

Lots of programmers, maybe even most of them, are so irrationally afraid of new syntax that they'd rather leaf through hundreds of pages of similar-looking object-oriented calls than accept one new syntactic construct.

My personal take will be to exploit all features the language has to offer. With a language like Ruby or Scala or Lisp, syntax extensibility is the natural model. While Java offers powerful OO abstractions - look at the natural difference of paradigms in modeling a Ruby on Rails application and a Spring-Hibernate application. This is one of the great eye-openers that the new dynamic languages have brought to the forefront of OO programmers - beautiful abstractions are no longer a monopoly of OO languages. Lisp tried to force this realization long back, but possibly the world was not ready for it.