The Art of Groovy Command Expressions in DSLs

Domain Specific Languages (DSLs) are often littered with the accidental complexity of the host language. Have you ever seen a supposedly “friendly” language expression like “ride(minutes(10)).on(bus).towards(Basel)”. The newest version of Groovy contains a language feature that aims to eliminate the noise of all those extra periods and parenthesis, so that your DSL looks more like “ride 10.minutes on bus towards Basel”. This article shows you step-by-step how to use Groovy Command Expressions and plain old metaprogramming to write just this DSL, and also offers advice on when, and when not, to use this new language feature.

The full article is available over on the Canoo Blog: http://www.canoo.com/blog/2011/12/08/the-art-of-groovy-command-expressions-in-dsls/

And of course you can upvote in all the usual places.

Until next time...

Gilad Bracha and the Java Post Mortem

At Jax.de this week I got to see two sessions with Gilad Bracha and sit on a panel with him (never would have imagined that).

I wrote my thoughts about the day up on the Canoo blog in a piece called "Gilad Bracha and the Java Post Mortem"

Upvotes always appreciated!

How to Tell Your Co-Workers Have Discovered Functional Programming

Three ways to tell that your co-workers are learning a Functional Programming language at home:

1. You've found the number of .class file in your project quadrupled in the last month.

Here is an output directory with 17 .class files, only 2 of which are not inner classes, most of which are anonymous.



See all those dollar signs indicating inner/anonymous classes? Uh oh, looks like someone discovered shitty lambdas. Your chaos meter should be increasing... will your project be maintainable of it's not idiomatic Java?

2. You've found a class named Pair

public class Pair<T, U> {

   private final T left;
   private final U right;

   public Pair(T left, U right) {
       this.left = left;
       this.right = right;
   }

   T getLeft() { return left; }
   U getRight() { return right; }

   static <T, U> Pair from(T left, U right) {
       return new Pair<T, U>(left, right);
   }
}

It's not a tuple, but it's about 2/3 of the way there! There's already jfun.util.Pair in JFun and gnu.lists.Pair in Gnu Lists, and now you've got your own too. Hooray. But wait, how will you maintain your code base if the declared name of your data structures don't tell you how it's meant to be used? Chaos... level... rising...

3. You've found a widely implemented one-method highly generic interface

An interface like this has over 100 implementations, many of which are anonymous:

public interface Transformer<T, U> {
   U transform(T input);
}

This is equivalent to F in Functional Java and Transformer5 in JConch.

At this point your chaos meter should be off the chart. These functional programmers are surely ruining your system and livelihood. So, I'm hesitant to mention what happens when you combine all of them together:

Transformer identityfunction =
   new Transformer<Pair<String, String>, Pair<String, String>>() {
       public Pair<String, String> transform(Pair<String, String> input>) {
           return input;
       }
   }
}

Voila, an anonymous function object that uses the Pair class to make a 1-arity method into a 2-arity method, returning multiple values.

These are the trials and tribulations of finding the appropriate use of new programming paradigms in old languages. Looking at version control history, the person that committed all these sins was... me. Will this be the Java code I write in two years time? Certainly not. But which of these example is the growing pains of learning FP and which of these will still be with me in the future?

Comments are open!

The End of Intuitive

We all want our interfaces to be intuitive, don't we? Sounds good to me. I know I've used this word in the past when passing judgment on a user interface: "I'm confused by the interface... it's just not intuitive."

To intuitively understand something means it is understood without prior exposure to the subject, without requiring a learning process, and without using rational thought. Running away from a bear in the woods: intuition, fight or flight. Drinking water from a lake: not intuition, prior experience with lakes teaches you to expect water. Shaking a coconut, realizing there is milk inside, smashing it and drinking it: not intuition, you inferred liquid from rationally thinking about the sound it made. Operating any user interface that requires a keyboard or mouse: not intuitive by any stretch of the definition. Somewhere, someone taught you how these things worked. So sorry, your cool new interface that passed usability testing with flying colors is not intuitive.

Your user interface might be familiar, and this is good because it reduces the learning curve of users who have already been trained in something similar.

Or your user interface might be easily habituated, and this is good because repeated use will increase the user's productivity and operation time.

But ease of use and speed of learning are certainly not the same thing as intuitiveness. As Jef Raskin points out in "The Humane Interface", "The mouse is very easy to learn... [It's] fast and easy, but it is neither intuitive or natural. No artifact is."

What you can do is make an interface self-teaching, meaning the user can always find proper explanations and instructions when they are needed.

So from here on, I pledge to stop saying "intuitive" when it comes to interfaces. Instead I'll stick with "familiar", "self-teaching", and "easily habituated".

AND WHILE WE'RE ON THE TOPIC... I often hear people say their co-workers are idiots. The word idiot traditionally, specifically describes people with an IQ of 0-25. It is one step below imbecile (26-50), and two below moron (51-70). Here is a handy table for comparison:

Term	IQ Range
Moron	51-70
Imbecile	26-50
Idiot	0-25

True idiots are as rare as the true genius. While it might be possible, however unlikely, that you work with a bunch of morons, it is almost unthinkably improbable that you work with a bunch of idiots. So from here on, I pledge to stop calling my co-workers of years past idiots, and instead will upgrade them to mere morons (I would never call any present co-workers either of course!).

Sadly, the work of Henry Goddard, who created this colorful scale of feeble-mindedness, is now politically incorrect, and since the turn of the century state governments have been moving to replace these terms with "mentally incompetent". This is discouraging for lovers of language, but at least we can take heart in the knowledge that at least 5 states do not allow idiots to vote. Surprisingly, none of those states voted Republican in the 2004 presidential election. Funny, I'd of thought it'd be the other way around.

Improve Groovy by Removing a Feature

Summer 1962... Nelson Mandela was being arrested, Algeria joined the Arab League, Spider-Man made his first appearance, and a working group convened to shepherd in the next release of ALGOL 60 (behold the milestones!). As C.A.R. Hoare writes of the effort:

"The group's main task was to design a subset of the language which would remove some of its less successful features. Even in those days and even with such a simple language, we recognized that a subset could be an improvement on the original."

Sage words from almost 50 years ago. Simple is better. Less is more. And C.A.R. Hoare is practically the grandfather of the simple language movement(Google for free version).

So what language feature would you remove from Groovy to make it a better language?

Now I realize there are good reasons why all the following features exist. The goal of Groovy is not simplicity; Java interoperability probably trumps simplicity in many cases. But considering the complexity of a language is a useful and fun thought experiment. Just don't take my list too seriously.

this, owner, delegate, self, super, it, huh? - It's hard to keep all the hidden variables straight in a closure. Is the 'it' default variable really needed? Delegate... seriously? You don't need delegate, and I guess you don't even need this, owner, and super. I certainly can't tell you what they'll reference at runtime without running a debugger session, especially since there seems to be a difference between a Script and a Class. Dropping almost all these terms leaves Groovy a simpler language with minimal lost productivity.

curry - I have a new mistress and her name is Functional Programming. I relish slipping away from objects and procedures to spend a few stolen moments with recursion and lambdas. But Groovy isn't a functional language, and curry adds little functionality that you couldn't achieve by simply wrapping a closure in a closure. Plus, many Java developers aren't familiar with the term. Drop it and rip those pages out of the Groovy books, making more room for something better.

Arrays - What a pain arrays are. Can't everything be a list? We all agree that primitive types make the language worse, but why aren't arrays grouped with int, float, and double as something we should just do without? So there'd be no way to invoke a Java method that was overloaded on List and Array. I'd easily trade that ability if I could get better implicit List to Array conversions within the runtime. Those "as Type" and "as []" statements are annoying... let's get rid of them.

GroovyInterceptable - invokeMethod is called for every method invocation. Oh, except sometimes when the object doesn't implement GroovyInterceptable. The flow of method dispatch is complex and ditching GroovyInterceptable would simplify the picture. While we're at it, ditching the weird Expando vs. MetaclassImpl issue would eliminate complexity too, but that seems more a bug fix than a language feature.

Implementing Interfaces with a Map - I'm not kidding about removing one of the best and most useful Groovy features. I love the syntax of creating partial interfaces, but have had issues with how the object is still a Map at runtime. Partial interfaces are a superb idea, and the syntax for creating them is wonderfully concise. But under the covers the thing is still a Map, not the type you are implementing. What happens when you pass it to a method that takes a Map? Have you ever gotten a NullPointerException because of a missing method on your Map-erface? You have to start thinking about the way Groovy is implemented to reason about this, which is a sign of edge cases and complexity. To make the language simple, I would keep the syntax for easy partial interface creation, but I'd use ASM to generate an actual object of the correct type under the covers.

Stack Traces - For application developers, Groovy stack traces contain a bunch of meaningless junk. I've held several team member's hands through their first foray into Groovy and apologize profusely on Groovy's behalf when the first stack trace is encountered. I'll almost claim that the excessive stack trace noise makes test driven development harder for newcomers, as they try to parse and diagnose what their failures are. This isn't a language feature though, just more of a complaint. Sorry!

So what would you remove from Groovy to make it a better language? And it's fine to argue against one of my items, but please suggest an alternative removal if you don't like one of mine!

Why Functional Programming?

Earlier this week Robert Fischer presented a session at the local Ruby group called "The OCaml Guide for Rubyists" (slides and video available soon). During Q&A, the last question was a simple, "Why?" Not a question so much as a lamentation. The real question being asked was, "Why in the world would you want to put up with horrible syntax, bizarre compiler errors, and unreadable type system REPL output just to accomplish the same thing I do regularly with a few lines of Ruby?" Robert's short answer was, "It will warp your mind," which might not have convinced anyone that OCaml was worth their time. I'd have given the same answer, though, only I would have made my response long, meandering, and possibly in need of a good edit. Welcome and let the journey begin!

I love programming because of the abstraction. Finding and removing repetition is a joyful act. In my C days (which were thankfully short, for both me and my customers), the abstraction came from carefully defined modules. The pinnacle of design was a finely crafted global library accessible to anyone who cared to include the proper header and debug the resulting link error. Entities and concepts were modeled in their own .c files, hierarchies and relationships were defined through directory structures, and exported abstractions made available in .lib files. Oh yeah, I really had this down, and I knew what programming was all about (including spending days finding memory leaks).

And when someone showed me C++ a lightbulb turned on. My basic unit of abstraction changed from the library to inheritence, and my mind was warped. Holy cow! Modeling everything as parent-child relationships opened up a whole new realm of software design. Everything is an object, and everything has shared code in its parent type. I could get wide-spread behavior changes in my program simply by changing some code way up high in the type hierarchy. And the way to do this was multiple inheritance, right? All my systems looked like inverted pyramids... there was one core implementation class with dozens of parent classes spread out above it. Genius, or so I thought.

And then I found out about the Gang of Four Design Patterns. In an act of supreme hubris, I remember arguing with the team architect about the definition of an interface. I firmly believed an interface was an interaction library, like the Win32 Application Programming Interface. I had no idea why anyone would create a parent type with no implementation, and thank you Gavin Ford (wherever you are) for handing me that book and telling me to bugger off. With design patterns and a new found mandate to prefer composition over inheritance, I was off and running with a primary unit of abstraction: the class. Mind warp #2, for those counting. I could never go back to C now.

And for a long time it stopped. Java and its awesome toolset let me create reams and reams of code all structured to one design pattern or another. And the Java libraries contained many of the patterns, so it must be the right way to design software. Java enabled me to overuse patterns, and I slowly realized that they weren't an end to themselves... but this didn't really qualify as a mind warp despite the hundreds of Internet posts proclaiming the death of patterns due to closures and dynamic typing.

People moved on. Some moved on to embrace meta-programming as the next mind warp. But is this really a means of abstraction? Does meta-programming allow you to construct more general and reusable structures? From a client perspective, writing code against ActiveRecord and GORM is certainly a time saving shortcut. But in my experience, meta-programming exposes you (as a writer) to a very low level of the language. Method dispatch, metaclasses, caching closures... this doesn't feel very abstract. It feels reusable as long as method calls and database columns and xml schema all line up. (Quick quiz, what's the difference between programming by convention and coincidence? One is the next big thing and the other is warned against in the Prag book.) But is metaprogramming layer of abstraction that allows you to specify behavior in a more general way? Perhaps not. It's a mind warp, but a very implementation specific warp.

Another direction people move towards is device development, most notably the iPhone. I spent waaay too long programming mobile devices and I cringe at the thought of downloading some iPhone SDK. Honing my programming skills towards learning a device API and graphics library is simply the wrong direction. It is a step closer to the hardware, the implementation, and a step away from a higher level of abstraction. No thanks. I'm quite skeptical that new software ideas will be the product of anyone moving into the device development space. Rediscovering pointer arithmetic and reimplementing existing frameworks on a smaller footprint seems a more likely outcome. Neither of which sounds appealing.

So I'm here to say that mindwarp #3 is discovering the function as the basic unit of abstraction. Jaw-droppingly beautiful abstractions and generalizations can be created out of just functions. You can rediscover the usefulness of partial functions and currying, which were techniques created in the 1800s. You can be in the direct lineage of Alan Turing, who used higher order functions in the 1930s to define his theoretical Turing Machine in his paper "On Computable Numbers, with an Application to the Entscheidungsproblem." And you can finally understand recursion in a deep and intuitive way, and you'll feel like you've looked into the abyss and somehow come back to tell everyone else about it. And maybe, just maybe, you can explain to me what a freakin' monad is.

And maybe (just maybe!) Ruby isn't the language to try this in. Ruby is awesome for a lot of things. Just check out how fast and furious one-line responses to Cedric's challenge appeared. The fact the Ruby solutions are denigrated as "a dime a dozen" says something about how fast and easy the language is to work in. But how easy are function objects in Ruby? Does it take 5 pages of text to fully explain the fine points of procs and blocks? I've been told several times to stop trying to do FP in Groovy because a better functional programming language will more easily reveal the beauty and simplicity of functions. You want to concentrate on the FP concepts and eliminate all the noise of how the language implements functions. Non-functional languages contain imperative and procedural noise that will distract you from having your mind warped. Just pick a more functional language.

And that is my answer of why Rubyists should learn OCaml: Your mind will warp when you discover functions are the next unit of abstraction, and OCaml's sort-of FP purity will have your mind warping sooner than it would with other languages.

Thus concludes my contribution to a well covered topic. Interested readers may wish to read the original Why Functional Programming Matters and Raganwald's thoughts on the same.

Thanks!

Functional Calisthenics

An overview of an essay from The ThoughtWorks Anthology caused quite a hoo-ha recently. (That's a cross between a brouhaha and a hoopla, see the Reddit comments and this great response).

Regardless of how you feel about the original article, it's safe to say that practicing the basics of a newly learned skill will improve your ability at it. And you should probably practice with some principles in mind or you might just be perfecting bad habits. As my guitar teacher used to say, "You can learn a bad habit in a week that takes you two weeks to unlearn." I still suck at guitar, but it seemed like sage advice when I was 12.

I'm not trying to learn OO programming, I get enough of that at my day job. But I am trying to learn functional programming. What would functional calisthenics look like? If you're trying to learn functional programming, what are some of the basic principles to try to apply? Here's what I'm trying:

Pick an interesting but small problem to work on. Mathematical problems seem to work well, like these. Problems tied to databases and filesystems don't seem to work as well.

Pick a language that supports first class functions and recursion (to some extent, at least).

And here is my list of principles to follow... please feel free to make your own suggestions:

• Immutable Objects - Don't allow an object's state to change after it is created.
• No State - Don't track state, either within an object or globally.
• No side effects - Don't allow functions to perform side effects.
• Favor Recursion - Favor recursive algorithms over iterative ones.
• Favor Higher Order Functions - Favor passing collaborating functions as parameters, returning functions from other functions, or composing new functions out of existing ones.
  

I have no idea if these principles are really the ones you want to follow, so I figured I'd try this out myself.

My problem: Maximum Segment Sum (MSS). Take a list of integers; the MSS is the maximum of the sums of any number of adjacent integer. I stole the problem from a cool paper I found when trying to figure out what Squiggol meant. For this list of integers:

31 -41 59 26 -53 58 97 -93 -23 84

the MSS is 187. That is the sum of the segment that omits the first two and last three elements of the list.

The language: I picked Groovy because that is my pet right now, but be warned that closure recursion within a script doesn't work well. I ended up having to define a Class to contain my functions. Meh.

The solution:

mss = max º sum* º segs

Simple right? Maybe not. Let's break it down. To solve this we need to generate a list of all possible adjacent values from the integer list, which we'll call "all possible segments", or just segs. Then we need to sum each of those segments, and then we need to find the highest value among those sums. So this:

mss = max º sum* º segs

reading left to right is "the mss is the max of the sum function mapped across the segments". The * is the map function, which in Groovy is Collection.collect(Closure). Call sum on each segment and take the max. The tough part of this is producing all possible segments from the integer list.

The keys to producing the segments are the functions inits() and tails(). inits() returns all the initial segments of a list in order of increasing length:

inits.[a1, a2, ..., an] = [[a1], [a1, a2], ..., [a1, a2, ..., an]]
or, in Groovy
inits([1, 2, 3]) == [[1], [1,2], [1,2,3]]

Tails() returns all the trailing segments of a list in order of decreasing length:

tails.[a1, a2, ..., an] = [[a1, a2, ..., an], [a2, ...an], ..., [an]]
or, in Groovy
tails([1, 2, 3]) == [[1,2,3], [2,3], [3]]

So if you want to produce all the possible segments of the list, all you need to do is map the tails function across the result of the inits function, flattening the result:

segs = flatten º tails* º inits

Let's get Groovy.

At the highest level, the solution is easily expressed: mss = max º sum* º segs

def solve = { list ->
 segs(list).collect { it.sum() }.max()
}
assert 187 == solve([31,-41,59,26,-53,58,97,-93,-23,84]

Generate a list of all possible segments from the initial list, loop over each sub-list summing the list contents, and then take the max value.

segs is also easily expressed: segs = flatten º tails* º inits

def segs = { list ->
 flatten(inits(list).collect { tails(it) })
}
assert segs([1,2,3]) == [[1], [1, 2], [2], [1, 2, 3], [2, 3], [3]]

Take the inits of the list, and loop across each entry collecting the tails of the entry. Then flatten the whole thing.

inits was simple:

def inits = { list ->
 (0..list.size()-1).collect {
   list[0..it]
 }
}
assert inits([1,2,3]) == [[1], [1, 2], [1, 2, 3]]

This is going to loop once for every element in the list, returning a sub-list from element 0 to 'count' on each loop.

Tails has a very similar implementation:

def tails = { list ->
 (0..list.size()-1).collect {
   list[it..list.size()-1]
 }
}
assert tails([1,2,3]) == [[1,2,3],[2,3],[3]]

This will loop once for every element, returning a sub-list from element 'count' to the end on each loop.

Flatten is where things get interesting. Collection.flatten() does us no good, because we want to reduce the result into a list of lists one element deep, not just flatten the lists into integers. My first iteration used inject() to accumulate the sum, but that seemed to much like tracking state. My interim solution was a function that just walked a nested list, flattening to 1 level. I got this working but realized it could be abstracted more by creating a higher order function.

What I needed first was a function that determined whether an element in a list was a list of primitives (in which case it was a leaf in my tree) or whether it was a list of more lists (in which case it needed flattening). This wasn't too hard:

def isPrimitiveList = {
 if (!it) return true
 if (!(it instanceof Collection)) return false
 if (it.head() instanceof Collection) {
   if (it.head().head() instanceof Collection) return false
 }
 true
}
assert isPrimitiveList([[1], [2]])
assert false == isPrimitiveList([[[1]], [[2]]])

If the parameter is empty or null, simply return true. (Remember, Groovy considers an empty list to evaluate to false). If the parameter isn't a collection then return false too. Otherwise, return true if the head of the head is not a collection (it is a list, but the first element is a primitive).

Then we just define a generic traversal function that will take isPrimitiveList as a parameter, stopping the traversal if that function evaluates to true when passed the current node:

def traverse = {isLeaf, collection ->
 if (isLeaf(collection)) return collection
 return traverse(isLeaf,
   traverse(isLeaf, collection.head()) +
   traverse(isLeaf, collection.tail()))
}  

If the element is a leaf already, then just return the element. Otherwise, return the results of traversing the head plus the results of traversing the tail. Here you see recursion (the function calls itself) and a higher order function (the function accepts another function (isLeaf) as a parameter). From here it is simple to define a flatten function:

def flatten = traverse.curry(isPrimitiveList)
assert flatten([[1],[2],[3]]) == [[1], [2], [3]]
assert flatten([[[1]],[[2]],[[3]]]) == [[1], [2], [3]]

Curry is the worst kept secret of Groovy. If you're not familiar, it simply wraps an existing closure in another one, binding the first parameter to the passed argument. flatten() is now a traverse() function where the isLeaf closure is always the isPrimitiveList function. There's more to curry than that but it's not worth rehashing.

And there you have it, an MSS solver with only immutable objects, no state, no side effects, recursion, and a higher order function. Took me about 6 hours. Without a doubt, I definitely know the Groovy Collection functions a lot better. It was also interesting to see how my first revision was much more imperative, and over time it was whittled down to something much more functional. I felt this exercise, given the constraints stated, was worthwhile. So whether you hate the idea of functional/object calisthenics or not, this helped me. Does anyone out there have other functional principles that should be included in future exercises?

Full source (and tests) available for download.

Why Groovy as a service layer

I've spent the last week working on getting an Axis2 and Groovy based web service up and running. Getting Groovy to work as an Axis2 service class was incredibly simple... the Groovy scripts compile to class files and then it's just dropping the Groovy jar in the classpath.

So the implementation is finished. Now comes the hard part: convincing my co-workers in a Java shop that Groovy is the right choice for a service layer. Everyone knows it's my pet technology, and so far my demos have resulted in a lot of eye rolling. But darn it, I think it's the right choice, and I'm using this post to explain why.

The design has the service layer interacting with the Java-based business logic facades. My intent was to have the service layer's concern be strictly focused. It moves incoming requests from untyped primitives (Strings and XML blocks) into typed Java objects consumable by the facade, and it moves typed Java objects produced by the facade back into untyped primitives (Plain Old XML) consumable by the client. (I'm pretty sure this is not proprietary information at this point, but don't ask for more details).

Groovy's Java integration makes it well-suited to do this. My service class just instantiates the Java objects it needs almost exactly like it would in Java. As a result, the impedance mismatch between a Java and Groovy service class is very low. Context switching languages can be tough, but this is one of the easier cases.

But that explanation answers the question, "Which dynamic language?" rather than the question, "Should we even use a dynamic language?", which is the real decision.

In my career I've spent an inordinate amount of time at type system boundaries. Most my time has been spent in C++ and Java, statically typed languages enforced with a compiler. And at the same time, most of my career has been spent working with untyped data: primitive types coming from the database, byte streams coming from sockets, character data coming from files. I've spent a lot of time writing code like this:

String name = resultSet.getValue("name")
  
char* name = resultSet[4]

I'm not saying my projects are littered with code like this. I'm saying the projects contain code like this, and this type of code is where I've spent too much time. It's error prone because the compiler can't check it for you. It's hard to test because you usually need an integration test to really check it. And it's hard to read because the offset 4 and the entire getValue() call are mostly just noise in the signal.

The statically typed languages I've used simply don't support untyped data very well. This is why a lot of the frameworks exist. Hibernate makes working with untyped databases easier. XPath and Castor make consuming untyped XML easier. But using a dynamic language like Groovy makes working with untyped data natural.

For instance, consuming XML is trivial with XMLSlurper:

def records = new XmlSlurper().parseText(CAR_RECORDS)
def carRecords = records.car
def allNodes = carRecords.depthFirst().collect{ it }

The signal to noise ratio here is much higher than the Java equivalent. There is no getValue() or getChilden() cruft to obscure the intent. In my experience, this is easier to read, a little easier to write once you're over the learning curve, and about the same to debug. Before the IDEA/Eclipse Groovy plugins it was harder to debug, but now I believe it is about the same.

Producing XML is trivial too, using the built-in MarkupBuilder:

def writer = new StringWriter()
new MarkupBuilder(writer).records() {
  car(make:'Holden') {
    country('Australia')
  }
  car(make:'Peel') {
    country('Isle of Man')
  }
}

I find this easy to read because, even in more complicated examples, the structure of the builder calls mirror the structure of the XML produced. This is also very easy to write. So easy that I've yet had to debug it. Overall, I've been really satisfied with the ability of Groovy to produce XML without additional libraries.

Back to my service layer... remember it's sole purpose is to move data back and forth between Java types and XML primitives. When choosing a language for this layer, we have to ask, "How well does it support untyped data? (Moving data in and out of XML)" The Groovy language has Java beat here. We also have to ask, "How well does it support typed data? (Moving data in and out of Java classes)" Groovy probably has Java beat here again, with its support for builders and identity blocks. Once you concerns are isolated to a single layer, it seems that Groovy is the clear winner for implementation language.

The decision to use Groovy has not yet been made... so what do you think? What are your reasons to use Groovy?

Aristotelian Metaphysics and Software Complexity

Metaphysics... it's the study of all that stuff "beyond" physics, right? Or is it that New Age-y study of crystals? I can never remember. I looked it up this weekend: the term Metaphysics was created by a first century translator of Aristotle. It's the title he gave to the untitled chapter that came after the chapter on physics. I always suspected that metaphysics contained a large amount of b.s. Ah, vindication is sweet.

Anyway, metaphysics (for better or worse) is the study of those things which transcend scientific observation. Questions like "Why does the world exist?" and "Does the world exist outside our mind?" are both firmly metaphysical questions. Have you and a friend ever discussed the possibility of the Matrix being real? Congratulations, you are an amateur metaphysicist! Time to update the resume. Go ahead and do it now, I'll wait.

Considered one way, judging software complexity is certainly not a metaphysical endeavor. We have lines of code, we have cyclomatic complexity, we have function point analysis. All of these are valid, scientific measurements of software complexity. You may disagree, but I assure you, somewhere in the world someone is willing to defend these measurements.

So let's cut straight to the point and ask, "What would Aristotle do?" And (indirectly) what does metaphysics tell us about these complexity measurements? Well... Aristotle believed that everything has a telos, an inner goal it is meant to attain. A pine cone has an inner goal of a pine tree. A coffee bean has a telos of a double latte. It is what the coffee bean was meant to be. A "Hello World" example written in Lisp also has a telos:

(print "Hello World")

It is meant to print "Hello World" to the console. The Java version is slightly longer, but does have the same telos:

class HelloWorld {
public static void main(String args[]) {
 System.out.println("Hello World");
}
}

On a larger scale, most software has a telos too. It might be meant to reduce the costs of your supply chain. Or maybe it is meant to allow you to shop for pet food from your computer. And the components within a system also have a telos. The Shopping Cart is meant to allow you to track ordered, but not purchased, items. It's not hard to argue that software components have a telos; it is considerably harder to do so about humans in general!

So the first question metaphysics asks is, "What's the point of all this?" Our answer is, "to print hello world!" Easy. The second question metaphysics asks is, "What makes this so?" What makes an elephant an elephant? What makes a duck a duck? Just as importantly, what makes an elephant not a duck? And just what makes our "Hello World example" a "Hello World example"?

Aristotle makes a distinction between two things: essential properties and accidental properties of an object. The essential properties are those traits that makes a thing that thing. A black elephant is still an elephant. A gray duck is still a duck. But if your elephant has feathers and a beak, then perhaps it's not really an elephant. Accidental properties, on the other hand, describe how a thing is rather than what it is. The color brown is accidental to a duck, but feathers and a beak are pretty essential.

And what of the Hello World examples?

What essential properties must a Hello World example contain? And what is just accidental? When you compare the two examples above, it seems that they both contain some sort of print/println command and the "Hello World" string. Those traits seem pretty essential. The rest is probably all just accidental properties: the weird parentheses required in Lisp, the class declaration and main() method required in Java.

Now, considering the two examples, what parts of the examples account for the most complexity? Isn't it the exact same list we created for the accidential properties? Lisp is complex (at first) because of all those danged parenthesis, and Java is complex (at first) because of all that danged OO cruft.

Let's consider another example and stop picking on Java. The example performs some simple math 40 - (5 + 10 * 2) and prints the result

In Lisp:

(print (- 40 (* 2 (+ 5 10))))

In Groovy:

println 40 - (5 + 10 * 2)

The Groovy version does seem more essential... it more closely models the way we are taught basic arithmetic. All those parentheses in Lisp are kinda tough to look at. They are the same thing though, right? They both print 10. Right? Ummm, actually, the Groovy version prints 15. Why? Java evaluates the * operator before the + operator, resulting in 40 - 25 instead of 40 - 30. I FULLY UNDERSTAND that println 40 - ((5 + 10) * 2) would have yielded 30, just like the Lisp version. The point is that the essential complexity of operator precedence is abbreviated in the Java/Groovy version, while it is visible in the Lisp version. Abbreviated to the point of hiding it within the language.

In this example, the Lisp version is superficially more complex. It shows all the essential properties of arithmetic while the Groovy/Java version abbreviates them away to nothingness. The Groovy/Java version is therefore more complex because it is missing essential features like operator precedence. Sometimes simplicity is more verbose than complexity, as this example shows, and sometimes complexity is more verbose, as the Hello World example shows.

How can this knowledge be applied to scientific methods of simplicity like lines of code, cyclometric complexity, and function point analysis?

Lines of code clearly has nothing to do with the amount of essential properties of a software component or system. This is a very poor measure of software complexity, and the definition of a complex system as one that contains many lines of code is a bit of circular logic. You are essentially saying, the software is complex because it is large and the software is large because it is complex.

Similarly, cyclometric complexity is also a poor measurement of software complexity because it does not address any of the essential properties of the system. Breaking down a component into small methods each with low complexity does not remove any of the essential complexity. You're left with a measurement of complexity with one and only one data point - the code you wrote. There is no way to compare your solution to a solution designed differently.

Function Point Analysis is probably the closest you'll come to measuring essential complexity. Yet, it would seem possible to make two wildly different implementations of a system with x number of function points, each with a wildly different accidental complexity. I have zero experience with function point analysis, but I'm suspect.

So what are we to do?

The software you write has a telos... it was meant to do something. Simplicity is modeling a solution that contains those traits essential to the telos. Complexity is a solution that contains only accidental traits. Hiding essential traits and showing accidental traits both lead to complexity.

As for how to measure complexity, isn't it a metaphysical and somewhat subjective activity best performed by groups in design and code reviews? Can anyone scientifically measure that which is essential and accidental to a software system? Sounds like software complexity really is all about metaphysics after all!

Project Retrospectives DOA?

Dr. Shepherd: [Packing away the defibrillators] "We did everything we could, but it wasn't enough... we lost him."
Dr. Grey: [Looking lost and emaciated] "The only thing left to do now is hold our project post mortem."
Dr. Shepherd: "Yes, let's start by talking about the things we did right, and then look for opportunities to improve in the future. There's no topic we can't broach given enough post it notes and white boards."

I'm sure that Grey's Anatomy is about as accurate a depiction of hospitals as Hackers is of my workplace (and if you haven't seen it then think Jeff Goldblum uploading a virus to on oncoming asteroid). But I still can't imagine a group of doctor's holding a post mortem and starting off by discussing what they did right. Somebody died! That's what post mortem means: "after death". (And if they do start this way that doesn't ruin the post... read on).

A post mortem is a very serious event. Someone has died. If it is within our control, then we need to make sure it never happens again. It isn't a time for constructive feedback and team building. It is a time to focus on the mistakes. Does anyone seriously thing that performing a trust fall exercise is an appropriate response to a death?

Project retrospectives are different. It's not a team of 2 surgeons, some nurses, and a handful of smart, flawed, yet sexy interns (that's you Christina Yang). It's a much larger team including developers, managers, quality assurance, tech writers, even sales. In the wake of a failed project you're probably demoralized, maybe bitter, definitely overworked. With retrospectives it's OK to pull out the index cards and pipe cleaners. Heck, order some pizza. Let's have some fun. But let's not call them post mortems. Our work is a lot different than surgeons. As developers we're known for hubris, but comparing ourselves to surgeons is a bit rich. Right?

Let's consider for a moment that there is a substantive difference between the situations that call for a post-mortem and those that call for a project retrospective. When do you cross the line from one to the other? It is easy to consider that a human life may be at one end of the spectrum and a small software project at the other end... but where do the two meet? Here's an interesting way to think about it: take all the jobs in America that require low skill. Now look at the hourly wage for each one. You'll see that as risk of death increases, so does hourly wage. Using these numbers it is possible to calculate the amount of money American workers will accept in exchange for a higher risk job, and statistically speaking, death. Timothy Taylor at Macalester College did this, and the answer is 5 to 7 million US dollars per human life.

So by one measure a human life is worth 5 to 7 million dollars. What is your project worth? 1 million? 2 million? 5 million? A human life? 2 lives? We're in a serious business here. And since that business is software we're allowed to be cold and calculating. Or maybe it's just me.

Anyway, my point is that not only is there no clear delineation between those projects that require post mortems and those that require retrospectives, there is actually no delineation at all. The only difference is what you call it (and that is a big difference). Using the post mortem metaphor for your project retrospectives invokes a response from the participants. One of seriousness, somberness, and possibly death. The term retrospectives invokes pipecleaners and pizza. Arguably more fun. Proven to be more effective. Seriously, go look at the research. Looking at what you did right over the course of the project is the best thing to do... even if someone died. There is no human-cost magic that precludes retrospective best practices on your project.

The first step towards an bad retrospective is calling it a post mortem. A meeting announcement called "Post Mortem" prepares the participants for a negative, serious, and somber event, which isn't an atmosphere conducive to organizational learning. It doesn't matter how much free soda and food you show up with.

But what does it matter? "Who cares what we call it," you're thinking. Well, imagine the following three couples in marriage counseling:

Couple 1: "We had a strong, healthy marriage. But now we have a sick relationship."
Couple 2: "We were crazy about each other. But now she's driving me out of my mind."
Couple 3: "She cast her spell over me, she had me hypnotized. But now the magic is gone."

Each couple uses a different metaphor for love. The first couple uses a patient metaphor, in which love is something that can be nurtured and healed over time. The second couple sees loves as a madness. Love is irrational at times, but madness in our society can be dealt with. Maybe medication is required, but there are still options. And the third sees love as magic. Unexplainable, unaccountable, and certainly something without morals.

Which of these three couples is most likely to salvage their marriage? Couple 1, right? Implicit in the language is their knowledge that relationships require work, and are subject to some forms of cause and effect. Which couple is most likely to split up? Couple 3, right? If love is something totally unexplainable, that is completely outside of your abilities and behavior to affect, then you're not going to even try to improve the marriage.

Since the 1970s, linguistic researchers like George Lakoff and Mark Johnson have been showing that language shapes our experience, rather than reflecting our experience. By thinking and speaking of love as a patient, couple 1 is increasing the likelihood that they will work to correct their relationship. They aren't using that language because that is how they behave... they are behaving that way because they are using that language. You don't refer to love as madness because you continually make irrational decisions; instead, you are making those poor decisions because you've mentally framed love as a madness, which is not subject to rationale and clear thinking. Has anyone ever told you that forcing yourself to smile will eventually make you a happier person? They might be right.

So when when your project "post mortem" is completely derailed, and people are yelling at each other, and there is enough blame to go around for several projects... maybe you shouldn't have called it a post mortem. Maybe you should be more careful with the language you use. And if someone really has died... hopefully you're a doctor and then hopefully you have insurance.

And at the end of it all, you're thinking, "Seriously? 1000 words to tell me to call it a retrospective and not a post mortem?" Yeah. It bothers me that much. That's why mine are called Futurespectives. But I wouldn't want to force that level of cheesiness on anybody.

Disclaimer: This post has a much longer history than my current employer and in no way reflects any real events in the last year.

Is it really a domain specific language?

We have now reached the point where the term Domain Specific Language is so overused that it has become meaningless. And if not, then we are quickly approaching it. In my mind, this Fall's No Fluff conference will be remembered as "the one about DSL". But we need to slow down... I feel like standing up and shouting, "Not everything is a DSL!" Excel functions? Yes, great example. EasyMock's humane interface? Eh, perhaps. Ordering hashbrowns at Waffle House? umm... maybe this one has questionable value.

It seems that several terms are being confused, and we should be specific about different ideas. A domain specific language is a vocabulary for discussing a specific problem you have. Excel functions are a darn good vocabulary for working with spreadsheet data. They may not be easy to learn, but they are an extremely powerful language to solve the problem of expressing complex formulas in spreadsheets. A fluent interface (or humane interface, or literate interface) is not the same thing as a DSL! A fluent interface is an API design that allows users to work with concepts in a way that closely matches most Western sentence structure, such as:

receipt = order.2.lattes.andPay($10)

Compare this API to the traditional Java style API:

Order order = new Order();
order.setQuantity(2);
order.setItem(latte);
Receipt receipt = order.pay(new Currency(10, DOLLARS));

Which API is easier to read, the first or second? Yes, fluent interfaces are easier to read (you answered "first", right?). Hopefully your native language follows the same subject-then-predicate order of English, or else the code example may confuse you, but that's an aside. I do think fluent interfaces are a good idea: they're easy to read and easy to write, despite sometimes being harder to debug because of their multi-step nature.

But just because something has a fluent interface doesn't mean it is a DSL. EasyMock has a fluent interface, but it is not a DSL. Consider the following unit test code:

MyObject mock = new EasyMock.createMock(MyObject.class);
EasyMock.expect(mock.foo()).andReturn(10).once();
new SystemUnderTest().perform(mock);
EasyMock.verify(mock);

This code, if you're at all familiar, creates a mock object and tells it to return 10 when the foo() method is called, and only do it once. The system under test is then exercised and the mock is verified to have been called correctly. This is a fluent interface. The methods are chained together on the 2nd line and it is sort of easy to read.

Let's revisit my definition of a DSL: "a vocabulary for discussing a specific problem you have". Is EasyMock a DSL for unit testing? EasyMock's fluent interface provides a nice way to set and verify expectations on mock objects. But setting expectations was not my problem, unit testing collaborating objects was my problem. And the two are different! EasyMock uses a record/playback metaphor to create mocks, and its interface is a DSL for record and playback. Using EasyMock's API in my project didn't solve the problem of writing better unit tests, it created a problem of how to work with record/playback frameworks. Since this fluent interface does not provide a vocabulary for unit testing, it is therefore not a DSL.

So a very important part of a DSL is that it solves the problem you have, not just any problem. Let's consider the famous Waffle House ordering "DSL" of "scattered, covered, and smothered" (these are three different ways to order hashbrowns, by the way). Is this a DSL? What problem does it solve? If you wait tables at Waffle House then it is a terser than English vocabulary used to communicate with the cooks, and therefore a DSL. But if you are a customer at Waffle House, what possible problem does it solve? Can you not say the words "with cheese"? I say ordering at Waffle House is harder than it needs to be because we are forced to adopt the language of the cooks (the implementers). Have you ever ordered a Philly Cheese Steak in Philadelphia? If you don't say it right then you don't get your food. It's nerve wracking for the tourists! Imagine this one... imagine a restaurant where the cooks are Mexican. Can you imagine that? I've just described a ton of places you eat, trust me. Wait if you had to order in Spanish because that is the DSL of the cooks? Well, that's what you're doing at Waffle House. And that is what I'm doing with EasyMock every time I set an expectation.

My preference is to define a DSL in relationship to a person and their particular problem. A language might be a DSL for one person and not another. If you're an API designer, simply saying that you're going to wrap your implementation in a fluent interface is not the same thing as creating a DSL, and you might be exposing the wrong level of abstraction to your users. Creating mini-languages cannot be our goal. We need to first think hard about which problem we need to solve. And providing a nice, fluent interface at just the right level of abstraction can then be a beautiful thing.