The Debate over Closures for Java

We've been continuing work on the closures proposal. I'd like to give you some feedback on what I think about the debate. Tomorrow I'll post an updated proposal, and the next day I'll outline an alternative version of the proposal that avoids function types and contrast it with the current draft; some people think that might be a better approach, though we have some doubts that I'll explain.

Our publication of the draft closures proposal resulted in a great deal of debate and excitement. Even if we end up doing nothing, I think that is very healthy. These kind of things rarely get debated in such public forums. We've been listening to the debate as it rages (on scores of blogs and message boards, and the occasional email), and we're trying to adjust our thinking accordingly. My next post will be an updated version of the proposal, and the following will be an extended discussion of one alternative direction we're considering that addresses some concerns.

There are a few different kinds of feedback we've received, both positive and negative. To those whose feedback has been mostly positive: you've been no help whatsoever. Just kidding. Seriously, thanks for the support. We don't think things are perfect yet, and maybe they will never even be good enough to put anything like closures in the Java programming language. We need to figure out what, if anything, to change to make it better, so your criticism is welcome. If you can quantify precisely how much more fun Java will be to program in, that would help too.

To those with mainly negative or mixed feedback, I'd like to try to classify some of the most common concerns and give general, blanket responses. Please don't take these to mean we're dismissing your concerns. On the contrary, we expect the proposal to evolve, and your concerns are a primary source of information on how the proposal needs to evolve. This classification doesn't cover all the constructive criticism we've received or all the issues we know we need to address, and there is a great deal of overlap between these topics.

Closures don't buy you anything beyond what you can accomplish with inner classes. That just isn't true. See my previous blog post about use cases. Interfaces and inner classes solve the asynchronous use cases just fine, mostly, but they don't touch the synchronous use cases. I don't mean to imply that they can't be extended to cover both sets of use cases, but inner classes in their current form don't do it.

The addition of function types to the language is too much of a price to pay for closures. We have a few reactions to this. First, you're likely to be surprised how much you actually get with closures, and how little it actually costs you in complexity. While it might be possible to get closures without function types - see the alternative proposal for one stab at that approach - what you lose doing things that way is not trivial.

The structural typing of function types doesn't feel right in Java, which normally does its typing via programmer-defined named types. Actually, function types dovetail nicely with interfaces due to the closure conversion. The structural typing you get with closures is much like you would get using generic interfaces and using wildcards heavily in your code, except much simpler.

The proposal introduces a second way to do things that folks already do with interfaces. The concern here is that the language should provide strong guidance on how to do any particular kind of thing, and Java has already chosen to pass snippets of code around as objects that implement interfaces, typically via anonymous classes. Moreover, some existing APIs might look silly doing things the "wrong" way once a new way is introduced. I don't think this concern holds water because virtually all of the existing APIs in the JDK that I'm aware of are best written using interfaces rather than function types. I've heard the debate regarding the JDK notification APIs (for example, in swing) and what to do about them. I think they should be left alone.

Ordinary Java programmers won't be able to understand this stuff. Many folks who say this don't make it clear which parts of the proposal they find most troubling, or perhaps they don't see the proposal as decomposable into troubling and acceptable parts. The Closures for Java draft specification was written to be more like a specification than a tutorial. It is not a suitable thing to read to learn the language construct, any more than reading the Java Language Specification is a suitable way to learn object-oriented programming. Rather you would learn the feature by examples of gradually increasing complexity. I haven't provided that, and I apologize. I've tried to remedy that in my later blog posts but I'm afraid I'm not very good at that kind of writing.

If you believe that closures are inherently a difficult concept for programmers to understand, or some kind of language "trickery", I suspect you probably haven't done much coding in a language that supports them (see Steve Yegge's blog post Language Trickery and EJB, and Curtis Poe's article Sapir-Whorf is not a Klingon). In practice they are pretty straightforward and obvious. Things are complex with anonymous inner classes, and any tutorial would need to explain them as well. I will present an alternative spec in a few days that does away with function types entirely, syntax and all; with that change the spec may seem simpler than the previous one, but I believe as a practical matter it will be harder to use. Also gone are local functions from our proposal entirely. Those who identified function types as the main problem might be satisfied with the version of the proposal lacking function types. But if you have a hard time understanding anonymous inner classes, closures won't make things any easier as we can't remove anything from the language. There also needs to be a prototype implementation that people can actually use to get a feel for what closures can do. I eagerly await an open-source javac on which we can prototype this.

How much more expressive does the language become given the complexity of the additions? The gist of this concern is that adding closures seems to increase the complexity of the language substantially. Is the additional expressiveness worth the additional complexity? The best result would that we end up with an expressive but simple programming langauge. The question is: how much more expressive can you get with minimal changes to the language? We believe that closures adds significantly to the expressiveness of Java in an area that the language is severly lacking. The asynchronous use cases are handled just fine today, but the synchronous use cases like control abstraction and some kinds of functional programming idioms are very difficult.

Are you just doing this to compete with C#/Ruby/Javascript or add the latest experimental/research language feature? Closures have been an idea simmering in the background since the earliest days of Java, and it is the subject of repeated RFEs. Closures predate Java by many years. Closures were in Simula (the original object-oriented language), Lisp, and Smalltalk. They really do make some things easy that are difficult or impossible to express without them. Anonymous inner classes were a half-measure that made some use cases easier, but left some use cases unaddressed. Competition from other languages has shaped the political landscape and the programmer mindset to make closures more acceptable than they were before, so the time is ripe to propose them.

Values of function type can't be made serializable. Agreed, a problem in the previous spec. We intend that you should be allowed to "implement" function types so that you can make them serializable.

I can't stand the syntax. We've heard this before about things that people ended up loving once they started using them. We're open to alternatives. The alternative of just using anonymous class instance creation expressions doesn't work. Closures are not just shorthand for making an anonymous class, because names appearing inside the closure are resolved in the context of the enclosing scope instead of in the context of the anonymous class being created. Perhaps there is a better way of writing a function type. Perhaps we can improve the syntax for writing a closure - but it does require its own distinct syntax in order to get the expressive power. Your suggestions and ideas are welcome.

The synchronous and asynchronous use cases are so different that they deserve completely different syntax. I'm not sure about this argument. First, in the proposal the two kinds of APIs do use completely different syntaxes: one uses function types and the other uses interfaces. A closure means precisely the same thing (if it is allowed) whether used in one kind of API or the other. Any difference in meaning is because of what the API elects to do with the closure (for example, invoke it or not invoke it). Therefore, to understand a use of a closure, you need to consult the documentation or specification of the method it is passed to. In this respect a closure is no different than a value of any other type. I suspect I don't fully understand this concern, and we'll keep listening.

Closures will be implemented using the JVM invokedynamic bytecode; won't that hurt performance? The short answer is: no, not necessarily, and there are alternative implementation strategies that use existing instructions.

Compatibility with the existing language demands an inelegant/kludgy design for closures. Frankly, I don't believe it. I believe that the Java language in JDK7 can have closures AND seem to have been designed with them from the beginning, with virtually all existing APIs using the most appropriate idiom. Some minor syntactic challenges remain: closures would probably better be written with a new keyword (lambda? fun?), but can't due to backward compatibility; a related issue is the difficulty of using "," or "|" in the syntax for disjunction types for type parameters representing thrown exception types. I believe we are likely to converge on a syntax that feels quite natural once people start trying to use it.

A related argument compares the Java programming language to a delicious Oreo® Cookie: you can add syntactic sugar to its outside and still have a nice Oreo cookie, but if you disturb its delicious design center it just won't be the same anymore. Implicit in this argument is a warning (or veiled threat) that hordes of Java programmers will abandon the platform if anything so horrible is done to the language. I think this line of reasoning is overly melodramatic. In fact, I believe the addition of closures will attract a number of programmers who have previously had no choice but to seek alternative languages due to limitations of Java.

Closures are too powerful; programmers should not have the freedom to effectively define their own control constructs, because they will write programs that are too hard to understand. Instead, the argument goes, we should add new statements to the language for each of the important use cases we know about and leave it at that. The gist of this concern is that while the Java programming language today lends itself to easily understood programs, the addition of closures will result in a language in which readers of programs have no firm ground to stand on, and will not be able to tell what a program means. I've only heard this concern raised by a small number of people and our response (thanks to James Gosling) is: by the same logic we shouldn't allow programmers to write their own methods either. If you're reading a program with invocations of unfamiliar methods, you may have to consult the documentation of those methods to understand what the code does; if the code is well-written the name of the method will give you a good hint. This is true whether or not the method receives a closure as an argument. Related to this argument is the concern that the addition of closures will make it possible to write programs that are really hard to understand, but in reality poor programs can be written in any language. This line of reasoning is made by those who consider themselves advocates of the B&D (Bondage and Discipline) school of language design, but I would characterize it as the S&M (Sadism and Masochism) school. The subtext is that we should not trust ordinary programmers with anything so expressive as closures.

Closures should work with classes that have a single abstract method, like java.util.TimerTask, in addition to single-method interfaces. OK, sure, that sounds like a good idea.

There have also been proponents of various other languages like Smalltalk, Ruby, and Javascript noting with smug amusement that the Java folks (or some of them, anyway) finally "get" closures. To them I say "Pfffft!"

Control abstraction

All of the examples of synchronous use cases for closures that I've shown so far fall into the general category of control abstraction. The idea is that you have a common pattern appearing in your code, and you want to abstract the common parts while allowing the caller to provide the parts that differ. You can think of these use cases as satisfying a desire to extend the set of statements available in the language. Java is already pretty good at data abstraction, but not so good for control abstraction.

Thanks to a blogger whom I can't identify, I have another simple use case for closures of the synchronous variety (still control abstraction, though). I would love to be able to use this in the implementation of my project, Google Calendar to simplify the code in hundreds of places.

Many places throughout our code we have this pattern appearing:

 
void someOperation() throws SomeException { 
    long t0 = System.nanoTime(); 
    boolean success = false; 
    try { 
        someOperationInternal(); // may fail, throwing SomeException 
        success = true; 
    } finally { 
        long elapsed = System.nanoTime() - t0; 
        logElapsedTime("someOperation", success, elapsed); 
    } 
}

The presence of this code helps us with realtime (i.e. elapsed time) performance measurement, but it clutters up our code and tends to obscure the logic of the application. We would prefer to write a timeOperation method once, thereby enabling us to write the above code as follows:

 
void someOperation() throws SomeException { 
    timeOperation("someOperation", (){ 
        someOperationInternal(); // may fail, throwing SomeException 
    }); 
}

Or, better yet, (presuming the language supports the abbreviated invocation syntax that I'll describe in a future post)

 
void someOperation() throws SomeException { 
    timeOperation("someOperation") { 
        someOperationInternal(); // may fail, throwing SomeException 
    } 
}

The timeOperation method can be written (once) as follows:

 
<E extends Exception> 
void timeOperation(String operationName, void() throws E block) throws E { 
    long t0 = System.nanoTime(); 
    boolean success = false; 
    try { 
        block(); 
        success = true; 
    } finally { 
        long elapsed = System.nanoTime() - t0; 
        logElapsedTime(operationName, success, elapsed); 
    } 
}

This is the best way I've seen to remove this kind of redundant code that is currently repeated hundreds of times throughout our application. This example should help illustrate why closures will help make code much easier to read. Certainly timing support could instead be done with an additional extension to the Java language: a new statement for timing blocks of code. Do you also add a new statement for closing your streams at the end of a block, and one for locking and unlocking your java.util.concurrent locks around a block of code? Once you start down that path, where does it all end? We'd prefer to give ordinary programmers the ability to write these kinds of methods (and perhaps place some of them into the JDK) rather than adding a series of language extensions to address particular use cases.

Besides control abstraction, there are also important synchronous use cases for closures inspired by functional programming idioms with a very different feel, and I'll blog about that separately.