Java Closures Prototype Feature-Complete

I'm pleased to announce that the Java Closures prototype now supports all of the features of its specification!

The complete source code, released under GPLv2, is in the project's openjdk repository. A binary build, suitable for use with an existing JDK6, is at http://www.javac.info/closures.tar.gz. Other related documents are on the website http://www.javac.info/

Although there is room for performance tuning, the prototype supports the full Closures (v0.5) specification. Based on your feedback, there are some changes in the prototype suitable for a future update of the specification:

• Renamed Unreachable to Nothing

We adopt the name used by Scala to represent the same concept.

• Removed support for the type null

We used null as a placeholder for an exception type when none can be thrown. The type Nothing now serves that purpose; null is no longer supported as the name of a type.

• Overhauled restricted versus unrestricted

In the specification, an interface is considered restricted if it extends a marker interface. Unfortunately, the specification only provides a syntax for function type interfaces that are unrestricted. We modified the syntax so that a function type written using the => token designates a restricted function type, while one written using the newly introduced ==> token represents an unrestricted function type. This allows programmers to easily write APIs that restrict (or don't restrict) the operations of closure expressions passed as parameters.

• Refined restrictions

We modified the distinction between restricted and unrestricted closures. As before, it is not legal to convert an unrestricted closure to a restricted interface type, nor is it legal to break, continue, or return from inside a restricted closure to a target outside the closure. However, a restricted closure is allowed to refer to a non-final local variable from an enclosing scope. In this case a warning is given unless one of the following conditions holds:

1. The variable is not the target of any assignment, or
2. The variable is annotated @Shared

It is possible to suppress the warning by annotating some enclosing construct @SuppressWarnings("shared").

• Relaxed the closure conversion

In response to user feedback, we've relaxed the relationship between a closure parameter's type and the target interface's parameter type. Rather than requiring them to be of the same type, they are now allowed to be related by an assignment conversion, including boxing or unboxing.

• for-qualified method declarations

The for keyword on a method declaration, meant to introduce a control abstraction method that works like a loop, is now treated syntactically like a modifier rather than appearing immediately before the method name. This helps make the declaration site more similar to the use site.

• Added support for method references

We added extensive support for treating a reference to a method as a closure using a newly introduced token #. The syntax is borrowed from the FCM proposal. The semantics are as follows:

A method reference written as

Primary # Identifier ( TypeList )

where the Primary designates an expression (as opposed to a type) is treated the same as a closure

{ Type x0, Type x1 ... => tmp.Identifier(x0, x1 ...) }

or

{ Type x0, Type x1 ... => tmp.Identifier(x0, x1 ...); }

Where tmp is a temporary value that holds the computed value of the primary expression. The former translation is used when the resolved method has a non-void return type, while the latter is used when the resolved method has a void return type.

If the primary resolves to a type, then this is translated to

{ Type x0, Type x1 ... => Primary.Identifier(x0, x1 ...) }

or

{ Type x0, Type x1 ... => Primary.Identifier(x0, x1 ...); }

when the resolved method is static, or

{ Primary x, Type x0, Type x1 ... => x.Identifier(x0, x1 ...) }

or

{ Primary x, Type x0, Type x1 ... => x.Identifier(x0, x1 ...); }

when the resolved method is an instance method.

In addition, optional explicit type arguments, between angle brackets, may be placed immediately after the # token. These are used directly in the translated method invocation to resolve the method to be invoked.

• Implemented a classfile format for the for qualifier

We've impleemnted a class file representation of the for qualifier to support separate compilation.

Closures: Control Abstraction, Method References, Puzzler Solution

The Java Closures prototype now supports control abstraction and implements restricted closures and function types. The syntax has changed slightly. Also, as hinted in the draft JSR proposal, there is now support for eta abstraction, which is called method reference in Stephen Colebourne's FCM proposal. We haven't updated the specification, so this will serve as a brief tutorial on the changes until we do. I don't know if this will be the syntax we will end up with, but it will do for now. Finally, we look at solutions to the closure puzzler in my previous post.

Control Abstraction

The first thing you'll notice when using the new prototype is that the compiler gives a warning when a closure uses a local variable from an enclosing scope:

Example.java:4: warning: [shared] captured variable i not annotated @Shared
        Runnable r = { => System.out.println(i); };
                                             ^

There are a few ways to make this warning go away:

• declare the variable final; or
• annotate the variable @Shared; or
• make sure the variable is not the target of any assignment expression; or
• put @SuppressWarnings("shared") on an enclosing method or class; or
• use an unrestricted closure, by using the ==> token instead of the => token (when possible).

The => token builds a restricted closure that triggers this warning. Restricted closures also do not allow a break or continue statement to a target outside the closure, nor a return statement from the enclosing method. You will rarely want to write an unrestricted closure; many (but not all) of the things you need to do with an unrestricted closure can be expressed more clearly with a control invocation statement instead.

You're not allowed to assign an unrestricted closure to a restricted interface. A number of existing JDK interfaces, such as java.lang.Runnable, have been modified to be restricted.

Error: cannot assign an unrestricted closure to a restricted interface type
        Runnable r = { ==> System.out.println(i); };
                       ^

In the less common case that you're writing a method intended to be used as a control API, you can write a function type with the (new) ==> token to designate an unrestricted function (interface) type. Let's do that to write a method, with, that will automatically close a stream for us. The idea is to be able to replace this code

FileInputStream input = new FileInputStream(fileName);
try {
    // use input
} finally {
    try {
        input.close();
    } catch (IOException ex) {
        logger.log(Level.SEVERE, ex.getMessage(), ex);
    }
}

with this

with (FileInputStream input : new FileInputStream(fileName)) {
    // use input
}

which is an invocation of the following method

public static void with(FileInputStream t, {FileInputStream==>void} block) {
    try {
        block.invoke(t);
    } finally {
        try {
            t.close();
        } catch (IOException ex) {
            logger.log(Level.SEVERE, ex.getMessage(), ex);
        }
    }
}

This is among the simplest control APIs, but it has some limitations:

• It works with the type FileInputStream, but not any other Closeable types
• It does not support exception transparency
• It does not support completion transparency

Completing the API by repairing these defects is left as an exercise to the reader. A solution will be discussed in my JavaOne talk Closures Cookbook.

Method References

A natural companion to closures is a way to refer to an existing method instead of writing a closure that accepts the same arguments and just invokes the method. This is sometimes known as eta abstraction or method references. We expect closures in their final form to include support for this convenient feature, which is why it is called out in the draft JSR proposal. The latest version of the prototype supports this, with a syntax based on javadoc conventions. Here are a few examples:

{ int => Integer } integerValue = Integer#valueOf(int);
{ Integer => String } integerString = Integer#toString();
{ int, int => int } min = Math#min(int, int);
{ String => void } println = System.out#println(String);
{ => String } three = new Integer(3)#toString();
{ Collection<String> => String } max = Collections#max(Collection<String>);
{ => Collection<String> } makeEmpty = Collections#<String>emptySet();
Runnable printEmptyLine = System.out#println();

Writing code as a method is sometimes more convenient than writing it as a closure:

void doTask() {
    // a complex task to be done in the background
}


Executor ex = ...;
ex.execute(this#doTask());

Puzzler Solution

A couple of weeks ago we looked at a Java puzzler involving closures, and a number of people discussed the underlying issue. My favorite is David's post "Color-flavor locking breaks chiral symmetry". Lessons include not exposing public fields (accessors are better) and being careful to avoid cyclic initialization dependencies.

The enum language feature provides support for one solution to the puzzle: specialize each instance of the enums.

import java.util.*;

enum Color {
    BROWN {
        public Flavor flavor() {
            return Flavor.CHOCOLATE;
        }
    },
    RED {
        public Flavor flavor() {
            return Flavor.STRAWBERRY;
        }
    },
    WHITE {
        public Flavor flavor() {
            return Flavor.VANILLA;
        }
    };
    abstract Flavor flavor();
}

enum Flavor {
    CHOCOLATE {
        public Color color() {
            return Color.BROWN;
        }
    },
    STRAWBERRY {
        public Color color() {
            return Color.RED;
        }
    },
    VANILLA {
        public Color color() {
            return Color.WHITE;
        }
    };
    abstract Color color();

}

class Neapolitan {

    static <T,U> List<U> map(List<T> list, {T=>U} transform) {
        List<U> result = new ArrayList<U>(list.size());
        for (T t : list) {
            result.add(transform.invoke(t));
        }
        return result;
    }

    public static void main(String[] args) {
        List<Color> colors = map(Arrays.asList(Flavor.values()), { Flavor f => f.color() });
        System.out.println(colors.equals(Arrays.asList(Color.values())));

        List<Flavor> flavors = map(Arrays.asList(Color.values()), { Color c => c.flavor() });
        System.out.println(flavors.equals(Arrays.asList(Flavor.values())));
    }
}

Another elegant solution, due to 5er_levart, uses closures:

enum Color {
    BROWN({=>Flavor.CHOCOLATE}),
    RED({=>Flavor.STRAWBERRY}),
    WHITE({=>Flavor.VANILLA});

    private final {=>Flavor} flavor;

    public Flavor flavor() { return flavor.invoke(); }

    Color({=>Flavor} flavor) {
        this.flavor = flavor;
    }
}

enum Flavor {
    CHOCOLATE({=>Color.BROWN}),
    STRAWBERRY({=>Color.RED}),
    VANILLA({=>Color.WHITE});

    private final {=>Color} color;

    public Color color() { return color.invoke(); }

    Flavor({=>Color} color) {
        this.color = color;
    }
}

In both solutions the idea is to compute the value lazily, a key technique to break dependency cycles.

Closures Puzzler: Neapolitan Ice Cream

People experience the world in different ways, using different senses. Some people view the world primarily through sight. Do you see what I mean? Some through sound. Do you hear me? Some experience the world only after thought and reflection. Do you know what I mean? Some point out "smells" in code as a way of criticizing its design. This puzzle explores the question of whether people experiencing the same thing through different senses are really experiencing the same thing at all. What is the program's output, and why?

This puzzle uses Java Closures, so if you want to try it you should download the prototype here. You can download the puzzle sources here. Comments on this blog post will not be posted to avoid spoiling the puzzle.

import java.util.*;

enum Color {
    BROWN(Flavor.CHOCOLATE),
    RED(Flavor.STRAWBERRY),
    WHITE(Flavor.VANILLA);
    final Flavor flavor;
    Color(Flavor flavor) {
        this.flavor = flavor;
    }
}

enum Flavor {
    CHOCOLATE(Color.BROWN),
    STRAWBERRY(Color.RED),
    VANILLA(Color.WHITE);
    final Color color;
    Flavor(Color color) {
        this.color = color;
    }
}

class Neapolitan {

    static <T,U> List<U> map(List<T> list, {T=>U} transform) {
        List<U> result = new ArrayList<U>(list.size());
        for (T t : list) {
            result.add(transform.invoke(t));
        }
        return result;
    }

    public static void main(String[] args) {
        List<Color> colors = map(Arrays.asList(Flavor.values()), { Flavor f => f.color });
        System.out.println(colors.equals(Arrays.asList(Color.values())));

        List<Flavor> flavors = map(Arrays.asList(Color.values()), { Color c => c.flavor });
        System.out.println(flavors.equals(Arrays.asList(Flavor.values())));
    }
}

Is the Java Language Dying?

We tend to think of programming languages in two categories: "living languages" in which we should seriously consider developing new code, and "legacy languages" that we mainly use, if at all, because we have to maintain an existing code base. The act of classifying a language into one or the other category helps us decide what, if anything, we might consider doing to change the language. If a language is primarily a legacy language, changes should be aimed at making it easier to maintain and modify existing bodies of code. A living language, on the other hand, also benefits from changes that make it easier to design, develop, and maintain new code. Living languages evolve to reduce accidental complexity.

"What does a high-level language accomplish? It frees a program from much of its accidental complexity. An abstract program consists of conceptual constructs: operations, datatypes, sequences, and communication. The concrete machine program is concerned with bits, registers, conditions, branches, channels, disks, and such. To the extent that the high-level language embodies the constructs wanted in the abstract program and avoids all lower ones, it eliminates a whole level of complexity that was never inherent in the program at all." -No Silver Bullet - Fred Brooks

Programs written in legacy languages tend to exhibit a high degree of accidental complexity [Code's Worst Enemy, Steve Yegge] [Mr. Yegge meets Mr. Brooks, Brian C Cunningham]. Early in the life of a language, the complexity of programs written in that language may appear to be essential, but as we learn more about software engineering and programming languages, we find patterns of complexity appearing in the code that can be eliminated by improved languages.

A good example of this is garbage collection. In C and C++, memory management is a pervasive concern. Smart pointers and destructors help, but they do not significantly reduce the complexity of memory management. In languages with garbage collection, most of the complexity of memory management is assumed by the implementation of the language. Most languages that have been introduced in the past ten years support garbage collection.

Another example is concurrency. The threads-and-locks-and-semaphores primitives of Java enable parallel programming, but require that programmers express concurrency at a fairly low level. This has been "good enough" for some time, as most programs are not deployed on highly concurrent hardware. But that is changing [The Free Lunch Is Over, Herb Sutter]. Libraries such as java.util.concurrent and Doug Lea's fork-join framework help somewhat, but in many cases they introduce complexities of their own. Other languages that support closures, such as Scala make fork-join-like libraries much easier to use. Scala supports control abstraction [Crowl and LeBlanc], which allows the libraries to manage much of the complexity associated with concurrency [Debasish Ghosh]. Support for the Actors model [Haller and Odersky], for example, can be expressed cleanly as a library in Scala

Besides raising the level of abstraction of concurrent code, control abstraction also raises the level of abstraction for sequential code by eliminating whole categories of boilerplate, which can instead be moved into common library code. This kind of boilerplate cannot be significantly reduced by adding one or two custom statements to the language, because such built-in forms necessarily make assumptions about the use cases that narrow their applicability. For example, ARM blocks don't document how they handle exceptions arising from the close() method. One example in the proposal suggests they are silently swallowed at runtime, which may work for many cases involving I/O streams, but another example given is a transactional API, in which ignoring such exceptions is precisely wrong. Without a specification for the syntax and semantics, the reader is welcome to imagine the most favorable treatment of each use case. But an attempt to reconcile these and other conflicting requirements may show the approach cannot be salvaged. Perhaps that is why no progress has been made since mid 2006.

What about Java? Is it a living language, or a legacy language like Cobol? This question underlies much of the debate about how to move the Java programming language forward, if at all. Carl Quinn asked at the December 14, 2007, JavaPolis Future of Computing Panel (to be published on http://www.parleys.com): "How can we address the issue of evolving the [Java] platform, language, and libraries without breaking things?"

Neal Gafter: "If you don't want to change the meaning of anything ever, you have no choice but to not do anything. The trick is to minimize the effect of the changes while enabling as much as possible. I think there's still a lot of room for adding functionality without breaking existing stuff..."

Josh Bloch: "My view of what really happens is a little bit morbid. I think that languages and platforms age by getting larger and clunkier until they fall over of their own weight and die very very slowly, like over ... well, they're all still alive (though not many are programming Cobol anymore). I think it's a great thing, I really love it. I think it's marvelous. It's the cycle of birth, and growth, and death. I remember James saying to me [...] eight years ago 'It's really great when you get to hit the reset button every once and a while.'"

Josh may well be right. If so, we should place Java on life support and move our development to new languages such as Scala. The fork-join framework itself is an example of higher-order functional programming, which Josh argues is a style that we should neither encourage nor support in Java. Is it really time to move on?

Personally, I believe rumors of Java's demise are greatly exaggerated. We should think of Java as a living language, and strive to eliminate much of the accidental complexity of Java programs. I believe it is worth adding support for closures and control abstraction, to reduce such complexity of both the sequential and concurrent aspects of our programs. At the same time, for completely new code bases, we should also consider (and continue to develop) newer languages such as Scala, which benefit from the lessons of Java.