JSR Proposal: Closures for Java

Summary: Add support so programs can operate on an arbitrary block of code with parameters, and simplify the use of methods that receive such blocks.

Section 1: Identification

Identification (JCP Member submitting this proposal): TBD
Name of Contact Person: TBD
E-Mail Address: TBD
Telephone: TBD
Fax: TBD

Initial Group Membership:
(Please provide company or organization names. Note that expert group members must have signed the JSPA.)
Neal Gafter, spec lead
Osvaldo Pinali Doederlein (SouJava)
David Flanagan
Larry Cable (BEA)
Doug Lea
Terence Parr
JetBrains

Supporting this JSR (alphabetically):
[Note: Google has not committed to support a Closures JSR]
Peter von der Ahé
BEA
Borland
Gilad Bracha
Larry Cable
Bruce Chapman
Ricky Clarkson
Stephen Colebourne
David Flanagan
Neal Gafter
Cay Horstmann
JetBrains
Doug Lea
Bob Lee
Martin Odersky
John Rose
Stefan Schulz
SouJava
Hani Suleiman
Joe Walker (DWR)

Section 2: Request

2.1 Please describe the proposed Specification:

This JSR provides support for operating on an arbitrary "block of Java code", or body, which is either a statement list, an expression, or a combination of both. We call the mechanism a closure expression. Wrapping statements or an expression in a closure expression does not change their meaning, but merely defers their execution. Evaluating a closure expression produces a closure object. The closure object can later be invoked, which results in execution of the body, yielding the value of the expression (if one was present) to the invoker. A closure expression can have parameters, which act as variables whose scope is the body. In this case the invoker of the closure object must provide compatible arguments, which become the values for the parameters.

In addition, this JSR may support a new invocation statement for methods that accept a closure to simplify their use in common cases.

2.2 What is the target Java platform? (i.e., desktop, server, personal, embedded, card, etc.):

The results of this JSR will be suitable for inclusion in the Java SE and EE platform specifications.

2.3 The Executive Committees would like to ensure JSR submitters think about how their proposed technology relates to all of the Java platform editions. Please provide details here for which platform editions are being targeted by this JSR, and how this JSR has considered the relationship with the other platform editions:

We will detail JLS and possibly JVM specification changes. As such, the changes can be implemented in a wide variety of environments.

2.4 Should this JSR be voted on by both Executive Committees?

No.

2.5 What need of the Java community will be addressed by the proposed specification?

The specification will improve the Java Programming Language by:

simplifying many kinds of programs that currently rely on anonymous class instance creation expressions;
enabling powerful libraries that can help solve common problems such as resource termination;
simplifying the use of the concurrency framework in java.util.concurrent;
supporting a new kind of concurrency framework for aggregate operations; and
enabling future API design to replace language design for extending the Java platform.

a. Closures can be used to simplify the clients of many existing APIs that are currently invoked by the creation of anonymous class instance creation expressions (JLS 15.9.5). Examples include the appearance of callbacks in various APIs such as javax.swing, and asynchronous tasks such as java.lang.Runnable and java.util.concurrent.Callable. Closures eliminate many of the pain points that are associated with anonymous inner classes:

an excessively verbose syntax;
restrictions on the use of local variables;
confusing and potentially dangerous scoping rules;
the inability to use them with flexible exception checking;
inflexible subtyping; and
restrictions on control-flow interactions with enclosing constructs.

b. Closures enable the development of powerful libraries that can help simplify the code required to solve common problems such as resource termination. For example, one pain point in Java is the verbosity and awkwardness of resource termination, such as unlocking a lock or closing a stream. With the addition of closures, methods can be defined to eliminate these pain points. These Control APIs are methods in which controlled statements are received as closures. The definition and use of such APIs may be improved by the addition of a shorthand invocation statement syntax in which a controlled statement appears last. The following are a number of specific methods that the expert group may consider adding to the JDK. These statements illustrate one possible usage syntax that is to be considered a tentative placeholder; the specific syntax and method names is subject to consensus of the expert group:

withLock(expression) statement
Analogous to the built-in synchronized statement, this method invocation executes the controlled statement while holding the java.util.concurrent.locks.Lock provided by the expression.
ex.execute() statement
Executes the controlled statement under control of the java.util.concurrent.Executor ex.
with (T t : expression) statement
For type T which is a subtype of Closable, and an expression of that type, executes the controlled statement in the scope of the variable t and then closes t. Similar APIs will be specified for interfaces in java.sql such as Statement and Connection.
for eachEntry(K k, V v : map) statement
Executes the controlled statement once for each mapping k,v in in the given map expression; within this loop break and continue statements behave analogously to the way they operate in language-provided loop statements.
AccessController.doPrivileged() statement
Executes the controlled statement as a privileged activity under control of java.security.AccessController.

c. Closures will improve the use of Java for concurrent programming using the java.util.concurrent framework, by allowing programs to create instances of interfaces such as java.lang.Runnable and java.util.concurrent.Callable, thus simplifying many applications that rely on existing concurrency APIs. Extensions of the existing concurrency APIs to take advantage of closures, such as the previously mentioned withLock method, will make some kinds of concurrent programming much simpler. For another example, we will specify APIs that enable concurrent loops to be written as easily as if concurrent loop statements had been added to the language.

d. A style of programming using aggregate operations will be convenient enough to become practical with the presence of closures in the language. Aggregate operations typically appear in functional programming languages, and involve the application of some operation to every member of a collection. Aggregate operations include map, reduce, fold, and filter. These typically support concurrency because many of the component operations may take place in parallel. Such applications often require no additional explicit synchronization by clients beyond the implicit fork and join of the aggregate operations, and this fact enables a more efficient implementation than with a more general concurrency framework. The expert group may specify a set of concurrent aggregate APIs that integrate smoothly with closures.

e. The addition of closures simplifies the evolution of the Java platform. Many existing language RFEs in Sun's public bug database can retargeted as API requests for methods that receive closures. Many future needs for additional statement forms can instead be met by the addition of library methods.

2.6 Why isn't this need met by existing specifications?

Closures have been shown to be a very powerful abstraction mechanism, enhancing the power of a programming language in ways not available to Java programmers today:

Closures can be used for factoring out repeated or nearly-identical code that appears as boilerplate in languages without closures. See, for example, http://gafter.blogspot.com/2007/03/closures-for-organizing-your-code.html, http://www.joelonsoftware.com/items/2006/08/01.html, http://ivan.truemesh.com/archives/000637.html, http://www.talios.com/dear_java_i_need_closure.htm, and http://blog.moertel.com/articles/2005/08/30/closures-and-the-professional-programmer.
Closures can be used to define methods that behave as control constructs in the language, as described in: Guy Lewis Steele Jr. and Gerald Jay Sussman, LAMBDA the Ultimate Imperative, Massachusetts Institute of Technology Artificial Intelligence Laboratory Memo No. 353, http://dspace.mit.edu/bitstream/1721.1/5790/2/AIM-353.pdf. Many object-oriented languages provide full support for closures in the language, enabling very flexible definition of such control APIs. See, for example, Smalltalk http://en.wikipedia.org/wiki/Smalltalk#Code_blocks, Ruby http://www.artima.com/intv/closures.html and Scala http://www.scala-lang.org/.
An analysis and metric for the kind of expressive power closures add to Java is described in: Matthias Felleisen, On the Expressive Power of Programming Languages, 3rd European Symposium on Programming, Copenhagen, Denmark, 1990, http://citeseer.ist.psu.edu/felleisen90expressive.html. See also http://gafter.blogspot.com/2007/03/on-expressive-power-of-programming.html.

When closures are integrated into an object-oriented type system with subtyping, the types used to represent them obey a relationship that enables them to be used quite flexibly. See, for example, Scala http://www.scala-lang.org/, section 3.2.9. This enables a flexibility available in untyped languages such as Scheme, Ruby, and Smalltalk and which is achieved only with extreme awkwardness in Java today. This does not require an extension to the type system, as Java's generics and wildcards already obey the necessary typing relationships.

2.7 Please give a short description of the underlying technology or technologies:

Because this is a proposed extension to the Java Programming Language, the primary base document is the Java Language Specification. For any required VM changes the base document is the Java VM Specification. In both cases, we will specify the required changes as addenda to these base documents sutiable for integration in a new edition. Changes to existing J2SE APIs will be based on the JDK7 platform specification. The prototype will be based on the openjdk platform implementation.

A description of the problem space to be addressed by closures appears at http://gafter.blogspot.com/2007/01/definition-of-closures.html. The base specification for new features is an existing proposal by Gilad Bracha, Neal Gafter, James Gosling, and Peter von der Ahé entitled Closures for Java (v0.5) (http://www.javac.info). An early prototype is available at http://www.javac.info/closures.tgz. The expert group may draw from these in developing a specification and reference implementation for this JSR. No draft TCK is publicly available at the time of this writing.

2.8 Is there a proposed package name for the API Specification?

The primary changes will be to the language; any classes and interfaces added for language support will likely appear in java.lang and its subpackages. Additional APIs using these constructs are likely to be specified in java.util, java.util.concurrent, and possibly java.io.

2.9 Does the proposed specification have any dependencies on specific operating systems, CPUs, or I/O devices that you know of?

No.

2.10 Are there any security issues that cannot be addressed by the current security model?

No. It is possible that as a result of the proposal some existing security APIs may be significantly easier to use.

2.11 Are there any internationalization or localization issues?

Trivial. Some new compiler error messages might need to be issued in different languages.

2.12 Are there any existing specifications that might be rendered obsolete, deprecated, or in need of revision as a result of this work?

The Java Language Specification is likely to require an addendum or revision if the results of this JSR are integrated into a platform specification. Additional minor changes may be required in the Java VM specification.

2.13 Please describe the anticipated schedule for the development of this specification:

The JSR will aim to release an EDR (early draft release) within the first six months and reach a maturity level appropriate for inclusion in a Java SE platform release one year after that. We anticipate completing this work in time for inclusion into Java SE Release 8.

2.14 Please describe the anticipated working model for the Expert Group working on developing this specification.:

Most interactions will take place via email over a publicly-readable list. Teleconferences and in-person meetings will be held on an as-needed basis.

2.15 It is important to the success of the community and each JSR that the work of the Expert Group be handled in a manner which provides the community and the public with insight into the work the Expert Group is doing, and the decisions that the Expert Group has made. The Executive Committees would like to ensure Spec Leads understand the value of this transparency and ask that each JSR have an operating plan in place for how their JSR will address the involvement of the community and the public. Please provide your plan here, and refer to the Spec Lead Guide for a more detailed description and a set of example questions you may wish to answer in your plan.

See 2.14. We will also maintain a separate open mailing list for discussions by other interested people outside the expert group. Drafts of proposed specification changes required to incorporate these features into the platform will be published, under the Gnu Free Documentation License version 1.2, for public review. Drafts of proposed TCK addenda required to incorporate these features into the platform will be published, under GPLv2, for public review. We will provide prototypes (as early as possible), distributed under GPLv2, reflecting the sense of the expert group, so that the community can try the proposed language constructs in realistic settings.

We will work with IDE vendors to demonstrate support in IDEs. The IDE support will not be subject to specification or TCK, but rather to enable community members to try the language features using their preferred tools, and to demonstrate that it is feasible to integrate the features in existing IDEs.

2.16 Please describe how the RI and TCK will be delivered, i.e. as part of a profile or platform edition, or stand-alone, or both. Include version information for the profile or platform in your answer.:

This JSR proposes changes to the Java language and thus can only be delivered in a Java SE platform release.

2.17 Please state the rationale if previous versions are available stand-alone and you are now proposing in 2.16 to only deliver RI and TCK as part of a profile or platform edition (See sections 1.1.5 and 1.1.6 of the JCP 2 document).:

This is a new specification.

2.18 Please provide a description of the business terms for the Specification, RI and TCK that will apply when this JSR is final.:

The Specification, Reference Implementation, and TCK will be made available as part of some future Java SE platform release. This JSR therefore will not be licensed separately and does not have its own separate business terms.

Section 3: Contributions

3.1 Please list any existing documents, specifications, or implementations that describe the technology. Please include links to the documents if they are publicly available:

See section 2.7, above.

3.2 Explanation of how these items might be used as a starting point for the work.

The existing Java platform documents provide a starting point, for which the expert group will specify the addition of features to satisfy the requirements of this JSR. The base Closures specification is one approach to providing the desired features. The expert group may draw from that base specification and others.

The expert group may additionally consider extensions that provide support for currying, and specifically the special case of treating bound method references as closures. For example, see the "eta expansion" of Scala (http://www.scala-lang.org/docu/files/ScalaReference.pdf, 6.25.5), and "invocable method references" in First-class methods: Java-style closures (version 0.5) by Stephen Colebourne and Stefan Schulz (http://docs.google.com/Doc?id=ddhp95vd_6hg3qhc).

The expert group may consider additional extensions that improve compile-time checking of the use of closures within the language. For example, concurrent aggregate operations are more easily parallelized when the constituent operations satisfy certain conditions such as being wait-free, free of side-effects, or free of interference. The expert group may consider marker interfaces or annotations, and related language constraints or annotation processors, that help check or enforce these constraints. For example, the base specification proposes the marker interface RestrictedFunction that enforces constraints mirroring those of Java's existing inner classes.

Additional extensions may be necessary in Java's reflection system, in JSR 269's annotation processor API, and/or in JNI to reflect the language constructs added by this JSR.

Section 4: Additional Information

4.1 This section contains any additional information that the submitting Member wishes to include in the JSR.

n/a