Widespread Architectural Change Part 1

Last month I introduced Four Categories of Architectural Refactoring:

• Substitute Architectural Decision
• Refactor Architectural Structure
• Widespread Architectural Change
• Other Architectural Changes

Today I want to start discussing options to perform Widespread Architectural Changes. This category contains all kind of small changes repeated throughout the whole code base. The number of these changes is high, being hundreds or sometimes even thousands of occurrences that have to be changed in a similar way. Examples of such modifications are:

• Migrating or upgrading (similar) APIs, frameworks or libraries
• Changing or unifying coding conventions
• Fixing compiler warnings and removing technical debt
• Consistently applying or changing aspects like logging or security
• Applying internationalization
• Migrate between languages, e.g. SQL vs. HQL

All of them do not change the overall structure, they just work on the implementation. Often substituting architectural decisions or altering architectural structure contain similar changes.

Options
I have been using some techniques since 2004 because as Code Cop I value code consistency. Two years ago I had the opportunity to discuss the topic with Alexandru Bolboaca and other experienced developers during a small unconference and we came up with even more options, as shown in the picture on the right. The goal of this and the next articles is to introduce you to these options, the power of each one and the cost of using it. A word of warning: This is a raw list. I have used some but not all of them and I have yet to explore many options in more detail.

Supporting Manual Changes With Fast Navigation
The main challenge of widespread changes is the high number of occurrences. If the change itself is small, finding all occurrences and navigating to them is a major effort. Support for fast navigation would make things easier. The most basic form of this is to modify or delete something and see all resulting compile errors. Of course this only works with static languages. In Eclipse this works very well because Eclipse is compiling the code all the time and you get red markers just in time. Often it is possible to create a list of all lines that need to be changed. In the past have created custom rules of static analysis tools like PMD to find all places I needed to change. Such a list can be used to open the file and jump to the proper line, one source file after another.

As example, here is a Ruby script that converts pmd.xml which is contains the PMD violations from the Apache Maven PMD Plugin to Java stack traces suitable for Eclipse.

#! ruby
require 'rexml/document'

xml_doc = REXML::Document.new(File.new('./target/site/pmd.xml'))
xml_doc.elements.each('pmd/file/violation') do |violation|
  if violation.attributes['class']
    class_name = violation.attributes['package'] + '.' +
                 violation.attributes['class']
    short_name = violation.attributes['class'].sub(/\$.*$/, '')
    line_number = violation.attributes['beginline']
    rule = violation.attributes['rule']

    puts "#{class_name}.m(#{short_name}.java:#{line_number}) #{rule}"
  end
end

The script uses an XML parser to extract class names and line numbers from the violation report. After pasting the output into Eclipse's Stacktrace Console, you can click each line one by one and Eclipse opens each file in the editor with the cursor in the proper line. This is pretty neat. Another way is to script Vim to open each file and navigate to the proper line using vi +<line number> <file name>. Many editors support similar navigation with the pattern <file name>:<line number>.

Enabling fast navigation is a big help. The power this option is high (that is 4 out of 5 on my personal, totally subjective scale) and the effort to find the needed lines and script them might be medium.

Search and Replace (Across File System)
The most straight forward way to change similar code is by Search and Replace. Many tools offer to search and replace across the whole project, allowing to change many files at once. Even converting basic scenarios, which sometimes cover up to 80%, helps a lot. Unfortunately basic search is very limited. I rate its power low and the effort to use it is also low.

Scripted Search and Replace using Regular Expressions
Now Regular Expressions are much more powerful than basic search. Many editors allow Regular Expressions in Search and Replace. I recommend creating a little script. The traditional approach would be Bash with sed and awk but I have used Ruby and Python (or even Perl) to automate lots of different changes. While the script adds extra work to traverse all the source directories and files, the extra flexibility is needed for conditional logic, e.g. adding an import to a new class if it was not imported before. Also in a script Regular Expressions can be nested, i.e. analysing the match of an expression further in a second step. This helps to keep the expressions simple.

For example I used a script to migrate Java's clone() methods from version 1.4 to 5. Java 5 offers covariant return types which can be used for clone(), removing the cast from client code. The following Ruby snippet is called with the name of the class and the full Java source as string:

shortName = shortClassNameFromClassName(className)
if source =~ / Object clone\(\)/
  # use covariant return type for method signature
  source = $` + " #{shortName} clone()" + $'
end
if source =~ /return super\.clone\(\);/
  # add cast to make code compile again
  source = $` + "return (#{shortName}) super.clone();" + $'
end

In the code base where I applied this widespread change, it fixed 90% of all occurrences as clone methods did not do anything else. It also created some broken code which I reverted. I always review automated changes, even large numbers, as jumping from diff to diff is pretty fast with modern tooling. Scripts using Regular Expressions helped me a lot in the past and I rate their power to high. Creating them is some effort, e.g. a medium amount of work.

Macros and Scripts
Many tools like Vim, Emacs, Visual Studio, Notepad++ and IntelliJ IDEA allow creation or recording Keyboard and or mouse macros or Application scripts. With them, frequently used or repetitive sequences of keystrokes and mouse movements can be automated, and that is exactly what we want to do. When using Macros the approach is the opposite as for navigation markers: We find the place of the needed change manually and let the Macro do its magic.

Most people I have talked to know Macros and used them earlier (e.g. 20 years ago) but not recently in modern IDEs. Alex has used Vim Scripts to automate repetitive coding tasks. I have not used it and relating to his experience. Here is a Vim script function which extracts a variable, taken from Gary Bernhardt's dotfiles:

function! ExtractVariable()
  let name = input("Variable name: ")
  if name == ''
    return
  endif

  " Enter visual mode
  normal! gv
  " Replace selected text with the variable name
  exec "normal c" . name
  " Define the variable on the line above
  exec "normal! O" . name . " = "
  " Paste the original selected text to be the variable value
  normal! $p
endfunction

I guess large macros will not be very readable and hard to change but they will be able to do everything what Vim can do, which is everything. ;-) They are powerful and easy to create - as soon as you know Vim script of course.

Structural Search and Replace
IntelliJ IDEA offers Structural Search and Replace which performs search and replace across the whole project, taking advantage of IntelliJ IDEA's awareness of the syntax and code structure of the supported languages.. In other words it is Search and Replace on the Abstract Syntax Tree (AST). Additionally it is possible to apply semantic conditions to the search, for example locate the symbols that are read or written to. It is available for Java and C# (Resharper) and probably in other JetBrains products as well. I have not used it. People who use it tell me that it is useful and easy to use - or maybe not that easy to use. Some people say it is too complicated and they can not make it work.

Online help says that you can apply constraints described as Groovy scripts and make use of IntelliJ IDEA PSI (Program Structure Interface) API for the used programming language. As PSI is the IntelliJ version of the AST, this approach is very powerful but you need to work the PSI/AST which is (by its nature) complicated. This requires a higher effort.

To be continued
And there are many more options to be explored, e.g. scripting code changes inside IDEs or using Refactoring APIs as well as advanced tools outside of IDEs. I will continue my list in the next part. Stay tuned.

Complete Cofoja Setup Example

Design by Contract
Have you heard of Design by Contract (short DbC)? If not, here is a good introduction from Eiffel. (In short, Design by Contract is one of the major mechanisms to ensure the reliability of object-oriented software. It focuses on the communication between components and requires the interactions to be defined precisely. These specifications are called contracts and they contain Preconditions, Postconditions and Invariants. Unlike using assertions to ensure these conditions, DbC considers the contracts important parts of the design process which should be written first. It is a systematic approach to building bug-free object-oriented systems and helps in testing and debugging.)

Cofoja (Contracts for Java)
Cofoja is a Design by Contract library for Java. It uses annotation processing and byte code instrumentation to provide run-time checking. It supports a contract model similar to that of Eiffel, with added support for a few Java-specific things, such as exceptions. In Cofoja, contracts are written as Java code within quoted strings, embedded in annotations. Here is some sample code (derived from lost icontract library): A basic stack with methods to push, pop and to see the top element.

import java.util.LinkedList;
import com.google.java.contract.Ensures;
import com.google.java.contract.Invariant;
import com.google.java.contract.Requires;

@Invariant({ "elements != null",
             "isEmpty() || top() != null" }) // (1)
public class CofojaStack<T> {

  private final LinkedList<T> elements = new LinkedList<T>();

  @Requires("o != null") // (2)
  @Ensures({ "!isEmpty()", "top() == o" }) // (3)
  public void push(T o) {
    elements.add(o);
  }

  @Requires("!isEmpty()")
  @Ensures({ "result == old(top())", "result != null" })
  public T pop() {
    final T popped = top();
    elements.removeLast();
    return popped;
  }

  @Requires("!isEmpty()")
  @Ensures("result != null")
  public T top() {
    return elements.getLast();
  }

  public boolean isEmpty() {
    return elements.isEmpty();
  }

}

The annotations describe method preconditions (2), postconditions (3) and class invariants (1). Cofoja uses a Java 6 annotation processor to create .contract class files for the contracts. As soon as Cofoja's Jar is on the classpath the annotation processor is picked up by the service provider. There is no special work necessary.

javac -cp lib/cofoja.asm-1.3-20160207.jar -d classes src/*.java

To verify that the contracts are executed, here is some code which breaks the precondition of our stack:

import org.junit.Test;
import com.google.java.contract.PreconditionError;

public class CofojaStackTest {

  @Test(expected = PreconditionError.class)
  public void emptyStackFailsPreconditionOnPop() {
    CofojaStack<String> stack = new CofojaStack<String>();
    stack.pop(); // (4)
  }

}

We expect line (4) to throw Cofoja's PreconditionError instead of NoSuchElementException. Just running the code is not enough, Cofoja uses a Java instrumentation agent to weave in the contracts at runtime.

java -javaagent:lib/cofoja.asm-1.3-20160207.jar -cp classes ...

Cofoja is an interesting library and I wanted to use it to tighten my precondition checks. Unfortunately I had a lot of problems with the setup. Also I had never used annotation processors before. I compiled all my research into a complete setup example.

Maven
Someone already created an example setup for Maven. Here are the necessary pom.xml snippets to compile and run CofojaStackTest from above.

<dependencies>
  <dependency> <!-- (5) -->
    <groupId>org.huoc</groupId>
    <artifactId>cofoja</artifactId>
    <version>1.3.1</version>
  </dependency>
  ...
</dependencies>

<build>
  <plugins>
    <plugin> <!-- (6) -->
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.20</version>
      <configuration>
        <argLine>-ea</argLine>
        <argLine>-javaagent:${org.huoc:cofoja:jar}</argLine>
      </configuration>
    </plugin>
    <plugin> <!-- (7) -->
      <artifactId>maven-dependency-plugin</artifactId>
      <version>2.9</version>
      <executions>
        <execution>
          <id>define-dependencies-as-properties</id>
          <goals>
            <goal>properties</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Obviously we need to declare the dependency (5). All examples I found register the contracts' annotation processor with the maven-compiler-plugin, but that is not necessary if we are using the Maven defaults for source and output directories. To run the tests through the agent we need to enable the agent in the maven-surefire-plugin (6) like we did for plain execution with java. The Jar location is ${org.huoc:cofoja:jar}. To enable its resolution we need to run maven-dependency-plugin's properties goal (7). Cofoja is build using Java 6 and this setup works for Maven 2 and Maven 3.

Gradle
Similar to Maven, but usually shorter, we need to define the dependency to Cofoja (5) and specify the Java agent in the JVM argument during test execution (6). I did not find a standard way to resolve a dependency to its Jar file and several solutions are possible. The cleanest and shortest seems to be from Timur on StackOverflow, defining a dedicated configuration for Cofoja (7), which avoids duplicating the dependency in (5) and which we can use to access its files in (6).

configurations { // (7)
  cofoja
}

dependencies { // (5)
  cofoja group: 'org.huoc', name: 'cofoja', version: '1.3.1'
  compile configurations.cofoja.dependencies
  ...
}

test { // (6)
  jvmArgs '-ea', '-javaagent:' + configurations.cofoja.files[0]
}

Eclipse
Even when importing the Maven project into Eclipse, the annotation processor is not configured and we need to register it manually. Here is the Eclipse help how to do that. Fortunately there are several step by step guides how to set up Cofoja in Eclipse. In the project configuration, enable Annotation Processing under the Java Compiler settings.

Although Eclipse claims that source and classpath are passed to the processor, we need to configure source path, classpath and output directory.

com.google.java.contract.classoutput=%PROJECT.DIR%/target/classes
com.google.java.contract.classpath=%PROJECT.DIR%/lib/cofoja.asm-1.3-20160207.jar
com.google.java.contract.sourcepath=%PROJECT.DIR%/src/main/java

(These values are stored in .settings/org.eclipse.jdt.apt.core.prefs.) For Maven projects we can use the %M2_REPO% variable instead of %PROJECT.DIR%.

com.google.java.contract.classpath=%M2_REPO%/org/huoc/cofoja/1.3.1/cofoja-1.3.1.jar

Add the Cofoja Jar to the Factory Path as well.

Now Eclipse is able to compile our stack. To run the test we need to enable the agent.

IntelliJ IDEA
StackOverflow has the answer how to configure Cofoja in IntelliJ. Enable annotation processing in Settings > Build > Compiler > Annotation Processors.

Again we need to pass the arguments to the annotation processor.

com.google.java.contract.classoutput=$PROJECT_DIR$/target/classes
com.google.java.contract.classpath=$M2_REPO$/org/huoc/cofoja/1.3.1/cofoja-1.3.1.jar
com.google.java.contract.sourcepath=$PROJECT_DIR$/src/main/java

(These values are stored in .idea/compiler.xml.) For test runs we enable the agent.

That's it. See the complete example's source (zip) including all configuration files for Maven, Gradle, Eclipse and IntelliJ IDEA.

IDE Shortcut Flashcards

From time to time I need to look up certain keyboard shortcuts in IntelliJ IDEA or PyCharm. (This would not be necessary would I use my proper keyboard always.) While there are official productivity guides, I usually just google the shortcut. There are plenty of pages listing the basic and more useful ones, nothing special indeed. But the one Google showed me the last time was special: 69 things you should know about IntelliJ IDEA by Krzysztof Grajek.

The list of IntelliJ IDEA commands was as expected, but at the end Krzysztof had put something new: flashcards. Flashcards are a great way to learn short facts. While I am aware of flashcards, I have not thought about nor used them since many years. I got curious and immediately downloaded them.

The package was an apkg, a file format used by Anki, a free flashcard application. According to Wikipedia Anki is most feature complete and available on many platforms, including smart phones. The apkg file is just a zip including a SQLite database, so it is pretty light-weight.

Unfortunately Krzysztof's original deck was created for the Mac version of IntelliJ, so I had to translate and verify it. Some shortcuts did not map to Windows keys (or were not available any more), so my IntelliJ IDEA shortcuts for Windows flashcards just contain 59 cards. There were several keys I did not know and a few I had never heard about. I recommend you instal the mobile version of Anki and start learning more shortcuts today!

There is also a community sharing Anki decks and I immediately looked for Eclipse shortcuts. I found a large deck of 91 cards for Eclipse, which - again - was created for the Mac version of Eclipse. I translated and verified them as well, which was much easier because I have been using Eclipse for more than ten years. Here are the Eclipse shortcuts for Windows flashcards.

Download 59 IntelliJ Shortcuts Win.apkg here. Download 91 Eclipse Shortcuts Win.apkg here.

Finally a Proper Keyboard

At last GeeCON I met Hamlet D'Arcy and he distributed keyboard stickers with IntelliJ IDEA's keymap. I had them lying around for some time because I did not want to put them on my primary keyboard (because it does not have any labels). Last week I cleaned my old Silicon Graphics keyboard and boosted it with these stickers. Doesn't it look great?


I had to customize the stickers to my German keyboard. The Shift and Enter keys are shorter and I had to cut their stickers. Then I thought about changing the keymap inside IDEA and to put the stickers on the keys where they would be on an English keyboard. But I got confused, and in the end I put the stickers on their according places. I only changed the binding for [ to ö and ] to ä because 8 and 9 where [ and ] are on, were already taken.

equals and hashCode Generation

Recently we discussed equals() and hashCode() implementations. A proper implementation is not trivial, as "father" Bloch showed us years ago. Back in 2004 I wrote a plugin for Eclipse 2 to generate these methods. (Unfortunately I never managed to publish it. I know I am weak ;-) My home grown solution would produce something like

public int hashCode() {
  long bits;
  int result = 17;
  result = 37 * result + (aBoolean ? 1231 : 1237);
  result = 37 * result + (int)
    ((bits = Double.doubleToLongBits(aDouble)) ^ (bits >> 32));
  result = 37 * result + (int) (aLong ^ (aLong >> 32));
  result = 37 * result + anInt;
  if (anObject != null) {
    result = 37 * result + anObject.hashCode();
  }
  if (anArray != null) {
    result = 37 * result + anArray.hashCode();
  }
  return result;
}
public boolean equals(Object obj) {
  if (this == obj) {
    return true;
  }
  else if (obj == null || getClass() != obj.getClass()) {
    return false;
  }
  final Homegrown o = (Homegrown) obj;
  return (aBoolean == o.aBoolean &&
         aDouble == o.aDouble &&
         aLong == o.aLong &&
         anInt == o.anInt &&
         (anObject == o.anObject ||
           (anObject != null && anObject.equals(o.anObject))) &&
         (anArray == o.anArray ||
           (anArray != null && anArray.equals(o.anArray))));
}

I know, I know the implementation for arrays is most likely not what you want. (Did I say that I am weak? :-) Since Java 5 one could use the java.util.Arrays class to fix it. Nevertheless, it served me well for some years. There are several other plugins for Eclipse, and Scott McMaster wrote about in 2006. Since version 3.3 (Europa) Eclipse can finally do it on its own:

public int hashCode() {
  final int prime = 31;
  int result = 1;
  result = prime * result + (aBoolean ? 1231 : 1237);
  long temp;
  temp = Double.doubleToLongBits(aDouble);
  result = prime * result + (int) (temp ^ (temp >>> 32));
  result = prime * result + (int) (aLong ^ (aLong >>> 32));
  result = prime * result + anInt;
  result = prime * result +
                   ((anObject == null) ? 0 : anObject.hashCode());
  result = prime * result + Arrays.hashCode(anArray);
  return result;
}
public boolean equals(Object obj) {
  if (this == obj) {
    return true;
  }
  if (obj == null) {
    return false;
  }
  if (getClass() != obj.getClass()) {
    return false;
  }
  final Eclipse33Java5 other = (Eclipse33Java5) obj;
  if (aBoolean != other.aBoolean) {
    return false;
  }
  if (Double.doubleToLongBits(aDouble) !=
      Double.doubleToLongBits(other.aDouble)) {
    return false;
  }
  if (aLong != other.aLong) {
    return false;
  }
  if (anInt != other.anInt) {
    return false;
  }
  if (anObject == null) {
    if (other.anObject != null) {
      return false;
    }
  }
  else if (!anObject.equals(other.anObject)) {
    return false;
  }
  if (!Arrays.equals(anArray, other.anArray)) {
    return false;
  }
  return true;
}

Using the ternary operator the hashCode method gets quite compact, but equals is a bit too verbose for my liking. IntelliJ IDEA could always generate these methods. IDEA 7.0 creates something like

public int hashCode() {
  int result;
  long temp;
  result = (aBoolean ? 1 : 0);
  temp = aDouble != +0.0d ? Double.doubleToLongBits(aDouble) : 0L;
  result = 31 * result + (int) (temp ^ (temp >>> 32));
  result = 31 * result + (int) (aLong ^ (aLong >>> 32));
  result = 31 * result + anInt;
  result = 31 * result +
                (anObject != null ? anObject.hashCode() : 0);
  result = 31 * result +
                (anArray != null ? Arrays.hashCode(anArray) : 0);
  return result;
}
public boolean equals(Object o) {
  if (this == o) {
    return true;
  }
  if (o == null || getClass() != o.getClass()) {
    return false;
  }
  Idea70Java5 original = (Idea70Java5) o;
  if (aBoolean != original.aBoolean) {
    return false;
  }
  if (Double.compare(original.aDouble, aDouble) != 0) {
    return false;
  }
  if (aLong != original.aLong) {
    return false;
  }
  if (anInt != original.anInt) {
    return false;
  }
  if (anObject != null ? !anObject.equals(original.anObject) :
                         original.anObject != null) {
    return false;
  }
  // Probably incorrect - comparing Object[] with Arrays.equals
  if (!Arrays.equals(anArray, original.anArray)) {
    return false;
  }
  return true;
}

Typical IDEA, with a little fix for +0.0/-0.0 and some warning concerning Arrays.equals, but else totally the same. In fact, all these implementations suck (including my own, which sucks most). All these result = prime * result ... and if ... return false; are definitely not DRY. I always favour Apache Commons Lang builders. A hand-coded solution using them would look like

public int hashCode() {
  return new HashCodeBuilder().
         append(aBoolean).
         append(aDouble).
         append(aLong).
         append(anInt).
         append(anObject).toHashCode();
}
public boolean equals(Object other) {
  if (this == other) {
    return true;
  }
  if (other == null || getClass() != other.getClass()) {
    return false;
  }
  ApacheCommons o = (ApacheCommons) other;
  return new EqualsBuilder().
         append(aBoolean, o.aBoolean).
         append(aDouble, o.aDouble).
         append(aLong, o.aLong).
         append(anInt, o.anInt).
         append(anObject, o.anObject).
         append(anArray, o.anArray).isEquals();
}

Well, that's much shorter, isn't it.