Showing posts with label unit testing. Show all posts
Showing posts with label unit testing. Show all posts

19 October 2023

Unit Testing Scheme

For more than four years this article has been waiting to be written. Now a rainy summer day can do wonders. Back in 2018, when I fell into Scheme, I needed a unit testing framework. I was digging into Scheme with coding exercises and wanted the quick feedback of my TDD cycle so see if I was on the right track. I did not know anything about modules, dependency management or how to get custom code in Scheme and searched the web for a small test framework. As minimalism is in the spirit of Scheme, I initially went with the Scheme Unit outlined on Cunningham's Wiki,
;
; Unit test framework for scheme
; see http://c2.com/cgi/wiki?SchemeUnit
;
(define (report-error msg)
    (display "ERROR: ")
    (display msg)
    (newline))
    
(define (assert msg b)
    (if (not b)
        (report-error msg)))
Evolution
Now that was minimalist indeed. But as good as it was, it lacked important features: I wanted more assertions, at least for all basic data types, and needed to see some colours. What is the point of a green bar, if it is not shown in green. I had created my own xUnit libraries before, one for good old Turbo Pascal and one for assembly (written in assembly - usually you would use C). While both were useful, I did not get famous for them, it seems that am focusing on "niche products" in the testing framework world. Creating my own xUnit was a kata in itself and always helped me while learning a new programming language.

I started adding more assertions already in 2015 and test cases in 2017. I kept extending it whenever I needed something new, always copying the latest file to the next code kata. In 2018 I decided to put my "SchemeUnit" up on GitHub under the name of assert-scm. Now my tests, for example the tests for the Parrot kata, look like
(include "assert.scm")
(include "parrot.scm")

(test-case "it gets the speed of an european parrot"
    (assert= 12.0
             (parrot-speed 'european-parrot 0 0.0 #f)))

(test-case "it gets the speed of an african parrot with one coconut"
    (assert= 3.0
             (parrot-speed 'african-parrot 1 0.0 #f)))
Features of xUnit
Which features are expected from a true xUnit framework? From my knowledge of JUnit, I derive the core elements of xUnit:
  • Assertions: First we need assertions. These are typically called assertSomething. Assertions are necessary to verify the actual result versus the expected one. If these values are not equal, the assertion should fail in some way. There should be assertions for equality of basic data types. In my Scheme xUnit there are (assert-true actual) and assert-false, assert= for integer numbers and symbols (and everything you can compare with = in Scheme), assert-char= and assert-string= for these primitives and assert-inexact= for floating point numbers which allows a delta for rounding errors. There are assert-null, assert-not-null, and more. As lists are the basic, all encompassing data structure in Lisp, and therefore Scheme, any testing framework for these languages needs support for comparing lists for equality: assert-list= and assert-list-deep= for flat and deep list comparison.

  • Failure Messages: Assertions need to fail with descriptive messages. For example, if two values expected and actual are not equal, I would like to see "expected: <expected value> but was: <actual value>". I hate testing frameworks which just stop with "Assertion failed." Creating good messages gets more interesting when comparing lists as they can be of different length and nested. After assertions, this is the second important thing to have.

  • Test Cases: In xUnit, test cases are often a bit weird, as they are classes containing multiple methods. Each of these methods is an individual test case because during test execution the class is instantiated for each method. In some frameworks test methods are named testSomething(), or annotations or other markers are used. In frameworks without classes, e.g. Jest or Pytest, each test function is a test case. A test case has a name, often the name of the method, some arrange code, some logic and one or more assertions. Test cases should be run independently of each other and report success or failure individually.
    (test-case "(test-case) allows several assertions"
        (assert-true #t)
        (assert-true #t))
    will print (test-case) allows several assertions .. OK.

  • Ignoring Test Cases: Sometimes I want to ignore a certain test case. Most frameworks offer ways to do that, e.g. @Ignore, @Disabled, @mark.skip or using other markers. I like the Mocha way of replacing it('') with xit('') and went for a different function ignored-test-case:
    (ignored-test-case "(ignored-test-case) is ignored, else it would fail"
        (assert-true #f))
  • Test Suites: Test suites are used to group test cases. Naturally these are Java classes, Python modules or Jest/Mocha describe blocks containing test methods. In Scheme 5 that would be files. Files can include other files which allows me to build arbitrary test suites. I rarely use test suites in any language, as I am running all tests most of the time.

  • Fixtures: Fixtures contain the necessary creation and release of resources needed for the test and make sure these are released even if the test failed. Older test frameworks allow setup and teardown methods or @Before/@After markers. Other approaches include injecting necessary dependencies, as for example JUnit 5 and Pytest do. Till now I did not need fixtures in my exercises. In small test sets, I am fine when tests stop at the first failing test.

  • Asserting on Exceptions: Few testing frameworks offer assertions for exceptions. For example in Java, before JUnit 5's assertThrows, there were 5+ ways to test that a method threw an exception. Maybe this is a special case, something that is rarely used. As I was building my assert-scm Scheme xUnit from scratch, I wanted to be sure the assertions work. How would I test for a failing assertion? I had to dig deeper into Scheme. Standard R5RS Scheme has no function to catch exceptions and different implementations handle this differently. Gambit Scheme, the Scheme I started, offers some proprietary extension for exceptions, whereas Chicken Scheme, another common Scheme, has some support for handling exceptions - called conditions. At least Chicken version 4 does, but it is outdated now. Portability is an issue between different Scheme implementations. It seems Scheme R6RS has standard support for exceptions, but its structure is way more complicated. I would like to avoid this for fun exercises.
If you are interested in features of xUnit and how to use them I recommend you work through my Unit Testing-Koans exercises.

Prime Factors
Exploring a new language is incomplete for me, until there is a nice TDD, step by step implementation of the Prime Factors kata. It is my goto exercise and I am collecting my Prime Factors solutions - which usually look the same. Here are the tests:
(include "prime-factors.scm")
(include "../assert-r5rs.scm")

(test-case "one"
    (assert-null (prime-factors 1)))

(test-case "two"
    (assert-number-list= (list 2) (prime-factors 2)))

; ...

(test-case "nine"
    (assert-number-list= (list 3 3) (prime-factors 9)))

(test-case "max"
    (assert-number-list= (list 2147483647) (prime-factors 2147483647)))
which execute in assert-scm's GitHub build,

assert-scm Prime Factors Test
and verify the code
(define (prime-factors n)
    (test-candidate n 2))

(define (test-candidate n candidate)
    (cond ((= n 1) (list))
          ((too-large? n candidate) (prime n))
          ((divides? n candidate) (keep-candidate n candidate))
          (else (next-candidate n candidate))))

(define (too-large? n candidate)
    (> candidate (sqrt n)))

(define (prime n)
    (list n))

(define (divides? n candidate)
    (= (modulo n candidate) 0))

(define (keep-candidate n candidate)
    (append (prime candidate)
            (test-candidate (/ n candidate) candidate)))

(define (next-candidate n candidate)
    (test-candidate n (+ candidate 1)))
This is neat, isn't it? If you fancy playing with Scheme and do not want to miss unit tests, check out assert-scm, a minimalist unit test framework for Scheme R5RS.

1 August 2023

Shut up and write a test!

Due to the hype around ChatGPT I wanted to create this text with it. I failed. I tried several versions of different prompts, and also asked the Bing variant, but it failed to create any text I liked. It could not simulate my personal style. I should have to train a model with all my blog posts ... It seemed easier to use with the good old method of writing my texts myself.

Should I Write a Test?
Around ten years ago I stumbled upon a diagram if and when to write a test. This was a repeating question in trainings, when should you write tests? Before or after writing the code? Only when you have time? Only when you feel like it? I loved the diagram because there was only one end state. Whatever the reason or excuse would be, just shut up and write a test! It does not matter if you are tired, or if it is a little change, it should be tested. Such a rule is helpful when you want to adopt Test Driven Development, as there are plenty of excuses why to skip it.

Shut Up and Write a Test!
The original diagram was created by Gemma R Cameron, a software engineer and event organiser from Manchester. She was inspired by a similar diagram about skating. Recently, when I was asked again when to test, I wanted to share the diagram and could not find its page. I thought it lost and planned to repost the diagram here. Eventually I discovered Gemma's page again, it had moved from a .me to a WordPress domain. I am glad that the original source is still available. You can find it here.

Java Version
The original version of the diagram was for Ruby, indicated by the use of Automock. What was bad with Automock? It provided some support for automated mocking of Rails application and is dead since 2016. Here I present a slightly modified version for Java, replacing Automock with PowerMock. And I do know what is wrong with PowerMock. It is a powerful mocking tool which uses a custom class loader and bytecode manipulation to enable mocking of static methods, constructors, final classes and methods, private methods and more. It is useful to bring legacy code under test, before seams can be introduced. The bytecode manipulation makes it slow compared to regular mocking. If used often, the whole test suite will run much longer than needed. Further, if code is hard to test, I change its design to make it testable by applying SOLID principles or by creating seams as outlined by Michael Feathers in his WELC book. Using PowerMock for tests does not improve the design. (I am not inherently against PowerMock, it is extremely powerful and as such dangerous. I needed something to replace Automock.)

Download and print the Java version of the Shut up and Write a Test Diagram. The fact that there are versions of this diagram for both Ruby and Java is a testament to the universality of its principle: Shut up and Write a Test! The Code Cop approves, and I am sure your future self will too.

19 August 2019

Y U NO TDD

Y U No TDDDuring this year's GeeCON the crew organised an Open Space evening. (An Open Space is a self-organising meeting where the agenda is created by the people attending.) I participated and ran a session on the question why we are not doing Test Driven Development. (Y U No Do TDD?) I am running TDD trainings from time to time and wanted to get more insight where people are stuck with TDD.

Context
As I said, an Open Space is self organising, and only people interested in TDD attended my session. This is a typical problem of communities of practice - only people interested in the topic attend - which results in us living in a bubble. For example long time TDD practitioner Thomas Sundberg and Shirish Padalkar, lead consultant at ThoughtWorks, participated in the discussion. Depending on the background of each individual participant, my original question was understood as:
  • Why are you not doing TDD on production work at all?
  • Why are you not doing TDD most of the time?
  • Why are you not doing TDD all the time?
I collected the reasons not to do TDD during the session which I want to share here. Text inside quotation marks, e.g. "hi" quotes exactly what people said. While the previous three questions are slightly different, the reasons seem to be similar. I grouped the answers. I did not want to contradict or debunk these answers and have to hold back not to do so ;-)

Prototyping
I am "experimenting with something", the "expected outcome is unclear" and "it's only a prototype". Obviously these are valid reasons as Spikes are outside of TDD. These answers usually coming up quickly makes me wonder if they are kind of excuses sometimes. Experimenting with new libraries and APIs is covered further down, so what are we experimenting with? I worked with many developers who would agree that the expected outcome of their current ticket was unclear - because they did not take the time to analyse the story and understand the solution they were supposed to build? Additionally most of our prototypes go to production after all, don't they ;-)

Time Pressure
Another reason - given by some of my clients too - is their need to go fast: "I need to go very fast", there is "no time for that" and I "believe to be faster without it". While they might be wrong in the long term I understand the effects of pressure. One person made it more explicit, while he has no strong deadline, he said "I have a huge backlog, I am stressed". Indeed when I am extremely stressed, I find it hard to maintain a structured approach, especially if a lot of task switching is involved. Besides the needed skill to apply TDD under high load, much discipline is required to endure pressure. In such situations Strong Opinions and Dogma might help.

Missing the Bigger Picture
I am just "writing a script for myself". Maybe there is no need for automated tests when writing a one time script for myself. TDD has a testing aspect - and it has many other aspects like designing software, fast feedback and working incrementally. TDD is not only about testing. Some people miss that or have only partial understanding of the benefits or do not care for these benefits at the moment. The opposite reason is "because I know how the class will look like". Yes TDD is about software design as I said before, and I would like my class to work, too. Some people only want the fast feedback, e.g. using REPL based development and "looking at UI is faster".

Missing Priority on Testing
When starting with TDD, the testing aspect is most visible. After all we have to write a reasonable test first. For teams and organisations with low or missing priority on testing, people are "looking down on testing" and I got answers like "testing is a culture thing", "testing is not a first class activity" and "I am not asked to create a test by my project manager". Indeed it is hard to keep following TDD if it is looked down upon and if there is no time for quality work.

Avoiding Context Switches
There is a certain amount of context switching involved in TDD. Similar to Edward de Bono's Six Thinking Hats, we have different states which we have to be mindful of and which call for different actions. George Dinwiddie created a TDD Hat to show that. Maybe this switching is "not natural for some people". "I don't want to interrupt creative design with verification" and "I prefer staying in building hat and not change to testing hat". Similar one participant said that it is "easy to write code, harder to write tests, so I do it afterwards". I understand and there is certainly an urge to jump into the code and get hacking. I rarely feel that urge and I enjoy pair programming using the Ping Pong style because it enforces the separation of states without any (inner) discussion.

Missing TDD Skills
This is obviously the largest area and there is nothing wrong with not knowing how to apply Test Driven Development: Honest people just say "I can't do it". Many are aware of this problem and seem to be disappointed with existing material and/or look for more material to study TDD: "It is not taught at universities", "there are no good books" and "I am missing real examples". I know from my own experience that TDD is not easy to learn and some people are "scared for life after a bad experience" with it. Now the best way to learn TDD is to have someone show you while pairing with you. Even if there is no pair programming in your workplace, you can still experience it during a Coding Dojo or Coderetreat. Short of that, I recommend Kent's Beck Test Driven Development by Example, which is a short and excellent introduction.

New Language or Library
When discussing TDD and unit testing with a client, he said I "don't know the target technology" and "React is a new technology for us". I had to laugh. To me this sounded like "I got a car and know how to drive forward, but am not able to drive backwards." On the other hand I live at the dead end of a road and I see drivers working really hard to avoid driving backwards. Are they not able to do it? So maybe stopping halfway in the game (of skill acquisition) is natural after all. When working with some new language or working with an unknown API, I specially rely on tests to support me, these are Learning Tests.

It's too hard to test
I agree some things are harder to test than others. "Android is hard to test", "Vaadin is hard to test" and "some libraries are hard to test". (I have not worked with Android or Vaadin, I quote people.) We might need to know more about design to decouple things. This is definitely true for legacy code, as "existing code is usually hard to test". Some people see the root cause, like in "I don't know how to manage boundaries". In such situations we need (to know) more tooling. We definitely "need more tooling to test the UI" as UI is traditionally considered hard to test from a TDD perspective. Still, Steve Freeman and Nat Pryce, authors of Growing Object-Oriented Software Guided by Tests, always start their TDD (outer) loop with an UI test. GOOS is a great book and I recommend reading it if you want to go deeper into TDD.

It's too simple to test
If there are things which are too hard to test, there must also be things which are too simple to test, right? It is "useless to test, it is so simple" and it "makes no sense to test it". Maybe a better description is that it is "unclear what is important to test". From a TDD perspective no such things exist and I guess these reasons arise from the test after process, when looking at each public method and thinking how to test it. Further excessive test isolation, see Solitary vs. Sociable Unit Tests, will cause that.

Barriers to TDD adoption
Here is Matt Wynne's summary of Barriers to TDD adoption from a session during Lean Agile Scotland 2016. I recommend checking out the Twitter thread as Matt added detail discussions on temptation of fast reward, permission and safety to learn, "the egotist" and other reasons not covered by me.

Barriers to TDD adoption #lascot16 (C) Matt Wynne
What about test-induced design damage?
Maybe the only real reason not to do TDD is to keep the design integrity of your system. This idea was started back in 2014 by David Heinemeier Hansson, also known as DHH, and led to the whole Is TDD Dead? debate. DHH said that when using TDD code sometimes suffers tremendous design damage to achieve two testing goals: Faster tests and easy-to-mock unit tested controllers and that the design integrity of the system is far more important than being able to test it any particular layer. It is ironic that this never comes up during any group discussion or team interview. Probably because it is an expert level reason. If you followed the debate, DHH knew TDD, he used it for some time and liked it. And then, only then, did he know when not to apply it.

29 June 2018

xUnit Testing Koans Reloaded

Koans are sequences of little exercises to help you "on your way to enlightenment". Testing Koans are similar exercises to help you learn unit testing. Usually the code is in place, but the assertions are missing. xUnit Koans are Testing Koans for SUnit inspired testing frameworks, like JUnit, minitest, NUnit, PHPUnit, xUnit.net and more. They are introductory exercises which cover the most basic usage of assertions and life cycle methods.

JUnit5 Koans RetrospectiveJUnit 4 and PHPUnit
Initially I created a few JUnit Testing Koans in 2012 as part of a JUnit training while I was with IBM. I wanted the training to be hands-on and interactive. (These koans are in the Unit Testing Koans repository.) In 2015 I picked up the idea and ported the code to PHP to run a PHPUnit workshop for my local community. (These koans are in the same repository.)

JUnit 5
In 2017, with the new version of JUnit, Görge Albrecht, the Code Mentor, helped me add koans for new features of JUnit 5. We used the koans as hands-on exercises which were a part of our JUnit 5 introductory workshop, delivered at Topconf Linz 2017 and other conferences. The exercises worked well and gave an overview of the (old and) new API and showed how to use it. People liked the workshop and had fun, see the points made during the retrospective of one of the workshops on the right. (The code is in the same repository.)

Python's unittest
It seems that the koans keep coming back to me ;-) - and I keep coming back to them, always adding more exercises and languages. This year I ported them to Python, which took me around five hours. I ran them as a short, Pragmatic Introduction to Python Unit Testing at PyDays 2018. I liked PyDays for its enthusiastic crowd and consider the workshop successful because in the final feedback round several participants reported happily that they had "written their first test" and that "it's not that hard!". Even experienced participants reported some learnings, like "I never used skip" or "I did not know how to check the message in asserting exceptions". (The latest and greatest koans are in the same repository.)

Teaching xUnit
Koans, i.e. learning progress verification by assertions, work well in both static and dynamic languages. There is no difference in workshops using Java and PHP or Python. Testing Koans are a suitable exercise to help people with their first steps in unit testing. In short two hours, people who have never seen a test will learn the concepts and flow of unit testing. I usually start with a few slides, about 20 minutes of theory about structure of xUnit tests before I ask people to work with the exercises.

JUnit5 handouts by GörgeLike with JUnit 5, Testing Koans also work well to introduce people to new testing framework APIs and their usage. People familiar with JUnit 4 will need no more than a migration guide, see Görge's handouts on the right, to get their first impression of the new features. Some more advanced aspects will require them to research the documentation, but that is part of the exercise.

People appreciate the "focused, step by step nature" of koans. The "exercise is clear" and "invites to explore xUnit". And most importantly "it was fun". In all the feedback I have collected for my xUnit Testing Koans, fun is dominant. And fun helps learning. (All quotes were taken as is from actual feedback I got.)

Moar Testing Koans!
Most xUnit frameworks are consistent with xUnit, and are similar in their usage. It is easy to port the koans to other languages, at least for basic examples. The same is true for newer versions. I should upgrade some koans to accommodate newer features of the used frameworks, on the other hand the basic features are still the same. I would love to see more ports and also Koans for different testing styles, e.g. RSpec or Jasmine Testing Koans.

Current List of Koans

28 December 2015

Testing Koans

Koans
Koans have been proposed as an effective way to learn a new programming language. But what exactly are Koans? According to Wikipedia a Koan (where the o has a macron, a straight bar placed above it - which my text editor refuses to produce) is a "case, story, dialogue, question or statement in the history and lore of Zen Buddhism". Huh? Reading the Wikipedia article did not help me at all. All I understand is that a Koan is something the Buddhist monks would work with, a mystical sentence or maybe a kind of poem, which does not make any sense, but somehow helps them on their way to enlightenment. It seems the metaphor has been transferred from Buddhism to software, e.g. Hacker Koans and Koans are related to the TAO of Programming. (Again no idea what TAO is supposed to mean here. This is like a recursive definition.)

Ruby Koans
As far as I know, the first Programming Koans were available in Ruby, created by the late Jim Weirich, a popular Ruby hacker. Ruby Koans consists of several little exercises, starting with basic things and building on each other to move to more advanced topics in the end. The goal is to learn Ruby, to walk the "path to enlightenment" as Jim put it. He also wanted to teach the Ruby culture. The Ruby community has a strong focus on testing, which is considered essential to "do great things in the language". In fact the exercises are a list of failing test cases, where tiny pieces of code have to be filled in to make them pass. For example, here is the exercise to learn accessing array elements,
def test_accessing_array_elements
  array = [:peanut, :butter, :and, :jelly]

  assert_equal __(:peanut), array[0]
  assert_equal __(:peanut), array.first
  assert_equal __(:jelly), array[3]
  assert_equal __(:jelly), array.last
  assert_equal __(:jelly), array[-1]
  assert_equal __(:butter), array[-3]
end
Doyle Spiral + InversionThe double underscore marks the place where the code has to be changed to make it work and pass the test. These tests are very simple and there is not much explanation. Maybe this is the connection to the Zen Koans: The Language Koans are a list of exercises to work through, to master the language (i.e. reach enlightenment). Each one is very small (i.e. a sentence) but does not make much sense on its own. The exercises are sorted by increasing difficulty (i.e. the path to walk). Following Jim's example, Koans are usually based on unit tests which you make succeed. Language Koans are available for many programming languages, see a list of Koans by Laura Diane Hamilton.

Testing Koans
I took the idea for Testing Koans from Carlos Blé's training JavaScript for Testers. He created some Koans for JavaScript with inverted work-flow. The code was already in place, but the assertions were missing. That was reasonable as the training was created for tester.

xUnit Koans
Earlier this year I ran an introductory unit testing workshop for the local PHP community. I expected a junior audience and aimed for the most basic exercise for xUnit assertions and life cycle methods. I wanted the participants to focus on PHPUnit alone. I created some sample code, together with unit tests, and then deleted the assertion statements. The first test looked similar to the following Java code:
import org.junit.Test;

public class Session1_GreeterTest {

  @Test
  public void shouldReturnHelloName() {
    Greeter greeter = new Greeter();
    // TODO check that "Hello Peter" is greeter.greet("Peter")
  }

  @Test
  public void shouldReturnHelloForNull() {
    Greeter greeter = new Greeter();
    // TODO check that "Hello" is greeter.greet(null)
  }

  // more tests skipped...

}
The participants went through the tests one by one, adding assertions or fixing incomplete statements, making the tests pass. While this looked like a very basic and short exercise, developers unfamiliar to PHPUnit (and xUnit in general) needed several hours to complete all my PHPUnit Testing Koans.

Due to the uniform nature of all xUnit ports, the style and structure of the exercise can be used for other programming languages. I ported the exercise to Java using JUnit, creating Java/JUnit Koans. Both Koans cover the basic functionality of PHPUnit and JUnit, e.g. assertions, testing for exceptions and before- and after-methods. More advanced features could be added. I will port the Koans to C#/NUnit and Ruby/minitest as soon as I will need them.

Conclusion
Koans are a great way to partition the process of knowledge acquisition into a series of little exercises. They verify themselves, giving you fast feedback but you can still learn at your own pace. Language Koans are established and available for many languages. These can be extended to any library or public API you want to master. Testing Koans work similar, just inverted. They are available for PHPUnit and JUnit for now. I would love to see more ports and also Koans for different testing styles, e.g. RSpec or Jasmine Testing Koans.

7 August 2015

How to Unit-Test Assembly

now i just need a pirate and a waterfall...After my last trip into Assembly I got the feedback that I should have used TDD, especially as it was supposed to be a code kata. (Thank you Emmanuel Gaillot for reminding me of my duties.) And yes, I should have. But as I said, I could not find any real unit testing support, at least not without resorting to C or C++ unit testing frameworks. On the other hand, creating an xUnit implementation is a great exercise to get to know a language. So I decided to create my own, minimal unit testing framework for Windows IA-32 Assembly using NASM. (The following code uses stdcall call convention because it is used in the Microsoft Win32 API anyway.)

Failure
So what are the most required features for unit testing support? I believe the most important feature is to mark a test as failed. I also want to stop executing this test in case of failure, like JUnit does. This is the fail() method,
%macro fail 0
        log msg_failed, msg_failed_len

        leave
        ret 0 ; test method never has any arguments
%endmacro

msg_failed:     db 'FAILED'
msg_failed_len  equ $ - msg_failed
In fact it is a NASM macro which gets copied into each invocation. For simplicity, it does not support a message argument right now, hence zero macro parameters. It uses the log macro, which prints the given string to Standard Out, just like _printGrid of my Assembly Minesweeper. Then it resets the stack frame (leave) and returns from the current function. I expect all test methods to have no arguments, and fail's code gets copied into each test method, this skips further execution of the test method and returns immediately to the calling code, i.e. the test runner. Adding the count of failed tests in fail is straight forward but I will leave that for later.

Assertion
Next I need assertions to check my expectations, at least an assert_equals,
%macro assert_equals 2
        cmp     %1, %2
        je      .%1_%2_end
        fail
.%1_%2_end:
        call    _show_progress
%endmacro
This has to be a macro as well so fail works correctly as explained above. But as the macro gets expanded into the calling code, the local label .%1_%2_end has to be unique. While different assertions are possible, a specific assertion like assert_equals eax, 2 can only be used once per test method. This is a weird constraint but I do not mind as my tests tend to have a single assertion per test method anyway. (Later I changed the macro to use a macro local label %%_end which removed this problem.)

The function _show_progress just prints a single dot to show the progress during test execution. It needs to preserve all registers because it is called in the middle of my test code.
_show_progress:
        push    eax
        push    ebx
        push    ecx
        push    edx

        push    dot_len
        mov     eax, dot
        push    eax
        call    _print

        pop     edx
        pop     ecx
        pop     ebx
        pop     eax

        ret

dot:    db      '.'
dot_len equ     1
Test Cases
For my test cases I want descriptive names. Using long, regular labels for function names and the assertion macros I write my first test
_should_add_one_and_one_to_be_two:
        create_local_variables 0

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        leave
        ret
The test succeeds and prints a single dot to Standard Out. The create_local_variables macro creates the stack frame and local variables if needed. If a test fails, e.g.
_should_add_one_and_two_to_be_three:
        create_local_variables 0

        mov     eax, 1
        add     eax, 2
        assert_equals eax, 2 ; (6)

        ; not reached
        hlt

        leave
        ret
it prints FAILED and stops execution in line 6.

Test Runner
Finally I want to run all my tests one by one. In this simple case I just call the test methods from the main entry point, followed by printing DONE at the end.
global  _main

_main:

        ; welcome message
        log     msg_hello, msg_hello_end - msg_hello

        call    _should_add_one_and_one_to_be_two
        call    _should_add_one_and_two_to_be_three
        ; (10)

        ; completion message
        log     msg_done, msg_done_end - msg_done

        jmp     _exit

msg_hello:      db 'HELLO asmUnit'
msg_hello_end:

msg_done:       db 'DONE'
msg_done_end:
Macro Metal CatThis gets the job done but I am not happy with it. I need to add new test method invocations by hand in line 10. By seeing the new test fail I cannot forget to add the call of the new test method, but automatic test case discovery would be more convenient.

Before and After
What else does a unit testing framework need? For complex cases, before and after test execution hooks might be necessary. A before macro defines the proper label and the begin_test macro just calls it.
%macro before 0-1 0
        %ifdef ctx_before
                %error "before used more than once"
        %endif

        %define ctx_before, 1
_before_hook:
        create_local_variables %1
%endmacro

%macro begin_test 0-1 0
        create_local_variables %1
        %ifdef ctx_before
                call _before_hook
        %endif
%endmacro
The after and end_test macros work accordingly, just for the test clean-up. The actual test code stays the same, the macro calls just changed.
_should_add_one_and_one_to_be_two:
        begin_test

        mov     eax, 1
        add     eax, 1
        assert_equals eax, 2

        end_test
While this works well, I am unhappy with all the (NASM specific) macros in the code. There remains little real Assembly, but I guess that is the way to go. There are also plenty of missing features but this minimal first version gets me started. I will see how it works out in my next Assembly kata. (The complete source of asmUnit is here.)

17 July 2015

Using Hamcrest Matchers With Composer

gotta matchTo help my friends of the Vienna PHP User Group to get started with Coding Dojos, I created a project template. Obviously the used programming language was PHP and PHPUnit, the PHP xUnit implementation, was required.

Composer
Setting up a new project almost always starts with dependency management and I used Composer to define my dependencies. Composer was configured by its composer.json,
{
  "name": "codecop/CodingDojo-PHP",
  "description": "Coding Dojo template",
  "license": "BSD",
  "require": {
  },
  "require-dev": {
    "phpunit/phpunit": "4.5.*"
  }
}
After verifying my composer installation with composer --version I downloaded PHPUnit with composer install. PHPUnit and its transitive dependencies were installed into the vendor directory as expected.

PHPUnit
Next I configured PHPUnit with its phpunit.xml,
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.5/phpunit.xsd"
    backupGlobals="false"
    colors="true"
    beStrictAboutTestsThatDoNotTestAnything="true"
    beStrictAboutOutputDuringTests="true"
    beStrictAboutTestSize="true"
    beStrictAboutTodoAnnotatedTests="true"
    verbose="true">

    <testsuites>
        <testsuite name="All Tests">
            <directory suffix="Test.php">test</directory>
        </testsuite>
    </testsuites>
</phpunit>
This told PHPUnit to load all the *Test.php files inside the test directory for test classes. I especially like the beStrictAbout* flags and enabled them all. These flags warn about smelly tests, e.g. test methods without any assertions. I ran PHPUnit with ./vendor/bin/phpunit to verify my setup. It did not show any tests - after all this was a new and empty project. I have seen people creating an alias to run PHPUnit, but I created a (Windows) script phpunit.bat in the local directory with the same effect,
@call "%~dp0vendor\bin\phpunit" %*
Now I was ready to go and wrote some unit tests, e.g.
<?php

require 'Hello.php';

class HelloTest extends \PHPUnit_Framework_TestCase {

    /** @test */
    function shouldReturnHelloName() {
        $greeter = new Greeter();
        $this->assertEquals("Hello Peter", $greeter->greet("Peter"));
    }

}
Hamcrest Matchers
In the Java community Hamcrest Matchers are popular and they even ship with the core JUnit framework. I like Hamcrest because it allows me to write my own matchers, which make assertions much more expressive than plain assertEquals. Luckily there were some ports of it and I was happy to see a Hamcrest PHP port. I added it to composer.json,
"require-dev": {
  "phpunit/phpunit": "4.5.*",
  "hamcrest/hamcrest-php": "1.2.*"
}
and updated my installation with composer install. Hamcrest offers global functions for its matchers, which allow for shorter syntax, especially when matchers are chained together. To enable this global functions, Composer has to auto load the main Hamcrest file, which is configured using autoload-dev in composer.json,
"autoload-dev": {
  "files": ["vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php"]
}
Using global functions has some drawbacks and is considered a bad practise in large scale projects. There are different ways to use Hamcrest PHP with Composer without loading the global functions, e.g. see Hamcrest PHP issues at GitHub. For a first time Coding Dojo I wanted to stay with the simplest way to use Hamcrest and kept the global functions.

So I was able to write my unit tests using Hamcrest matchers, e.g.
<?php

require 'Hello.php';

class HelloTest extends \PHPUnit_Framework_TestCase {

    /** @test */
    function shouldReturnHelloName() {
        $greeter = new Greeter();
        assertThat($greeter->greet("Peter"), equalTo("Hello Peter"));
    }

}
While the test above succeeded, PHPUnit refused to show me a green bar. This was because Hamcrest's assertThat was not recognised as assertion and PHPUnit marked the test as erroneous. With a heavy heart I had to remove one of PHPUnit's beStrictAbout flags,
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/4.5/phpunit.xsd"
    backupGlobals="false"
    colors="true"
    beStrictAboutTestsThatDoNotTestAnything="false"
    beStrictAboutOutputDuringTests="true"
    beStrictAboutTestSize="true"
    beStrictAboutTodoAnnotatedTests="true"
    verbose="true">
That was it. Finally I was ready to rock, matchers included!

(The complete project as zip is here.)

20 April 2015

Maven Integration Tests in Extra Source Folder

On one of my current projects we want to separate the fast unit tests from the slow running integration and acceptance tests. Using Maven this is not possible out of the box because Maven only supports two source folders, main and test. How can we add another source and resource folder, e.g. it (for integration test)? Let's assume a project layout like that:
project
|-- pom.xml
`-- src
    |-- main
        `-- java
    |-- test
        `-- java
            `-- UnitTest.java
    `-- it
        `-- java
            `-- IntegrationIT.java
We use the regular Maven Surefire to execute our unit tests, i.e. all the tests in the src/test/java folder. The plugin definition in the pom.xml is as expected.
        <plugin>
            <!-- run the regular tests -->
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.18</version>
        </plugin>
And we use Maven Failsafe to execute the integration tests. If you do not know Failsafe, it is much like the Surefire plugin, but with different defaults and usually runs during the integration test phase.
        <plugin>
            <!-- run the integration tests -->
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.18.1</version>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
By default the Failsafe plugin does not look for class names ending with Test but ending with IT, making it possible to mix both kind of tests in the same src/test/java folder, but we do not want that. The "trick" to have another source folder for integration tests is to use the Build Helper Maven Plugin and add it as test source.
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>1.9.1</version>
            <executions>
                <execution>
                    <id>add-integration-test-source-as-test-sources</id>
                    <phase>generate-test-sources</phase>
                    <goals>
                        <goal>add-test-source</goal>
                    </goals>
                    <configuration>
                        <sources>
                            <source>src/it/java</source>
                        </sources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
Now src/it/java is added as test source as well, as seen during Maven execution. After the compile phase Maven logs
[INFO] [build-helper:add-test-source {execution: add-integration-test-source-as-test-sources}]
[INFO] Test Source directory: .\src\it\java added.
There is still only one test source for Maven but at least we have two folders in the file system. All test sources get merged and during the test compile phase we see
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Compiling 2 source files to .\target\test-classes
showing that all classes in both src/test/java and src/it/java are compiled at the same time. This is important to know because class names must not clash and tests still need different naming conventions like *Test and *IT. Now mvn test will only execute fast unit tests and can be rerun many times during development.
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory: .\target\surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running UnitTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in UnitTest

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
Only if we want, using mvn verify, the integration tests are executed.
[INFO] [failsafe:integration-test {execution: default}]
[INFO] Failsafe report directory: .\target\failsafe-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running IntegrationIT
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in IntegrationIT

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] [failsafe:verify {execution: default}]
[INFO] Failsafe report directory: .\target\failsafe-reports
Now we can run the slow tests before we check in the code or after major changes a few times during the day.

3 May 2014

The Life of (an) Ant

I already mentioned that Daniel impressed me by implementing Conway's Game of Life in MSBuild with MSBTest. Being a Java developer for most of my career, Apache Ant was the proper choice to build the cellular automaton using a build tool. I have worked with Ant before, so I was confident I would be able to pull it off.

Bring Out Your DeadIdiomatic Ant
I wanted a challenge and decided to avoid Ant extensions whenever possible. For example there are if and for tasks in Ant-Contrib which enable an imperative, procedural coding style. Obviously embedding any kind of script was also considered cheating.

An Ant's Universe
Wikipedia says that the universe of the Game of Life is an infinite two-dimensional orthogonal grid of square cells. How could I represent such a grid using basic Ant data structures? Which types does Ant support at all? Ant offers immutable properties and property sets, files, directories and paths and supports text file filtering using regular expressions. Which data structure could I use for the universe? How about directories? The universe is a two-dimensional grid, containing rows and columns, which might be defined by the names of nested folders. A single row in the grid is a folder which contains the folders of all its columns, e.g.
universe/
    row_folder_1/
      column_folder_1/
      column_folder_2/
      ...
    row_folder_2/
      column_folder_1/
      column_folder_2/
      ...
    row_folder_3/
    ...
That looks promising. But how to find the previous and following columns given a current column (folder) name? There are no arithmetic functions available in Ant, so there is no way to increase or decrease a number by one. But it is possible to append a character to a property creating a new one, allowing to reach the cells (folders) right and down of the current one, e.g.
universe/
    row_folder_x/
      column_folder_x/
      column_folder_xx/
      ...
    row_folder_xx/
      column_folder_x/
      column_folder_xx/
      ...
    row_folder_xxx/
    ...
To access the left and upper neighbours of a cell the name has to be shortened. Ant is not able to shorten String properties but the basename task is able to strip the extension off a file or directory name. So the row and column folders are called '.y' and '.x'.
universe/
    .y/
      .x/
      .x.x/
      ...
    .y.y/
      .x/
      .x.x/
      ...
    .y.y.y/
    ...
Testing Framework
To do it "right", I needed a testing framework. Although there is Apache AntUnit, I used a simpler solution to test my build files. Inspired by my former colleague Andy Balaam's Test-Driven Ant macros I created the necessary assertion macros on the way. For example the following code asserts that a certain property is set:
<macrodef name="assert-propery-set">
    <attribute name="propertyname" />
    <sequential>
        <fail message="property '@{propertyname}' not set."
              unless="@{propertyname}" />
    </sequential>
</macrodef>
Tell me about your neighbours
Having the data structure and assertions in place, it was time for the first test. I started with tests for counting the neighbours of a given cell, expecting its implementation in target count_neighbours of cell-build.xml.
<import file="lib/asserts.xml" />
<import file="cell-build.xml" />

<target name="test-count-neighbours-in-middle-when-no-neighbours">
    <make-tmp-dir />
    <antcall target="create_3x3_grid" />
    <touch file="${test.temp_dir}/.y.y/.x.x/cell" />

    <antcall target="assert-number-of-neighbours">
        <param name="p_cell_location" value="${test.temp_dir}/.y.y/.x.x" />
        <param name="p_expected_neighbour_count" value="0" />
    </antcall>
</target>

<target name="create_3x3_grid" if="test.temp_dir">
    <mkdir dir="${test.temp_dir}/.y/.x" />
    <mkdir dir="${test.temp_dir}/.y/.x.x" />
    <mkdir dir="${test.temp_dir}/.y/.x.x.x" />
    <mkdir dir="${test.temp_dir}/.y.y/.x" />
    <mkdir dir="${test.temp_dir}/.y.y/.x.x" />
    <mkdir dir="${test.temp_dir}/.y.y/.x.x.x" />
    <mkdir dir="${test.temp_dir}/.y.y.y/.x" />
    <mkdir dir="${test.temp_dir}/.y.y.y/.x.x" />
    <mkdir dir="${test.temp_dir}/.y.y.y/.x.x.x" />
</target>

<target name="assert-number-of-neighbours" depends="count_neighbours">
    <assert-propery-equals propertyname="cell.neighbour_count"
                           expected="${p_expected_neighbour_count}" />
</target>
First I fulfilled the test with a fake solution.
<target name="count_neighbours">
    <property name="cell.neighbour_count" value="0" />
</target>
After eight more tests,
<target name="all-test-targets" description="run all test targets">
    <antcall target="test-count-neighbours-in-middle-when-no-neighbours" />
    <antcall target="test-count-one-neighbour-left" />
    <antcall target="test-count-one-neighbour-right" />
    <antcall target="test-count-one-neighbour-up" />
    <antcall target="test-count-one-neighbour-down" />
    <antcall target="test-count-one-neighbour-up-left" />
    <antcall target="test-count-all-neighbours-around" />
    <antcall target="test-count-neighbours-in-left-up-when-no-neighbours" />
    <antcall target="test-count-neighbours-in-lower-right-when-no-neighbours" />
count_neighbours was finished.
<property name="cell.marker_alive" value="cell" />

<target name="count_neighbours" if="p_cell_location"
      description="sets property 'cell.neighbour_count' to the count of living
                   neighbours of a cell in the given location 'p_cell_location'.">

    <dirname property="cell.base_row" file="${p_cell_location}" />
    <dirname property="cell.base" file="${cell.base_row}" />

    <pref_self_next dim="y" folder="${cell.base_row}" /> <!-- (1) -->
    <pref_self_next dim="x" folder="${p_cell_location}" /> <!-- (2) -->

    <resourcecount property="cell.neighbour_count"> <!-- (3) -->
        <fileset dir="${cell.base}">
            <include name="${cell.prev_y}/${cell.prev_x}/${cell.marker_alive}" />
            <include name="${cell.prev_y}/${cell.self_x}/${cell.marker_alive}" />
            <include name="${cell.prev_y}/${cell.next_x}/${cell.marker_alive}" />

            <include name="${cell.self_y}/${cell.prev_x}/${cell.marker_alive}" />
            <include name="${cell.self_y}/${cell.next_x}/${cell.marker_alive}" />

            <include name="${cell.next_y}/${cell.prev_x}/${cell.marker_alive}" />
            <include name="${cell.next_y}/${cell.self_x}/${cell.marker_alive}" />
            <include name="${cell.next_y}/${cell.next_x}/${cell.marker_alive}" />
        </fileset>
    </resourcecount>
</target>
Line (1) defines the properties for the previous, current and following row, i.e. in the y dimension and line (2) does the same for columns, i.e. in the x dimension. Both sets of properties are similar, so I extracted a macro to remove duplication. (Remember: Red - Green - Refactor)
<macrodef name="pref_self_next"
      description="defines properties 'cell.self_X', 'cell.pref_X' and 'cell.next_X' for
                   given dimension/orientation 'dim' from center of current 'folder'.">

    <attribute name="dim" />
    <attribute name="folder" />

    <sequential>
        <basename property="cell.self_@{dim}" file="@{folder}" />
        <property name="cell.next_@{dim}" value="${cell.self_@{dim}}.@{dim}" />
        <basename property="cell.prev_@{dim}" file="@{folder}" suffix=".@{dim}" />
    </sequential>
</macrodef>
After determining the names of the previous and following neighbouring cells (folders), counting the live ones (files) was be done with Ant's resourcecount. Line (3) counts the number of files called 'cell' in the neighbouring folders.

To Be or Not To Be
With the number of living neighbours I was ready to implement the rules of the game right away. The target apply_rules uses this number to determine if a cell lives on or dies. With four tests,
<target name="all-test-targets" description="run all test targets">
    ...
    <antcall target="test-rules-with-two-neighbours-lives-on" />
    <antcall target="test-rules-with-one-neighbour-dies-on" />
    <antcall target="test-rules-empty-with-two-neighbours-stays-dead" />
    <antcall target="test-rules-empty-with-three-neighbours-comes-alive" />
apply_rules evolved to
<target name="apply_rules" depends="count_neighbours" if="p_cell_location"
      description="sets property 'cell.will_live' only if a cell in the given
                   location 'p_cell_location' will live on for the next generation.">

    <condition property="cell.will_live">
        <or>
          <and>
              <available file="${p_cell_location}/${cell.marker_alive}" />
              <equals arg1="${cell.neighbour_count}" arg2="2" />
          </and>
            <equals arg1="${cell.neighbour_count}" arg2="3" />
        </or>
    </condition>
</target>
Target apply_rules expects the property p_cell_location to point to a given cell (folder) in the universe. It first calls the target count_neighbours via depends and uses the cell.neighbour_count to determine if the given cell will be reborn, live on or die. This implements the rules of the game and ends this post. The next one will use apply_rules to run a full Game of Life on arbitrary universes. Stay tuned!

18 June 2013

TDD Exam Questions

Last year I ran a three day in-house training on unit-testing and TDD using Eclipse. I wanted the course to be as interactive as possible to keep the attendees engaged all the time. Besides my slides I carried out exercises like Jason Gorman's TDD from Hell and played short movies to keep the content varied. At the end of each chapter I showed a slide with a few questions about the topic just covered. These questions gave the audience a break to recall the content just studied and reflect on it before moving on. I searched for such questions but did not find any, so I had to come up with my own.

These are my sanity-check questions. They are not complicated or long-winded, just simple questions to verify if the content was understood. Some questions also served as a base for further discussion. The developers in the audience were exposed to TDD for the very first time, so I kept them simple. If you are a seasoned developer I am sure these questions will not challenge you at all.

Writing ExamsUnit Testing with JUnit
  • What is a test fixture?
  • Which assert is used to compare the values of objects?
  • What about assertTrue(true);?
  • What about assertTrue(a.equals(b));?
  • How to test for an expected exception?
  • What about System.out.println(...); in tests?
  • How to data-drive tests?
  • How to time-out tests?
  • How to test the contents of a private field? (This is a trap, the answer is not to test it but to change the design.)
  • Does full code coverage mean the code is fully tested?
Test Driven Development
  • What are the benefits of using TDD?
  • Is TDD primarily a testing technique?
  • When do you write tests? (This is a joke and the expected answer is "all the f*cking time" according to Bryan Liles' TATFT.)
  • What should you do when you cannot add another test?
  • If the code passes all the unit tests why should you still change (refactor) it?
  • Why not refactor on a red bar?
Mocking and Test Doubles
  • Why use stubs or mocks?
  • How are objects called that are passed around but never actually used?
  • How are objects called that can return fixed values in answer to calls made.
  • How are objects called that can verify the behaviour of the system under test, in addition to using asserts to verify its state.
  • What is used for testing state?
  • What is used for testing behaviour?
  • What is a spy?
ExamRefactoring
  • What is your favourite Refactoring?
  • How do we refactor? (This is a word play and the expected answer is mercilessly.)
  • Should Refactoring and adding new code be done at the same time?
  • During Refactoring, how long does your code not compile?
  • During Refactoring, when should we run the tests?
  • What Refactoring can be used when a method is too long, has duplicated code or Feature Envy?
  • What Refactoring can be applied to long parameter lists?
  • In Eclipse, what do the short-cuts Alt-Shift-R, Alt-Shift-L and Alt-Shift-M do?
Working with Legacy Code
  • What is Legacy Code?
  • How can you break dependencies?
  • What are Singletons and why are they evil?
  • Do you write them?
Using these questions, small exercises and short movies I kept the participants fully engaged for a whole day of TDD theory. It was awesome!

8 May 2013

XSLTunit Ant Support

some antsLast year I did the Prime Factors Kata in XSLT. Using XSLTunit I created a test case that applied the template to a number to calculate its prime factors. I focused on the coding problem and ignored the testing infrastructure, calling the XSLT processor from the command line and looking into the generated XML test result to see if any assert had failed. When I felt like playing with XSLT again, I first had to fix the infrastructure and make it ready for Continuous Integration.

Ant Support
CI tools like Jenkins are able to call any script, but my first idea for build automation is always Ant. I have some history with Ant but most likely I use it because I am an old school Java developer ;-). Fortunately Ant has build-in XSLT support which I can use to apply the XSLTunit template.
<property name="test_prefix" value="tst_" />
<property name="test_result_suffix" value=".test_result.xml" />

<!-- apply style to XMLs in suite -->
<xslt basedir="${suite}" destdir="${suite}" style="${suite}/${style}">
  <include name="*.xml" />
  <exclude name="*${test_result_suffix}" />
</xslt>

<!-- apply test -->
<xslt basedir="${suite}" destdir="${suite}" style="${suite}/${test_prefix}${style}"
      extension="${test_result_suffix}">
  <include name="*.xsl" />
  <exclude name="${test_prefix}*.xsl" />
</xslt>

<!-- create readable HTML test report -->
<xslt basedir="${suite}" destdir="${suite}" style="${basedir}/lib/xsltunit_report.xsl">
  <include name="*${test_result_suffix}" />
  <param name="testname" expression="${style}" />
</xslt>
My approach contains a lot of "convention over configuration". For example I assume that each template ${style} is located inside its own folder ${suite}. The given Ant target first transforms sample data with the template under test for manual review, then applies the test case to the template and finally generates a readable report out of the test result. The first and last step is optional but useful during development. In the end the test result is checked for the text "failed" which shows a failed assertion.
<dirname property="suite" file="${testResult}" />
<loadfile property="test_failed" srcFile="${testResult}" />
<fail message="Test ${suite} failed!">
  <condition>
    <contains string="${test_failed}"
              substring="outcome=&quot;failed&quot;" />
  </condition>
</fail>
Testing and checking the result is done for all templates in the source folder ${src} by using <foreach> from Ant-Contrib.
<foreach target="-testFolder" param="foreach">
  <path>
    <fileset dir="${src}">
      <include name="**/*.xsl" />
      <exclude name="**/${test_prefix}*.xsl" />
    </fileset>
  </path>
</foreach>

<foreach target="-verifyResult" param="foreach">
  <path>
    <fileset dir="${src}">
      <include name="**/*${test_result_suffix}" />
    </fileset>
  </path>
</foreach>

FizzBuzz Kata
After the unit tests were executed by Continuous Integration for each commit, I went for some XSLT exercise. Unfortunately adding the Ant support had taken too much of my scheduled learning time and I had to work with the smallest kata available, the FizzBuzz question: Write a program that prints the numbers from 1 to 100. But for multiples of three print "Fizz" instead of the number and for the multiples of five print "Buzz". For numbers which are multiples of both three and five print "FizzBuzz". I wrote the following template by following a few XSLTunit tests.
<xsl:template match="value" mode="fizzbuzz">
  <xsl:choose>
    <xsl:when test="number(current()) mod 15 = 0">FizzBuzz</xsl:when>
    <xsl:when test="number(current()) mod 3 = 0">Fizz</xsl:when>
    <xsl:when test="number(current()) mod 5 = 0">Buzz</xsl:when>
    <xsl:otherwise><xsl:value-of select="." /></xsl:otherwise>
  </xsl:choose>
</xsl:template>
To solve the complete question I applied the template to the values of 1 till 100,
<numbers>
  <number><value>1</value></number>
  <number><value>2</value></number>
  <number><value>3</value></number>
  <number><value>4</value></number>
  ...
  <number><value>100</value></number>
</numbers>
which generated a HTML file with the answer of the FizzBuzz question.
  • for 1 say 1
  • for 2 say 2
  • for 3 say Fizz
  • for 4 say 4
  • ...
  • for 100 say Buzz

23 November 2012

BaDaDam Testing Framework

DIY Cable ReleaseSome time ago I had a look into BDD - no, not Bullshit Driven Development ;-) - but Behaviour-Driven Development. I started with JBehave but it seemed heavyweight to me. Further it had more than ten dependencies to other libraries, some with possibly problematic licences. Still I wanted to know how that kind of framework worked, so instead of learning by taking it apart I decided to build my own.

For some time I struggled to find a nice name for the project and discussed my problem with Michael. I was mumbling as usual and he mistook BDD as BaDaDam. Voila - I present to you BaDaDam, a minimalistic BDD framework for Java. Obviously it follows the spirit of JBehave, is lightweight, self sufficient and depends only on JUnit. It allows you to write stories in plain text, implement them in Java classes and run them using JUnit. The first version is finished since some time and available in my Maven repository here.

28 May 2012

Eclipse Plugin Unit Testing

I will describe the set-up of a unit testing infrastructure for a big RCP application consisting of several plugins. In the first part I will write about how to run JUnit tests inside Eclipse.

PlugsTest Plugins
First we need a place to put the tests. We do not want to put the tests in the same project as the production code because we do not want to ship the tests. So for every plugin which we want to write tests for, we create a separate plugin with the same name suffixed with ".test". For example let's assume we want to write a JUnit test for a class in the company.product.general plugin. First we look for a plugin named company.product.general.test. Does it exist? Maybe we never used it and need to check it out from version control. Still nothing? OK, so we need to create it.

For a new test plugin we create a fragment project. We do that by selecting File > New > Project > Plug-in Development > Fragment Project. A fragment allows us to add code to another plug-in, known as the host. The fragment is merged with its host and has full access to all the packages declared in its host, not only the exported ones. Among other ways to set up tests for plugins, fragments generally are the best solution. So our unit tests have access to all classes of the host plugin. The fragment inherits all dependencies from the host. The only explicit required dependency of the test plugin is org.junit4. (Remember that various files in the build system might need to be updated for new (test) plugins to be part of the build. These might include some kind of loading rules, certain Ant build files or test features.)

The above set-up is enough to run simple JUnit tests which do not use any feature of Eclipse's OSGI container. To enable PDE JUnit tests the host plugin needs to define Eclipse-ExtensibleAPI: true in its MANIFEST.MF file. This allows the fragment to access classes from the host plugin and tells the PDE tooling to add the fragment to the classpath of the host plugin.

FragmentsTest Packages
We use the plugin's name as base package of the plugin's source code to avoid name collisions. For example all packages of the plugin company.product.general are sub packages of company.product.general. This is not the case for fragments, as fragments are merged with the host. All packages of the company.product.general.test fragment are still sub packages of company.product.general.

The package with the same name as the test fragment is used for common test set-up or custom assertion logic that might be used for more than one test. For example the package company.product.general.test might contain a class called UsLocale, which sets the current locale to US, as needed to test some business logic. Common test code can be reused by other test plugins if it is exported from the fragment. This works the same way as for normal plugins.

Warning about "Add Required Plugins"
Sometimes it is necessary to add new plugins to a launch configuration in order for the application to start. When you do that by using the "Add Required Plugins" button in the launch configuration, make sure that all the test plugins are not selected afterwards. We had some issues when test-only functionality somehow "leaked" into production.

Simple JUnit Tests
When the test plugin is ready, we can start writing simple JUnit tests. They will not necessarily be "simple", they are just called like that to distinguish them from PDE JUnit tests which have a more complex set-up. Even when developing RCP applications, these simple JUnit tests have their use. We can use them to test any class that is not dependent on the Eclipse Platform API. They are particularly useful for testing business logic and model classes.

JUnit tests go into the same Java package as the class under test. So the test plugin has the same packages as the host plugin. Test classes are named like the class under test suffixed with "Test". For example let's assume we want to test the class company.product.general.api.CatchHandlers in the plugin company.product.general. The related test is company.product.general.api.CatchHandlersTest in the company.product.general.test plugin.

FakeMocking
Unit testing is supposed to test classes or methods in isolation but classes usually depend on other classes. For more complex scenarios we want to use different objects than the live ones to get rid of these dependencies. These objects are provided to the class under test. By using them we can verify logic independently of the depended-on objects. Such objects are called Test Doubles.

Dummy objects, fakes and stubs are created easily with custom classes. But true mocks need some more logic to verify the order of method invocations (which needs some kind of state machine). Fortunately there are several frameworks that bring mocks to Java, for example EasyMock. To use it in our RCP application we need to create on OSGI bundle (i.e. an Eclipse plugin) from its jars. Let the plugin company.product.test.mock contain the mocking framework's jars. To use its classes in tests add it to the dependencies of the test fragment.

PDE JUnit Tests
PDE or Plugin JUnit tests are executed by a special test runner that launches another Eclipse instance and executes the test methods within that workbench. So PDE JUnit tests can be used for classes that are dependent on Eclipse Platform API. Parts of the Eclipse platform are available outside a running workbench as well, so a normal JUnit test might still be an option. Additionally some other parts, for example the extension registry can be mocked. But there are cases where a running workbench is necessary. For example you may assume a particular behaviour of Plugin.getStateLocation(), which would not be available if you were just using an ordinary JUnit test. Besides that the tests themselves look exactly like any other test.

When automating the build these PDE tests should be distinguished from basic ones. So we follow a different naming convention and call them *PlatformTests. Also note that the usual class extension mechanism might not work inside the workbench. You need to introduce interfaces for the types you want to mock.

5 September 2010

Maven Plugin Testing Tools

Obviously your MOJOs (read Maven Plugins) need to be tested as detailed as all your other classes. Here are some details on using the Maven Plugin Testing Tools and how I used them for the Macker Maven Plugin.

Dog in harnessUnit Testing
To unit test a MOJO create a JUnit test case and extend AbstractMojoTestCase. It comes with the maven-plugin-testing-harness:
<dependency>
  <groupId>org.apache.maven.plugin-testing</groupId>
  <artifactId>maven-plugin-testing-harness</artifactId>
  <version>1.2</version>
  <scope>test</scope>
</dependency>
Note that version 1.2 of the maven-plugin-testing-harness is still Maven 2.0 compatible, as will be version 1.3. The alpha release of version 2.0 is already based on Maven 3. In the test case the MOJO is initialised with a pom fragment and executed. The basic usage is well documented in the Maven Plugin Harness documentation. The usual folder structure for plugin unit tests is
.
 |-- pom.xml
 \-- src
     |-- main
     \-- test
         |-- java
         |   \-- MojoTest.java
         \-- resources
             \-- unit
                 |-- test-configuration1
                 |   \-- resources for this test if any
                 |-- test-configuration1-plugin-config.xml
                 \-- test-configuration2-plugin-config.xml
The class MojoTest contains methods testConfiguration1() and testConfiguration2(). There are several Maven projects using this approach, just do a code search.

Stubs
If your MOJO needs more complex parameters, e.g. a MavenProject or an ArtifactRepository, these have to be provided as stubs. Stubs are simple implementations of real Maven objects, e.g.
public class org.apache.maven.plugin.testing.stubs.ArtifactStub
  implements org.apache.maven.artifact.Artifact
and have to be configured in the pom fragment (test-configuration1-plugin-config.xml) as described in the Maven Plugin Testing Harness Cookbook. See also MackerMojoTest in the Macker Maven Plugin for an extensive example. There are several stubs to simulate Maven objects such as ArtifactHandler or ArtifactResolver. Creating stubs gets cumbersome when you need more of Maven's internals. Every object and every method has to be stubbed out.

Integration Testing
Integration testing is done with the maven-plugin-testing-tools:
<dependency>
  <groupId>org.apache.maven.plugin-testing</groupId>
  <artifactId>maven-plugin-testing-harness</artifactId>
  <version>1.2</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.apache.maven.plugin-testing</groupId>
  <artifactId>maven-plugin-testing-tools</artifactId>
  <version>1.2</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.codehaus.plexus</groupId>
  <artifactId>plexus-utils</artifactId>
  <version>1.5.6</version>
  <scope>test</scope>
</dependency>
Note that the testing tools 1.2 specifically need plexus-utils version 1.5.6. The usual folder structure for plugin integration tests is
.
 |-- pom.xml
 \-- src
     |-- main
     \-- test
         |-- java
         |   \-- MojoIT.java
         \-- resources
             \-- it
                 |-- test-case1
                 |   |-- pom.xml
                 |   \-- main
                 |       \-- classes and data for this test
                 \-- test-case2
The class MojoIT contains the test methods testCase1() and testCase2(). Folder test-case1 contains a full Maven module that uses the MOJO under test in some way.

safety netThe test uses PluginTestTool, BuildTool and the other tools from maven-plugin-testing-tools to create a local repository, install the Maven module under test into it and execute the Maven build of test-case1/pom.xml. The Plugin Testing Tools site provides more detail about this process and the various tools.

Setup
The Plugin Testing Tools do not provide an abstract test case. Each test has to create it's own AbstractPluginITCase. A good example is the AbstractEclipsePluginIT of the Eclipse Plugin. It contains methods to build the module, execute poms against it and verify created artefacts. As far as I know this is the only example available. AbstractOuncePluginITCase is a modified copy as well as AbstractMackerPluginITCase.

Execution
Integration tests should be executed in the integration-test phase.
<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <executions>
        <execution>
          <phase>integration-test</phase>
          <goals>
            <goal>test</goal>
          </goals>
          <configuration>
            <includes>
              <include>**/*IT.java</include>
            </includes>
            <excludes>
              <exclude>specified only to override config
                       from default execution</exclude>
            </excludes>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>
alternative routeThe PluginTestTool modifies the pom.xml to skip all tests, so the tests are not invoked again when the module is build by the integration test.

Help Mojo Workaround
Unfortunately there is a problem with the above approach. It works fine for modules which do not contain Maven Plugins. But it fails during integration test preparation when the help-mojo of maven-plugin-plugin is executed. Fortunately the guys from the Eclipse Plugin found a workaround:
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-plugin-plugin</artifactId>
  <!-- lock down to old version as newer version aborts
       build upon no mojos as required during ITs -->
  <version>2.4.3</version>
  <executions>
    <!-- disable execution, makes IT preparation using
         maven-plugin-testing-tools fail (see
         target/test-build-logs/setup.build.log) -->
    <execution>
      <id>help-mojo</id>
      <configuration>
        <extractors>
          <extractor />
        </extractors>
      </configuration>
    </execution>
  </executions>
</plugin>
But now the plugin can't be built from scratch any more. So they used a profile to run the integration tests and disable help-mojo execution.
<profiles>
  <profile>
    <id>run-its</id>
    <build>
      <plugins>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          ...
        </plugin>
        <plugin>
          <artifactId>maven-plugin-plugin</artifactId>
          ...
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>
Wrap-up
So far so good. The BuildTool has to activate the profile run-its (as it has to skip help-mojo execution). This could be done by setting a certain property, let's call it ProjectTool:packageProjectArtifact. Then the profile would only be activated during integration test preparation.
<profiles>
  <profile>
    <id>run-its</id>
    ...
    <activation>
      <property>
        <name>ProjectTool:packageProjectArtifact</name>
      </property>
    </activation>
  </profile>
</profiles>
I've submitted a patch for that, but in the meantime I had to copy the BuildTool into my own plugin, ugh. (I was working towards a clean solution throughout this post but in the end all gets messed up.) The whole plugin testing can be seen in action in the Macker Maven Plugin.

Acknowledgement
Experimenting with the Maven Plugin Testing Tools was part of a System One Research Day. Thank you System One for supporting Open Source :-)