Showing posts with label Ruby. Show all posts
Showing posts with label Ruby. Show all posts

20 March 2021

11 Years of Prime Factors Kata

In this post I want to celebrate the Prime Factors Code Kata. Prime Factors is a small coding exercise first used by Robert C. Martin in 2005. It is my favourite code kata and it has been almost nine years since I last wrote about it - time to change that. Weird enough, the first code kata I ever worked on - outside of university assignments that turned out to be katas later - was Roman Numerals in 2007. The first time I did the Prime Factors was during Christmas holidays 2009 in Java and Ruby:
import java.util.ArrayList;
import java.util.List;

public class PrimeFactors {
  public static List<Integer> generate(int n) {
    List<Integer> primes = new ArrayList<Integer>();

    for (int candidate = 2; candidate <= n; candidate++) {
      for (; n % candidate == 0; n /= candidate) {
        primes.add(candidate);
      }
    }

    return primes;
  }
}
Now the Java code is exactly the code Robert Martin showed, I was following his example. The Ruby version from back then looks pretty similar, just using while instead of for.
module PrimeFactors
  def generate(n)
    prime_factors = []

    candidate = 2
    while n > 1 do
      while n % candidate == 0 do
        prime_factors << candidate
        n /= candidate
      end
      candidate += 1
      candidate = n if candidate > Math.sqrt(n) # performance fix
    end

    prime_factors
  end
end
If you are wondering how I am still able to find the code, I organise my code katas to allow lookup and comparison. Since then I did the kata 141 times and it has many uses.

Learn a language
Prime Factors is one of the first pieces of code I write - Test Driven of course - when revisiting old languages, like Commodore BASIC or looking at a new language, like Forth, using Gforth 0.7:
: prime_factors ( n -- n1 n2 n3 n4 )
  DUP 1 > IF           \ test for ?DO
    DUP 2 ?DO
      BEGIN
        DUP I MOD 0 =  \ test candidate I
      WHILE
        I SWAP I /     \ add candidate, reduce number
      REPEAT
    LOOP
  THEN
  DUP 1 = IF DROP THEN ;

T{ 1 prime_factors -> }T
T{ 2 prime_factors -> 2 }T
T{ 3 prime_factors -> 3 }T
T{ 4 prime_factors -> 2 2 }T
T{ 6 prime_factors -> 2 3 }T
T{ 8 prime_factors -> 2 2 2 }T
T{ 9 prime_factors -> 3 3 }T
Gforth came with a modified testing framework based on John Hayes S1I's tester.fs, defining the functions T{, -> and }T for testing. Note that the given function prime_factors is not realistic as the number of returned arguments is not known by the caller.

When I had a look at Scala, of course I did Prime Factors:
import java.lang.Math.sqrt

object PrimeFactors {
  def generate(number: Int): List[Int] = {

    def fold(current: Pair[Int, List[Int]], candidate: Int): Pair[Int, List[Int]] = {
      if (current._1 % candidate == 0)
        fold((current._1 / candidate, candidate :: current._2), candidate)
      else
        current
    }

    val (remainder, factors) =
      (2 to sqrt(number).intValue).foldLeft((number, List[Int]()))(fold)

    if (remainder > 1)
      (remainder :: factors).reverse
    else
      factors.reverse
  }
}
This is crazy. The code creates a sequence of all candidate primes and folds it starting from left by dividing by the candidate recursively, appending to the begin of the list, which is cheap. Because of that the list is reversed at the end. I have no idea why I created this, probably I was playing around with foldLeft. This is not a good example, please do not copy it. After all these years, the Forth solution seems easier to grasp. ;-)

So which languages are missing? PowerShell looks much like my PHP (shown below) and my Python Prime Factors looks similar too, just with Python specific range(2, number + 1) and //= inside. And of course JavaScript is missing:
PrimeFactors = function() {
  this.factors = [];
};

PrimeFactors.prototype.generate = function(num) {
  var candidate;
  for (candidate = 2; candidate <= num; candidate += 1) {
    num = this.tryCandidate(num, candidate);
  }
  return this.factors;
};

PrimeFactors.prototype.tryCandidate = function(num, candidate) {
  while (num % candidate === 0) {
    num = this.reduceByFactor(num, candidate);
  }
  return num;
};

PrimeFactors.prototype.reduceByFactor = function(num, factor) {
  this.factors.push(factor);
  return num / factor;
};
Isn't that lovely? Again this is not good code, please do not copy it. At least I showed some creativity using prototype functions.

Learn a testing framework
Using TDD to write some known code is a perfect way to learn more about a testing framework. So I explored XSLTunit using Prime Factors in XSLT or NUnit in C#:
using NUnit.Framework;

[TestFixture]
public class PrimeFactorsTest
{
  [TestCase(new int[0], 1)]
  [TestCase(new int[] { 2 }, 2)]
  [TestCase(new int[] { 3 }, 3)]
  [TestCase(new int[] { 2, 2 }, 4)]
  [TestCase(new int[] { 2, 2, 2 }, 8)]
  [TestCase(new int[] { 3, 3 }, 9)]
  public void TestFactors(int[] expected, int number)
  {
    CollectionAssert.AreEqual(expected, PrimeFactors.Generate(number));
  }

  [Test]
  [Timeout(100)]
  public void TestLarge()
  {
    CollectionAssert.AreEqual(new int[] { int.MaxValue },
                              PrimeFactors.Generate(int.MaxValue));
  }
}
Test your own testing framework
Sometimes, when I need to create my own unit testing framework, e.g. TPUnit for old Turbo Pascal, assert-scm (Scheme R5RS) or ASM Unit for Windows Assembly, I use Prime Factors as test cases:
_prime_factors:
  mov     esi, 0          ; esi = number of factors
  mov     edi, ebx        ; edi = address of factors
  mov     ecx, eax        ; ecx = current number
  mov     ebx, 1          ; ebx = candidate

  jmp .not_diviseable

.loop_over_candidates:
  inc     ebx             ; next candidate

.break_if_candidate_is_larger_than_square:
; if candidate * candidate <= number then try candidate
  mov     eax, ebx
  mul     ebx
  cmp     eax, ecx
  jbe     .try_candidate

; else number is a (large) prime and we are done
  mov     [edi + esi * register_size], ecx
  add     esi, 1
  jmp     .done

.try_candidate:
; if number % candidate == 0 then add candidate
  mov     eax, ecx
  xor     edx, edx
  div     ebx
  cmp     edx, 0          ; remainder is 0
  jne     .not_diviseable

.is_diviseable:
  mov     [edi + esi * register_size], ebx
                          ; store candidate in factors
  add     esi, 1          ; we found a factor
  mov     ecx, eax        ; number is remainder of division
  jmp     .try_candidate  ; try current candidate again

.not_diviseable:
; if number > 1 then try next candidate
  cmp     ecx, 1
  jne     .loop_over_candidates

.done:
; return number of factors
  mov     eax, esi
  ret
This piece of assembly calcultes the prime factors of the number passed in EAX into in the dword array address EBX.

TDD Demo
In 2012 I started practising Prime Factors as kata performance, minimising the number of keys I pressed. I ran it around 50 times. In the end I used the practice to calm down when I was anxious - it was like mediation. I still have to perform it somewhere, adding music and all... I have used it demoing TDD in uncounted presentations - actually around 40: during my university guest lectures, user group meetings and at my clients. Most demos were in Java and some in PHP,
<?php
class PrimeFactors {
  static function generate($n) {
    $factors = [];
    for ($candidate = 2; $candidate <= $n; $candidate += 1) {
      while ($n % $candidate == 0) {
        $factors[]= $candidate;
        $n /= $candidate;
      }
    }
    return $factors;
  }
}
and a single demo of test driving R code,
primeFactors <- function(number) {
  factors <- vector(mode="integer")

  candidate <- 2
  while (candidate <= sqrt(number)) {
    while (number %% candidate == 0) {
      factors <- c(factors, candidate)
      number <- number / candidate
    }
    candidate = candidate + 1
  }

  if (number > 1) {
    factors <- c(factors, number)
  }

  factors
}
It seems, most programming languages look the same. The last sentence is not true for NATURAL, Cobol's cousin, which is ugly. I will not show it here as it would destroy this lovely post.

Conclusion
By writing this post, I learned that I still need to create Prime Factors in the programming languages Go, Kotlin, OpenOffice Basic, Oracle PL/SQL and of course TypeScript - I could - and I will, it is just a matter of time. So Prime Factors - in fact any small enough code kata - is a great tool for exploring, studying and practising programming languages, testing frameworks, IDE tools and Test Driven Development in general. I will leave you with my latest addition to my collection of Prime Factors, using C99. Have fun!
#include <math.h>

typedef struct {
  unsigned char count;
  unsigned int values[31];
} PrimeFactors;

void PrimeFactors_init(PrimeFactors* factors)
{
  (*factors).count = 0;
}

void PrimeFactors_add(PrimeFactors* factors, const unsigned int factor)
{
  int count = (*factors).count;
  (*factors).values[count] = factor;
  (*factors).count = count + 1;
}

void generate(const unsigned int number, PrimeFactors* factors)
{
  PrimeFactors_init(factors);

  unsigned int remaining = number;
  for (unsigned int candidate = 2; candidate <= sqrtl(remaining); candidate += 1) {
    while (remaining % candidate == 0) {
      PrimeFactors_add(factors, candidate);
      remaining /= candidate;
    }
  }

  if (remaining > 1) {
    PrimeFactors_add(factors, remaining);
  }
}

17 August 2019

In Memory of Ruby

Meet Ruby, our dog. Ruby was a Great Dane. If you are not into dogs, Great Danes are known for their size, being one of the largest dog breeds in the world. Great Danes are also known for their friendly nature and are often referred to as a "gentle giants".

Great Dane Ruby
Ruby lived well beyond the average lifespan of her breed of six to eight years and passed away last month with an age of ten. She died of old age in our arms. This was - and still is - a sad and painful situation which only dog lovers can understand.

(And of course, Ruby was named after the Ruby programming language, my favourite programming language of that time.)

12 March 2013

Complexity Slope

GeeCON is a great conference with many good presentations. Last year, a talk by Keith Braithwaite on Measuring the Effect of TDD on Design particularly piqued my interest. He talked about Cyclomatic Complexity and that code with tests is more strongly biased towards simpler methods than code without tests. Keith started this interesting research in 2006 and you can read everything about it in a series of blog posts about complexity and test-first. Unfortunately there is no recording of the session Keith gave at GeeCON, but he gave a similar talk at QCon 2008. I will not repeat his research here, so you should read the series of posts or at least watch the recording to continue.

Complexity Slope of Apache Harmony 1.6M3Measure It!
Keith wrote a tool called Measure to harvest the distribution of Cyclomatic Complexity throughout a code base. It is based on Checkstyle and analyses Java source code. It worked out of the box, so I gave it a try.

Some Numbers
My current project, a large RCP application, has a complexity slope as bad as 1,75. It has really bad code, there are many large and complex methods and no tests at all. (There were no tests when I joined the project last year. Now with me on the project, the number of tests is growing slowly but steadily ;-) I am wondering if the used framework has an impact in the complexity slope as well. RCP is known for its complex dependency structure by its overuse of the singleton patten.

Another project that I worked on some years ago was a large web application and it was probably as bad as the current one. After five years of heavy refactoring and retrofitting with unit tests up to 60% code coverage, it scored almost two (1,99). It seems to be a special case. The application was never test-first and as such should score below two. The refactoring activities did not target complexity but were guided by simple metrics and layering rules.

Usage
If you want to see the complexity slope of your projects download measure-0.3.1.zip and unpack it. It contains everything you need to run Measure. There is a bash for Linux and a I added a batch for Windows. If you run one of them they will print a short help message. I ran it against my little BDD testing framework and it showed me a nice score of almost three (2,94). This was expected as I had used strict TDD while developing it. After my latest refactoring where I reduced the complexity of the story parser considerably, it even scored beyond three (3,12).

Complexity Slope of BaDaDam Testing-Framework
What about Ruby?
Keith's Measure supported Java by using Checkstyle under its hood. In fact it did little more than parse the Checkstyle complexity report with a threshold of zero, so reporting all complexity values. I used Saikuro, a Ruby Cyclomatic Complexity Analyser to implement the same for Ruby. First I created a Saikuro Rake task and set Saikuro's complexity threshold to zero,
state_filter = Filter.new(0)
  state_formater = Saikuro::StateHTMLComplexityFormater.new(STDOUT, state_filter)
  ...
  idx_states, idx_tokens = Saikuro.analyze(@files, state_formater, nil, @output_dir)
  write_cyclo_index(idx_states, @output_dir)
Then I ran Saikuro to generate the complexity report into some folder,
rake -f ruby\saikuro_task.rb dir=some_folder
added a Saikuro parser to Measure,
measure -k Saikuro -l some_folder
and wrote some glue code to bring all these things together,
measure_ruby -t JavaClass -d E:\Develop\Ruby\JavaClass\lib

Complexity Slope of JavaClass (Ruby)
Note that the scripts are only available for Windows, but the Rake task and Java code work on all platforms. Also note that Saikuro messes up RDOC, so both cannot be in the same Rake file.

19 December 2011

JavaClass 0.4 released

Much has happened since I created JavaClass almost three years ago. I started with version 0.0.1 because much was missing in there. Since then I managed a small release once a year. Version two added class names and references. Last year I added classpath abstractions and moved the project from Rubyforge to Google Code (and then in the future to Bitbucket and eventually to GitHub). This year I finished version 0.0.4. Now the Java class file parser supports everything except fields and methods. Adding them would be no big deal, I just had no need to do it till now. This fourth release feels more complete and should rather be named version 0.4 than 0.0.4. So the next version will be 0.5.

Coffee BeansNew in JavaClass 0.4
I added support for Java 5 Enums and Annotations to JavaClass::ClassFile::JavaClassHeader and write code to retrieve all interfaces implemented by a class. Further I created Maven- and Eclipse-aware classpath abstractions. If you point JavaClass to the folder of a whole Eclipse workspace, you get a JavaClass::Classpath::CompositeClasspath containing all projects found in that workspace. Finally I fixed some defects, e.g. a character to byte conversion problem found in Ruby 1.9 and a printf inconsistency between Windows and Linux platforms. See the history of JavaClass 0.4 for the complete list of changes.

RDOC documentation was improved and several examples have been added. In fact these examples are real world scripts that I use to analyse code bases. They are just formatted nicely and commented in very detail.

Find Unused Classes
For example let us look for unused classes. By "unused" I mean classes that have no incoming reference inside a code base. Such classes might still be referenced by Java Reflection, but more likely are just dead code. I have seen legacy code bases where more than 10% of all classes were unused and their deletion was a welcomed reduction of the maintenance burden. A list of classes that can (potentially) be deleted is a valuable asset for the next clean-up cycle. Still, each class on the list needs to be double checked because classes with main methods, Servlets and other framework classes might show up on the list but still be used.
require 'javaclass/dsl/mixin'
require 'javaclass/classpath/tracking_classpath'

location = 'C:\Eclipse\workspace'
package = 'com.biz.app'

# 1) create the (tracking) composite classpath of the given workspace.
cp = workspace(location)

# Create a filter to limit all operations to the classes of our application.
filter = Proc.new { |clazz| clazz.same_or_subpackage_of?(package) }

# 2) load all classes of the application, this can take several minutes.
classes = cp.values(&filter)

# 3) for all classes, mark all referenced types as accessed.
cp.reset_access
classes.map { |clazz| clazz.imported_types }.flatten.
  find_all(&filter).
  each { |classname| cp.mark_accessed(classname) }

# 4) also mark classes referenced from config files.
scan_config_for_3rd_party_class_names(location).
  find_all(&filter).
  each { |classname| cp.mark_accessed(classname) }

# 5) find non accessed classes.
unused_classes = classes.find_all { |clazz| cp.accessed(clazz) == 0 }
The above example uses EclipseClasspath and TrackingClasspath under the hood. TrackingClasspath adds functionality to record the number of usages of each class. The classes that are not used by any imports (step 3) or any hard coded configurations (step 4) are potentially unused.

Java Class Literals in Ruby
Inspired by date literals in Ruby I thought about Java class name literals in Ruby. This feels a bit crazy so it is not enabled by default. After including JavaClass::Dsl::JavaNameFactory you can write
clazz = java.lang.String    # => "java.lang.String"
Note the absence of any quotes around the right hand side of the above assignment. The returned class name is not just a plain String, but also a JavaQualifiedName with some extra methods to work on Java names, like
clazz.to_java_file          # => "java/lang/String.java"
clazz.full_name             # => java.lang.String"
clazz.package               # => "java.lang"
clazz.simple_name           # => "String"
Adding these rich strings caused me a lot of trouble, but that is another story.

22 October 2011

Freeze Custom Ruby Strings When Used as Keys in Hash

Last week I spent quite some time chasing a single issue in my JavaClass Ruby gem. It really annoyed me and I could not find anything useful even using Google. I had to dig deep. Read what happened: I began with some kind of rich string, quite similar to the following class:
class RichString < String
  def initialize(string)
    super(string)
    @data = string[0..0] # some manipulation here
  end
  def data
    @data
  end
end

word = RichString.new('word')
puts word               # => word
puts word.data          # => w
That was not special and worked as expected.

Lost ... !!Then I happened to use instances of RichString as keys in a hash. Why shouldn't I? They were still normal Strings and their data should be ignored when used in the hash.
map = {}
map[word] = :anything

word_key = map.keys[0]
puts word_key           # => word
puts word_key.data      # => nil
The last line warned me "instance variable @data not initialized". Oops, my little @data went missing indicated by the bold nil in the last line. First I did not know what was causing the problems. I was baffled as all tests were green and had a good coverage. I spent some time digging and rewriting a lot of functionality until I found that Hash#keys() caused the trouble when given my RichStrings as hash keys.
puts word == word_key   # => true
puts word.object_id == word_key.object_id  # => false
Aha, Hash changed the keys. It's reasonable to prohibit key changes, so a String passed as a key will be duplicated and frozen. (RTFM always helps ;-) But how did it do that? It did not call dup() on the RichString. As Hash is natively implemented, I ended up in the C source hash.c.
/*
*  call-seq:
*     hsh[key] = value        => value
*     hsh.store(key, value)   => value
*/

VALUE
rb_hash_aset(hash, key, val)
  VALUE hash, key, val;
{
  rb_hash_modify(hash);
  if (TYPE(key) != T_STRING || st_lookup(RHASH(hash)->tbl, key, 0)) {
    st_insert(RHASH(hash)->tbl, key, val);
  }
  else {
    st_add_direct(RHASH(hash)->tbl, rb_str_new4(key), val);
  }
  return val;
}
So when the key is a String and not already included in the hash, then rb_str_new4 is called. (I just love descriptive names ;-) Furthermore string.c revealed some fiddling with the original key.
VALUE
rb_str_new4(orig)
  VALUE orig;
{
  VALUE klass, str;

  if (OBJ_FROZEN(orig)) return orig;
  klass = rb_obj_class(orig);
  if (FL_TEST(orig, ELTS_SHARED) &&
      (str = RSTRING(orig)->aux.shared) &&
      klass == RBASIC(str)->klass) {
    long ofs;
    ofs = RSTRING(str)->len - RSTRING(orig)->len;
    if ((ofs > 0) || (!OBJ_TAINTED(str) && OBJ_TAINTED(orig))) {
      str = str_new3(klass, str);
      RSTRING(str)->ptr += ofs;
      RSTRING(str)->len -= ofs;
    }
  }
  else if (FL_TEST(orig, STR_ASSOC)) {
    str = str_new(klass, RSTRING(orig)->ptr, RSTRING(orig)->len);
  }
  else {
    str = str_new4(klass, orig);
  }
  OBJ_INFECT(str, orig);
  OBJ_FREEZE(str);
  return str;
}
Frozen StringI didn't quite understand what was going on in rb_str_new4(), but it was sufficient to read a few lines: If the original string was frozen, then it was used directly. I verified that.
map = {}
map[word.freeze] = :anything

word_key = map.keys[0]
puts word_key           # => word
puts word_key.data      # => w
Excellent, finally my @data showed up as expected. Fixing the problem added some complexity dealing with frozen values, but it worked.

Freeze your custom Ruby strings when you use them as keys in a hash (and want to retrieve them with Hash#keys())

21 March 2010

Horror of Time Accounting

The Passage of TimeRecently Uncle Bob mentioned that there is so much time wasted by developers on "fiddling around with time accounting tools". That's so true. During my 11 years of professional software development I had the opportunity to witness different approaches used by companies. I understand the need to monitor and control spent times but I hate wasting my own time on time accounting stuff. So I always try to minimise the time needed by automating the process of filling in the forms (if possible). Remember Terence John Parr's quote: "Why program by hand in five days what you can spend five years of your life automating?" :-)

Spread Yourself
Of course, if you have to track times of your tasks, you have to write them down. If you work on different tasks during the day then spreadsheets work best for that, especially if you spend some time automating them further. For example you may use VBA to write some Excel Macros. (Or even use some Ruby OLE magic.) Early in my career I was lucky to meet Andreas, who provided me with his detailed Excel sheet.

Since then I have been using it. It's just great. The most useful feature is adding a new line with one key shortcut (CTRL-T). The new line is filled with the current date and the current (rounded) time or the ending time of the last task. This makes entering new lines extremely fast: CTRL-T, fill project and subproject number, add description. Andreas' sheet worked so well for everybody that it became the official time accounting tool for all employees later. (Thank you Andreas for your great spreadsheet!)Todo Macro SheetAnother company I worked with did not have any kind of time accounting. That was nice for a change, but I did not believe in it and kept using my macro infested Excel sheet. Later, when they started using their own tool based on Oracle Forms, I wrote a little application that imported the lines from my sheet into the database. It was quite simple (less than 100 lines) to read a CSV, do some translations and insert the data into a database. Unfortunately companies usually do not allow employees to write directly into their time accounting database (for obvious reasons).

Timed Overkill
Last year, while working for a big bank, I experienced the overkill of time accounting. There were three different systems. Project management used XPlanner to track task progress. XPlanner is a web application. I could have "remote controlled" it using plain HTTP GETs and POSTs, but that would have been cumbersome. So I used Mechanize to create a small API to access XPlanner. The API is outlined by the following code, which was written for XPlanner version 0.6.2. To save space I removed all error handling. (The current XPlanner version features a SOAP interface, so remote controlling it gets even simpler.)
class XplannerHttp

PLANNER = "#{HOST}/xplanner/do"

def initialize
@agent = WWW::Mechanize.new
@agent.redirect_ok = true
@agent.follow_meta_refresh = true
end

# Login to XPlanner web application.
def login(user, pass)
login_page = @agent.get("#{PLANNER}/login")
authenticate = login_page.form_with(:name => 'login/authenticate')
authenticate['userId'] = user
authenticate['password'] = pass
authenticate['remember'] = 'Y' # set cookie to remain logged in.
authenticate.click_button
end

# Book for a task with _taskid_ (5 digits).
# _date_ is the date in "YYYY-MM-DD" format.
# _hours_ is the time in decimal hours in format "0,0".
def book_time(taskid, date, hours)
task_page = @agent.get("#{PLANNER}/view/task?oid=#{taskid}")

# open edit time
add_link = task_page.links.find do |l|
l.href =~ /\/xplanner\/do\/edit\/time\?/
end
edit_page = add_link.click

# add new times and submit
timelog = edit_page.form_with(:name => 'timelog')
c = timelog['rowcount'].to_i-1

timelog["reportDate[#{c}]"] = date
timelog["duration[#{c}]"] = hours.to_s
timelog["person1Id[#{c}]"] = '65071' # id of my XPlanner user
timelog.click_button
end
end
Once the API was in place, a simple script extracted the values from the spreadsheet (in fact from an FX input field) and booked it. I just used XPlanner ids as subprojects and copied the cumulated sums of my Excel into the input field.
bot = XplannerHttp.new
bot.login(USER, PASS)
text.split(/\n/).each do |line|
parts = line.scan(SUM_LINE).flatten
parts[1] = parts[1].sub(/,/, '.').to_f
bot.book_time(parts[2], parts[0], parts[1])
# => bot.book_time(65142, '2009-01-22', '1,5')
end
The second application was used by the HR department to monitor working and extra hours. This was a web application using frame sets and Java Script. Mechanize does not (yet?) support Java Script, so I had to look somewhere else. I started remote controlling my browser by OLE automation, which turned out to be too difficult (for me). Fortunately other people have done a great job creating Watir, which does exactly that. So here we go (again without any error handling):
class StimeIe

def initialize
@browser = Watir::Browser.new
end

def login(user, pass)
@browser.goto("#{STIME}")
@browser.wait

login_frame = @browser.frame(:index, 4)
login_frame.text_field(:name, 'Num').set(user)
login_frame.text_field(:name, 'Password').set(pass)
login_frame.button(:name, 'Done').click
@browser.wait
end

# Book for a day with _date_ in format "DD.MM.YYYY".
# _from_ is the time of coming in "HH:MM" format.
# _to_ is the time of leaving in "HH:MM" format.
def book_time(date, from, to)
navigate_to 'Erfassung/Korrektur'
input_frame = @browser.frame(:index, 4)
if input_frame.text_field(:name, 'VonDatum').text != date
input_frame.text_field(:name, 'VonDatum').set(date)
@browser.wait
end

# add new times and submit
input_frame.text_field(:name, 'VonZeit').set(from)
input_frame.text_field(:name, 'BisZeit').set(to)
input_frame.button(:id, 'Change').click
input_frame.button(:id, 'Done').click
@browser.wait
end

# Logout and close the browser.
def close
navigate_to 'Abmelden'
@browser.close
end

def navigate_to(link_name)
nav_frame = @browser.frame(:index, 3)
nav_frame.link(:text, link_name).click
@browser.wait
end
end
A little script would extract the first and last time of a day from the spreadsheet and call the bot:
bot = StimeIe.new
bot.login
times = ...
times.keys.sort.each do |date|
from_until = times[date]
bot.book_time(date, from_until.from, from_until.until)
# => bot.book_time('09.03.2009', '09:00', '18:00')
end
bot.close
A Nut to Crack
The third application was the hardest to "crack": A proprietary .NET application used by controlling. I spent (read wasted) some time reverse engineering server and database access. (.NET Reflector is a tool worth knowing. It is able to decompile .NET DLLs.) Then I started scripting the application with AutoIt. AutoIt is a freeware scripting language designed for automating Windows GUIs. Using win32ole AutoIt can be scripted with Ruby:
class Apollo

APOLLO_EXE = 'Some.exe'
APOLLO_WIN_NAME = 'APOLLO'

CTID = '[CLASS:WindowsForms10.SysTreeView32.app.0.19c1610; INSTANCE:1]'

def initialize
@autoit = WIN32OLE.new('AutoItX3.Control')
end

# Open with local Windows credentials.
def login
@autoit.Run(APOLLO_EXE)

# Wait for the exe to become active.
@autoit.WinWaitActive(APOLLO_WIN_NAME)
end

# Book for an _apo_ (5 digits and decimals).
# _text_ is the name of the sub-item.
# _date_ is the date in "DD.MM.YYYY" format.
# _hours_ is the time in full hours.
# _mins_ is the time in remaining minutes.
def book_time(apo, text, date, hours, mins)
# select the tree control
@autoit.ControlFocus(APOLLO_WIN_NAME, '', CTID)

# select the APO number
@autoit.Send(apo)

@autoit.Send('{RIGHT 5}')

...
# continue with a lot of boring code
# selecting controls and sending keystrokes.
end

def close
@autoit.WinClose(APOLLO_WIN_NAME)
@autoit.WinWaitClose(APOLLO_WIN_NAME)
end
end
Velvet MiteMite
Recently I happened to use mite. It claims to be an "advanced, yet simple tool to help you get things done". Well that's true. It's easy to use and has only the most basic features needed for time accounting. More important there is an API for developers. Good. Even better there is an official Ruby library for interacting with the RESTful mite.api: mite-rb. Excellent.

No need to change my good old spreadsheet. Using a new class MiteBot instead of XplannerHttp and friends the driver script looked quite similar.
class MiteBot

# Mapping spreadsheet activities to mite project names.
CATEGORIES = {
:default => 'Main Product',
'Meeting' => 'Meeting',
...
}

def initialize
Mite.account = ACCOUNT # your mite. account
end

# Login to the mite web application with _key_
def login(key)
Mite.key = key
end

# Book a task in _category_ with detailed _comment_
# _date_ is the date in "DD.MM" format.
# _hours_ is the time in decimal hours in format "0,0".
def book_time(date, hours, comment, category='')

key = if category =~ /^.+$/ then category else :default end
proj_name = CATEGORIES[key]

# parse date
day, month = date.split(/\./).collect{ |i| i.to_i }.to_a
timestamp = Date.new(YEAR,month,day)

# parse hours
minutes = (hours.to_s.sub(',', '.').to_f * 60).to_i

# get project
proj = find_project_for(proj_name)

# create time entry
e = Mite::TimeEntry.new(:date_at => timestamp, :minutes => minutes,
:note => comment, :project_id => proj.id)

# add new times
e.save
end

# Find the project with _name_ and return it.
def find_project_for(name)
Mite::Project.all(:params => {:name => name}).first
end
end

# usage with data read from spreadsheet/FX input field
bot = MiteBot.new
bot.login(USER_KEY)
...
# => bot.book_time('15.12', '2,25', 'Introduction of Radar', '')
Conclusion
So what is the point of this blog post besides giving you some ideas how to automate time accounting? I want you to stop moaning about it! Take the initiative. Do something about it. Trust me, developing little automation scripts is fun and rewarding on its own.

2 May 2009

Fragments of cool code

From time to time I stumble over a piece of code, that just looks "cool". For example date literals in Ruby:
class Fixnum
def /(part)
[self, part]
end
end

class Array
def /(final_part)
Time.mktime(final_part, self[1], self[0])
end
end

13/11/2008 # => Thu Nov 13 00:00:00 +0100 2008
or a proper name for a throw:
catch (Exception up) {
// log the error
throw up;
}
The best name for a JUnit 4 test fixture (free to choose) I ever saw was in a presentation given by Peter Petrov:
@Before
public void doBeforeEachTest() {
A good name for a Ruby binding was
module Regenerated
Inside = binding
end
eval(script, Regenerated::Inside) where the evaluation is done within the scope of another module Regenerated which acts as a namespace.

5 April 2009

Sony Ericsson Theme File Format

Recently I wanted to pimp my Sony Ericsson mobile and add some nice themes (.thm files). There were plenty of them around and I just downloaded a collection of some hundred free ones. Unfortunately most came without a preview image and going through these on my mobile to find nice ones was painstaking and slow: I had to activate each of them, return to the standby mode, check out the main image, return to the theme manager, and start over again. What I needed was some kind of theme browser. I did not trust the Sony Ericsson Themes Creator application to provide browsing and did not bother installing it.

Sony Ericsson G705 - Keypad As being a hard core developer I had a look into the THM files. I STFW but could not find any resources explaining the file format. So I explored it a bit myself. Here is what I found together with some Ruby code.

Sony Ericsson Theme File (.THM) Format
The file contains several files concatenated together with some kind of header before the data.
  • At 0x0000 a null-terminated string with the name of the original file. Remaining bytes are all null. This is read with name = data[ofs..ofs+0x63].strip.
  • At 0x0064 several null-or-blank-terminated strings containing octal numbers. The number fields may contain blanks which are treated as separators as well. Parse them with
    numbers = data[ofs+0x64..ofs+0x100].
      scan(/[0-7]+/).
      collect { |n| n.to_i(8) }
  • The only important number is the fourth (body_len = numbers[3]), which is the length of the data body in bytes. If this number is not positive then the file is corrupt.
  • At 0x0101 there is most of the time the string ustar and at 0x0129 nogroup, but we do not need them.
  • At 0x0200 starts the data: body = data[ofs+0x200...ofs+0x200+body_len].
  • After 0x0200+body_len the space is filled with nulls until the next address continuing a zero low-byte. We just skip them until data[ofs] != 0.
  • Repeat these steps until EOF.
Usually the first item in the theme file is called Theme.xml. This is an XML configuration file and looks something like
<Sony_Ericsson_theme version="4.5">
  <Background Color="0xb5f8fd"/>
  <Background_image Source="desktop.png"/>
  <Desktop Color="0xb5f8fd"/>
Sometimes it's not called in this way, but any XML file will do. The Color looks like a regular HTML Colour Code. The Source is a reference to an image stored inside the theme file. The important image types are
  • Standby_image - main image.
  • Desktop_image - background in the menu.
  • Popup_image - background of popup.
  • There are a lot more, but often these three images are reused in smaller themes.
Knowing the standby image name, it is possible to extract it to the file-system (File.open(filename,'wb') {|io| io.write body}). Then browsing these preview images with some image utility and deleting unwanted ones is a piece of cake. The whole Ruby program is here. Have fun!

23 April 2008

Redland RDF library and Ruby

Ever wanted to use the Redland RDF library with Ruby? In case you do not know Redland, it's a library to manipulate RDF triples and perform SPARQL queries as needed for semantic web applications. Redland is written in C++ and provides bindings to other languages like Java or Ruby.

Although current version is 1.0.7, windows binaries are only available for 1.0.3. Ok, it's my fault, I could always compile the stuff myself, but as a true Java person, I try to stay away from such obscenities. (This is probably another argument for Joel that Java coders can't be real programmers).

So I took version 1.0.3 and hoped that everything would be fine. When testing some of its command line tools, it turned out that still some libraries were missing: msvcp71.dll and msvcr71.dll. This should have been a hint to me, obviously these files belong to Microsoft's Visual C++ 7.1, but I did not bother and was happy to find them online in some dll archive.

Bound But the Ruby binding still needed another library (read a dll) to run, which was nowhere to find. So I was forced to fool around with Devccp (Mingw) myself to compile it. Unfortunately Ruby's extconf.rb works only with MSVC, so the auto-generated make file was of little use. After a painful day I finally managed to compile the given redland_wrap.c into a nice little redland.dll.

So far so good. Redland's Ruby API looked easy and some unit tests showed the way. But why did some tests work and some failed with a segmentation fault. In fact, why did most Redland functions crash for URIs longer than 20 characters? So after 2 days wasted, this looked like obviously incomplete compilation or linkage problems.

You might think that the Mingw compiled code might not be fully compatible with Ruby's core libraries, which are supposed to be MSVC6 stuff on Windows systems. Well, there are some issues about this topic, but here it was not the case. Much later I was able to access an old MSVC6 installation and compile the Redland Ruby binding again. This time the Ruby's extconf.rb worked nicely and make produced a redland.so. Still the functions crashed the same.

In the end it turned out, that the Redland 1.0.3 binaries for Windows where compiled using Visual C++ .NET 2003 (aka VC 7.1) , which needs the dotNET 1.1 framework runtime, which I choose not to install. Couldn't the system tell me that some runtime dependency is missing instead of producing segmentation faults all the time?

Downloads
In case you face similar problems, here are the results of my "hard" work: