Patching Io Addons

This project is getting out of hand. I just wanted to use Regular Expressions in Io. This is the forth part of me trying different extensions of Io. First I installed addons without native code, then I compiled native code of addons and in the third part I:

• Checked package.json and build.io for hints about needed third party libraries and headers.
• Found ported libraries for Windows in GnuWin32, e.g. ReadLine.
• Compiled Io addons with dependencies.
• Fixed the undefined reference to 'IoState_registerProtoWithFunc_' error, which occurs using addons created for older versions of Io.
• Worked around conflicting headers included by IoObject.h or IoState.h.
• Finally compiled with dependencies on the native code of another addon.

On the way I added minor fixes to several addons, see my (forked) Io repositories and today I cover addons which needed more extensive modifications.

JSON Parsing
Io has "half" JSON support, most objects provide an asJson() method to represent the contained data in JSON. Some addons (like Docio) require Sequence.parseJson() but omit a specific dependency for that and my Io (iobin-​win32-​current.zip) misses this method. These addons are for a newer version of Io, which has not been released (yet?). Writing a JSON parser is a nice exercise and there is already a JSON parser for Io. It is neither an addon nor an Eerie package. Preparing some arbitrary Io code as addon means moving the existing files into the proper folder structure, and adding the files needed to run as addon (i.e. proto and depends files). I even added a test. My Io JSON addon can be cloned directly into the addons folder %IO_HOME%\​lib\​io\​addons.

Missing Io Core Functions
I started changing addon source code to include the JSON parser addon, at the same time I wanted to keep my changes as little as possible. The Io Guide said that if you have a .iorc file in your home folder, it will be eval'ed before the interactive prompt starts.. In my %HOMEDRIVE%%HOMEPATH%\.iorc I added all definitions of missing Io core functions:

false isFalse := method( true )
true isFalse := method( false )

// load parseJson
Sequence hasSlot("parseJson") ifFalse(
    Json
)

false asJson := "false"
true asJson := "true"
nil asJson := "null"

This made me ask myself, what else was added to Io since my Windows binary was built in 2013? I cloned Io and compared the proper tag with master. There were many changes and I filtered out formatting and comments. The final result was a short list of additions and modifications like

• Addon and AddonLoader extensions to use Eerie.
• Core tildeExpandsTo() using UserProfile instead of HOME on Windows.
• Sequence power() and powerEquals().
• TestRunner run() returning the number of errors.

Docio
With JSON support and Markdown working, I can run Docio. Docio is the documentation generator for Eerie packages. It extracts tagged comments from C and Io code and generates Markdown and HTML documentation like this. To make Docio work, I had to work around several issues:

1. Docio's repository name is lowercase. To install it as addon the folder name has to be uppercase, as for Kano:

   > cd "%IO_HOME%\lib\io\addons"
   > git clone git@github.com:IoLanguage/docio.git Docio

2. Docio does not depend on native code, and it has Markdown as its sole dependency.

   > echo Markdown > Docio\depends
   > echo Docio > Docio\protos

3. Docio loads Eerie eagerly in the first line of Docio.io. It uses Eerie to query its template path. I lack Eerie and the eager loading fails, so I remove that. With Eerie out of the picture, I always have to provide the full path to the template (%IO_HOME%\lib\io\addons\Docio\template). Now I can initialise Docio with io -e "Docio println". Success.
4. Docio has a starter script in bin/docio which needs a Windows version, i.e. a bin/docio.bat which calls the original starter script,

   io "%IO_HOME%\lib\io\addons\Docio\bin\docio" %*

5. With the starter script Docio is available as a command line tool. One thing to know is that its help is wrong. It says

   docio package=/path/to/package [template=/path/to/template]

   while it really needs two dashes for each option,

   docio --package=/path/to/package [--template=/path/to/template]

   Both paths need to be absolute, otherwise documentation files appear in strange places.
6. There is a syntax error in DocsExtractor.io line 65 and following.
7. Copying of binary resources, i.e. fonts, corrupts the files. You will have to copy them manually. This is because Io's low level C code does fails to set the binary flag when opening files. On Windows the C library functions consider text files to be finished on the first byte 0x1A. I am unable to fix that right now.

After fixing all that, Docio provides help, even inside the Io REPL, which comes handy:

Io> Docio printDocFor("ReadLine")
Binding to GNU readline.

Docio will generate the documentation for any package on the fly which has a package.json with a name field. Bundled addons miss that file, and I create empty ones containing each addon's name for addons where I want to generate documentation. Here is my own Docio with all the fixes.

Socket
Finally I am going for the Socket addon. I was scared of Socket, some sources stated that Io socket support has always been tricky. Socket depends on libevent 2.0.x, an event notification library. Surprisingly autogen.sh, configure and make install compile and link libevent without any problems. Now I have some libevent_*.dll files. This is way too easy.

But of course Socket's C code does not compile. The header sys/​queue.h is not available on Windows. Fortunately libevent comes with its own compat/​sys/​queue.h. Some header files, e.g. IoEvConnection.h, IoEvHttpServer.h and several others need the conditional include:

#include "Socket.h"
#if !defined(_WIN32) || defined(__CYGWIN__)
#include <sys/queue.h>
#else
#include <compat/sys/queue.h>
#endif
#include <event.h>

Further UnixPath.c fails compilation, and I drop it from the list of files to compile. Even if it would compile, it would not work as indicated by the error returned from IoUnixPath for defined(_WIN32) || defined(__CYGWIN__): Sorry, no Unix Domain sockets on Windows MSCRT.

Linking the compiled files produces an undefined reference to 'GetNetworkParams@8', which is part of the Windows GetNetworkParams interface in iphlpapi.dll, as listed on build.io together with ws2_32. It seems that after my exploration of building addons with native dependencies like ReadLine or Markdown, compiling Socket as difficult than that. Here are the important pieces of the steps:

> ...
> cd source
> gcc -fPIC -D _NO_OLDNAMES -D IO_WINSOCK_COMPAT -c ...
> gcc -liovmall -lbasekit -levent -liphlpapi -lws2_32 ^
      -shared -Wl,--out-implib,libIoSocket.dll.a ^
      ... -o libIoSocket.dll
> ...

My fork of Socket contains the required fixes for Windows.

Using the Socket addon on current Ubuntu
In my product development Coderetreat template, I have GitHub actions to verify the sample code. Based on Sandro's installation instructions I first install the required dependencies:

sudo apt-get install libpcre3-dev libevent-dev

(This might be unecessary as GitHub's Ubuntu image has both of them installed.) The current branch of libevent is 2.1.x and the none of the older 2.0.x versions do compile due to wrong dependencies. At the same time, the Linux version of Io contains Socket but needs libevent 2.0.5 specifically. I have no idea if that is the way to fix these kind of issues, but it works, so I link the installed version (2.1.7) as the required (2.0.5) one.

ls -la /usr/lib/x86_64-linux-gnu/libevent*.so.*
sudo ln -s /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7.0.1 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5

(I leave the ls is the action to show the current version when the GitHub runner changes and there is a newer version of libevent.) After that I download the "latest" version of Io and install it:

wget -q http://iobin.suspended-chord.info/linux/iobin-linux-x64-deb-current.zip
unzip -qq iobin-linux-x64-deb-current.zip
sudo dpkg -i IoLanguage-2013.11.04-Linux-x64.deb
sudo ldconfig
io --version

The Linux version of Io comes with almost all addons pre-compiled, and there is no need for any compilation. Success. Because this version is without Eerie, custom addons have to be installed manually into the System installPrefix folder, e.g. adding Docio

git clone https://github.com/codecop/Docio.git
io -e "System installPrefix println"
sudo mv Docio /usr/local/lib/io/addons/
io -e "Docio println"

Now the runner is ready to execute some tests which is usually done with io ./tests/correctness/run.io. The full GitHub action for Io is here.

Random Facts about Addons
During my exploration I learned more things about Io's addon structure:

• The protos file does not need to contain the name of the "main" prototype, which is also the name of the addon, and often it does not. I put the name of all (exported) prototypes there to simplify my scripts. Then package.json, the Eerie manifest, does contain all prototypes.
• In the beginning I though depends was some kind of manifest, listing dependencies. But it is only needed for dependencies of native code, so AddonLoader loads dependent addons before they are used in native code. For non native addons, Io will load whatever is needed when parsing any unknown type name. Till now the only addon I have seen which needs that feature is Regex.
• When an addon is loaded all its files are evaluated. Only then is it registered as active. This can lead to addon initialisation loops. The initialisation order seems to be by file name. Some addons with complex inter dependencies - like Socket - prefix Io files with A_0, A_1 and so on to ensure ordered initialisation. (This is a bit annoying for tooling as the prototype name usually equals the file name in Io.)

Summary: My Feelings Towards Io
While Io is dead since more than ten years, working with it feels bleeding edge, even more it is outside the frontier. You are at your own, there is no help. Nobody is coming to save you. The latest article I found was written in 2019. There are less than ten (!) active blog posts: Blame it on Io (2006), Io language on Windows (2014), Io Basics (2015) and Io Programming Language (2019) - to list the best ones. There are a handful of Stack Overflow questions and a few repositories on GitHub - which are sometimes incompatible with the "latest" Io. ChatGPT understands the language but fantasises the libraries, so no help from AI neither. I am used to modern languages with a rich ecosystem, e.g. Java, C#, Python and this is an unfamiliar feeling. At the same time it is a refreshing puzzle. Maybe I will come back for a vacation in uncharted territory.

Integrating Io Addons

This is the third time I write about making existing Io extensions work under Windows. While I still struggle with Socket - a crucial dependency - I install and compile random addons to see what could go wrong. Almost every addon I touch has a different issue that I need to resolve for my Io Windows build dating back to 5.11.2013. Till now I have covered the following situations:

• Clone current addons right into the addons folder, see part 1, Kano.
• Rename addon folders to start with an uppercase letter.
• Create the required protos file with a list of all exported prototypes and depends file with a list of required prototypes, which is usually empty.
• Create the missing Windows starter bat for tools like Kano.
• If the name of the addon is missing in the package.json, Docio will be unable to process the package. I will cover Docio next time.
• Compile the native code using MingW GCC, see part 2, Rainbow.
• Fix linker issues regarding one versus two underscores in name decoration.
• Copy native headers to _build/​headers, which I will need today.
• Resolve initialisation recursion issue when loading addons.

I walk through the installation of addons with different needs and increasing difficulty using concrete examples. As many addons are wrappers around C libraries, which "may not be installed on your system" - I will work on integrating these libraries today.

Addons with (Pre-Compiled) Dependencies: ReadLine
The Io ReadLine addon is a binding to GNU readline. I have no use for it but it is small and a good example. I start as usual with cloning, creating the init file and required files:

> cd "%IO_HOME%\lib\io\addons"
> git clone git@github.com:IoLanguage/ReadLine.git
> io generate.io . ReadLine
> cd ReadLine
> touch depends
> echo ReadLine > protos
> cd source
> gcc -fPIC -D _NO_OLDNAMES -c IoReadLine.c IoReadLineInit.c
IoReadLine.c: fatal error: readline/readline.h: No such file or directory

This is expected as ReadLine binds to libreadline. For more complex cases it helps to look at the Eerie package manifest package.json:

{
  "name": "ReadLine",
  ...
  "dependencies": {
    "libs":     ["readline"],
    "headers":  ["readline/readline.h", "readline/history.h"],
    "protos":   [],
    "packages": []
  }
}

I need libreadline and two of its headers. The easiest way to build a Linux library for Windows is not build it. There is GnuWin32 @ SourceForge. It was last updated 2010, which is around the same time that active Io development stopped. Lovely there is readline-5.0-1-bin.zip which is all I need. Now I can continue building the addon.

> gcc -liovmall -lbasekit -lreadline5 ^
      -shared -Wl,--out-implib,libIoReadLine.dll.a ^
      IoReadLine.o IoReadLineInit.o -o libIoReadLine.dll
IoReadLine.o: undefined reference to `IoState_registerProtoWithFunc_'

Sigh, this was supposed to be the easiest case. I have seen this method call before. Older addons from the 2009 source release used this function, while addons bundled with my 2013 version use IoState_registerProtoWithId_. The required change in source/​IoReadLine.c, line 47 is

  using_history();

-  IoState_registerProtoWithFunc_((IoState *)state, self, protoId);
+  IoState_registerProtoWithId_(state, self, protoId);

  IoObject_addMethodTable_(self, methodTable);

Now compilation and linking succeeds.

> gcc -fPIC -D _NO_OLDNAMES -c IoReadLine.c IoReadLineInit.c
> gcc -liovmall -lbasekit -lreadline5 ^
      -shared -Wl,--out-implib,libIoReadLine.dll.a ^
      IoReadLine.o IoReadLineInit.o -o libIoReadLine.dll
> del *.o
> move /Y lib*.* ..\_build\dll
> copy /Y *.h ..\_build\headers
> cd ..\..
> io -e "ReadLine readLine() println"

The last step is to copy readline5.dll into the %IO_HOME%\​bin folder, where Io's DLLs are located. This makes running Io independent from my build environment. Here is my Io ReadLine with the necessary fixes. A nice surprise after installing ReadLine is that the Io CLI remembers my input.

Addons with Dependencies: Markdown
Io's Markdown addon is next in line. It is a Markdown parser for Io, based on discount. Its package.json states that it needs libmarkdown and a header mkdio.h. I need to compile the library first. The addon comes with the library's source code, and even a compiled version for Windows x64. It has a build.io, which will rewrite Eerie's AddonBuilder to represent a receipt for your package. Eerie should be able to build it. What does it say?

AddonBuilder clone do(

  srcDir := Directory with(Directory currentWorkingDirectory .. "/source/discount")

  compileDiscountIfNeeded := method(
    if((platform == "windows") or(platform == "mingw"),
      appendLibSearchPath(Path with(Directory currentWorkingDirectory, "deps/w64/lib") asIoPath)
      appendHeaderSearchPath(Path with(Directory currentWorkingDirectory, "/deps/w64/include") asIoPath)
    ,
      prefix := Directory currentWorkingDirectory .. "/_build"
      Eerie sh("cd #{srcDir path} && " ..
               "CC='cc -fPIC' ./configure.sh --prefix=#{prefix} && " ..
               "make && " ..
               "make install" interpolate)
      appendHeaderSearchPath(Path with(Directory currentWorkingDirectory, "_build/include") asIoPath)
      appendLibSearchPath(Path with(Directory currentWorkingDirectory, "_build/lib") asIoPath)
    )
  )

  compileDiscountIfNeeded

  dependsOnLib("markdown")
  dependsOnHeader("mkdio.h")
)

I have little experience with building C, still the bold line above gives me enough information to build discount. Tweaking its makefile and librarian.sh a bit does the trick of creating DLLs. It would have been nice to use Eerie instead... I follow the usual steps as for ReadLine above to build the addon and in the end io -e "Markdown toHtml(\"# A1\") println" displays <h1>A1</h1>.

Addons with (Conflicting) Dependencies: UUID
There waits a different problem when installing UUID, Io's wrapper around libuuid. To get started there is a MinGW compatible libuuid, thank you Alessandro Pilotti. The typical commands sh.exe ./configure, make && make install work. Between configure and make I had to #define HAVE_USLEEP 1 in the configured config.h. Having built libuuid, I can work on Io's UUID. The problem is that both libuuid (in uuid.h) and MinGW (in basetyps.h) define the type uuid_t differently. I dislike messing with MinGW include files.

Maybe I can avoid including basetyps.h from the code I want to compile? Io's source contains more than 40 headers, one for each Io object. Even the smallest addon needs to include IoObject.h and IoState.h, which provides access to the whole Io virtual machine - the largest headers in the source - which include other headers on the way. I start copying required includes into a stand-alone header. It is a boring and repetitive work, and brute force wins the day. I end up copying 500 lines of structs, method declarations, and macros from various headers. Using that isolated header instead of the provided ones, addon compilation and linking works as expected.

Addons with Io and Third Party Dependencies: Regex
My whole exploration of Io addons started with the Regex package. I really wanted to use Regular Expressions. I am a big fan of Regex, and Mastering Regular Expressions is one of my favourite books since 2006. I still consider it one of the most exciting technical books of all times. The Regex addon supports Perl regular expressions using the PCRE library. There is a suitable GnuWin32 PCRE build. Maybe I am lucky today.

This addon is special, it depends on a third party library libpcre3, and it depends on the native code of another Io addon, Range. It is shown inside build.io:

AddonBuilder clone do(
  dependsOnLib("pcre")
  dependsOnHeader("pcre.h")

  dependsOnBinding("Range")

  debs atPut("pcre", "libpcre3-dev")
  ebuilds atPut("pcre", "pcre")
  pkgs atPut("pcre", "pcre")
  rpms atPut("pcre", "pcre-devel")
)

I will have to consider this when building it.

> cd "%IO_HOME%\lib\io\addons"
> git clone git@github.com:IoLanguage/Regex.git
> io generate.io . Regex
> cd Regex/source
> gcc -fPIC -D _NO_OLDNAMES -I ..\..\Range\_build\headers ^
      -c IoRegex.c IoRegexInit.c IoRegexMatch.c IoRegexMatches.c Regex.c
> gcc -liovmall -lbasekit -lpcre3 -L ..\..\Range\_build\dll -lIoRange ^
      -shared -Wl,--out-implib,libIoRegex.dll.a ^
      IoRegex.o IoRegexInit.o IoRegexMatch.o IoRegexMatches.o Regex.o -o libIoRegex.dll
> del *.o
> move /Y lib*.* ..\_build\dll
> copy /Y *.h ..\_build\headers
> cd ../..
> io -e "Regex; \"11aabb\" matchesOfRegex(\"aa*\") asStrings println"
list(aa)

If there is a linker error,

Creating library file: libIoRegex.dll.a
IoRegexMatches.o: undefined reference to `_imp__IoRange_new'
IoRegexMatches.o: undefined reference to `_imp__IoRange_setRange'

it is the same name decoration problem as before. The solution is to move out libIoRange.dll.a from Range\​_build\​dll temporarily.

If io -e Regex fails during runtime with Exception: Error loading object 'lib\​io\​addons\​Regex\​_build\​dll\​libIoRegex.dll': 'Did not find module' or similar, then Range functionality was not loaded in advance. Make sure that depends contains Range and that it does not end with a newline at the end.

The next episode (part 4) will deal with patching some broken addons like Docio.

Compiling Io Addons

This is the second part of my journey into installing Io addons. A quick recap: Io "is a dynamic prototype-based programming language". It was made more? widely known by Bruce Tate's book Seven Languages in Seven Weeks. Io's extensions, i.e. libraries or packages however you like to call them, are called addons and follow the same structure. Eerie, Io's package manager, fails on my platform and I have to add addons manually. It is the main purpose of these posts to show how it can be done. I explain the installation of addons with different needs and increasing difficulty using concrete examples. In part one I covered Kano as an example for installing addons without any native code..

Compiling an Empty Addon on Windows: Kano
As an example, I want to compile an addon without any native code. I noticed that all installed addons, even the one without any native code, e.g. CGI, do have an init file and a dll. Maybe it is not strictly necessary. AddonLoader, a helper to the importer, checks for more than one C file - but I will need it later anyway. How does such an empty init file look like? For example, in the distribution IoCGIInit.c is

#include "IoState.h"
#include "IoObject.h"

__declspec(dllexport)

void IoCGIInit(IoObject *context)
{
}

Easy enough to create such a file Kano/​sources/​IoKanoInit.c. In the source distribution which I will mention later, there is generate.io in the root of the addons, which does exactly that. The script accepts a path to the addons dir and the name of the addon:

> cd "%IO_HOME%\lib\io\addons"
> io generate.io . Kano
> cd Kano\source
> gcc -fPIC -D _NO_OLDNAMES -c IoKanoInit.c
IoKanoInit.c:1:21: fatal error: IoState.h: No such file or directory

Of course I need headers. Where is the source distribution?

Io Source Distribution
The latest binary I downloaded from the Io site is iobin-​win32-​current.zip. Little version information there. Io reports io --version a version of 5.9.2011 but the executables and libraries were in fact created 5.11.2013. The nearest source release is 4.12.2013 which seems to be compatible. (The older 5.9.2011 source release is definitely incompatible with the addons included in iobin-​win32-​current.zip.) The source distribution is also interesting because it contains all sources of the standard library in libs\​iovm\​io\*.io.

The required headers are in libs\​iovm\​source\*.h and libs\​basekit\​source\*.h. After setting proper include paths (C_INCLUDE_PATH), compilation is possible. The switch -fPIC seems to be redundant on Windows, but it is set in one of Io's makefiles, so I keep it. I have no idea what I am doing ;-) The switch -D _NO_OLDNAMES defines _NO_OLDNAMES which solves the error: conflicting types for 'SSIZE_T' which occurs when old names are allowed. Sigh. Again, no idea what I am doing.

> cd Kano\source
> gcc -fPIC -D _NO_OLDNAMES -c IoKanoInit.c
> gcc -liovmall -shared -Wl,--out-implib,libIoKano.dll.a ^
      IoKanoInit.o -o libIoKano.dll
> mkdir ..\_build\dll
> move lib*.* ..\_build\dll
> cd ..\..
> io -e "Kano supportedFiles println"
list(make.io, Kanofile, kanofile, Kanofile.io, kanofile.io)

Compiling Native Code: Rainbow
Next I tackle addons with native code but without any third party dependencies. Most addons of this type are included in the distribution, because their compilation does only depend on Io itself, e.g. Blowfish, Box, LZO, MD5, Random, Range, etc. An addon I found which is missing is Rainbow, which "allows you to colourize command line output." Rainbow does not have a protos file and it has no package.json neither, but there is an eerie.json which does not say anything about prototypes. The addon contains a single Io file Rainbow.io and a single C header IoRainbow.h. I guess this is the prototype then - see line (1) in the following shell commands:

> git clone https://github.com/IoLanguage/Rainbow
Cloning into 'Rainbow'...
> touch Rainbow\depends
> echo Rainbow > Rainbow\protos && rem (1)
> io -e "Rainbow println"
Exception: Failed to load Addon Rainbow - it appears that the addon exists but needs compiling."

This is a useful error message. Io's AddonLoader checks native sources when loading an addon. Compile it. First

> io generate.io . Rainbow

creates a different IoRainbowInit.c file this time,

#include "IoState.h"
#include "IoObject.h"

IoObject *IoRainbow_proto(void *state); // (2)

__declspec(dllexport)

void IoRainbowInit(IoObject *context)
{
    IoState *self = IoObject_state((IoObject *)context);

    IoObject_setSlot_to_(context, SIOSYMBOL("Rainbow"), IoRainbow_proto(self)); // (3)
}

I have marked relevant references to the addon a.k.a. the "primary" prototype in bold face. Line (2) is the forward reference to code in Rainbow.c, which will create a new prototype. Line (3) sets this prototype under the symbol of its name into a slot of the context. (Io stores all variables in slots, many functions in the standard library work with slots, e.g. slotNames() or getSlot​(name).) There is some more information about writing C addons for Io in the Io Wikibook. For example it explains the addon structure - which I already knew but at least my "research" is validated. ;-)

Back to the compilation of Rainbow. Continue after line (1) in the previous script:

> cd Rainbow\source
> gcc -fPIC -D _NO_OLDNAMES -c IoRainbowInit.c IoRainbow.c
> gcc -liovmall -shared -Wl,--out-implib,libIoRainbow.dll.a ^
      IoRainbowInit.o IoRainbow.o -o libIoRainbow.dll
Creating library file: libIoRainbow.dll.a
IoRainbow.o:IoRainbow.c:(.text+0x5c): undefined reference to `_imp__IoTag_newWithName_'
... more errors omitted

Of Course, Linker Errors
These errors are strange and I spent a long time investigating. The libiovmall.dll.a exports __imp​__IoTag_​newWithName_, with two underscores in the beginning but the linker only looks for one. I verified this with nm libiovmall.dll.a | grep __imp__. This is due to some compiler inconsistency between Linux and Windows and/or GCC and/or MingW regarding function name decoration. GCC Options "-fleading-underscore and its counterpart, -fno-leading-underscore, change the way C symbols are represented in the object file." I tried both of them but then other symbols like fprintf are not found any more. There might be a linker option -Wl,--add-stdcall-underscore but it is not recognised by my MingW GCC. Sigh.

The solution is to not link against the link libraries (*dll.a), but against the dlls directly. (Either delete the *.dll.a from the io/lib folder or change your LIBRARY_PATH to use io/bin instead.) Summary: When your linker looks for a single underscore in dll exported symbols, and the library exports correctly, link against the dll instead of the lib..

The change to the library path fixes the linking and libIoRainbow.dll is created. The dll must be moved to the build result folder ..\​_build\​dll as before. In addition all headers should be copied to _build\​headers, as other addons might depend on them.

> mkdir ..\_build\dll
> move lib*.* ..\_build\dll
> mkdir ..\_build\headers
> copy *.h ..\_build\headers
> cd ..\..
> io -e "Rainbow println"
Rainbow_0x29ba488

All addons contained in the Windows distribution have the _build\​headers folder, but there are no headers. I will need these headers to compile Regex in part three. Let's iterate all addons and copy the headers:

set ADDONS=%IO_HOME%\lib\io\addons
dir /A:D /b "%ADDONS%" > addons.txt
for /F %a in (addons.txt) do ^
    if exist "%ADDONS%\%a\source\*.h" ^
        copy /Y "%ADDONS%\%a\source\*.h" "%ADDONS%\%a\_build\headers"
del addons.txt

Addon Initialization Recursion Issue
One recurring issues I faced was that AddonLoader was recursing endlessly trying to load something. For example, when I created IoRainbowInit.c by hand, I made a mistake with the symbol in line (3). On loading the addon, AddonLoader entered an endless loop,

> io -e "Rainbow println"
IOVM: Received signal. Setting interrupt flag.

current coroutine
---------
Object Rainbow            Rainbow.io 14
FileImporter importPath   Z_Importer.io 49   <-
FileImporter import       Z_Importer.io 125    |
List detect               Z_Importer.io 125    |
true and                  Z_Importer.io 125    |
Object Rainbow            Rainbow.io 14        |
Importer import           Z_Importer.io 138    |
FileImporter importPath   Z_Importer.io 49   --
...

This was because the proto was not registered under its correct name, see SIOSYMBOL in line (3) above, and the importer tried to import it again, found the addon (again), and repeated. Tip: When addon importing enters a loop, double check the symbol name in the C init.

A similar error happens when the a prototype of an addon uses another prototype of that addon during code loading. For example Eerie comes with two prototypes: Eerie and SystemCommand. The first line of Eerie.io imports SystemCommand. Now when I evaluate Eerie println, the AddonLoader finds the Eerie addon (by name) and loads the file Eerie.io to load that prototype. During import it sees SystemCommand as a dependency and looks for it. It finds it in the proto file of Eerie and tries to load it. When loading the addon, it loads the prototype of the same name, i.e. Eerie.io again.

> io -e "Eerie println"
IOVM: Received signal. Setting interrupt flag.

current coroutine
---------
Object SystemCommand        Eerie.io 4
Addon load                  AddonLoader.io 124  <-
AddonLoader loadAddonNamed  Z_Importer.io 97      |
FolderImporter import       Z_Importer.io 125     |
List detect                 Z_Importer.io 125     |
true and                    Z_Importer.io 125     |
Object SystemCommand        Eerie.io 4            |
Importer import             Z_Importer.io 138     |
Addon load                  AddonLoader.io 124  --
...

Cyclic imports are a bad idea anyway, it is fair that this does not work. Still I have not seen any warning about it in the Io documentation. Tip: Avoid top level "imports" of prototypes from the same addon.

Compiling Native Code: Thread
Another addon without dependencies is Thread. It has a proto, depends and package.json file. Using generate.io I follow all the steps from Rainbow above, just with different C files and compilation and linking succeeds. Unfortunately Thread createThread("1+1"), as shown in the documentation, terminates the Io VM when evaluating the expression string (1+1). There are no further hints or messages. I have no idea why and no means to debug the Io virtual machine.

Follow along in Part three about compiling addons with third party dependencies.

Installing Io Addons

Io is an interesting programming language. I had not planned to learn a new language in 2024. Then, in a quiet moment, I reached for Seven Languages in Seven Weeks by Bruce Tate. It is an older book, published in 2010 and I know some of the languages described there. I like studying programming languages. Playing with basic concepts without the need to finish larger tasks is relaxing. It should be an easy read.

The Io Language
Io "is a dynamic prototype-based programming language." It seems more or less dead since 2010, basically since the book mentions it. Its site states "actively developed until 2008". While there are a few updates from last year, the latest available build for Windows dates to 4th of December 2013, more than ten years ago. Much of Io is written in C and one should be able to build it from source using cmake. While I like the retro character of plain C, similar to old Pascal, I am un-experienced in the C ecosystem. I am using Windows and building C has been proven to be a hassle due to dependencies and compiler inconsistencies. So I went with the provided Windows build.

Io is prototype based, like JavaScript and looks different than your usual C-styled languages. The core library iovm comes with a unit testing framework. The UnitTest is an old-school xUnit, e.g.

FizzBuzzTest := UnitTest clone do (

  setUp := method(
    super(setUp)
    self fizzbuzz := FizzBuzz clone
  )

  // ...

  testFizz := method(
    result := fizzbuzz single(3)
    assertEquals("Fizz", result)
  )

  // ...

  testTo5 := method(
    result := fizzbuzz upto(5)
    assertEquals(list("1", "2", "Fizz", "4", "Buzz"), result)
  )

)

The corresponding Fizz Buzz is

FizzBuzz := Object clone do (

  single := method(n,
    if (n % (3 * 5) == 0, "FizzBuzz",
      if (n % 5 == 0, "Buzz",
        if (n % 3 == 0, "Fizz",
          n asString)))
  )

  upto := method(limit,
    Range 1 to(limit) map(n, single(n))
  )
)

This could be written more concise, but such was my first Io code. I should probably do a Prime Factors, too. It seems that I am obsessed with unit tests, as it is usually the first thing I look for in a new language, e.g. Scheme or Assembly. If you want to know more about Io, read the guide or the second chapter of Seven Languages in Seven Weeks. While the language is dormant, it will change the way you see programs - exactly as Bruce promises.

Io Addons
The Io distribution contain a bunch of extensions collected under the IoLanguage GitHub organisation. Many extensions are wrappers around well tested C libraries, but these "may not be installed on your system already." And this is probably the reason why the Windows distribution only contains 28 out of the 80+ available addons. While I ignore MySQL or graphics library wrappers, basic functions like socket communication or regular expressions would be nice. There is Eerie, the Io package manager, which should be able to deal with compiling addons, but its installation fails on my system - because of a dependency that needs a missing C library (recursion ;-). Then I try to add addons manually. It is the main purpose of this post to explain how it can be done.

Let me start with some general structure of Io addons: Installed or bundled addons are located in %IO_HOME%\​lib\​io\​addons and share the same folder structure as expected by Io's AddonLoader. Assume an addon named Foo, then the following folders and files (could) exist:

• _build: Several folders with the build results of C code, mostly empty.
• _build\dll\libIoFoo.dll and libIoFoo.dll.a: Compiled C code, its dynamic library and GCC import library file.
• _build\headers: Empty for bundled addons, but should contain the C headers of exported functions, i.e. the Io*.h files from source.
• bin: Starter scripts for addons which are command line tools. e.g. Kano.
• io\*.io: Main Io source code, i.e. new prototypes. There is at least a Foo.io.
• samples\*.io: Optional sample code.
• source\*.c|h: C source code, at least IoFooInit.c. The empty folder must exist.
• tests\correctness\run.io and one or more *Test.io: Unit tests, the run.io runs all tests in the current folder.
• tests\performance\*.io: Optional performance tests, sometimes with run.io.
• depends file: a list of other prototypes (or addons) this addon depends on.
• protos file: a list of all prototypes this addon provides.

Addons are activated in Io by using the prototype with the name of the addon. In an Io REPL or script, Foo will find the addon and import it making all its prototypes available. This works for all provided prototypes given the manifest (i.e. the protos file) is correct.

Io Package Management: Eerie
While working with the different types of addons, see this and the next article, I seem to have reverse engineered much of Io's build and package infrastructure ;-). Knowing about it upfront would have helped. Eerie is the package manager for Io. Unfortunately its installation failed on my system - and it seemed way too complicated for me because it created multiple environments, much like Python's virtualenv. Its documentation is "coming soon", and not helping. Some addons on GitHub are Eerie packages and miss necessary files like protos. While sometimes these files are still there - an oversight as it seems, Eerie packages contain other files:

• package.json is the Eerie manifest. It contains the description of the package and its dependencies. This file is useful.
• eerie.json is some kind of manifest, used in the Eerie Package Storage Database. Only two addons have it and the database is empty.
• build.io will "rewrite AddonBuilder to represent a receipt for your package." Most addons have this and it contains information about needed native libraries and transitive dependencies.

It seems that Eerie's PackageInstaller extractDataFromPackageJson() method creates the protos, depends and build.io files from package.json. I will have to do this by hand. Unfortunately some package.json are incomplete and some miss important information.

Now I explain the installation of several addons with different needs and increasing difficulty using concrete examples:

Addons without Any Native Code: Kano
There are some addons without any C source, e.g. CGI, Continued​Fraction, Rational and integrations like Bitly, Facebook, etc. All of these are included in the distribution. When looking for addons, I checked all addon repositories for ones without native code and found Docio, Eerie and Kano. I guess these were left out because they are part of Eerie. Let's look at Kano. Kano is a simple Make/​Rake inspired tool for Io. It uses a Kanofile to declare tasks. Let's get it running:

> cd "%IO_HOME%\lib\io\addons"
> git clone https://github.com/IoLanguage/kano
Cloning into 'kano'...
> io -e "Kano println"
Exception: Object does not respond to 'kano'

This is broken. io -e "<expression>" runs Io and passes the command, much like Ruby does. Passing the prototype name asks Io to load it and it is not found. By Io convention prototypes start with an uppercase letter while instances (and fields and variables) start with a lowercase one. Somewhere (in the source of Importer maybe) I read that the names must match for importing. Also this is one of only three repositories which lowercase name, i.e. docio, eerie and kano. Tip: Watch out for addon repository names matching the primary prototype.

> ren kano Kano
> io -e "Kano println"
Exception: unable to read file '...\lib\io\addons\Kano\depends'

Ah progress. It is missing the depends and protos files I mentioned earlier. It has a package.json instead:

{
  "version": 0.1,
  "author": "Josip Lisec",
  "description": "Rake-like utility for Io.",
  "readme": "README.textile",
  "category": "Utility",
  "dependencies": { },
  "protos": ["Kano", "Namespace", "Namespaces", "ObjectDescriber"]
}

No dependencies and four prototypes.

> touch Kano\depends
> echo Kano Namespace Namespaces ObjectDescriber > Kano\protos
> io -e "Kano println"
Exception: Unable to open directory ...\lib\io\addons\Kano\source

Progress again but why does Io look for C sources? In some addons, I see an empty source folder with a .gitisdumb file in it, to keep the folder in version control. Maybe it is needed.

> mkdir Kano\source
> io -e "Kano supportedFiles println"
list(make.io, Kanofile, kanofile, Kanofile.io, kanofile.io)

Nice, no error. Kano works. To be fully useable, there are a few more things to fix:

1. Kano is a tool, it comes with bin/kano starter script. Each starter script needs a Windows version, i.e. a bin/kano.bat which calls the original starter script,

   io "%IO_HOME%\lib\io\addons\Kano\bin\kano" %*

   And both scripts need to be in the PATH. The easiest way is to copy kano.bat to %IO_HOME%\bin, which contains io.exe.
   
   
2. Kano offers kano [-T|V|help|ns] [namespace:]taskName arg1 arg2... as help. -T lists all tasks defined in the local Kanofile and -V shows its version. At least it tries to do that. It calls Eerie to get its version. Now Eerie depends on Kano and Kano depends on Eerie, and I do not have Eerie, this is all bad. If you want everything to work, you can replace line 44 in Kano's Namespaces.io:

    V := option(
      """Prints Kano version."""
   -  version := Eerie Env named("_base") packageNamed("Kano") \
                       config at("meta") at("version")
   +  version := File thisSourceFile parentDirectory parentDirectory \
                      at("package.json") contents parseJson at("version")
      ("Kano v" .. version) println)

3. Much later I noticed that Kano's package.json does not have a name field. This is needed for Docio to generate the documentation. Sigh. (Here is my patched Kano.)

Follow along in part two.

This is the last time

It is August, it is hot and it is time to do some alternate work:


Quick Fix
Looking at the picture made me think. Every industry seems to have its quick fixes: Engineers use duct tape and WD-40. In software development, we take short cuts by violating existing structure, e.g. adding another conditional resulting in convoluted logic. And in the construction industry it clearly is the usage of polyurethane or spray foam. As every tool it has its use, and like in the picture it is especially useful for temporary work. Unfortunately most professional construction workers I commissioned wanted to use it for everything - like duct tape: A hole in the wall? Why use brick and mortar when it is faster to put some foam into it. A gap in the boards? Why care to work accurately when it is easier to close the gap with foam afterwards. Not enough steam brake to cover the ceiling? Let's fill the remaining area with foam. You get the idea. While some people like the speed such quick fixes provide, e.g. Joel Spolsky praises the Duct Tape Programmer, I do not agree. Anyway, this project must come to an end and I sincerely hope it is the last time. ;-)

Freeze Custom Ruby Strings When Used as Keys in Hash

Last week I spent quite some time chasing a single issue in my JavaClass Ruby gem. It really annoyed me and I could not find anything useful even using Google. I had to dig deep. Read what happened: I began with some kind of rich string, quite similar to the following class:

class RichString < String
  def initialize(string)
    super(string)
    @data = string[0..0] # some manipulation here
  end
  def data
    @data
  end
end

word = RichString.new('word')
puts word               # => word
puts word.data          # => w

That was not special and worked as expected.

Then I happened to use instances of RichString as keys in a hash. Why shouldn't I? They were still normal Strings and their data should be ignored when used in the hash.

map = {}
map[word] = :anything

word_key = map.keys[0]
puts word_key           # => word
puts word_key.data      # => nil

The last line warned me "instance variable @data not initialized". Oops, my little @data went missing indicated by the bold nil in the last line. First I did not know what was causing the problems. I was baffled as all tests were green and had a good coverage. I spent some time digging and rewriting a lot of functionality until I found that Hash#keys() caused the trouble when given my RichStrings as hash keys.

puts word == word_key   # => true
puts word.object_id == word_key.object_id  # => false

Aha, Hash changed the keys. It's reasonable to prohibit key changes, so a String passed as a key will be duplicated and frozen. (RTFM always helps ;-) But how did it do that? It did not call dup() on the RichString. As Hash is natively implemented, I ended up in the C source hash.c.

/*
*  call-seq:
*     hsh[key] = value        => value
*     hsh.store(key, value)   => value
*/

VALUE
rb_hash_aset(hash, key, val)
  VALUE hash, key, val;
{
  rb_hash_modify(hash);
  if (TYPE(key) != T_STRING || st_lookup(RHASH(hash)->tbl, key, 0)) {
    st_insert(RHASH(hash)->tbl, key, val);
  }
  else {
    st_add_direct(RHASH(hash)->tbl, rb_str_new4(key), val);
  }
  return val;
}

So when the key is a String and not already included in the hash, then rb_str_new4 is called. (I just love descriptive names ;-) Furthermore string.c revealed some fiddling with the original key.

VALUE
rb_str_new4(orig)
  VALUE orig;
{
  VALUE klass, str;

  if (OBJ_FROZEN(orig)) return orig;
  klass = rb_obj_class(orig);
  if (FL_TEST(orig, ELTS_SHARED) &&
      (str = RSTRING(orig)->aux.shared) &&
      klass == RBASIC(str)->klass) {
    long ofs;
    ofs = RSTRING(str)->len - RSTRING(orig)->len;
    if ((ofs > 0) || (!OBJ_TAINTED(str) && OBJ_TAINTED(orig))) {
      str = str_new3(klass, str);
      RSTRING(str)->ptr += ofs;
      RSTRING(str)->len -= ofs;
    }
  }
  else if (FL_TEST(orig, STR_ASSOC)) {
    str = str_new(klass, RSTRING(orig)->ptr, RSTRING(orig)->len);
  }
  else {
    str = str_new4(klass, orig);
  }
  OBJ_INFECT(str, orig);
  OBJ_FREEZE(str);
  return str;
}

I didn't quite understand what was going on in rb_str_new4(), but it was sufficient to read a few lines: If the original string was frozen, then it was used directly. I verified that.

map = {}
map[word.freeze] = :anything

word_key = map.keys[0]
puts word_key           # => word
puts word_key.data      # => w

Excellent, finally my @data showed up as expected. Fixing the problem added some complexity dealing with frozen values, but it worked.

Freeze your custom Ruby strings when you use them as keys in a hash (and want to retrieve them with Hash#keys())

Maven Plugin Harness Woes

Last year I figured out how to use the Maven Plugin Harness and started using it. I added it to my Maven plugin projects. Recently I started using DEV@cloud. DEV@cloud is a new service which contains Jenkins and Maven source repositories. CloudBees, the company behind DEV@cloud, offers a free subscription with reduced capabilities which are more than enough for small projects. I set up all my projects there in no time, but had serious problems with the integration tests.

Using a local Maven repository other than ~/.m2/repository
You don't have to use the default repository location. It's possible to define your own in the user's settings.xml or even in the global settings. But I guess most people just use the default. On the other hand in an environment like DEV@cloud, all the different builds from different users must be separated. So CloudBees decided that each Jenkins job has its own Maven repository inside the job's workspace. That is good because the repository is deleted together with the project.

Problem
The Testing Harness embeds Maven, i.e. forks a new Maven instance. It fails to relay the modified settings to this new process. During the execution of the integration test a new local repository is created and the original local one is used as a remote one (called "local-as-remote"). But without any hints, Maven uses ~/.m2/repository. So the true local repository is not found and all needed artefacts are downloaded again. This takes a lot of time (and wastes bandwidth). Dependencies that exist only in the local repository, e.g. snapshots of dependent projects, are not found and the integration test fails.

Solution
RepositoryTool.findLocalRepositoryDirectoy() uses an instance of MavenSettingsBuilder to get the settings. Its only implementing class is DefaultMavenSettingsBuilder and it tries to determine the repository location from the value of the system property maven.repo.local. Then it reads the user settings and in the end it uses ~/.m2/repository. The solution is to set the maven.repo.local system property whenever the local repository is not under ~/.m2/repository. Add -Dmaven.repo.local=.repository into the field for Goals and Options of the Jenkins job configuration.

Using additional dependencies while building integration test projects
After the plugin under test is built and installed into the new local repository the Maven Plugin Harness runs Maven against the integration test projects inside the src/test/resources/it folder. The approach which I described last year forks a Maven with pom, properties and goals defined by the JUnit test method.

Problem
The integration tests know the location of the new local repository (because it is set explicitly) and are able to access the plugin under test. But they know nothing about the local-as-remote repository. They can only access all artefacts which have been "downloaded" from the local-as-remote repository during the build of the plugin under test. So the problem is similar to the previous problem but occurs only when an integration test project needs additional artefacts. For example a global ruleset Maven module might consists of XML ruleset configuration files. The test module depends on the Checkstyle plugin and executes it using the newly build rulesets. So the object under test (the rules XML) is tested indirectly through the invocation of Checkstyle but the ruleset module itself does not depend on Checkstyle.

Solution
All POMs used during integration test have to be "mangled", not just the POM of the plugin under test. The method manglePomForTestModule(pom) is defined in the ProjectTool but it's protected and not accessible. So I copied it to AbstractPluginITCase and applied it to the integration test POMs.

Using settings other than ~/.m2/repository/settings.xml
If you need artefacts from repositories other than Maven Central you usually add them to your settings.xml. Then you refer to them in the Jenkins job configuration. Behind the scenes Jenkins calls Maven with the parameter -s custom_settings.xml.

Problem
Similar to the repository location, the custom settings' path is not propagated to the embedded Maven and it uses the default settings. This causes no problems if all needed artefacts are either in the local-as-remote repository or can be downloaded from Maven Central. For example a Global Ruleset might contain some Macker Architecture Rules. The snapshot of the Macker Maven Plugin is deployed by another build job into the CloudBees snapshot repository. The test module depends on this Macker plugin and runs it using the newly built rulesets.

Solution
AbstractPluginITCase calls BuildTool's createBasicInvocationRequest() to get an InvocationRequest and subsequently executes this request. Using any system property the InvocationRequest can be customised:

if (System.getProperty(ALT_USER_SETTINGS_XML_LOCATION) != null) {
   File settings =
      new File(System.getProperty(ALT_USER_SETTINGS_XML_LOCATION));
   if (settings.exists()) {
      request.setUserSettingsFile(settings);
   }
}

Then the value of the used system property is added into the field for Jenkins' Goals and Options: -s custom_settings.xml -Dorg.apache.maven.user-settings=custom_settings.xml.

Alas I'm not a Maven expert and it took me quite some time to solve these problems. They are not specific to CloudBees but they result from using non default settings. Other plugins that fork Maven have similar problems.

ASUS Eee PC and TRIM

Last year, after commuting for more than ten years I got tired of reading on the train. I wanted to make better use of the time and got myself one of these small sub-notebooks. I chose an ASUS Eee PC S101. Although it's not very powerful it is able to handle small Eclipse projects. It's a slick device and I love it.

The Problem with SSDs
It contains a ridiculously small SSD hard drive, an "ASUS-JM S41 SSD". Recently after the drive was full for the first time, disc performance degraded catastrophic. Whenever the disc was accessed the computer would freeze one or two seconds. The whole device got totally unusable. I was already fearing the worst.

TRIM to the Rescue?
When searching the web I learned that all SSD share the same problem of free space management which is fixed by the TRIM command. TRIM is a standard command defined by the (S)ATA specification. Unfortunately Windows 7 is the first version to make use of TRIM and there are no plans to port it back to earlier versions. (I also found references to WIPE but I don't know if it's a command some drives implement or just another name for the process of trimming.)

Vendor Tools
Some SSD vendors have noticed the need for SSD TRIM and provide tools of their own. Some vendors provide firmware upgrades like OCZ. Others offer special tools. For example there is a tool called wiper.exe provided for the G.SKILL Falcon drives and maybe some other drives by G.SKILL. Unfortunately wiper crashes when run on the Eee PC. Intel offers its own Intel SSD Toolbox but the TRIM option is not available when run on the Eee PC. These two were the only tools supporting TRIM that I could find. Bad luck here too.

I could not believe it. I didn't own the first SSD on this planet. How was I going to fix it? Format the SSD and install the OS all over? Not if I could help it. Probably I had not searched the web long enough...

From 0x00 to 0xFF
One entry in a forum suggested that Piriform CCleaner's Secure Wipe trims the disc. Well it doesn't but it seems that some SSDs reclaim a block when it's filled with some data and that's what Secure Wipe is doing. It overwrites all empty blocks. Someone has written a little program to do exactly that: "AS FreeSpaceCleaner with FF" (aka "AS Cleaner 0.5") is meant to work around not having TRIM and is like a generic wiper that works on any drive. It creates a big file and uses it to write zeros in all empty blocks. It has one option, to use 0xFF instead of 0x00 to fill these blocks. Some forum entries suggested that people have successfully used the 0xFF option to trim their SSDs.

Finally Whole Again
The short story is that I managed to restore performance of the SSD in my Eee PC using FreeSpaceCleaner.exe. The long story is as follows:

• First I did a BIOS upgrade. Disc access might benefit from a newer BIOS. I'm not sure if it's part of the solution but now that it's done it's done.
  
  
• Then I reduced disc writes as much as possible. I turned off the index service, removed the swap file and disabled the recording of the last file access. This is not related to restoring SSD performance, but it's supposed to keep it performing a bit longer.
  
  
• After that I uninstalled all programs which I didn't need and used CCleaner to remove all temporary crap. I think it's vital to have as much free space as possible so the SSD doesn't run out of "clean" blocks too soon. Some forum entries suggested that it's beneficial to have at least 50% of the disc free.
  
  
• In the end I used FreeSpaceCleaner with the FF option to wipe the SSD and it worked! At least it did something as SSD performance definitely improved after using it but I doubt it was able to do a full TRIM on the disc because I have to use it quite often.

So thanks to FreeSpaceCleaner the problem was solved. (Download FreeSpaceCleaner with FF)