BHL to PDF workflow

Just some random thoughts on creating searchable PDFs for article extracted from BHL.

BHL, DjVu, and reading the f*cking manual

One of the many biggest challenges I've faced with the BioStor project, apart from dealing with messy metadata, has been handling page images. At present I get these from the Biodiversity Heritage Library. They are big (typically 1 Mb in size), and have the caramel colour of old paper. Nothing fills up a server quicker than thousands of images.

A while ago started playing with ImageMagick to resize the images, making them smaller, as well as ways to remove the background colour, leaving just black text and lines on white background.

I think this makes the page image clearer, as well as removing the impression that this is some ancient document, rather than a scientific article. Yes, it's the Biodiversity Heritage Library, but the whole point of the taxonomic literature is that it lasts forever. Why not make it look as fresh as when it was first printed?

Working out how to best remove the background colour takes some effort, and running ImageMagick on every image that's downloaded starts putting a lot of stress on the poor little Mac Mini that powers BioStor.

Then there's the issue of having an iPad viewer for BHL, and making it interactive. So, I started looking at the DjVu files generated by the Internet Archive, and thinking whether it would make more sense to download those and extract images from them, rather than go via the BHL API. I'll need the DjVu files for the text layout anyway (see Towards an interactive DjVu file viewer for the BHL).

I couldn't remember the command to extract images from DjVu, but I did remember that Google is my friend, which led me to this question on Stack Overflow: Using the DjVu tools to for background / foreground seperation?.

OMG! DjVu tools can remove the background? A quick look at the documentation confirmed it. So I did a quick test. The page on the left is the default page image, the page on the right was extracted using ddjvu with the option -mode=foreground.

Much, much nicer. But why didn't I know this? Why did I waste time playing with ImageMagick when it's a trivial option in a DjVu tool? And why does BHL serve the discoloured page images when it could serve crisp, clean versions?

So, I felt like an idiot. But the other good thing that's come out of this is that I've taken a closer look at the Internet Archive's BHL-related content, and I'm beginning to think that perhaps the more efficient way to build something like BioStor is not through downloading BHL data and using their API, but by going directly to the Internet Archive and downloading the DjVu and associated files. Maybe it's time to rethink everything about how BioStor is built...

Towards an interactive DjVu file viewer for the BHL

The bulk of the Biodiversity Heritage Library's content is available as DjVu files, which package together scanned page images and OCR text. Websites such as BHL or my own BioStor display page images, but there's no way to interact with the page content itself. Because it's just a bitmap image there's no obvious way to do simple things such as select and copy some text, click on some text and correct the OCR, or highlight some text as a taxonomic name or bibliographic citation. This is frustrating, and greatly limits what we can do with BHL's content.

In March I wrote a short post DjVu XML to HTML showing how to pull out and display the text boxes for a DjVu file. I've put this example, together with links to the XSLT file I use to do the transformation online at Display text boxes in a DjVu page. Here's an example, where each box (a DIV element) corresponds to a fragment of text extracted by OCR software.

The next step is to make this interactive. Inspired by Google's Javascript-based PDF viewer (see How does the Google Docs PDF viewer work?), I've revisited this problem. One thing the Google PDF viewer does nicely is enable you to select a block of text from a PDF page, in the same way that you can in a native PDF viewer such as Adobe Acrobat or Mac OS X Preview. It's quite a trick, because Google is displaying a bitmap image of the PDF page. So, can we do something similar for DjVu?

The thing I'd like to do is something what is shown below — drag a "rubber band" on the page and select all the text that falls within that rectangle:

This boils down to knowing for each text box whether it is inside or outside the selection rectangle:

Implementation

We could try and solve this by brute force, that is, query each text box on the page to see whether it overlaps with the selection or not, but we can make use of a data structure called an R-tree to speed things up. I stumbled across Jon-Carlos Rivera's R-Tree Library for Javascript, and so was inspired to try and implement DjVu text selection in a web browser using this technique.

The basic approach is as follows:

1. Extract text boxes from DjVu XML file and lay these over the scanned page image.


2. Add each text box to a R-tree index, together with the "id" attribute of the corresponding DIV on the web page, and the OCR text string from that text box.


3. Track mouse events on the page, when the user clicks with the mouse we create a selection rectangle ("rubber band"), and as the mouse moves we query the R-tree to discover which text boxes have any portion of their extent within the selection rectangle.


4. Text boxes in the selection have their background colour set to an semi-transparent shade of blue, so that the user can see the extent of the selected text. Boxes outside the selection are hidden.


5. When the user releases the mouse we get the list of text boxes from the R-tree, and concatenate the text corresponding to each box, and finally display the resulting selection to the user.

Copying text

So far so good, but what can we do with the selected text? One obvious thing would be to copy and paste it (for example, we could select a species distribution and paste it into a text editor). Since all we've done is highlight some DIVs on a web page, how can we get the browser to realise that it has some text it can copy to the clipboard? After browsing Stack Overflow I came across this question, which gives us some clues. It's a bit of a hack, but behind the page image I've hidden a TEXTAREA element, and when the user has selected some text I populate the TEXTAREA with the corresponding text, then set the browser's selection range to that text. As a consequence, the browser's Copy command (⌘C on a Mac) will copy the text to the clipboard.

Demo

You can view the demo here. It only works in Safari and Chrome, I've not had a chance to address cross-browser compatibility. It also works in the iPad, which seems a natural device to support interactive editing and annotation of BHL text, but you need to click on the button On iPad click here to select text before selecting text. This is an ugly hack, so I need to give a bit more thought to how to support the iPad touch screen, while still enabling users to pan and zoom the page image.

Next steps

This is all very crude, but I think it shows what can be done. There are some obvious next steps:

• Enable selected text to be edited so that we can correct the underlying OCR text.


• Add tools that operate on the selected text, such as check whether it is a taxonomic name, or if it is a bibliographic citation we could attempt to parse it and locate it online (such as David Shorthouse's reference parser).


• Select parts of the page image itself, so that we could extract a figure or map.


• Add "post it note" style annotations.


• Add services that store the edits and annotations, and display annotations made by others.
  

Lots to do. I foresee a lot of Javascript hacking over the coming weeks.

BHL and the iPad

@elyw I'd leave bookmarking to 3rd party, e.g. Mendeley. #bhlib specific issues incl. displaying DjVu files, and highlighting taxon namesless than a minute ago via Tweetie for MacRoderic Page
rdmpage

Quick mock-up of a possible BHL iPad app (made using OmniGraffle), showing a paper from BioStor(http://biostor.org/reference/50335). Idea is to display a scanned page at a time, with taxonomic names on page being clickable (for example, user might get a list of other BHL content for this name). To enable quick navigation all the pages in the document being viewed are displayed in a scrollable gallery below main page.



Key to making this happen is being able to display DjVu files in a sensible way, maybe building on DjVu XML to HTML. Because BHL content is scanned, it makes sense to treat content as pages. We could extract OCR text and display that as a continuous block of text, but the OCR is sometimes pretty poor, and we'd also have to parse the text and interpret its structure (e.g., this is the title, these are section headings, etc.), and that's going to be hard work.

DjVu XML to HTML

This post is simply a quick note on some experiments with DjVu that I haven't finished. Much of BHL's content is available as DjVu files, which contain both the scanned images and OCR text, complete with co-ordinates of each piece of text. This means that it would, in principle, be trivial to lay out the bounding boxes of each text element on a web page. Reasons for doing this include:

1. To support Chris Freeland's Holy Grail of Digital Legacy Taxonomic Literature, where user can select text overlaid on BHL scan image.


2. Developing a DjVu viewer along the lines of Google's very clever Javascript-based PDF viewer (see How does the Google Docs PDF viewer work?).


3. Highlighting search results on a BHL page image (by highlighting the boxes containing terms the user was searching for).

As an example, here is a BHL page image:

and here's the bounding boxes of the text recognised by OCR overlain on the page image:

and here's the bounding boxes of the text recognised by OCR without the page image:

The HTML is generated using a XSL transformation that take two parameters, an image name and a scale factor, where 1.0 generates HTML at the same size as the original image (which may be rather large). The view above were generated with a scale of 0.1. The XSL is here:

<?xml version='1.0' encoding='utf-8'?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:output method="html" version="1.0" encoding="utf-8" indent="yes"/>

    <xsl:param name="scale"/>
    <xsl:param name="image"/>

    <xsl:template match="/">
        <xsl:apply-templates select="//OBJECT"/>
    </xsl:template>

    <xsl:template match="//OBJECT">
        <div>
            <xsl:attribute name="style">
                <xsl:variable name="height" select="@height"/>
                <xsl:variable name="width" select="@width"/>
                <xsl:text>position:relative;</xsl:text>
                <xsl:text>border:1px solid rgb(128,128,128);</xsl:text>
                <xsl:text>width:</xsl:text>
                <xsl:value-of select="$width * $scale"/>
                <xsl:text>px;</xsl:text>
                <xsl:text>height:</xsl:text>
                <xsl:value-of select="$height * $scale"/>
                <xsl:text>px;</xsl:text>
            </xsl:attribute>

            <img>
                <xsl:attribute name="src">
                    <xsl:value-of select="$image"/>
                </xsl:attribute>
                <xsl:attribute name="style">
                    <xsl:variable name="height" select="@height"/>
                    <xsl:variable name="width" select="@width"/>
                    <xsl:text>margin:0px;padding:0px;</xsl:text>
                    <xsl:text>width:</xsl:text>
                    <xsl:value-of select="$width * $scale"/>
                    <xsl:text>px;</xsl:text>
                    <xsl:text>height:</xsl:text>
                    <xsl:value-of select="$height * $scale"/>
                    <xsl:text>px;</xsl:text>
                </xsl:attribute>
            </img>

            <xsl:apply-templates select="//WORD"/>

        </div>
    </xsl:template>

    <xsl:template match="//WORD">
        <div>
            <xsl:attribute name="style">
                <xsl:text>position:absolute;</xsl:text>
                <xsl:text>border:1px solid rgb(128,128,128);</xsl:text>
                <xsl:variable name="coords" select="@coords"/>
                <xsl:variable name="minx" select="substring-before($coords,',')"/>
                <xsl:variable name="afterminx" select="substring-after($coords,',')"/>
                <xsl:variable name="maxy" select="substring-before($afterminx,',')"/>
                <xsl:variable name="aftermaxy" select="substring-after($afterminx,',')"/>
                <xsl:variable name="maxx" select="substring-before($aftermaxy,',')"/>
                <xsl:variable name="aftermaxx" select="substring-after($aftermaxy,',')"/>
                <xsl:variable name="miny" select="substring-after($aftermaxy,',')"/>

                <xsl:text>left:</xsl:text>
                <xsl:value-of select="$minx * $scale"/>
                <xsl:text>px;</xsl:text>
                <xsl:text>width:</xsl:text>
                <xsl:value-of select="($maxx - $minx) * $scale"/>
                <xsl:text>px;</xsl:text>
                <xsl:text>top:</xsl:text>
                <xsl:value-of select="$miny * $scale"/>
                <xsl:text>px;</xsl:text>
                <xsl:text>height:</xsl:text>
                <xsl:value-of select="($maxy - $miny) * $scale"/>
                <xsl:text>px;</xsl:text>

            </xsl:attribute>

            <!-- actual text -->
            <!-- <xsl:value-of select="." /> -->
        </div>
    </xsl:template>

</xsl:stylesheet>