lesscode.org

Bob Ippolito, speaking about the upcoming flashticle, offers proof in the pudding for things I wrote a week ago:

This is a total of a whopping 117 lines of very liberally spaced Python code that defines all three database tables and fully implements every feature of both sample applications.

The PHP version of the pizzaService backend […] is 138 lines of code, is one big security flaw (doesn’t escape SQL properly, big surprise), is MySQL specific, and it doesn’t include the DB schema and can’t create any tables for you (thank you tg-admin sql create).

The Java version of the BirthdayOrganizer backend, which I won’t even bother linking to, is well… ginormous. 246 lines of XML configuration sludge in 5 files, 29 lines of SQL schema in 1 file, and 3004 lines of Java code in 45 files. Holy crap. If you add that all up, the trivial BirthdayOrganizer example is only a hair shorter than all of flashticle! And the BirthdayOrganizer example builds on top of 15 Java dependencies and requires a J2EE server plus ant. Friends don’t let friends code Java. Oh, and it’s also MySQL specific, but you better be damn well sure that there’s a stub implementation you can subclass and an XML file you can mangle in order to support something else if you so choose to write that 200 lines of code to support a single three column table database.

Here’s a slightly elided log line from a system I work with:

WARN 2005-09-28 14:38:38 StandardContext[/pmr]  {”mid”:”123″, “type”:”XYZ”, “msg”:”foo”, “T”:”2005-09-28 14:38″, “L”:”warn”}

It identifies an XML document passing through the system using some keys from the envelope. It may seem a strange looking way to log an event, but between the angle brackets lies a useful side effect - it’s a Python dictionary. Hence this (vastly oversimplified) example log processing code, that runs eval() on each log line to load the dictionaries into memory:

Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>> dicts =[]
>>> import re
>>> p = re.compile(”.*({.*?}).*”)
>>> file = open(”log.txt”).readlines()
>>> for line in file:
…     m = p.match(line)
…     if m:
…             d = eval(m.group(1))
…             dicts.append(d)
…
>>> for d in dicts:
…    print d
…
{’L': ‘warn’, ‘mid’: ‘123′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer  found’}
{’L': ‘warn’, ‘mid’: ‘124′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer  found’}
{’L': ‘warn’, ‘mid’: ‘125′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer  found’}
{’L': ‘warn’, ‘mid’: ‘126′, ‘type’: ‘XYZ’, ‘T’: ‘2005-09-28 14:38′, ‘msg’: ‘frobnizer  found’}
>>>

What’s nice about doing this is that it’s cheap to write the log line in a way that maximises usefulness later on - logs are often an afterthought, but good logging is critical to operating production systems. And it’s one example where creating metadata is cheap.

As well as programming data structures (changing the above to a Ruby hash is straightforward), you can get log lines to output something like Turtle, which is a greppable RDF format. What’s good about Turtle in particular is that you can dump or stream a log file straight into an RDF query engine, such as ARQ without any transform step and start writing queries against the data. Or how about writing down inferencing rules that help indicate the business consequences of a software error? The log lines themselves don’t carry enough information, but combined with rules and some context as to where the log event came from you can do interesting and useful things - the biggest issue with any monitoring software is gathering up all those machine level events and contextualising them so that data (noise) can be turned it into information (signal). Think of a machine readable data stucture inside logfiles as just another microformat.

On December 3rd, a mess of fine Chicago area Ruby, Python, and Linux user groups are to meet up at DePaul University for a Django / Rails love-fest. I’ve heard there might also be time for debate between songs and hand-holding. Adrian Holovaty, award winning journalist and distinguished gentlemen, will be fielding questions for the Pythonistas while David Heinemeier Hansson, Extraordinary Alien / “one man” trying to save the world (hey, I owe him one), will be answering for the Rubyists.

Even better, the Snakes and Rubies website is collecting questions for Adrian or David or both. Get over there and get your questions in before it’s too late.

If there is any way someone could capture the audio for this event, it would be very much appreciated.

Rather than reinventing everything or trying to clone some other framework, TurboGears takes advantage of the great tools that are already available!

The tools:

• MochiKit is a clean and powerful JavaScript library.
• Kid is a designer- and programmer-friendly template system… written by lesscode’s fearless leader.
• CherryPy
• SQLObject

I’ve written previously on the trade-offs that microformats make in vocabulary design. I’m still not sure how I feel about the short-string issue, but it appears no one is waiting for me to make up my mind, so I figured I’d try it out.

So far I like it that microformats depend on well-understood technology, and that a lazy dev could point their HTML rendering component at the page if they didn’t feel like writing support. OTOH, parsing can be a bear. I think I may have cheated a little in that I insisted on XHTML, but even that had its own problems, since it’s tied to a DTD. Either you write your own entity resolver or the XML parser downloads three DTD files from w3.org. Rotten. I let my example client do just that in the interest of brevity.

On the lesscode side of things, the draft includes Python implementations of a client and server. With margins, the IETF txt format ends up allowing something less than the usual number of columns, so it cramped my Python style a little bit. I think they ended up pretty short, but one of the nice things about Python is that someone can always show you a way to do it that’s shorter and clearer. So, let’s have it! (no generators allowed, to keep things adaptable to those other languages :)