What Engineers Do

Wayne Allen writes...

What is it that we are envious of from the "other" engineering disciplines?

I think we envy the respect and prestige more than anything else. The nature of software is different though. We have to build software worthy of respect, and it's very easy to build software that is not. We have unprecendented freedom, but...

Chui Tey writes in a comment...

I don't think the software that runs a satellite or one that powers weaponry would be written any less formally than the effort involved with engineering a bridge. Even at these lofty levels, there are elements of craft at the design level, one which comes from experience, and not one that comes from formal learning.

I think we have more freedom to ignore useful formalisms. Part of the challenge is to understand our formal tools and to push them into new capabilities. Test-first programming is an example of a formal tool not widely understood today. Model checking concurrent systems is an example of a current push into new capabilities.

Generally speaking though I think it is the softer communication issues among a development team that typically leads to failure.

More on common language runtimes

Ian Bicking from comments...

You mean, tried it like Guile? That was the whole premise of Guile, that you could translate other languages into Scheme. They never even got as far as translating elisp (which should have been the easy part) so that they could move Emacs to Scheme. That might not really be fair -- honestly, I don't know if anyone cared enough to try, Guile was born of a flame and never had a real champion...

I did not know that was a goal. I've neither tried Guile nor even looked at it. Sounds like more of an organizational thing than a technical issue. The history of Lisp is full of people writing compatibilities to move their code from one (usually older) Lisp system to another. This ultimately led to Common Lisp to avoid doing this so much.

Someone also tried translating Python to PLT Scheme, and they ended up giving up and just embedding Python in the Scheme interpreter...

Guile was crappy to debug Scheme in, but it offered absolutely nothing for any other language built on top of it. Can you annotate your Scheme source with information that allows you to reconstruct the native language environment when there's an error? Many languages deal poorly with incorrect programs, but Scheme implementations tend to be particularly bad, as is any implementation that comes out of an academic background (rather than being designed by a programmer that actually wants to *do* things with their implementation).

Gambit is a good choice because of its C interface, compiler to C, and good debugging capability. Again I am shooting from the hip, but for annotating Scheme source with information for reconstructing the source language information, the nice thing about this is it is Scheme and you can define all kinds of "meta-data" that does something or does nothing execution-wise, but just keeps information around (or eliminates it when desired). Everything is a list, remember. 8^) Just define syntax and/or structure that keeps the information you want, e.g. source text spans for highlighting an expression in a debugger or whatever. Seriously for non-Lisp programmers: everything is not a list. Scheme has "records" ("structs" in C), objects, vectors, etc. I would try defining records for the "meta-data" and the Scheme code would include references to these records.

And that doesn't even deal with the object mismatch issues. You don't actually have to resolve those issues if you just want to take advantage of the performance of the underlying implementation, but if you want to leverage it for interlanguage communication, translating it all to Scheme doesn't actually accomplish a whole lot.

I personally would not want to try reconciling object models too much. I would have each language keep its own definitions and see how easily the translation to Scheme data can support those languages. I would define some kind of interchange but that would not have to be too elaborate. Maybe start with YAML and leave it at that.

I'm highly sceptical of any academic projects, as I they tend to be mismotivated.

Yes, this is just a thought experiment for me. Like the Python and PLT Scheme observation, it may simply be unnecessary when the original language implementations can simply communicate with each other well enough.

The one time I could see attempting something like this would be when defining a new language from scratch. Lisp has always been a good vehicle for experimenting with new languages. They've tended to be more academic and less production too, but something like Gambit for performance is intriguing.

Craftiness

My response to a comment on Wayne Allen's blog...

True, craftiness appeals to many developers. But what is an example of cut and dry standard bits that don't involve craftiness?

Developing TV's? Cars? There is a craftiness to each of these. But less so. And that is where the communication comes into play with the many more informalities. And it is not just color or layout... it's also "What does a user want to do with this purchase order?" There are formal things to do with it, but also informal.

I think software is informal all the way down to the bits on the wire. Only then does it become informal. TV's and cars stay pretty much the same with inflection points here and there (e.g. Tivo, HDTV's, hybrid cars).

Factory, Engineering, Craft, or What?

As James Robertson points out, there are reasons to favor the Software as a Craft metaphor over the Software as a Factory metaphor. But he writes "software is still a craft" (emphasis mine).

That's almost an apology. I'm not sure how intentional his words align with my interpretation, but the notion is commonly expressed, as if someday it won't be a craft but for now it is. Usually the notion is that someday software will be more like "real engineering".

I've taken another view on these metaphors. Let's forget about factories because 10 seconds of thought informs us this does not fit well at all. So the candidates seem to be "craft" or "engineering".

My view is there are things software developers can learn from various kinds of engineers. Maybe there is something to be learned from legal engineering certification processes. But the lack of doing so is not what makes software development seem craft-like.

I think the "informality" of software makes it feel craft-like. If only we had more formalisms. Could be, but we'll always have "informalities". Every engineering effort, and every creative effort in general, involves many informal conversations.

The difference is perhaps software depends more on informal conversations than do many of the traditional engineering disciplines. The requirements in those disciplines are more formal. Most of software development is really requirements development. The interpretation of formal requirements is most easily expressed in software as tests and code that (correctly one hopes) runs those tests.

I think the key difference between software development and traditional engineering disciplines is not primarily the lack of discipline. Rather it is the inherent predominance of informal conversation and that software development fails mostly due to lack of communication. There is room to improve our formalisms for sure. But those improvements will not replace the focus on informal conversation the way standards and formalisms have in traditional engineering disciplines.

Those software development improvements have to support informal conversation. That's why the agile approaches have been successful... when they have improved communication for a specific team, they have improved the results without any new technology or formalisms. Of course some of that communication is in the form of "knowledge transfer" of how to use the available formal tools better. But in the aggregate it is *all* sorts of communication among the whole team that has made agile approaches successful. And when they have not been successful, in my experience it is not from failing to use a formalism well, it is failure to communicate well.

I'm a Broken Record

Update: Luke Gorrie comments...

Can you point me towards any examples that have gone "all the way" with a production language implementation based on translation to Scheme? The approach looks really appealing but I'm becoming skeptical of it.

I don't know. Kelsey went far enough to get a PhD. Feeley (with Erlang to Scheme) went far enough to "prove" the concept. Neither could be considered close to production worthy. The closest thing I can cite would be the compilers for Fortran, Ada, Pascal, C, and Prolog for the Lisp Machine. I'm not sure if they compiled to Lisp or to the Lisp Machine assembly language (which could be seen as a kind of Lisp as much or more than a typical "machine" language.)

I guess the point is not that this is proven, but that today, in 2005, why has it not been explored? People are still beginning VM projects left and right in the 1970's approach. Where does that leave us? Why not try something else, at a higher level, to *see* if it works?

End Update

As per James Robertson, Sriram Krishnan and some folks are building a new VM. It sounds ambitious and two minutes of looking around did not clarify for me why they want to build something from scratch. So rather than suggest they shouldn't, I'll try to be more positive and suggest some reading material for someone interested in these things, then speculate on what I'd do, then admit I have no intention of doing such a thing myself but you might listen to me anyway. Of course I think these ideas are worth considering, unless you *really* want to work on the lower level bits.

Right now, we're looking to copy from existing VMs as much as we can since we really don't have time to design a lot of stuff. This doesn't mean that we don't want to innovate - just that we don't want to reinvent every wheel. I'm personally looking forward to figuring out how to implement continuations inside the VM itself.

You could do worse than to look at the source code for Gambit Scheme and read the 20+ years of technical reports on compiling Scheme. By Sriram's second post, it looks like he is on this trail...

Everything worth inventing has been invented by the Lisp guys 2 decades ago. People like Guy Steele have done some really cool work

I guess if I wanted to work on a language project it would go more like this... start with the implementation of Gambit Scheme because you probably won't do better with less than 2-5 years of research and development. That's probably optimistic, based on the fact that Gambit has a decade or so of concentrated effort from a very bright person, and there's decades of work and discussions that went into Marc Feeley's own thought processes. You'd better talk with those guys because you'll be recreating their mistakes at best.

Then I'd build the translators for the languages I want to run to compile from source to Scheme. Along the way I'd do something interesting to demonstrate the payoff of using what's been done so well already: e.g. I might write translators for partial definitions of a couple languages , perhaps subsets of Smalltalk, Python, and Ruby, and then combine Scheme, Smalltalk, Python, and Ruby in a simple continuation-based modal web server based on Chris Double's tutorial.

The would demonstrate a better way of programming web applications on a well-designed kernel, supporting multiple popular languages. It would also demonstrate the value of continutations in languages that do not have them, e.g. Python (other than Stackless, I think). Pehaps others would want to add to the language fidelity. Another benefit of this approach then is that those folks don't have to understand my VM design, or track its quality, or track its design changes. All they need is to understand Scheme and write a translator from their language to Scheme, or enhance one of the initial prototypes.

GNU Smalltalk is fairly simple... a simple initial translator to Scheme does not seem too hard. I'm not sure what starting points to use for Python or Ruby. But I'm not proposing to do this... only suggest it to someone. I think it would be useful but it's not high on my personal list.

Just some thoughts if there's a Masters student looking for an interesting project.

Shortness

Jim Weirich writes about web apps, Ta Da, and Ruby on Rails...

Ta-Da lists uses XMLHttpRequest to interact dynamically with the host. Beautiful.

And to top it off, it is a Ruby-on-Rails application. Written in 579 lines of Ruby code, that’s less than the size of the XML config files used in many J2EE applications.

And John Wiseman writes about Groovy, Common Lisp, and the shortness of code for an RSS reader...

I had forgotten about when I said that the only way someone would be able to compete with the Groovy app was to integrate with Java or Cocoa, or use CLIM: He created a webapp and generated HTML.

It took me about 3 hours to write. The initial quick hack version took only about an hour and was about 120 lines, but it was not as nice as this one :).

The final version of the code, at about 200 lines, looks pretty good and wouldn't be hard to turn into a desktop GUI app.

The shortness is due to Ruby, Lisp, (or Python or Smalltalk or...) being able to hide any kind of functionality behind a few simple lines of code. We should expect these languages to provide roughly the same amount of abbreviation vs. more elaborate notations.

Maybe if you like writing code these languages aren't for you after all. 8^)

WTF? Solved: burnatonce

Update: The winner is burnatonce. I downloaded the software and had the CD burned in less than 1/2 hour. I got my refund from the 24-hour company and will apply it as a donation to burnatonce.

Not sure why Microsoft doesn't make their CD burner more easily available to XP users than to download a 12MB Windows Server 2003 package, while suggesting XP users download some guy's freeware that doesn't work. End of update

OK. I want to burn an ISO image onto a CD using Windows XP. The information from Microsoft says they don't have that capability, but to try some guy's free (beta) software! Thanks a lot. Glad my money's being put to good use up there in Redmond.

Hey, I've got an idea. Maybe I should just run some guy's free software from the ground up and skip this whole Windows thing. But OK, I install this guy's free software. But when I burn the CD an error immediately pops up... error code 1800183298837 or something. This guy must have worked at Microsoft recently. No wonder they recommend his stuff.

Then I try Nero Buring ROM. (Ha! Ha! I get it.) Problem is the software sucks so bad out the user interface that I want nothing more than to burn the Nero software itself.

Next I try MagicISO. Installs quick. Simple interface. BAM! The trial cannot burn an ISO greater than 300MB. Thanks. Seems useful enough that I decide to spend the $30 USD right then and there. The order process goes OK.

I get the email confirming my purchase. Great. But there is no key to register the software in the email. And there is no URL in the email to get a non-trial version.

Let's actually read the email.

Your order will be fulfilled directly by company that develops the software. Orders are generally fulfilled very quickly, however, please allow up to 24 hours.

%@($#^!& What is this? A 24-hour turn-around for an electronic purchase? (I chose not to spend the extra $10 USD to get a CD!)

I guess this time tomorrow I will be burning an ISO image on a CD.

With any luck.