Fuzzy Egg Sandwich

I have somehow become a legend in my home for my egg sandwiches. My belated Christmas present to my readers is the recipe. You of course will not be able to replicate exactly, but you can try ;)

1. One pad of butter to the frying pan (lowish heat)
2. One organic egg cracked open to the frying pan (sunny side up) - cover
3. 2 pieces of Dave's Killer Bread to the toaster
4. When toast is done, egg is generally done. Turn off heat to egg
5. One slice of Horizon Organic Cheddar Cheese to the top of the egg - cover
6. As the cheese is melting (1 minute), start the toaster again to maintain crucial bread-heat-ratio
7. Remove egg and place on one piece of toast
8. Place ample amounts of hot sauce (either Marie Sharp's Habanero or Bali Spice Hot Chile Sauce) on the other side of toast
9. Put hot sauce side on top of the egg side
10. Slice in half
11. Hand out to your family / devour
12. Bask in the glory of your egg-sandwich-making-prowess

Remember the Milk = Neat

People who have worked with me may remember me getting all excited about various task management approaches. I liked Getting Things Done and other techniques before that.

I typically go through phases of being very organized & very disorganized as I'm in various states of work flow/learning levels. Well I'm in an organized state right now as I have a lot to do for CSI and am learning at a high rate (Ruby, JRuby, the Rails way, etc).

I saw on somebody's del.icio.us a link to Remember the Milk. I remember hearing about it from Jive's Bill Lynch in the spring.

I gave it a try the other day and am pretty happy with it so far.

Here is why:

• You can have as many lists as you want. Categories don't work for me.
• It can IM you when things are due
• It has an offline mode with Google Gears
• It works well with my phone
• You can email it tasks - for instance I was on a plane yesterday and I mailed it like 20 tasks. I categorized them today
• You can render it by URL (like Evolution's task manager)
• Very searchable
• Integrates your task lists into views across them well

Time will tell if I stick with it, but so far so good on Remember the Milk.

Update 30-DEC

I forgot to mention perhaps the best part: Keyboard Shortcuts.

Changing the rules of enterprise software

I had a good conversation with Matt Asay yesterday about open source moving up the stack & CSI. I got to know him a bit over the last few years via blogs, OSCON, OSBC, & Alfresco. He's a lot of fun to talk to about open source & everything else.

2008 is going to be an exciting year.

Starting to think about 2008 Predictions

I had fun making some predictions last year.

I plan to do the same thing this year in a week or so.

It is a bit funny looking back at what I thought for 2007.

I was spot on for some of it, but way off for others. I voted with my feet with one of them ;)

Recruiting == Lots of Work

One of the things that I like to think I'm good at is putting together lethal software teams.

Whenever I have had the opportunity to build a team, it consumes me entirely until the team is in place. My logic is you can't do jack until you have the team. Once you identify the need for people, there is no priority that is more important than getting the best people you can find.

I think I'm close to being done for this round, but I suspect we'll be doing a lot of hiring in the future.

Stay tuned.

I hope that this round is nearly over. Although I like putting together teams, it exhausts me.

JRuby Book

I read Practical JRuby on Rails Web 2.0 Projects: Bringing Ruby on Rails to Java over the past couple 2-3 days.

It is a pretty good book. It was released in September and is already a bit out of date. JRuby & the Ruby world in general seem to move pretty fast.

The topics I liked the best were:

• Nuts & bolts of Java & JRuby integration
• Talk of REXML limitations and the (nice) option to just bang out to the multitude of Java XML APIs should you need extra features
• JRuby & JMX
• JRuby & MOM (Message Oriented Middleware) - ActiveMessaging

After reading it and messing with JRuby a bit, I'm more certain of what I thought in May (ok I didn't say that exactly & I've lost track of Scala - focus on the JVM :) ).

There clearly is a very bright future for Ruby in general & in particular Ruby on the JVM.

What (obviously) makes JRuby + Java so compelling is:

• There is a Java API for *everything*. I hope to stay in Ruby code as much as possible, but it sure is nice knowing that the APIs I have been using for the past 10 years are easily available
• The majority of companies run Java in their data center & are very comfortable deploying to it. Many are also not comfortable adding platforms (e.g., C Ruby)
• The JVM has had a tremendous amount of engineering put into it
• Ruby is a fantastic language that truly is a joy to work with
• Ruby has heaps of momentum & enthusiasm behind it

Job Opportunity

Please contact me if you are interested & have the skills (mike at-sign csinitiative dot com:

Collaborative Software Initiative

Web application developer(s) to join an all-star team building a high quality, commercial open source, platform neutral, public health application utilizing state-of-the-art FOSS tools. This is an opportunity for a self-starting individual to participate in a ground-breaking effort that combines non-technical subject matter experts with skilled professional developers in a major, well-funded, open source community and project.

Job Details

Reports to: Project Program Manager

* Compensation: Salary & Benefits or Contract

* Location: North America work from home (if out of Portland, OR) with occasional US travel

Responsibilities

* Develop high quality test cases and application code

* Contribute to release planning, iteration planning, and retrospectives

* Communicate clearly via strong oral and written communication skills

Technical Skills

Required

* Ruby

* Java

* Testing frameworks

* Relational databases (Postgres experience preferred)

* XHTML

* Javascript

* CSS

* XML

* Ajax

* Linux

Desired

* JRuby

* RESTful Web Services

* Java GUI / Java Web Start / Eclipse RCP

* Systems integration

Methodology

* Open Source Software Development Best Practices

* Lean Software Development / Agile

* Test Driven Development

Update 11-DEC-2007 I got several questions on preference of Ruby vs. Python. Ruby is the preference. To make that clear I removed Python and Jython. Sorry for the confusion. I also added systems integration as there will be a fair amount of that as well.

Rails 2.0 - gem install actionwebservice

I'm kicking it in Sunriver before starting full time at Collaborative Software Initiative Monday.

Rails 2.0 is out.

From the link:

ActionWebService out, ActiveResource in

It’ll probably come as no surprise that Rails has picked a side in the SOAP vs REST debate. Unless you absolutely have to use SOAP for integration purposes, we strongly discourage you from doing so. As a naturally extension of that, we’ve pulled ActionWebService from the default bundle. It’s only a gem install actionwebservice away, but it sends an important message none the less.

Take that WS-* !!

Next Chapter

I am going to start the next chapter of my career at Collaborative Software Initiative.

As I get my feet under me there in the coming weeks I'll say more about it.

I have had a fantastic time in my current position & have learned a ton. I have worked with wonderful & brilliant people. Leaving is definitely bittersweet.

But this opportunity is where my heart is. I am passionate about open source and more specifically open source in vertical markets & government.

You only live once & you only get a shot at fulfilling your dreams a couple times. When opportunity knocks you must jump with both feet.

Wish me luck.

Ubuntu Thinkpad

I spent part of my evening installing Ubuntu on my new Thinkpad.

The download of Ubuntu went fine. Burning a CD/DVD took 6 tries!

After that, though, it was smooth sailing. Ubuntu found my wireless network & prompted me to install the proprietary NVIDIA driver for my graphics card.

Yay.

Disconnected Client Alternatives

I want to build a *simple* cross platform fat client application. It will allow users to enter data while disconnected and then upload it to a server when they have a connection.

I'd prefer not to use Java (black magic) or AIR (not ready?).

I'd prefer to hurt the web as little as possible (i.e., the more web technology in it the better for skills transfer)

I'd prefer it to be open source.

I'd prefer to not have to hire specialist ninjas to be able to build it. Every day web programmers should be able to get up to speed with it.

What are some good choices?

XUL? Mozilla Prism (some day)? Google Gears?

Update - Nov 27

This post got some helpful comments - thanks! And a nice post from Erik Onnen - thanks! There is much love for AIR. I went to the Adobe Bus Tour and generally like Adobe. The proprietary nature of it is tough pill to swallow - opening Flex was a step in the right direction, but I'd like to see more more more.

Release It!

What Steve Vinoski said:

Release It! is truly excellent.

REST UML Drawing

Nice REST UML drawring style by Benjamin Carlyle.

Kenny and Zukes

I went to Kenny and Zukes yesterday. I had a pastrami sandwich. It was freakishly good.

Erik Onnen claims it is better than NYC. I have to (gasp) agree. Dang good. And right down the street from me. Yum.

RESTful Web Apps

Nice post by Matt Raible on RESTful Struts. One thing Matt, code gen is a bug ;)

I have been thinking about this for a couple 2-3 weeks. How do you build apps that are apps and RESTful services vs. building an app and then a RESTful service bolt-on-side-car?

Pete Lacey had a nice demo with a Ruby on Rails demo of this type of thing last week.

I find it really entertaining that more and more people are having the REST light bulb go off in their heads. It is like - wait! I have been hurtn' the web for how many years!?!? You mean it could be this simple and I can have all this reliable, scalable, security booya because the web works so well? Holy s%*#!!!!

Java based Atompub Options

I know of the following Java based Atompub impls:

• Apache Abdera
• Atomojo
• Rome Propono

The Restlet folks have also talked of implementing basic Atompub support which is appealing.

Am I missing any other impls?

Anybody have any happy experience with these or other impls?

We looked at several a couple 2-3 months ago, but all have changed since then.

QCon REST

I had a great time at QCon Connecting SOA and the Web: How much REST do we need?.

The presentations / discussions were great. I learned a couple new things. It was definitely the best conference I have been to in a long time.

Having dinner with Dan Diephouse, Stefan Tilkov, Steve Vinoski, Sanjiva Weerawarana, Pete Lacey, Glen Daniels, Jim Webber & Patrick Logan was also a lot of fun. I have known many of them virtually for a long time. It was nice to talk in person and share some great laughs.

Stefan Tilkov - REST Talk

Good stuff.

Ducks

For whatever reason I stopped watching sports when I moved to Oregon 6 years ago. I think it was because I started getting into my own sports more here (fly fishing, hiking, biking) & it was hard to see the teams I liked on TV here.

I have followed Oregon State and Oregon a little bit, but not very much.

Yesterday was the first time I felt myself becoming a Duck Fan.

My dad called me from the Michigan - Michigan State game to report that Michigan State was coming back - I was watching the Oregon game completely unaware. I grew up a fairly big Michigan State fan.

I think that the fact that Oregon is doing so well is a big part of my interest. But as I make more and more Oregonian friends (who are fanatics for Oregon), I find myself caring a bit more. It is funny how the longer you live somewhere the more it feels like home.

That Dixon dude is ridiculous - very exciting QB to watch. I hope his knee isn't canned.

Update PS. I am still very sad that Michigan beat Michigan State. I hate it when MSU loses to Michigan or Notre Dame.

PM

Good stuff on Project Managers & Project Administrators by Dave Christiansen. He links to Are you a leader or a tracker? by Scott Berkun.

See also Dave's post on The Meaning of Rigor in Project Management

28-OCT-2008 Atompub Server Count Over Under

How many open source & proprietary Atompub servers will be available October 28, 2008?

If I was the house and I was setting the over-under, I'd say 10 & 6 respectively.

Update

Yikes: James Snell: We (IBM) probably have at least that many already in production within our firewall with more on the way.

I'm glad this is make believe over-under.

To clarify, I meant general purpose - integration focused Atompub servers. I predict that Atompub will result in 10 OSS and 6 proprietary general purpose - integration focused - Atompub servers. I'll go ahead and set the "Atompub built into your existing commercial product/service (e.g., Lotus Connections)" at 350.

Atompop

Bill de hÓra talked about a potential Atompop RFC a month ago.

He said: Messaging is a big deal and having something run over Atompub would justify an RFC.

+1 to that

Otherwise a lot of people are more or less going to reinvent it anyway.

More and more I'm seeing HTTP as the preferred protocol and protocol gateways to messaging. As we are using a bit of messaging and a bit of Atompub, it would be nice to have an RFC that would define how to do this one way. It might not be necessary (or achievable), but it could also be useful to include how to go from messaging to Atompub (i.e., the messaging side). I mean specify the "server-side gateway" & the "client-side gateway".

HTTP

I first started to really know HTTP when I was the technical lead of a Netegrity SiteMinder implementation in 2001. Prior to that, I just built web applications on top of it & didn't really know a ton of details about what was really happening on the wire.

I ran into so many bizarre problems with obscure web containers that I was forced to do all sorts of HTTP traces, repeatedly read the specs, etc. A memorable problem was a version of Chilisoft truncating SiteMinder's ginormous SMSESSION cookie. Identifying that defect took some work. Sun was awesome to deal with though - I remember them getting us a patch in 2 days.

Anyway, its an older book, but I'm reading HTTP: The Definitive Guide. I am surprised at how much I still don't know about HTTP. Anyway, I really like the book - it makes me like HTTP even more.

Enterprise 2.1

Mark Masterson is working on consolidating some enterprise memes into a list called Enterprise 2.1.

The whole naming thing is a little silly (e.g., Web 2.0, Enterprise 2.0), but I suppose it does have its place in the communication of ideas.

Anyway, here is the list:

• The impact of social software on the enterprise (the original Enterprise 2.0 meme)
• The superiority of ROA over other distributed system design styles
• The need to provide for reflective, adaptable behavior of business users rather than lock them into static, predefined processes
• The desire to reduce (and control) complexity and system entropy through the rigorous application of ideas like lesscode, convention over configuration, and favoring simple, lightweight solutions over large system designs.

It might not fit - or might be too much of "the how" rather than "the what", but something about open standards and open code on the list might be appropriate.

Creswell on Architecture

Dan Creswell says stuff about architecture: Consider how prevalent the use of frameworks is within our industry and think about the fact that in many cases one simply writes a POJO or two and leaves the rest to the framework. The framework makes life easier, it solves the big problems but it also exerts force on the design of our software as after all we must write it to follow the appropriate conventions, implement the appropriate methods etc.

The very worst example of the framework trend is seen in the decision to purchase a mammoth framework offering that provides everything in one box as an “integrated solution”. A huge stack that gets connected into everything and exerts massive gravity on our architecture. Everything becomes an exercise in warping aspects of our system to fit with this stack and the assumptions of its creators. Essentially we’ve bought “architecture in a box”.

---

The older I get, the simpler I like things. I don't have anything against frameworks, but they often become massive technical debt by themselves.

Einstein is over quoted, but more and more my views on software architecture are centered on Make everything as simple as possible, but not simpler.

This means writing the tiniest amount of code you can. For instance, use HTTP clients, use JMS clients, write a security impl., perhaps a simple feed reader. Deploy some really good web architecture. Some good monitoring foo. And then build in unit, integration, and acceptance testing from the beginning. And yer done. At least if you want to take advantage of all the engineering that already went into the web - which at this point is kindove lot.

Google Reader

I started using Google Reader a couple 2-3 weeks ago rather than Bloglines.

It is darn-tootn' good. And you can just export your feeds from Bloglines and import into Google Reader. 2 minutes, done.

Thanks for the good year (or so) Bloglines - you were very good to me . . .

Wake up folks - architecture is YOUR responsibility

If you want to make SOA work in a heterogeneous environment, why would you want to limit yourself to one technology, one approach? You’re buying right into their strategy of locking you into a platform. That’s only good if you sell platforms. Wake up folks - architecture is YOUR responsibility, not that of some vendors hawking middleware!

Ron Schmelzer of ZapThink

Simple

I descended into a simple blog link vortex - a couple notables:

Bill de hÓra: All successful large systems were successful small systems

Dare Obasanjo: My Website is Bigger Than Your Enterprise

Ryan Tomayko

Sam Ruby: Radical Simplification

Proposed Standard

Oh just a proposed standard huh?

So what does "Proposed Standard" mean? That might not sound very stable - "Proposed" - but that term does have special meaning in the IETF. The best way to explain it is that there are plenty of other protocols that you use every day that are "Proposed Standards", for example: WebDAV, TLS, LDAP, SMTP, SIP, and IMAP. Of course, if you want to know more about the IETF's standards process, they have it documented, in RFC 2026

Joe Gregorio talks (now at Google) about GData and how says: This is great news for us because the Google Data APIs are built on the Atom Publishing Protocol. The current Google Data APIs are based on early versions of the AtomPub specification and now that AtomPub is a Proposed Standard work will begin on getting all of our Google Data APIs compliant with RFC 5023.

Push & Pull Naming Standard & the Uniform Interface

The uniform interface is great. Resource name + GET, POST, PUT, DELETE & you are done. This is an amazingly simple / powerful constraint of the REST architectural style.

I spent a little time today thinking through a naming standard for our integration architecture. We need to support Push & Pull. This is essentially REST/AtomPub & Messaging.

For REST, I liked what RESTful Web Services had to say:

. . . So which is it? URI as UI, or URI opacity? For once in this book I'm going to give you the cop-out answer: it depends. It depends on which is worse,for your clients: a URI that has no visible relationship to the resource it names, or a URI that breaks when its resource state changes. I almost always come down on the side of URI as UI, but that's just my opinion.

It references URL as UI (Jakob Nielsen) and Universal Resource Identifiers -- Axioms of Web Architecture (Tim Berners-Lee)

URI Templates seem like a natural fit for specifying URI standards & URIs that are in use.

What about on the Push side (i.e., traditional MOM/XMPP) - what is the analogue? Is there one?

I toyed with the idea of using the same verbs (GET & POST at least), but I think that the semantics are different enough with messaging that it is either too clever or just lame to use a subset of the same verbs. The main issue is that GET isn't idempotent in messaging. I guess you could use GET for Browse, DELETE to pop off a queue, and POST to send a message, but it just seems too contrived to me. These are different styles - why break people's head with foolish consistency?

In the past I have had good luck with NOUN.VERB in messaging which is similar to REST in a way if you standardize on it.

Other ideas are NOUN.INPUT, NOUN.OUTPUT; NOUN.REQUEST, NOUN.RESPONSE.

Any other ideas / experiences?

Sweet Slides: The Web: Distributed Objects Realized!

OOPSLA 2007 The Web: Distributed Objects Realized! (Stu Charlton and Mark Baker)

Good paper(s) too:

REST is clearly not the "end of history" with regards to network-based software architecture. At smaller scales, with somewhat different desirable properties, other styles, such as Remote Data Access, or Event Based Integration, continue to solve important organizational and technological challenges. Beyond the current iteration of the Web, future requirements will require a re-evaluation of REST's constraints and desirable properties for a new set of requirements. This may lead to relaxing some of REST's constraints, or an introduction of new constraints (Extending REST for Decentralized Systems). Regardless of the direction this may take, a continued means to successful future systems architecture will be the discipline of objectively evaluating constraints, and the properties they induce, for the new generation of global scale, networked-based software systems.

Via Stefan Tilkov

Update fixed broken link to presentation. Also added this image as it caught my eye:

I distinctly remember the "Object Web" phase of distributed computing. I remember being a participant in it. I bought into CORBA. I remember saying things in 1998 like, "we need more - we can't be constrained to HTTP GET and POST in the browser. We need to be able to bust a socket and talk to an ORB for a richer experience" I was kinda clueless then. At least I had a lot of company - everybody including Lotus Domino which I was integrating with when I made that comment was busily adding CORBA capabilities to there whos-its and whats-its. Well everybody besides MSFT and friends, but that is a different story.

Almost 10 years later, we still just have GET and POST in the browser, but XHTML 5 will eventually fix that. It turns out you can get pretty far with just GET and POST. You can get even farther by just discarding distributed object / RPC technology all together and using the REST uniform interface.

QCon - Connecting SOA and the Web: How much REST do we need?

Like Patrick, I'm very excited to head to SF for QCon.

It will be great to meet Stefan Tilkov, Steve Vinoski, Pete Lacey, and Dan Diephouse in person. We'll no doubt have a lot to discuss.

History of HTTP

While at Powell's Technical this morning, I picked up HTTP The Definitive Guide.

It has a couple of good links on the history of HTTP:

Why a new protocol? Tim Berners-Lee (1991)

A Little History of the World Wide Web

Web Architecture from 50,000 feet

What the software of the future looks like

A nice post from Jacob Kaplan-Moss:

Let me try to explain what I’m talking about. You want an API without having to write a line of code? It’s called curl, and it ships with your MacBook. And just look at how simple the APIs are in your favorite language.

The wins go deeper than APIs, though. Think about what it would take to add load-balancing to CouchDB. I’ll give you a hint: perlbal. Or what about adding a transparent caching layer? Try Squid.

HTTP is the lingua franca of our age; if you speak HTTP, it opens up all sorts of doors. There’s something almost subversive about CouchDB; it’s completely language-, platform-, and OS-agnostic.

Look, CouchDB may succeeded, and it may fail; who knows. I’m sure of one thing, though — this is what the software of the future looks like.

---

Patrick and I were talking about this a bit yesterday. It is tough for some in the middleware world to get this. It will come with time. It is a bit nuts that your integration architecture really is just the judicious use of HTTP, well designed URIs, and data formats. How that architecture is realized (CouchDB, Squid, perlbal, etc. etc.) really won't matter over time.

If you can't answer yes, yes, and yes to: completely language-, platform-, and OS-agnostic when it comes to your integration architecture, you are doing something very wrong.

Via Simon Willison

Ubuntu 7.10 Install Today + Garden Shop + Semantic Web

I am going to do a couple 2-3 things today:

1. Install Ubuntu 7.10 (Gutsy Gibbon) Check out the screen shots
2. Go to the garden shop and get some ferns
3. Buy & start to read one of these if they are not at Quimby Warehouse (code for not in stock at Powell's Technical down the street)

XMPP Slides

Check out the nice slide deck on Peter Saint-Andre's blog.

Bloatware vs. Pairing Knife

An interesting post from Eric Newcomer on the HPTS (High Performance Transaction System) Workshop:

For example, one of the Web guys commented that they did not want to have to buy the whole set (i.e. complete product) when all they really needed was a paring knife, yet the only choice they are offered is to buy the whole set.

. . .

"Bloatware" is definitely part of the issue. Ironically so are many of the things a lot of us have worked to define, create, and promote during the past two or three decades around guarantees of atomicity and isolation. None of the Web companies use distributed two-phase commit, for example. They only use it to deque an element and update a database in the same local transaction. So much for all that work on WS-TX! ;-)

. . .

It will be very interesting to observe over the next few years the extent to which the ideas and techniques in the custom built solutions become more widely adopted and incorporated into commercial products. One of the inevitable questions, as raised during the discussions, is how broad the market is for such things as Google's file system and big table, or Amazon's S3 and Dynamo.

---

The next few years will indeed be interesting as we enter the post WS-* era. It will be interesting to see which middleware companies survive. For one I wonder if Google & Amazon will more completely enter the enterprise integration business by productizing some of their core technology.

PDX Software

The Silicon Forest blog has a brief interview with Jim Zemlin (The Linux Foundation - formally OSDL):

You have to look at the open source movement as a global phenomenon. My day literally starts with a call from Moscow and ends with a call from Beijing. From that perspective, geography is less important in this incredible global collaboration over the Internet.

Having said that, I'd like to look specifically at Oregon and where I believe Oregon provides a leading role...starting with Linus...I think it's important to Oregon that Linus lives there. I think that is a real indication of the importance of Portland as a place where real leaders in the open source community reside and do their work.

. . .

There is opportunity for places like Portland because of this fact, that it is a global phenomenon, to really attract top people to the area, either to be close to their peer group, or just based on the fact that it's a more affordable and superior standard of living.

---

It will be interesting to see what Portland looks like in 20 years. Come to think of it, it will be interesting to see what lots of places look like in 20 years. I am naturally most interested in Portland since I will in all likelihood still be here involved in the creation of software.

I see more small - mid sized software companies in Portland's future. Also small satellite development centers for mid-large companies as the world continues to flatten. I don't see many major companies moving here or anything like that. Portland has a decent amount of talent in open source & software more broadly.

The biggest thing Portland really has going for it is people (especially creative people) will typically move here in a heart beat as it is such a great / unique place to live.

I don't see Portland ever being a Tier 1 city or major tech hub. But clearly over time (20 years or so) SF, Portland, and Seattle will continue to morph together. I see a very bright future for software in Portland.

Distributed Scrum

InfoQ has a nice presentation:Planning and Maintaining the Rhythm of Distributed Scrum.

Summary At Agile2007 we heard the tale of a distributed Scrum project with 50 people in 4 continents. BMC Identity Management decided to build their next generation product, including architectural changes and component integration, using Scrum to handle the uncertainty of their product's requirements.

Will the technical debt consume us?

Pete Lacey quotes Anne Thomas Manes:

The problem is caused by the root culture of IT — project-driven funding models, a cobbler’s kids perspective on investing in infrastructure that helps IT (rather than a particular project), and a propensity to never decommission applications. IT systems have grown organically for the last 40 years. They’re a mess. It requires a fundamental change in the way IT operates as a service provider within the organization.

Here is the post on the Yahoo service-oriented-architecture list.

She is of course correct.

And all the while the technical debt is piling up. Maintenance is accounting for more and more of the IT budget. Soon there will be nothing left for new development.

There are of course solutions to this problem, but they require a lot of discipline and vision.

I think the discipline and vision are much less costly & more effective than the annual madness of budgeting in your typical large company.

Update 12-OCT-2007

I don't know why I didn't think of this last night, but Patrick quoted 12 Questions with Mary Poppendieck just yesterday in our internal wiki:

The first step in moving from forecast-driven projects to feedback-driven... is to change the measurements. The book "Rebirth of American Industry" by William Waddell and Norman Bodek makes a good case that the measurements imposed by traditional cost-accounting methods are the biggest impediment to the successful implementation of lean manufacturing. Similarly, I believe that the measurements imposed by traditional project management methods are the biggest impediment to the successful implementation of lean development. In particular, instead of measuring variation from plan, we need to start measuring the delivery of realized business value.

AtomPub RFC - RFC 5023

James Snell notes that AtomPub is now RFC 5023.

I think that AtomPub has heaps of potential.

Ubuntu 7.10

It's coming soon.. 11 days.

Ubuntu rocks. The new version is going to be interesting to watch. Ubuntu is already fantastic. But it seems like this could be a break out release with all they eye/feature candy.

ESB Spaz Out Excerpts

A couple 2-3 excerpts from the ESB spaz out that caught my eye. I have largely sat this one out since playing a small part in starting it. Perhaps I'll say something soon, but for now, here are a few quotes that caught my eye. I'm too spent from a long week to say much more.

Steve Vinoski: Reactions to the ESB Question

Eric Newcomer comment: I tend to think of the IT world as divided between systems designed before the Web, and those designed to include it. The mindsets are very different, as you point out, but I would add that the pre-Web mindset is kind of driven by mainframe centric designs - I like to think that the issues with existing middleware (and perhaps binary languages) is due to the fact we always felt we had to design in all the features/functions of mainframe based systems in order to entice enterprises to move those apps to standards based systems.

There is a lot of truth to this. I live with a lot of big iron. Interestingly, AtomPub & the whole pull based feed approach to integration is somewhat similar to batch processing. Perhaps it can help bridge that gap a little bit? Hey, that would be cool - Atom feeds from VSAM files & AtomPub from CICS. Maybe, but it is the batch cycle itself that has a strangle hold. Batch cycles that have been nurtured for 30+ years are very difficult to untangle. Things are really bound up there.

The Enterprise Cool URI will save us yet.

Steve Vinoski: The Degenerating ESB Discussion

Dan Hatfield:Honestly, I see the ESB as primarily a political thing. It allows for a greater degree of control on delivered solutions. In large companies, we don’t often do architecture - we do politecture…The politics drive the architecture. Not the way it should be…but that’s the way it is.

Politecture! Ouch. Sad but true. I walk this line on a regular basis. I push the envelope as far as it can go, but politics are ever present.

More Vinoski: Another non-technical way to look at it is from the viewpoint of Clayton Christensen’s classic book, The Innovator’s Dilemma. For quite a few years now, we’ve seen a series of sustaining innovations in the “object/service RPC” line of descent originally popularized by CORBA and COM, both of which built on earlier RPC, distributed object, and TP monitor technologies. RMI, EJB, SOAP, WS-*, and ESB are all offspring in that line, and there are surely more to come. I feel that REST, on the other hand, fits the definition of a disruptive innovation perfectly (and if you’re too lazy to read the book, then please at least follow the link, otherwise you won’t understand this at all). The proponents of the sustaining technologies look at REST and say, “well it can’t solve this and it can’t solve that” and voice numerous other complaints about it, precisely as Christensen predicts they would. But Chistensen also explains why, at the end of the day, any real or perceived technical shortcomings simply don’t matter (and in this case, they’re mostly perceived, not real). HTTP-based REST approaches have a lower barrier to entry and are less complex than anything the sustaining technologies have to offer, and REST is disrupting them, whether all the smart folks pushing ESBs like it or not. It’s not a technical issue, and there’s no amount of technology the non-REST tribe can throw at it to stop it because it’s based on how markets work, not on the technical specifics.

Internet Machine

Tim Bray: The Intimate Internet

But when the next big thing comes along (and I love this business, because I know it will) you won’t have to rely on the professional noticers to tell you because it’ll touch your life directly.

I love this business too.

This internet-machine is wicked cool.

Pretty Good ESB Spaz Out Underway

The "always-insightful" Patrick Logan has triggered a nice sized blog spaz out on ESBs it seems.

Good stuff is being discussed. Cool. The ESB Question - Steve Vinoski of former IONA fame.

More insight from Patrick: Properly Striking a Balance Between Shared Agreement and Decentralized Execution

There is plenty more, but you can find the rest pretty easily.

Camel Cast

James Strachan has a Camel Cast.

Ed (co-worker) and I messed with it a bit a couple weeks ago. It is approachable for newbies in that it is just code. I haven't really gotten that excited about the whole DSL thing, but an Enterprise Integration Patterns DSL sounds nice.

When Ed and I messed with it, it seemed young, but headed in a good direction. There is a lot to be said for simple - just download a jar file or 2 vs. adopting a whole platform. I know for a fact that there are a lot of people out there that would like to stop writing this type of code, but either can't or don't want to adopt a whole platform to do it.

It loks like the Camel docs might be getting rev'd a bit. I particularly liked seeing links to unit tests in the docs.

Good docs rock. Matt Asay says it results in more sales for open products.

These abstractions make it easier, see.

I came across this on time spent in "modern" Java web dev. Funny stuff.

I rat lol'd at the Spring bit and the Hibernate bit.

Via Buko Obele

Cool Enterprise URIs & Mule

Patrick Logan expounds on my Enterprise Cool URIs post.

Ironically, Ross Mason commented on Patrick's post & Patrick responded with what we've been discussing lately at work. The ironic part is I spent the better part of yesterday messing with Restlet and Mule (Ross's product).

My point of view on this topic has changed significantly over the years like Patrick's. I have been pro-CORBA, pro-EAI, tolerant of WS-* (couple months where I built a couple of them in 2001), pro-message broker, pro-ESB, pro-JavaSpaces, and most recently pro-please don't hurt the web

I think that if you spend enough time with middle-ware, you eventually conclude that the best approach is to not hurt the web as much as you can and embrace a more-or-less-peer-to-peer model (i.e., how the web works). A year and a half ago, I preferred the message-centric to the service-centric approach. I have concluded that at the time I was misguided in how I thought I was dealing with the fallacies of distributed computing. I don't think there is anything wrong with messaging & there is a place for it for sure - I just think that the dominant integration protocol should be HTTP. I used to see messaging in the center, now I see it on the edges. HTTP should be in the center.

It is probably important to note that I am talking about coarse grained integration ... the integration of large domains (e.g., sales, manufacturing). I've heard this called the "federated ESB" approach although I wouldn't call it that. I've also heard it called the "ABC" model (application, business unit/domain, corporate). I'm focused right now on the communication at the corporate level - across domains rather than within them. I think that HTTP makes sense everywhere, but for lower level integration (A & B) there may be other choices (I still think that conceptually, you can't beat the Tuple Space concept for A - large applications).

Mule is an ESB, but that term is essentially meaningless today. MuleSource also refers to it as an "integration platform" and an "Enterprise Service Network". Mule allows and encourages any-2-any protocol "mediation". This is good. Mule can certainly be used in a way that does not hurt the web.

I have more learning on Mule to do. I was a little overwhelmed in working with it yesterday (lots of jar files, lots of XML config, examples that didn't help with what I wanted to do, docs that didn't answer my questions - typical newbie stuff). I did succeed in getting it configured to do what I wanted it to do after going down a number of rat holes & now have something to build on this week.

Mule may be a good runtime choice, but I agree with Patrick that an ESB isn't your architecture. Your architecture is things like reference data, meta data, the services & events you expose, your URI naming standard, security, and what is on the wire. You want to build these things so they can stand the test of time.

The appealing thing to me about hiding as much of this as possible behind a good URI naming scheme that doesn't change & HTTP is that I know for a fact that the URI & HTTP will be thriving in 10 years. I can't say that about any integration vendor let alone any integration vendor's current product set.

Taming the Traps of Traditional Thinking

Nice article: Mind of the Innovator: Taming the Traps of Traditional Thinking. It talks about the "seven sins of solutions" & how to tame them.

This is tragic & very true. We all see this:

IDEA loops. IDEA is an acronym for Investigate, Design, Execute, Adjust. it’s a codification of the human learning cycle...the one that starts disappearing around age 5, once we enter the formal school system. that’s when it becomes about the right answer and not the right question.

I have been fortunate in that I have been affected by this less than most. Some of it is personality type, most of it is probably due to really good parents who encouraged me to try things and not be afraid to fail.

We need to stop thinking about innovation as an outcome, and start thinking about innovation as a process. we need to move from innovations to innovation. Because as a practical matter, innovation, problem-solving and learning employ the same iterative process—blending supposition, logic, creativity and reflection

Experiments show that creative revelations come when the mind is engaged in an activity unrelated to the issue being addressed, and that pressure is not conducive to creative thought. recent research demonstrates that the ultimate break—sleep—actually changes our mind’s perspective.

There are pros and cons to being a light sleeper (me). I sleep 6 hours a night unless I'm exhausted. I personally need to be more disciplined in turning things off. Most of the time, software related "work" doesn't seem like work to me because I genuinely enjoy it.

There is this too: Enter the irrational fear of failure. Backing off is counterintuitive. it somehow feels wrong, like preemptive surrender. it’s scary to ease up, because we may lose our steam, or we may abandon hope. we get anxious when the answers aren’t so forthcoming, and we begin to doubt our creativity, abilities and intelligence, fearing that if we take our eye off the problem even for a moment, we may lose the energy we’ve invested.

I definitely have the fear of losing steam - I'm generally not afraid of failure though. I'm a "damn the torpedoes" type. This too has pros and cons.

decisions: "follow the decisions you made" vs. "they were made"

I saw a link to this on Dan Creswell's del.icio.us:

Train-Wreck Management by Mary Poppendieck.

The article is referenced on InfoQ. The Poppendiecks are my favorite methodologists. I have yet to be disappointed with Lean thinking.

I highlighted some sections that stood out to me:

Exhorting people to "be careful," "try harder," and "work smarter" is not useful if individuals have little effect on results. Rewarding or punishing people for outcomes that are not under their control can only result in discouragement - or in gaming the system. Instead, chronic problems must be fixed by finding their underlying causes and addressing these effectively. As Deming points out, this usually involves changing the system - the way things are done. And according to Deming, it is management's job to change the system.

Deming: All of the empowered, motivated, teamed-up, self-directed, incentivized, accountable, reengineered, and reinvented people you can muster cannot compensate for a dysfunctional system.... A well-run organization with well-functioning systems allows people from top to bottom do work of which they can be proud.

"There is something called standard work, but standards should be changed constantly. Instead, if you think of the standard as the best you can do, it's all over. The standard work is only a baseline for doing further kaizen. It is kai-aku [change for the worse] if things get worse than now, and it is kaizen [change for the better] if things get better than now. Standards are set arbitrarily by humans, so how can they not change?

"When creating Standard Work, it will be difficult to establish a standard if you are trying to achieve 'the best way.' This is a big mistake. Document exactly what you are doing now. If you make it better than it is now, it is kaizen. If not, and you establish the best possible way, the motivation for kaizen will be gone.

"We need to use the words 'you made' as in 'follow the decisions you made.' When we say 'they were made' people feel like it was forced upon them.

Standards are not about how work should be done, but how work is being done. You don't want the standard to be too perfect, because that leaves no incentive for workers to improve their standards.

That is, workers - led by their team leader - do many rapid experiments, find a better way, agree on the improvement, quickly document the new way, and use it. When a standard is improved, the decision for the change must be made by the people doing the work, so they won't feel it is being forced upon them.

When Deming said "change the system", he was talking about changing the complex, interrelated processes used to get work done. Deming believed that changing the system is management's primary job, and in order to do this, managers need competency in four areas: 1. Appreciation for the overall system in which work is done 2. An understanding of variation - and the true relationship between cause and effect 3. Constant pursuit of learning (improvement) through designed experiments 4. An understanding of the psychology of people When all of these areas are balanced and working together, great things can happen.

Ready. Fire. Aim

David Christiansen sent me a good article by Tom Peters on systems thinking and innovation. I couldn't agree more with it. It is astounding how much time is wasted with excessive planning.

Tom's post reminds me of the book The Myths of Innovation. I read it earlier this summer.

I'm a big believer in failing fast and learning. If you want to innovate, you plan a little bit, create a whole team, give them a wiki, foster a collaborative/transparent culture, set a short deadline, try something, measure it, and do it again until you are done.

I whole heartedly agree with this excerpt: I am an unabashed fan of MIT Media Lab guru Michael Schrage—particularly his book Serious Play. His principal axiom: "You can't be a serious innovator unless you are ready and able to play. 'Serious play' is not an oxymoron; it is the essence of innovation." And, in turn, the heart of his serious play is ... fast prototyping: "Effective prototyping may be the most valuable core competence an innovative organization can hope to have." His intriguing connection, which makes all the sense in the world to me, is that true innovation comes not from the idea per se, though it guides the work, but from the "reaction to the prototype." In fact, in a surprising number of cases (the majority?) the collective responses to a host of fast prototypes reshape the original idea beyond recognition—or lead one down an entirely new path.

A more succinct way to sum up how innovation works is Ready. Fire. Aim. Tom's article says he heard a Cadbury exec call his approach to product development. This phrase is often used as what not to do when it is exactly what you should do.

Update 01-OCT-2007 Tom has some follow up posts that are well worth reading: Systems Thinking II: My Summer Vacation and Systems Thinking III.

Here is a great quote:

"For C-sakes, quit drawing those f-ing maps and run some experiments, quick and dirty, and see if anything you are babbling on about actually works or makes the slightest bit of sense in the real world as we know it. And after you've done your real work, then you are welcome to write your 'complete theory of everything.'" (That was close to the actual script, minus many more f%^*s and about 25 minutes of elaboration.)

Enterprise Cool URIs

Cool URIs don't change.

We are working on some fairly large scale integration at work.

As you might expect, we are trying to not hurt the web.

As most older "enterprises", we have many many aging systems. But the "domains" or large groupings of our systems are well known & shouldn't change too much over the next 10-20 years.

I'd rather bet on the URI than any ESB vendor.

I started working on a naming scheme for our cool URIs today. I think that if we get that right, we'll be heroes.

With a URI naming scheme that doesn't change you get a very different, very simple view of your systems. Sure, there may be madness today behind those URIs, but over time that madness will hopefully start to go away. But your URIs will stay the same. A simple vaneer on madness that slowly becomes sane. Yay Cool URIs that don't change.

Pair Atom Restlet

I rewarded myself after a long week of teeing-up the next phase of a big project with some pair-programming with Patrick.

We did some messing around with Restlet & Atom.

It was a pleasant afternoon.

Restlet is a great little API.

Atom is a great little format.

Linux and open source software pay off for PayPal

Linux and open source software pay off for PayPal: When Scott Thompson left Visa to take the CTO role at PayPal in 2005, the Web company's data centre surprised him. "Wait a minute," he recalls saying, "they run a payment system on Linux?"

"I was pretty familiar with payment systems and global trading systems, but I just scratched my head when I came here," Thompson says. With his history of working on IBM mainframes and large Sun Solaris systems, the PayPal approach to computing seemed alien, especially for a company whose core mission was dealing with money.

PayPal runs thousands of Linux-based, single-rack-unit servers, which host the company's Web-presentation layer, middleware and user interface. Thompson says he quickly saw the economic, operational and development advantages of open source and Linux technology. He now sees no other way to do it.

"When you're buying lots of big iron, as I did in other places I've worked, your upgrade path is $2 million, $3 million at a clip. You just had to buy big chunks of stuff to scale," he says. "Here at PayPal, our upgrade path is 10 $1,000 no-name servers, slapped into the mid-tier of the platform. And we just keep scaling it that way. It's unbelievably cost-effective."

Via Matt Asay.

GOSCON 2007 in Portland

I got a note from Deb Bryant regarding GOSCON. I went last year and it was great. I'm shocked at how much has happened in a year.

• The week after I heard Larry Augustin speak at GOSCON, Compiere was forked. Less than a year later, a VP from Oracle took over Compiere hoping to turn it around.
• JanRain was just getting started - I saw Jason McKerr speak at GOSCON. He had recently left OSL which was pretty high flying at the time. I remember him saying, "I'm speaking at an OSS conference, but our products aren't OSS. We did a lot with OSS at OSL." I don't remember much more about what he said. A year later, their CEO has moved on. I have no idea how they are doing besides that. But when that happened, it reminded me just how hard it is to start a company & how hard you have to work to make good software products that the market needs & will pay more $$ than it costs to make. Looking at the JanRain website, I don't know where the $$ will come from. I do think that OpenID is as good an idea as any though for identity. I get really tired of registering and remembering my password. I just don't know how they will monetize it given some of the stuff going on in that community. I hope they do though - clearly some talent there. And I have a natural bias towards all Portland tech companies.
• Bradley Wheeler gave the closing keynote at GOSCON - it was incredible.
• Stuart Cohen gave a speech as the CEO of OSDL (now The Linux Foundation). He now runs The Collaborative Software Initiave (CSI) - it is conceptually similar to Brad Wheeler's success with universities.
• Open Source was a key focus of my daily work (along the lines of Brad Wheeler & CSI)- now it is just one fairly small aspect of it. Open Source has come a long way in a year yet has a ways to go. IMHO it is an unstoppable force. But lest we forget, it only makes up 2% of the software market. My personal opinions around open source have changed a ton in the last year. I remain convinced that it is the #1 disruptive force in the software market, but am a little more realistic about where it is and where it is leading.

Anyway, GOSCON is on again this year (year 3) GOSCON 2007, Oct 15-16 in Portland. Deb says: This year's keynotes include: Andrea DiMaio, Vice President and Distinguished Analyst, Gartner Research; Jim Zemlin, Executive Director, Linux Foundation; Skip McGaughey, Director of Eclipse Ecosystem, Eclipse Foundation.

Roy Fielding Video

Stefan Tilkov links to Roy Fielding presentation video. It is about 30 minutes long & well worth being late for work over ;)

I wish I could go to ApacheCon to see the latest in person.

Message Queue Panic

Now Dare Obasanjo is joining the spaz-out re: messaging, push vs. pull and such.

Also, Stefan Tilkov is momentarily a binary messaging bigot.

XMPP is IETF'd. It has been around for years (started in 1998 according to Wikipedia).

AMQP is a young upstart. Perhaps it will break out. I'm just saying it will be hard. It is really hard to get a protocol going. Time will tell I suppose.

At the end of the day, IBM MQ Series/WebSphere MQ still has over 90% of the messaging market last time I heard so in terms of application integration within the "enterprise", I'm putting my money on that continuing for the foreseeable future.

Perhaps overtime, AtomPub, XMPP, and HTTP can put a dent in that.

The whole pub/sub business may be an opening. MQ Series isn't known for that. They have a new 100% Java version that I presume does a better job. XEP 0060, AtomPub, etc. could be alternatives.

In the short term, I think that ActiveMQ has the best chance of taking on IBM MQ Series/WebSphere MQ in terms of the more classical messaging stuff. But then again, I guess that may be ok with IBM as they support Geronimo which includes ActiveMQ.

Seagull Architect

Speaking of David Christiansen, he recommended a good book recently that I have been reading: Manage It! Your Guide to Modern, Pragmatic Project Management.

He dropped "Seagull Architect" in some meeting we were in a few weeks ago. He got that from the book.

A few excerpts from Johanna Rothman (author):

I've worked on several projects where the architect was like a seagull. He swooped in, dumped a lot of poop in the form of PowerPoint pictures of the architecture, and left as soon as possible. He didn't stick around for the hard part of the project: making the product work in the architecture ore evolving the architecture so that the product could work by the time of release.

. . . But if your architect is overly fond of drawing programs and not fond of writing code and can't really answer the developers' questions about how to make the parts fit into a coherent structure, you don't have a real architect. Eliminate that person from your project . . .

Yep, PowerPoint doesn't compile.

I have been accidentally guilty of this before. My reason was being spread thin, not that I can't sling code. I hated doing that to the team at the time. And it was true, the team really suffered. Ultimately I removed myself from the project and they were successful.

The $100MM Canonical Model

A co-worker of mine in Indianapolis, IN David Christiansen has a good post with The $100,000,000 Canonical Model.

Lest we forget, there are no sliver bullets.

There are two extremes in my mind when it comes to data formats and integration architecture:

1. Big Ball of Mud
2. A canonical model / data exchange architecture will solve everything

The pragmatic approach, IMHO, is to start small and build these assets over time via projects. The wrong approach is to go off for months/years to create it. Clearly, an asset like this needs to be as simple as possible, embrace change, be accessible, and based on Atom. (ok the last bit was bias).

GData AtomPub Podcast

I just listened to The world of Google data APIs. It is 42 minutes long. I took notes.

Here is the agenda:

• What "Google data API" actually means (the parts and pieces)
• What Atom, Atom Publishing Protocol, and other tech behind GData are all about
• What GData adds to the mix on top of Atom and APP
• How Atom compares to RSS
• What are ETags? And how can they help me?
• Why REST, the style, was chosen for these APIs
• Where REST makes sense, and where it doesn't. Resource driven vs. RPC.
• What the first GData APIs were
• How the killer app of syncing data with Google Calendar
• How you actually use the APIs? What do they need to learn? What tools do we give them?
• Can you write APIs that implement the same GData APIs?

Notes: Atom is an IETF standard in case you didn't know. RSS isn't.

Google has been working with Atom for over 2 years.

AtomPub provides a basic REST API - Google thought this was a great starting point. Atom leaves query to the student. Google uses URLs to do this. You can have output sorted etc. Atom doesn't say anything about this.

ETags were unfortunately not implemented. They chose a version number in the URL so that multiple people can write to the same entry. Plan to implement ETags in the future.

Are ETags magic? No, a lot simpler than they sound. A little string that tells you the version of the entry. Just a great way of making caching work & making the web more efficient.

Chose Atom because of momentum. And sold on REST. Not SOAP because the web is based on REST. Much easier for devs to learn REST than SOAP. SOAP require tooling. REST is simply manipulating tables of entries.

What are disadvantages of REST? AtomPub is just an implementation of the REST style. Difficult to map certain types of operations to REST - translation API example: send text & send back a list of options & then feedback how to improve the translations. Document centric request / response that was never saved on the server. Just a straight RPC call. REST is about manipulating resources.

How about transactions? Each request is essentially transactional. REST not concerned with multi-resource transactions.

How do devs use? Can use curl if they want to - some do create apps with shell scripts and curl. Have a Java, .NET, PHP, Python, Objective C API. Contributed APIs: Lisp (Patrick!), Ruby, Flash in development. It's just XML over HTTP - it's NOT THAT HARD!!

Authentication - "Client Login" is pretty straight forward auth. URL for uname/passwd where you get a token back that is your identity. Authentication for Web Apps - "Auth Sub" used for "on behalf of" type stuff. You can grant access to other web sites. You control who has access to.

Can I use Google Auth? They are meant to be open standards. Google legal hasn't signed off yet, but licensing will be worked out soon.

What is this "Kinds" business? Entries get passed around a lot - kinds concept is Atom categories to tag each entry with a "kind". Just gives more semantic information to computers (clients).

AtomPub Google Interop Event? 12 devs from different orgs came. Great success. Google's basic AtomPub worked fine. Google custom auth scheme was more problematic. After that everything worked smoothly. A lot of the impls built around AtomPub introspection document - Google doesn't use them much. Will in the future.

Any GData / AtomPub tips? Google has a particular approach to designing APIs. AtomPub good at certain APIs. Things that map well - RPC not one of them. On occassion Atom Entry is just a pointer to the real data (e.g., photos). Prefer to put data in entry as much as possible. There is a lot of art/style/parsimony to it. You can achieve a lot with 1 feed with a lot of query parms. Have to review APIs & ensure that they have good clean concepts: this feed clearly includes these types of elements.

WADL? Haven't found the need for it.

Atom can be tough to get at first, but once you do it is amazingly simple & then applicable to many. Very good programmers who don't understand AtomPub gargen/language can. Concepts are very simple. Feeds, entries, links to other entries. Very simple mental model once you get it. All APIs make the same. Very powerful.

Google working with the IETF on improving Atom/AtomPub. Introduced batch model that increases efficiency a lot. Auth. Teams in Google are very ambitious - hope to make publically available as drafts that can become standards.

Some talk of JAY-SAHN (JSON).

Viking Bus

Erik Onnen (will answer to "Viking") has a good post on Apache HTTPD & using it as your integration bus.

Day by day I am making a cleaner break with my MOM bigot history.

There is clearly a place for messaging, but I think it's future is on the web. XMPP is of course perfectly positioned for this.

Fixed

Matt Asay writes about a case against a SI (actually my first employer - it was called Andersen Consulting then):

[T]he U.S. Department of Justice is suing Accenture for allegedly receiving kickback-like payments from technology suppliers it recommended and/or implemented at DOJ. The alleged fraud was a collusion with big-name IT suppliers (e.g., HP, Sun) and smaller vendors (e.g., Vignette) to defraud the Government.

IT land is a very interesting place. It never ceases to amaze me. There is very interesting behavior in this market. There are all sorts of vendors all chasing the same Fortune 500 - 2000. There is tons of money, yet there is none. Everything is easy, yet everything is impossible. To survive/thrive in IT you need to develop many different types of skills. You must, however, maintain your integrity above all else.

Note: Obviously I have no idea if there is any merit to this case. All of the people I worked with and Andersen Consulting had heaps of integrity. I just felt the desire to beat the integrity drum today and saw this and leapt on it.

Viking Async Sighting

The Viking (aka Erik Onnen) spied AsyncWeb.

From AsyncWeb home page (bold emphasis mine):

The majority of today's popular java HTTP engines are built around the J2EE Servlet Specification - and are inherently synchronous in their operation. The typical order of play is that an individual HTTP connection is mapped to a thread of execution which reads data from the underlying socket, parses the request and forwards it up through the web container to the target Servlet - blocking until a response is completed.

As processing latency (as described above) increases, a real scalability problem is encountered: To increase throughput, it is necessary to increase the number of connection threads. This, however, is not the best way to scale - and depending on the JVM / operating system employed, having a very large number of threads can become a big problem very quickly.

A real problem then needed to be solved: How can we obtain a very high throughput - whilst dealing with high processing latency - without requiring a potentially unbounded number of threads? An initial area of investigation was to provide a NIO transport connector for an existing HTTP engine. However, existing popular HTTP engines built around the J2EE Servlet Specification have a blocking synchronous architecture throughout. Swapping in a NIO transport to such an engine does not solve the problem: We'd still need to scale processing threads in order to push requests up through the container.

What was needed was an http engine which provided non-blocking behaviour throughout - and so work began on writing such an engine.

Looks like AsyncWeb has been heading for Apache MINA for some time. Apache MINA is a network application framework which helps users develop high performance and high scalability network applications easily. It provides an abstract · event-driven · asynchronous API over various transports such as TCP/IP and UDP/IP via Java NIO.

The things I drawr don't come true!?

Wait, those gigantic diagrams on my wall aren't going to compile with the next version of my bitchn' uber IDE? But the vendors said it was just around the corner!!

"Words and diagrams are all we have to work with. Mr. Klein. We need to live within their limitations. Seek not a single diagram to capture your enterprise architecture. Seek not a single sound bite. It is a hyper-dimensional thing you are grappling with. All you can do is capture fleeting glimpses of it with words and 2 dimensional views. Do not start with the wiring diagram. Mr. Klein. End with that as the least important view of your enterprise architecture. One view amongst many and a not-very-important view at that." Sean McGrath

Via Stefan Tilkov.

Dan Diephouse & Abdera (AtomPub)

Yay cool beans Dan.

Spring & Abdera

Both please

Dan Creswell picks up on the INATT business.

The problem is myopic views that focus on either only technology or only business. My point is that the lines are blurring. And the right approach is a balanced view.

If you spent millions and a year on a "build it and they will come" approach to deploying an ESB for your SOA platform, drink 3 shots of "It's Not Not About the Technology KOOL-AID" ASAP.

If you are ignoring the fundamental changes occurring in technology & the relationship society has with technology, drink 3 shots of "It's Not Not About the Technology ALONE KOOL-AID" ASAP.

CouchDB

I can't remember who's blog pointed me at this, but it is very interesting: CouchDB: Thinking beyond the RDBMS.

There clearly is a need for something like CounchDB. All the impedance of mapping document data to relational is quite tiresome.

I don't think that we'll ever get rid of relational databases all together, but at the very least it would be nice to have something for the cases where it makes no sense to map to a relational database and you want to effectively retain your documents. This could be great just to effectively store service interactions. Is it really just JSON or can I stick XML in there too? JSON is great and all, but there are an awful lot of XML documents out there.

I thought document storage like this was a great idea going back to 2000-2001. I was working at eXcelon (now Sonic Software) at the time. They had a product named XIS that was an XML database built on top of eXcelon Objectstore. It's issue in general at the time was scalability - I am not quite sure what happened to it after Sonic Software (Progress) bought eXcelon.

REST Conversation

Benjamin Carlyle has a good post on simplifying the introduction to REST here.

I love the question/answer style blog post. I have to do that sometime.

INATT

INATT - It's Not Not About the Technology

INATT is pessimistic and self-defeating, even if it's not intended to be. It denies that there can be improvements, incremental or radical, in the ability of technologies to accomplish important goals. I disagree categorically with this. . . .

I think I may have even said this last week. In that case it really wasn't about the technology. But more often than not, It's not about the technology alone. as Andrew McAfee says.

IMHO technology is going to steam roll a lot of people and companies in the coming decade. Many are in denial. We are in an age where generations and generations of platforms have accumulated and we are drowning in technical debt. And we have generations of people that are accustomed to doing everything on computers, cell phones, and the Internet. These people will be the majority in 10 years.

Of course business execution aside from technology is crucial. The problem is that technology is becoming more and more crucial to business execution. INATT denies the freight train coming towards many businesses.

I predict that more and more companies will be coming to terms with the fact that it is about the technology like Amazon did. What happens when there is the Amazon of your industry?

Via Stefan Tilkov.

Atom & Pub/Sub Revisited

Tim Bray picked up on my Pub/Sub vs. Atom & AtomPub? post.

During the last year, I have not been doing that much messaging, but the previous 7 years I did a substantial amount of it - both point-2-point and lots of pub/sub. I absolutely don't think that Atom / AtomPub by itself will replace messaging. I don't know enough about XMPP to know if it will replace traditional messaging some day. I am a big fan of AMQP, but don't know if it is viable yet or will be viable soon. Tim is dead on that PSB is an impressive corpus of work

My point, was only that Atom & AtomPub appears to be a very good format / protocol to do business events with a polling model. Having done a lot of pub/sub, I have done a lot of clustering. The buffer that Tim describes is not so simple when you get lots of clients and lots of consumers. That is where flow control starts to appear (if you really want to guarantee message delivery) and where the dark corners of fail over and clustering rear their heads. It is nothing against messaging per say - just how it is. My point really was that even "guaranteed" messaging is often not truly guaranteed. And that for the right usage scenario the poll model is simpler and more appropriate.

Furthermore - push models can be appealing to developers because it feels cool (has felt very cool to me) - even simple. My point is just that this isn't typically really the case.

It really just comes down to your requirements. For example message rate, how quickly your event sinks need to process, etc. For where I am thinking right now, high level business events don't typically need to be delivered more than every 10 minutes - perhaps ever minute. I'd much rather deploy some simple HA web infrastructure & Atom/AtomPub than AMQP, TIBCO, SonicMQ, ActiveMQ, etc. to meet those requirements.

In my experience with pub/sub style integration architectures when you are pushing information amongst major domains (e.g., party/customer information), there is always the nagging question if you missed an event - or how do I deal with drift. This is because even though it is supposedly guaranteed messaging, there is always a risk. I've seen people try to capture events and persist them so that they can be replayed in the event of this type of a problem. Or more commonly, check the master system every month to ensure that the systems are synchronized. An Atom feed seems like a good way to avoid this failure all together - in that rather than pushing a copy of the event to each interested sink, the sink just reads the feed. Sure the client has to be a little bit more intelligent, but it just seems more deterministic to me. But I have more to learn.

Atom Service Directory

I re-read some of the Atom specifications on a plane today.

I haven't been as excited about a data format since creating a mostly proprietary, but somewhat based on ACORD one at a Canadian insurance company in 1999-2000.

I still have a lot to learn about ATOM, but it seems to address all of the pain points I have seen in integration data formats so I'm optimistic.

I obviously can't be the first one to think of it, but wouldn't Atom / AtomPub make a nice simple Service Directory (think UDDI, but with legs)? Just a feed with entries for each service instance? Categrories and other meta data? You could dynamically register services via AtomPub - perhaps services would have to re-register every so often. Is there something like this already?

I also looked back at some of the blogosphere spaz-outs regarding Atom yesterday. I have more to learn, but perhaps it is best that Atom get's a head of steam before embrace and extend sets in anyway.

Here's to hoping that Atom stays simple - seems to be a beautiful format.

XML Schema Languages

Ok, so Atom for the document format, what about the XML Schema Language?

According to Tim Bray (and others), it is no contest - RELAX NG wins.

The last two canonical models I was involved in used XSD. I have a hate hate relationship, with XSD, but I'm not yet convinced that RELAX NG is the path forward for me right now. I need to get my learn on. The biggest issue, really, is that there is a lot of XSD out there - a lot of schemas that I may want to fork into my canonical format are already in XSD.

In my experience, the basic 25% of XSD is ok - it is the other 75% that has a lot of very dark corners. That and the never ending evil of code generation from schema. Here are some other benefits of RNG over XSD according to Wikipedia.

I also have to figure out where Schematron fits (or doesn't fit).

What tools are people using with RELAX NG and Schematron that they are happy with? I see Topologi. Last time I was involved in this we used XML Spy. It is a pitty that it does not seem to support RNG?

What about runtime support - what works well for Java, .NET, Ruby, and z/OS (the last one should be fun).

Any experience in the trenches?

Canonical Model Format & Atom

It has been awhile since I thought a lot about canonical models. I have been involved in a few over the years.

I'm thinking about them tonight.

In creating one today, is there a reason why you wouldn't use Atom instead of say SOAP or more likely your own proprietary solution? I re-read Pete Lacey's classic The S stands for Simple to remind myself why I left SOAP behind.

SOAP isn't even on my radar these days, but perhaps someone can convince me otherwise on why it belongs in the heart of your integration architecture? Of course you have to interoperate with it, but you can certainly keep it at the edges (i.e., integrate with legacy web services, packaged apps that only expose a SOAP interface that you want to use).

We are talking about your lingua franca here - there are few things in my opinion, more important to an integration architecture than a canonical model - you want to get this right.

SOAP started simple enough - it just became an industry and now it is fairly hard to separate the wheat from the chaff. Here's to hoping that Atom stays simple.

I kinda doubt that many people are using SOAP as a part of their canonical model. I would have to bet that they are using something they made up. From what I can tell, Atom seems to give you what you need out of the box - most people will need to add some more standard elements to it, but it has all of the basics so why reinvent the wheel?

Jive Funding

Wow, Jive took some VC funding.

A couple of these guys are my friends - they work their butts off. Great guys, great company.

Congratulations Jive!

Amnesia & Terminal Gravity

These brewers are not easy to find. But they are incredible.

Terminal Gravity (Enterprise, OR).

Amnesia Brewing (Portland, OR)

After a long day yesterday, I had one of each at Henry's Tavern last night :)

Weak Sauce: RESTful Flex Angst

I think this report from Erik Onnen on Flex and REST is pretty weak sauce.

I'm surprised Flex doesn't have a real HTTP client built into it. You can't do REST with just GET and POST. Being dependent on the browser is pretty lame - is this driven by some sort of other requirement or some security sand box what-nots?

I guess that is what we get for hurting the web.

How are others dealing with this?

Pub/Sub vs. Atom & AtomPub?

I came across an interesting article by Sean McGrath entitled I'll push and you pull. The mashup approach to application integration.

I posted about Push vs. Pull in the spring. I'm a sucker for Lean thinking so the title of this article got my attention.

Sean says:

Now here is the kicker. The web - with all its concomitant bits'n'bobs from XML to RSS/Atom to AJAX - is an extremely good platform for pull-centric design. On the Web, if you try to pull some piece of information and something goes wrong, well you just pull again and again until you get it or give up. Nothing fancy. Just brutish repetition. Something machines are extremely good at. If you want to look at information from yesterday, you just go to the URL that contains yesterday's information. Nothing fancy. Just a simple naming convention that includes dates in URLs.

I have done a fair amount with pub/sub over the years (complete with clustering etc., fail over strategies, etc.). I like it, but it is not without it's challenges. If only there was real guaranteed delivery etc. If only there wasn't "flow control". If only fail over always worked. If only messages didn't get trapped, etc. All I'm saying is having implemented several large implementations of pub/sub (EDA whatever you like), I know that it isn't easy - that nothing is really "guaranteed".

Clearly Atom can't replace ALL pub/sub use cases, but for every day integration architecture where you want business events / EDA why can't we use Atom feeds? In an extreme case, you might have an event sink requesting the feed every 10 seconds - in most cases every 10 minutes would likely be fine?

Who is doing this today? Any lessons from the trenches?

Update I meant to read the article earlier linked off of this post by James Snell. Just did. Dang. It is a sample of using Apache Abdera.

Update 20-AUG Bill de hÓra has a pretty $$ response - thanks Bill.

Update 21-AUG Dan Creswell has some thoughtful comments. When I blog I often do the x vs. y thing - but clearly the truth is in the middle. I used to be a push bigot and have just learn the hard way how difficult it is to achieve. Clearly, not everything can be pull - of course it is a mix. In case anyone is curios, I'm thinking of inter-domain integration (e.g., getting customer additions/updates/deletions to many interested systems) with this line of thinking rather than some sort of intra-domain integration (e.g., trading system where there is massive high performance pub/sub). You have to choose the right tool for the job. More and more for me, simplicity is winning out. This is just an evolution of my thinking - a year ago I was still a MOM bigot.

Bell's Oberon

Speaking of beer, my family had a fair amount of Bell's beer in Glen Arbor.

Oberon is delightful.

Bell's is far and away my favorite non-west-coast-microbrew.

Deschutes Brewery Down the Street

My favorite beer of all time is Mirror Pond.

It is certainly the beer I have consumed the most since I got out of college. Sadly, the beer I probably have consumed the most of is The Beast.

I am pretty happy that there will be a Deschutes Brewery down the street from me this coming Spring.

Note: the picture above is their new label - finally they are getting rid of the crappy current one. It isn't that great either. Oh well you don't drink the label very often. I still think the first 2 generations were best: (source)

Second Generation

Original