Atom & Pub/Sub Revisited

Tim Bray picked up on my Pub/Sub vs. Atom & AtomPub? post.

During the last year, I have not been doing that much messaging, but the previous 7 years I did a substantial amount of it - both point-2-point and lots of pub/sub. I absolutely don't think that Atom / AtomPub by itself will replace messaging. I don't know enough about XMPP to know if it will replace traditional messaging some day. I am a big fan of AMQP, but don't know if it is viable yet or will be viable soon. Tim is dead on that PSB is an impressive corpus of work

My point, was only that Atom & AtomPub appears to be a very good format / protocol to do business events with a polling model. Having done a lot of pub/sub, I have done a lot of clustering. The buffer that Tim describes is not so simple when you get lots of clients and lots of consumers. That is where flow control starts to appear (if you really want to guarantee message delivery) and where the dark corners of fail over and clustering rear their heads. It is nothing against messaging per say - just how it is. My point really was that even "guaranteed" messaging is often not truly guaranteed. And that for the right usage scenario the poll model is simpler and more appropriate.

Furthermore - push models can be appealing to developers because it feels cool (has felt very cool to me) - even simple. My point is just that this isn't typically really the case.

It really just comes down to your requirements. For example message rate, how quickly your event sinks need to process, etc. For where I am thinking right now, high level business events don't typically need to be delivered more than every 10 minutes - perhaps ever minute. I'd much rather deploy some simple HA web infrastructure & Atom/AtomPub than AMQP, TIBCO, SonicMQ, ActiveMQ, etc. to meet those requirements.

In my experience with pub/sub style integration architectures when you are pushing information amongst major domains (e.g., party/customer information), there is always the nagging question if you missed an event - or how do I deal with drift. This is because even though it is supposedly guaranteed messaging, there is always a risk. I've seen people try to capture events and persist them so that they can be replayed in the event of this type of a problem. Or more commonly, check the master system every month to ensure that the systems are synchronized. An Atom feed seems like a good way to avoid this failure all together - in that rather than pushing a copy of the event to each interested sink, the sink just reads the feed. Sure the client has to be a little bit more intelligent, but it just seems more deterministic to me. But I have more to learn.

Nice

This just in from the nicest person I know (my mother):

Part of a prayer I read this morning:

............"that I will be well in my own soul and part of the world's healing this day." Amen

SOAP over JMS

Hey, looks like SOAP really does work over any protocol. Well over JMS anyway (not really a protocol). Saw this here and here. I'm sure all of these vendors (I know Sonic does) provide their own proprietary SOAP to JMS mapping. I suppose there is some value in them mapping it the same way.

The chances of me using this? Zero.

With SonicMQ, however, I have happily used their RESTish HTTP Adapter that just maps destination names into the URL & looks for HTTP Headers for any messaging specifics. Want to check for a message? Check the response queue via a different HTTP GET. Certainly not perfect, but can be useful when working with a technology that does not have a richer client available.

I took a quick scan of this document & remembered a Tim Bray post from last year.

I guess I won’t offer any further commentary on the contents of this document. I can’t help but feel for the H/I/I/M staff who are going to have to do the work, and am reminded of Fritz Lang’s immortal Metropolis, where Maria says: “Nobody cared about the slaves who died laboring to raise the Tower of Babel.”

Not sure if they already are (don't see any of their names listed), but it would be nice if these vendors got involved in AMQP. That seems to have some promise.

Why AMQP? Though many networking protocol needs have been addressed, a large gap exists in common guaranteed-delivery messaging middleware. AMQP fills that gap...

What is AMQP? AMQP enables complete interoperability for messaging middleware, both the networking protocol and the semantics of broker services are defined in AMQP.

AMQP Model The AMQP model explicitly defines the server's semantics because interoperability demands the same semantics for any server implementation.

Wire-level Format To enable technology-neutral interoperability, AMQP defines an efficient wire-level format with modern features.

Advanced Message Queue Protocol

My friend Erik pointed me at Advanced Message Queue Protocol a year ago, but I assumed it died.

Well, he pointed me at it again and it looks like it may go somewhere.

This would be fantastic. While I love messaging, bridging messaging systems really limits their reach.

I wonder how they are going to get everyone to play along (IBM, Tibco, Sonic, etc.). Hopefully, RedHat's effort will help: RedHat is currently working on an implementation that will be built into the operating system, making AMQ as free and available as SendMail, and accessible from any technology API such as JMS.

If this happens, it would be a major step forward in my opinion and a major enabler for large scale event driven architecture.

Temporary MOM bigot

I'm currently a MOM bigot, but only because I haven't seen another viable way to do extensive service oriented integration today.

I yapped about Message-Centric vs. Service-Centric a month or so ago.

I heard that SOA Software acquired Blue Titan this week. This is an interesting development. Here is a good post on why this happened (via Stefan Tilkov).

As I tried to get going on the Yahoo REST message board and in various posts in this blog, this is where I'd like to see things go. Leverage HTTP, but add:

• guaranteed delivery
• durable destinations
• event driven support (i.e., pub/sub)

It is ok to support SOAP, WS-* for the true believers, but do not mandate it. If anything, tolerate SOAP/WS-*, but prefer POX and REST. I wouldn't buy something that preferred SOAP/WS-* and had an extension or subset of support for POX and REST.

Now, I have no idea if Blue Titan even does this (I haven't used it) - it sounds like it does some of this or will. Anyone worked with it?

Ideally, I'd like to see OSS impls or at least dual licensing of this type of thing. This type of infrastructure is too critical to rely on a small vendor for. I can't afford another vendor-pire.

Message-Centric vs. Service-Centric

Stumbled across this on Dave Orchard's blog. It talks about leaky abstractions in SOAP. Also discusses how SOAP hides the underlying protocol. This is the main reason that REST or POX/HTTP appeals to me more then SOAP - they say you have to embrace the network and not hide it. SOAP just brings too much unnecessary complexity to the table for my taste. It truly is the doorknob to hell.

When I was working on "Aspira" - the EDA (Event Driven Architecture) bus that never was, we dealt with this by intentionally forbidding any:any transport. You couldn't for instance have an HTTP endpoint talk directly to a Socket endpoint. Every transport was designed to have an adapter. That adapter would deal with the transport in its native form and get the event to the persistent / trusted / consistent messaging layer as fast as possible. In short, it was "Message-Centric" rather then "Service-Centric". Decent write-up on the differences by CapeClear advocating the opposite of what I think.

The nice thing about the "Message-Centric" approach is most of your services end up being vanilla - they use one of the standard service types, which, deal directly with the messaging layer for its inputs and outputs. They get the benefits of persistent messaging, clustering, consistent security, fail over, durable subscriptions, standard error handling (e.g., rollback and pause, route to error queue). Yes, they do hide the network, but that is ok, because network problems have been accounted for. And the services don't care about the payload - so long as it is XML. The result is your service impls are stupid simple. They just do their little bit.

Dave Orchard's post prescribes some proposals for fixing the problems he sees. Sounded good enough for me. I just think the REST or POX/HTTP approach is the better way to go. WSDL/SOAP intended to ease pain, but sadly, I think they make things more complicated then they need to be. I think it is far simpler to just embrace various protocols directly and stay in control. Yeah, at the outset, it might feel a bit more complicated, but in the long run, you save yourself pints of blood.

Anyone care to talk me out of the "Message-Centric" approach?

Sonic Software - Actional in Seattle

I went up to Seattle today to listen to Sonic Software and Actional (Progress/Sonic recently acquired Actional) talk about SOA/Event Driven Architecture best practices and maturity models and such.

I had to get up at the butt-crack of dawn to get there in time. It is a 3 hour drive from Portland to Seattle - the meeting started at 8:30. Not fun. So early that I forgot my belt. Good thing is I look particularly handsome without a belt unlike most people.

I also participated in a forum talking about lessons learned and SOA challenges. It was interesting to hear other points of view, etc.

My company uses SonicMQ as our JMS provider. One of the better ones out there. Coolest thing they do is "shared subscriptions". Basically lets you scale a subscription horizontally. You just use their clever little shared subscription syntax when you register the subscription. Then, multiple connections on N hosts are considered one logical entity and they shared the burden of processing the messages for that entity. Basically, you get the benefit of competing consumers on the Topic. It is just like it is a Queue, but you can have other shared subscriptions and normal subscriptions listening to the same topic at the same time.

I didn't know much about Actional until recently. I had lunch with Sonic's new CTO, Dan Foody and some other Sonic guys. Dan was the CTO at Actional. They have some cool products. Using their products, you can monitor a SOA/EDA deployment end to end. It traces transactions through all your services. Apparently they do this by some byte code magic where they listen at all the appropriate times. They append headers to HTTP/XML requests, JMS, CORBA, RMI, etc. that allows them to trace the logical transaction where it goes on the SOA/EDA. Wicked cool.

We do this by hand currently - as event bounce around, we scribble which hosts/Destinations they were routed on, which services processed them, etc to a "event history" message property. This information is really useful when events end up on error queues etc. Also nice for debugging and bringing new developers up to speed on the system. Self documenting and what not. We just do this for JMS though ... we don't have the same visibility for HTTP based services. Anyway, the Actional stuff seems pretty useful. I want to give it a spin.

Update 17-JUL-06 - Now that it is possible to determine my employer by reviewing my blog posts, I must follow my employer's blog policy.

This is not an endorsement by Liberty Mutual, but my personal view. Vendors are NOT free to quote with attribution.

Initial Thoughts on JEP-0060 Pub/Sub

I re-gandered the JEP-0060: Publish-Subscribe spec. It is a pretty cool specification. See a lot of similarities to JMS, but a fair amount of differences. I don't think that it is going to cut it though for what I am after, however. What I would like to see is a RESTful (or Web Style as Tim Bray is now calling it) event driven - pub/sub.

The JEP-0066 messages for registering a subscription are very concise and relatively simple which I like:

Example 1. Entity requests a new node with default configuration.

<iq type="set"
    from="pgm@jabber.org"
    to="pubsub.jabber.org"
    id="create1">
    <pubsub xmlns="http://jabber.org/protocol/pubsub">
        <create node="generic/pgm-mp3-player"/>
        <configure/>
    </pubsub>
</iq>

Example 2. Server replies with success

<iq type="result"
    from="pubsub.jabber.org"
    to="pgm@jabber.org"
    id="create1"/>

What I don't like, however is that just like SOAP, the actual message has the meta data in XML:

Example 9. Entity publishes an item with an ItemID

<iq type="set"
    from="pgm@jabber.org"
    to="pubsub.jabber.org"
    id="publish1">
  <pubsub xmlns="http://jabber.org/protocol/pubsub">
    <publish node="generic/pgm-mp3-player">
      <item id="current">
        <tune xmlns="http://jabber.org/protocol/tune">
          <artist>Ralph Vaughan Williams</artist>
          <title>Concerto in F for Bass Tuba</title>
          <source>Golden Brass: The Collector's Edition</source>
        </tune>
      </item>
    </publish>
  </pubsub>
</iq>

Example 10. Server replies with success

<iq type="result"
    from="pubsub.jabber.org"
    to="pgm@jabber.org"
    id="publish1"/>

While this is much better then what you get with WS-*, I still think there is a better way. I'm thinking RESTful messages to create subscriptions (PUT) (would include URL where subscriber is listening), remove subscriptions (DELETE) (etc.).
And then for the actual events that are delivered to subscribers, the payload could be anything ... XML, JSON, etc. The message id, correlation id, redelivered, etc., custom message properties ... anything for the "Header" would be HTTP HEADERS rather then embedded in the payload.

The power I see in this is that you could easily implement something like this on many different platforms. You could have web server plugins, app servers, whatever. These components would do the guaranteed delivery, durable subscriptions, routing to N subscribers of a topic, etc. and the service impls would be able to be writen in any language. And then we'd have event driven architecture on the web vs. just in one off messaging systems (e.g., JMS).

HTTP/POX XML Based EDA / Pub Sub ??

I googled around a bit.

Found mod_pubsub, but it didn't look terribly active (WIKI down). Anyone worked with that?

Also there is JEP-0060: Publish-Subscribe. I just glanced at it - looked like it manages the subscriptions via XML (non-SOAP) messages. I guess that would work too, but assume that is basically the WS-* sans the SOAP / cruft.

I need to catch up with what is already out there.

It would be great if there was a decent / simple spec. for this and there were either web server plugins, app servers, etc. for many platforms. For all I know, I have been drinking the JMS Kool-Aid & muttering things about WS-* (what I considered the somewhat available alternative) for too long and there is a viable solution to HTTP/POX XML Based EDA Pub Sub today.

Anybody have any experience using HTTP / POX XML (be it via HTTP Headers or something like JEP-0060) for EDA / Pub Sub?

27-MAR-06 Update, found this: The Goal A Jabber manifesto of sorts

Started a thread over at Jive - figure they might have some thoughts. Jive Thread

04-APR-06 Update, found this on the Yahoo SOA Mailing list: Reach Reliable Messaging Transport Protocol (RRMTP/1.0)

HTTP Extensions

Don Box mentioned HTTP Extensions I think that this would have had a much better chance of actually being used by people that have complex integration problems to solve then the mess we now find ourselves in with WS-*.

For one, there is no SOAP or XSD. Chance of success just went up 10x because of that alone. Second, all the header cruft would not be in the XML document. Developers wouldn't be impaling themselves with generated code, schema problems, parser problems, etc.

Those headers would be *much* easier to parse (one would assume). Much less chance to cut ourselves on interop.

I would assume that some faction of REST/POX will eventually want to do something like this. In a previous post on HTTP, XML, REST, and $100, I mentioned event driven architecture (EDA). I don't see that happening without either this or the equivalent of what the WS-* world has (I saw on Tim Bray's blog EDA is being revisited in WS-* land). Including the headers in the XML document is what makes WS-* awful IMHO. With POX + HTTP Extensions (or something like it), the service developer would have the power of owning the XML contract outright (just has to be a well formed XML doc). You could add complexity as you went (via HTTP extensions), and the XML wouldn't change. Setting/Reading the headers could be set via configuration in relatively simple (compared to WS-*) frameworks. Seems like appropriate separation of concerns to me.

Obviously, web servers would have to evolve (plugins), but this path actually seems plausible - I just don't see WS-* resulting in a workable integration solution.

What's the catch?

HTTP, XML, REST, and $100

In response to: Don Box's Spoutlet: HTTP, XML, REST, and $100 I’d target event driven architecture & or pub/sub using HTTP,XML,REST. Not sure how good Microsoft MQ is these days , but last time I checked it was pretty weak in the pub/sub area compared to good JMS impls. This has been my biggest hang up with the state of the art of web services. I think that currently, you really can’t build an event driven architecture with web services. And most of the WS-* specs have an extreme P2P bias which in a large integration will just collapse like it always has (pointed haired n(n-1) problem). Yeah there is WS-Notification, but are there really any decent impls? Do the interoperate? I don’t want all that cruft in my events anyway. Not sure if you are already have something like this I have been away from Microsoft projects for some time. I have a hard time believing that event driven architecture can even work over HTTP (yeah you could have a service or too that work that way, but I wouldn’t bet my business on all of them working that way). My general background … I am a B player Microsoft developer (at best). I am conversant in .NET because I went to a training at Microsoft for a week and I did a couple C# projects with web services and messaging. My experience with WS-* was very unpleasant. I worked with CORBA, DCOM, and RMI before that and they were almost easier to make interoperate. I am a little dated though. I gave up on web services in late 2003 and haven’t really looked back. I do follow the arguments though and find myself in the Lo-Rest/POX corner. I make my living integrating large financial systems in an event driven architecture/Lo-Rest that is Java based. I read most of some write up on Indigo on a plane a couple weeks ago. Like I was with .NET, I was impressed. I said to myself, “This is about as good as anyone is going to make WS-* -- it is a pity that WS-* is so fatally flawed that it doesn’t really matter.” I was confronted though knowing that Microsoft would salvage something from it ;)