The Programmable Web is getting more complex

OSCON is a conference which has various amorphous tracks and most of those relate to the "trends" of tomorrow. Some of the people whom I met were complaining about the lack of Java's presence in the conference. Well, as Tim Bray says, cool kids are not using Java these days. Personally I was a bit disappointed with the lack of presence of the Scala community in OSCON. In fact, except in one of Tim Bray's keynotes, Scala, I guess, never figured in the radar of this year's conference. There were tutorials and BOFs on actor models, where also Scala got only a passing reference. But I digress ..

Web Architecture is getting more complicated!

There is no doubt about the fact that we are looking at more and more options to structure services (and data) delivered over the Web. While we are all huge fans of RESTful APIs delivering the goods at optimal payloads for many of today's Web 2.0 sites, things may start changing with the scale of data that we are witnessing today as a result of frivolous cross pollination of data streams. Inundating message streams, flooding of tweets, flickr uploads and continuous exchange of other social objects over the Web have started challenging the inherent polling model for some of the usecases that REST based architecture employs.

In the session titled Beyond REST? Building Data Services with XMPP PubSub, Rabble and Kellan made a great presentation of how FriendFeed polled Flickr 2.9 million times on one day to check on updates for 45 thousand users, of whom only 6.7 thousand were logged in at any one time. With Flickr publishing feeds as the only way to advertise presence, Friendfeed had no other alternative but to "poll". Kellan called this "Polling sucks!". The alternative is asynchronous message passing with shared nothing architectures and non-blocking event loop based processing. Then Kellan and Rabble goes on to present how the pub-sub model over XMPP/Jabber can be hijacked to implement Web scale message passing and thereby getting rid of the polling nightmare. When we are talking XMPP, we are talking XML payloads, and this makes perfect sense when you are operating with undefined (or loosely defined) endpoints, high latency and low bandwidth systems. Again a good mix for all the social mashups out there ..

Take a hard look at your usecase ..

While polling sucks for the FriendFeed-Flickr usecase that Rabble and Kellan discussed, polling has its legitimate and rational usecases too. RSS and Atom feeds are ubiquitous today, while very few clients support XMPP. After all, when I have a huge number of clients interested in my feed, it makes perfect sense to go for the polling model for all my clients. Google Reader has been doing this day in and day out.

In another related presentation Open Source XMPP for Cloud Services, Matt Tucker of Jive Software, also discussed the virtues of XMPP based message passing paradigm for handling complex cloud services. While comparing alternatives like SOAP, he talked about performance optimizations that XMPP based services offer through long lived persistent connections as opposed to overhead of establishing encryption and repeated authentication in polling based systems. Well, I am not sure, what impact will there be on the system through 1 million persistent connections - I would love to get some ground realities from people who are actually using it. Matt demonstrated how easily an XMPP based system can be implemented using open source OpenFire XMPP server and a host of client APIs available for a multitude of languages.

But that's really one part of the story, that, at best, adds another dimension to our thought process when designing services for the Web. You need to define what scalability and performance optimizations mean for you. What Google can achieve through Bigtable and MapReduce makes no sense for Twitter. DareObasanjo has a great piece on this subject of ubiquitous scalability.

Whether it's 100% RESTful or not is purely driven by requirements ..

Architecture changes when you have defined end points, low latency and high bandwidth networks to play with. You can afford to enforce constraints over your end points and have binary payloads like Protocol Buffer or Thrift that need a specific subset of programming languages and runtimes. Google and Facebook have been using them with great success. When you need to process lots of data and perform data intensive computations, it makes sense to use the binary wire format instead of XML.

So options start getting multiplied as to whether you need to use polling based RESTful Web services, or message oriented XMPP services or the more classical distributed computing model of using RPC through binary serialization. OSCON 2008 had quite a few sessions that discussed many of these cloud computing infrastructure components - the key message was, of course, one size doesn't fit all. And the Web architecture is getting complicated by the day.

OSCON 2008 (Day 1 and 2) - Great Tutorial Sessions

It has been a typical bootstrapping for the first 2 days at OSCON 2008. The tutorial sessions have started as per schedule on the usual low-key note, the irc channels yet to set up the buzz tone and all the pubs yet to get heated up with the latest of the conference musings. For the early comers, this is the time to start getting soaked into the spirit of the conference and build up the setting for the next 3 days of fun, frolic and frenzy.

I had registered for 4 tutorial sessions.

OSCON day 1 for me began with the early registration, the usual conference style breakfast and then a jump into the "Python in 3 hours" tutorial. This session was perfect in setting up the tone for Python non-programmers. The initial part of the tutorial was quite slow paced and focused much more on the basics of programming than what the time demanded. As a result, the tutorial faced lots of time pressure towards the end - hence we missed out on the more Pythonesque stuff that I had expected to see as part of this tutorial. Otherwise, it was managed well, Steve Holden is a nice speaker and had all things under control for tutoring on the basics of a language. It's not always that easy to control the entropy of an audience so varied in profile and skill level while teaching on the basics of a programming language.

The second pick for me was "Memcached and MySQL: Everything You Need to Know", which ended up being a great one for me. Both the speakers had been working on memcached codebase for quite some time, I guess. Hence lots of implementation issues were discussed and sorted out with expected seamlessness. In fact after the tutorial I heard some people complaining about the too-much-implementation-level-detail orientation of the session. I liked it though and it gave pointers to many of the memcached internals viz. memory allocation (slab allocators), consistent hashing (the secret sauce of memcached) and lots of tuning and stats tips.

OSCON day 2 began with the tutorial on "Ubiquitous Multithreading for a Multicore World", where the Intel guys took us through the pains of managing shared state concurrency. They introduced Threading Building Blocks (TBB) that offer parallelism within the C++ programming model. TBB offers a higher level task based parallelism that abstracts platform dependencies and threading implementations away from the programmer. TBB is offered as a library with GPLv2 licensing and comes bundled with lots of C++ compilers today. Being a higher level of abstraction, TBB will make concurrent programming easier in C++ than what it is today. But given the clumsiness and verbosity of C++ syntax, and the emergence of newer languages and concurrency paradigms, I have strong doubts on its acceptability to mainstream development community. One other interesting stuff that the Intel guys discussed was the lambda expression implementation in Intel C++ compiler, also part of the official C++ specification coming soon to an implementation near you. A lambda expression in C++ will have to be implemented as an object, which can contain references to local and global variables of the enclosing context. C++, not being a garbage collected language, needs to implement a copy-semantics for all such context elements. It is not yet clear to me how non-copyable objects will be handled in such cases .. need to look up more for implementation details. Overall the tutorial session was informative and it was good to see C++ getting a facelift through some positive movement forward.

The second tutorial that I attended was "An Introduction to Actors for Performance, Scalability and Resilience", conducted by Steven Parkes. The session began with the discussion of the actor model as implemented in Erlang. Steven was thorough in his exercise of designing real life problems modeled with actor based concurrency, implemented first using Erlang and then in Ruby using Dramatis actor library. A straight mapping of actors to native threads does not scale - Dramatis implementation also does not usually dedicate a thread to an actor. The receive is implicit, much like Erlang/OTP's gen_server, though it was not clear from the discussion regarding the exact payload of the actor model implementation in dynamic OO languages like Ruby and Python. Steven also mentioned that there is nothing implicitly functional about actors and that he chose to implement the imperative way with a nice OO scaffolding around the actor interface. Scala also provides a similar interface of objects with event based implementation - it will be a very interesting exercise to benchmark how much all the actor implementations scale under heavy concurrent throttling. The Erlang implementation still stands out tall and large as far as the seamless distribution and fault tolerance of the actor model is concerned. None of the other implementations can be made to distribute as easily as the Erlang model.

My Picks for OSCON 2008

Yay! I will be in OSCON 2008. Along with the growth of the open source ecosystem, OSCON has also grown up to the diversities of multitude of technologies. And this year, they are celebrating the tenth anniversary of promoting open source.

I will be in Portland from 21st till 25th of July as a part of this celebration. I will be attending 4 tutorials and a number of regular sessions. Each of them will be great inductions to some of the newer technologies that I have been planning for quite some time to get into. However, I will be focusing on 2 primary areas as my main takeaways from this conference ..

Scaling out with large datasets

Indoctrinated in the basics of relational databases, normalization and joins, I find that almost all of the Web 2.0 scalability stories are taking alternate routes. Have a look at Google BigTable, Amazon SimpleDB and Facebook Cassandra - all of them have got around the scalability problems of normalized relational databases through denormalization and usage based data models, replication among multiple nodes and delayed eventual consistency. Processing large tables with SQL based queries are being replaced by map-reduce jobs that operate on tuple models and enforce data integrity through code rather than database constraints.

OSCON 2008 has a number of sessions that deal with this topic of scalability in the cloud ..

1. Cloud Computing with bigdata


2. CouchDB from 10,000 ft


3. Processing Large Data with Hadoop and EC2


4. HDFS Under the Hood


5. Cloud Computing with Persistent Data: Pushing the Envelope of Amazon Web Services


6. Scale into the Cloud with Open Source

Asynchronous Messaging as the enabler of enterprise architectures

Today's common wisdom is that asynchronous messaging enables loose coupling in distributed architectures. And this has led to the recent popularity of the actor model of programming which relies on message-send and receive as the basic constructs of the programming model. Erlang has been encouraging this model for years, though only recently has started getting the mindshare and marketshare of the mainstream. Scala also offers great support for actor model of programming that makes your application scale so easily.

Along with the growing popularity of message queueing frameworks, we are looking at recent implementations of the Extensible Messaging and Presence Protocol (XMPP) and the Advanced Message Queueing Protocol (AMQP) that offer multiple messaging solutions to connect across distributed systems.

OSCON 2008 has multiple sessions and BOFs that discuss this paradigm at length ..

1. Beyond REST? Building Data Services with XMPP PubSub


2. Beautiful Concurrency with Erlang


3. XMPP/Open Source Components for Cloud Services

And this is just the beginning of the fun. There will be a number of events, parties, fun and frolic centered around open source innovation. I am really excited to be a party to it.

See you all in Portland !

Scaling out messaging applications with Scala Actors and AMQP

We have been sandboxing various alternatives towards scaling out Java/JMS based services that we implemented for our clients quite some time back. The services that form the stack comprise of dispatchers and routers meant to handle heavy load and perform heavy duty processing on a huge number of trade messages streaming in from the front and middle offices. I have been exploring lots of options including some of the standard ones like grid based distribution of processing and some wildly insane options like using Erlang/OTP. I knew Erlang/OTP is a difficult sell to a client, though we got some amazing results using OTP and mnesia over a clustered intranet. I was also looking at clustered Scala actors as another option, but I guess the Terracotta implementation is not yet production ready and would be a more difficult sell to clients.

Over the last week, I happened to have a look at RabbitMQ. It is based on Erlang/OTP and implements AMQP, a protocol, built on open standards, designed by the financial industry to replace their existing proprietary message queues. AMQP based implementations create full functional interoperability between conforming clients and messaging middleware servers (also called "brokers").

People have been talking about messaging as the front-runners in implementing enterprise architectures. And RabbitMQ is based on the best of the implementations, that, messaging over a set of clustered nodes, has to offer. Erlang's shared-nothing process hierarchies, extreme pattern matching capabilities and wonderful bit comprehension based binary data handling implement high performance reliable messaging with almost obscene scalability. And with RabbitMQ, the wonderful part is that you can hide your Erlang machine completely from your application programmers, who can still write Java classes to blast bits across the wire using typical message queueing programming paradigms.

AMQP (and hence RabbitMQ) defines the set of messaging capabilities as a set of components that route and store messages and a wire level protocol that defines the interaction between the client and the messaging services. The application has to define the producers and consumers for the messaging engines. And in order to ensure linear scalability, these components also need to be scalable enough to feed the Erlang exchange with enough bits to chew on and harness the full power of OTP.

Linear scalability on the JVM .. enter Scala actors ..

Over the weekend I had great fun with Scala actors churning out messages over AMQP endpoints, serializing trade objects at one end, and dispatching them to the subscribers at the other end for doing necessary trade enrichment calculations. Here are some snippets (simplified for brevity) of the quick and dirty prototype that I cooked up. Incidentally Lift contains actor-style APIs that allows you to send and receive messages from an AMQP broker, and the following prototype uses the same interfaces ..

The class TradeDispatcher is a Scala actor that listens as an AMQP message endpoint. It manages a list of subscribers to the trade message and also sends AMQP messages coming in to the queue/exchange to the list of observers.

// message for adding observers
case class AddListener(a: Actor)

// The trade object that needs to be serialized
@serializable
case class Trade(ref: String, security: String, var value: Int)

case class TradeMessage(message: Trade)

// The dispatcher that listens over the AMQP message endpoint
class TradeDispatcher(cf: ConnectionFactory, host: String, port: Int)
    extends Actor {

  val conn = cf.newConnection(host, port)
  val channel = conn.createChannel()
  val ticket = channel.accessRequest("/data")

  // set up exchange and queue
  channel.exchangeDeclare(ticket, "mult", "direct")
  channel.queueDeclare(ticket, "mult_queue")
  channel.queueBind(ticket, "mult_queue", "mult", "routeroute")

  // register consumer
  channel.basicConsume(ticket, "mult_queue", false, new TradeValueCalculator(channel, this))

  def act = loop(Nil)

  def loop(as: List[Actor]) {
    react {
    case AddListener(a) => loop(a :: as)
    case msg@TradeMessage(t) => as.foreach(_ ! msg); loop(as)
    case _ => loop(as)
    }
  }
}


and here is an actor that gets messages from upstream and publishes them to the AMQP exchange ..

class TradeMessageGenerator(cf: ConnectionFactory, host: String,
         port: Int, exchange: String, routingKey: String) extends Actor {

  val conn = cf.newConnection(host, port)
  val channel = conn.createChannel()
  val ticket = channel.accessRequest("/data")

  def send(msg: Trade) {

    val bytes = new ByteArrayOutputStream
    val store = new ObjectOutputStream(bytes)
    store.writeObject(msg)
    store.close

    // publish to exchange
    channel.basicPublish(ticket, exchange, routingKey, null, bytes.toByteArray)
  }

  def act = loop

  def loop {
    react {
      case TradeMessage(msg: Trade) => send(msg); loop
    }
  }
}


The next step is to design the consumer that reads from the exchange and does some business processing. Here the consumer (TradeValueCalculator) does valuation of the trade and has already been registered with the dispatcher above. Then it passes the message back to the dispatcher for relaying to the interested observers. Note that the TradeDispatcher has already passed itself as the actor while registering the object TradeValueCalculator as consumer callback in the snippet above (class TradeDispatcher).

class TradeValueCalculator(channel: Channel, a: Actor)
    extends DefaultConsumer(channel) {

  override def handleDelivery(tag: String, env: Envelope,
               props: AMQP.BasicProperties, body: Array[byte]) {

    val routingKey = env.getRoutingKey
    val contentType = props.contentType
    val deliveryTag = env.getDeliveryTag
    val in = new ObjectInputStream(new ByteArrayInputStream(body))

    // deserialize
    var t = in.readObject.asInstanceOf[Trade]

    // invoke business processing logic
    t.value = computeTradeValue(...)

    // send back to dispatcher for further relay to
    // interested observers
    a ! TradeMessage(t)

    channel.basicAck(deliveryTag, false)
  }
}


I have not yet done any serious benchmarking. But the implementation, on its face, looks wicked cool. Erlang at the backend, for high performance reliable messaging being throttled out by Scala actors in the application layer. Every actor opens up a new channel - the channel-per-thread model that AMQP encourages for multi-threaded client applications and scales so well in RabbitMQ.

Integrating the above classes into a small application prototype is not that difficult. Here is a small service class that uses the above framework classes to have the scala actors flying to talk to RabbitMQ ..

class SampleTradeListener {
  val params = new ConnectionParameters
  params.setUsername("guest")
  params.setPassword("guest")
  params.setVirtualHost("/")
  params.setRequestedHeartbeat(0)

  val factory = new ConnectionFactory(params)
  val amqp = new TradeDispatcher(factory, "localhost", 5672)
  amqp.start

  class TradeListener extends Actor {
    def act = {
      react {
      case msg@TradeMessage(contents: Trade) =>
        println("received trade: " + msg.message); act
      }
    }
  }
  val tradeListener = new TradeListener()
  tradeListener.start
  amqp ! AddListener(tradeListener)
}


Instantiate the above SampleTradeListener class and write a sample message generator facade that sends trdae messages to the consumer TradeMessageGenerator designed above.

Meanwhile here are some other related thoughts behind an AMQP based architecture ..

• One of the areas which always makes me sceptical about linear scalability of applications is the single point of dependency on the relational database store.


• If the application involves heavy relational database processing, does it make sense to make use of mnesia's easy distribution and fault tolerance capabilities to achieve overall scalability of the application ? Harness the power of durable and reliable transacted message processing of RabbitMQ and integrate RDBMS storage through write-behind log of AMQP activities.


• Financial services solutions like back office systems typically need to talk to lots of external systems like Clearing Corporations, Stock Exchanges, Custodians etc. I think AMQP is more meaningful when you have the end points under your control and operate over a low latency, high bandwidth wire. When we talk about messaging over the internet (high latency, low bandwidth), possibly XMPP or Atompub is a better option. RabbitMQ has also released an XMPP gateway, for exposing a RabbitMQ instance to the global XMPP network through an ejabberd extension module. Looks like it's going to be messaging all the way down and up.

Erlang's concurrency model on the JVM - Can we have (at least a subset of) OTP in Scala ?

Jonas Boner is working towards an OTP implementation in Scala. Definitely sounds to be a great exercise and may well prove to be a viable and scalable implementation on the JVM. We are seeing more and more implementations of highly scalable systems using Erlang/OTP - Facebook, ejabberd, CouchDB, Mochi* to name a few. And not without a reason. Erlang/OTP provides an awesome stack that fits the scalability-reliability-distribution space like a glove.

Scala offers shared-nothing asynchronous message passing paradigm in its programming model, very much similar to Erlang. As a language, however, Scala is quite different - statically typed, functional-OO hybrid with type inferencing and a lot more core features than Erlang. The beauty of Erlang is in its small simplicity, often looked at as being syntactically weird. But the amount of research that has gone into the concurrency and distribution model which Erlang implements is truly truly phenomenal. Here are some thoughts that come to my mind when thinking about Scala's contribution and possibilities in this space ..

Scala actors are very similar to Erlang processes as abstractions of composability. Erlang/OTP offers gen_server, gen_fsm, gen_event etc. along with robust fault tolerant hierarchical supervision of processes across nodes and clusters as combination of generic servers and pluggable callbacks. More than the language, OTP offers the platform on which Erlang processes can play with gay abandon. And the callbacks that the client need to implement can be absolutely oblivious of concurrency, process spawning, failover and clustering issues - they are written as purely sequential functions that can be easily hot swapped in and out of live installations. Can we have the same reliability of implementation in Scala ? I don't know, but even if we can have a meaningful proper subset, then it will be enough to safeguard most of the corporate investments that have been drained on to the Java Virtual Machine.

Scala needs a port of mnesia, Erlang's distributed, fault tolerant database that offers fast lookup, dynamic reconfiguration capabilities, Erlang data types all the way down (implying zero impedance mismatch) and seamless distribution and partitioning semantics. qlc, the data query language based on Erlang's list comprehension syntax is the DSL that makes Erlang almost a database programming language. For Scala, it should not be very difficult to come up with something similar to qlc, built upon its for-comprehensions. mnesia is lightweight, can be easily replicated and partitioned. And with today's trend of traditional relational databases starting to get a beating from map-reduce jobs, tuple models, and tablestores that can't even do joins, a lightweight, loosely coupled, easily replicated engine like mnesia can be a very good starting point towards persistence as a service paradigm. Memory replicated mnesia data store can be backed up with transparent persistence services offered by today's grid platforms.

Erlang offers utterly immutable variables, and Erlang's reliability as a platform is almost a mathematical corollary of this theorem. In Scala you can make things immutable, but the VM does not enforce it. With the JVM, leave immutability to the creativity of your programmers. Not all pure stuff gets blessed by the masses, we have seen this with Smalltalk and C++. And here, we can hope for the best.

People often boast of the abundance of Java libraries as the shining part of Scala's ecosystem. With respect to implementation of the actor model and the OTP paradigm, this is where Scala comes down hard. All Java libraries are baked with imperative mutable structures that will land you in the blues of synchronization and locks, that you have been trying to get away from, with the shared nothing process model. And for the OTP implementation, almost all of these Java libraries will be useless.

It is quite easy to get distribution across nodes/clusters with Erlang/OTP applications. They rely upon mnesia's serialization of tuples. Scala can be more efficient in this space using Terracotta's selective differential serialization techniques. Scala actors can be clustered using Terracotta and this can be one area where Erlang capabilities may pale out to Scala and JVM power.

In my last post, I was wondering about distribution concerns that need to be considered while designing APIs in Erlang. But if you use OTP, many of the concerns get addressed by the platform. As a client you are left with implementing callbacks that can be plugged in and out. Scala's actor model has many of the promises that Erlang offers. And as I mentioned above, if it can implement at least part of what OTP does today, Scala can play a *good enough* coup over Erlang that C++ played over Smalltalk.

What else you need to consider for designing well-behaved Erlang APIs

When you design a framework or a library in an implementation language, you need to play to the rules of the game. While implementing a library for a functional language, you need to design functions that are referentially transparent, side-effect-free and easily composable with the existing platform. Similarly for an object oriented implementation, there are well-published idioms and design patterns (factories, builders etc.) that the language espouses, which your contracts need to honor. e.g. designing a module in a statically typed language like Java, you need to design abstractions based on interfaces that can be easily injected using dependency injection. While DI is not a particularly forceful consideration for implementing the same functionality in Ruby that offers classes as soft abstractions that can be gutted out very easily at runtime.

The more succinct a language is, the more subtle it is to design usable APIs, conforming to the best practices of the language. And it becomes increasingly difficult when the language starts offering more and more intellectual surface area along multiple orthogonal axes of variability. You need to think in terms of adapting your APIs to all the axes and try to make them extensible across all of them.

Erlang is one language that makes you think in at least one additional dimension while designing new functionalities. Being a functional language, dynamically typed, you need to think of all the usual stuff (as mentioned above) to come up with an extensible design of your module. Like other functional languages, functions are the basic units of abstraction in Erlang. But there is also a dynamic view of how functions live within processes in Erlang. Functions are static entities that get their lives when passed around in graphs of linked processes spawned by the Erlang virtual machine. Just as in an OO language like Java, you have objects as the means to encapsulate state and identity of the abstraction, in Erlang you design processes that provide encapsulation of states through published message interfaces. And processes can run anywhere - in your local machine, in another virtual node in the same machine, in another machine on the same subnet / domain, or in some place else anywhere within the internet.

When you design APIs in Erlang, you need to think of distribution and concurrency as well. This is because transparency of distribution is baked into the language itself. And your module needs to be compliant with all the characteristics of a distributed ecosystem.

Designing APIs in Erlang needs you to make the additional consideration for transparent distribution.

I was going through this thread in the Erlang mailing list over the last weekend. The thread discusses alternatives for designing a mocking framework in Erlang to improve testability of Erlang programs.

In Erlang the basic unit of execution is a process, which is supposed to be an ultra-lightweight abstraction that can be spawned in millions on your commodity hardware. A typical program uses spawning of processes as ..

Pid = spawn_link(fun banana:loop/0).


Erlang supports hot code swapping, whereby you can change your code and immediately test impact of your code changes, without recompilation and without a millisecond of server downtime. The running process receives new messages on the fly, loads the new code and manages multiple versions simultaneously - a feature of the Erlang virtual machine that is fairly unique, and will remain so, till we see the OSGi stuff maturing into a comparable offering.

But the problem with the above is that the function name is hardcoded while spawning the process. Here the function name acts as the interface that scaffolds native Erlang message processing within it. Hot swapping of code is possible by sending new messages to processes and the VM will allow dynamic loading of the new code to perform the new task. But the interface module hiding the message processing makes it impossible to change the function name and replace it with a mock.

One approach discussed is to purge the existing real banana module and replace it with the mock using code:load_abs/1. However that will not allow a generic enough mock and will have to be another banana. And the approach will suffer from the downside that tests can't be run safely in parallel with the real module in the same node, since module namespace is node-wide. Finally Christian suggested implementing a module having an interface similar to erlang:error_handler and redefined undefined_function and undefined_lambda to set the mock module into process dictionary and load it on error (similar to method_missing of Ruby). Have a look at the thread for the gory details ..

Anyway, this post is not about hot code swapping in Erlang. The interesting bit about the above mentioned thread is how distribution-transparency features as a concern in a discussion that apparently does not seem to be related to mocking abstractions to improve testability.

In languages like Java, distribution is a totally extraneous concern to the language features and is addressed as a separate deployment concern using grid computing frameworks. In some respect, the POJO model of Java development shines in the fact that you need to consider only the domain problem at hand viz. mocking, in this case. Additional frameworks take care of distribution and concurrency issues - a good example of separation of concerns. But, at the same time, introducing new frameworks has its own downsides, the impedance mismatch that it brings and makes distribution concerns look like bolted from outside. With Erlang, it is a unified thought process, since the language is designed for distribution and reliability, and with a little thinking ahead, you can design robust, distributable frameworks *only* with the help of the native language features and libraries.

Still thinking, which one is better ..