Steve Vinoski's Blog

Speaking of Reg Braithwaite, in this post he touches on team size and productivity. There are so many gems there, I could comment on just about every sentence, but I think this is my favorite:

If we know that bug per line of code remains amazingly constant, why do we try to scale code out in verbosity rather than up in abstraction?

I’ve very often wondered the same thing.

When I’ve given talks about dynamic languages and productivity, I’ve sometimes brought up the “bugs per line of code” constant. With dynamic languages, I generally write far fewer lines to solve the problem than I would in Java or C++. While that alone doesn’t guarantee correctness, of course, in my experience it sure seems to lead to solutions that are completed faster and with fewer bugs. Yet in those presentations, I’ve always gotten pushback on that idea from someone in the audience, typically based on the “static typing will save you” argument, which I’ve never fully understood despite having written, read, and fixed tons and tons of statically-typed code over the course of my career. First, the notion that you’re generally likely to have more bugs if you have more lines of code just seems like common sense, since there are just more chances to be wrong. Second, static typing does nothing to help ensure that you’re applying suitable algorithms and are correctly solving the problem, yet that’s where the bugs tend to be.

But it goes beyond bug counts — sometimes it even comes down to getting a solution vs. not getting one at all. The fact that you can quickly visualize and produce a solution in Python or Ruby, for example, means you’re a lot more willing to actually sit down and crank out that solution in just a few days, rather than agree to take on a multi-month multi-person task to try to drive that nail with a Java or C++ hammer. Not to abuse another cliché, but I truly have been there and done that, believe me.

Postings like these often seem to attract comments of the form, “Well, dynamic languages are just too dangerous; we have several hundred programmers and we can’t watch everything that every one of them does.” I think that’s precisely Reg’s point! But this also gets to another related issue, which is that if you build teams that you can’t or won’t trust, you’re simply creating a self-fulfilling prophecy. You’re essentially expecting your teams to fail, so they will. In that type of environment, you’re just not going to be able to attract or keep anyone that you actually can trust. Furthermore, if you’re that type of manager or leader, not only is the likelihood that you’ll be able to scale your organization quite low, but you have nobody but yourself to blame for the eventual failures you’re sure to encounter. Issues related to dynamic languages and static typing are the least of your worries. Rather than tying your developers’ hands, it’s far better to devote continual effort to instilling common values and culture across the organization so that your teams can work toward common goals without requiring constant centralized oversight. Read Peter Senge’s The Fifth Discipline for great insight into this area.

For me, dynamic languages mean increased productivity. It’s as simple as that. It’s because the programming languages I use actually can do this, and because succinctness is power. And if you’ve never actually tried developing real software in such languages, that’s too bad, because that power — that productivity — really is quite amazing.

One of my favorite bloggers is Reg Braithwaite. He writes extremely well, and every one of his posts, even when he posts only links, is worth reading.

Judging from this comment Reg made on another blog, it seems that he’s being accused of being an absolutist, in this case regarding static typing. In that comment he writes:

I am going to challenge you now. I am not trying to provoke a fight, I honesty want YOU to go away and say hmmm. Why is it that you are trying to paint me as such an absolutist, that when I say “I like static typing but I’m currently achieving my personal goals with a dynamic language” that I must be against static typing?

and he summarizes his comment with this:

But I would encourage you to consider the possibility that others, such as myself, are not as rigid in their mindset. I’m very proud of the work I’ve done using some extreme “type torture” in C++ to impose constraints on programs. I’m also very proud of some of the work I’ve done in Ruby. I have no difficulty embracing each in its own way.

After all, one roadhouse can host both kinds of music, Country AND Western.

Reg’s comments really struck a chord with me; not only do I completely understand where he’s coming from, but I especially like the Blues Brothers reference as well. :-) Seems like anytime I post about my preferences for REST and dynamic languages, I receive a number of blog comments that are nothing more than unfriendly flaming rants accusing me of absolutism. It kinda makes me laugh, given that I use 3, 4, or 5 different programming languages almost every day, and can develop a wide variety of systems, especially distributed systems, using a fairly broad range of different techniques. Simply put, I’ve been in this career for way too long to be an absolutist, except perhaps for one thing: I absolutely believe every developer should be strong in multiple programming languages and multiple approaches because there’s no single language or technique that can do it all.

Actually, now that I think about it, here’s a couple more absolutes:

• I moderate absolutely all blog comments.
• I’m happy to post all reasonable comments that actually make a useful point, regardless of whether the writer agrees or disagrees with me. But if you try to post a rant, or try to turn a technical discussion into a pointless argument about absolutes, I can absolutely guarantee you that your comment won’t appear here.

Sergey Beryozkin, a former coworker at IONA, has posted some questions specifically for me. Questions are good; it’s almost like being interviewed, and it’s certainly better than being called sad and depressing! Let’s see if I can provide some answers.

But first, I want to address something I keep hearing when I post things like the answers below: “Steve, you’ve changed your whole story ever since you left IONA!” While the technical opinions I’ve been expressing in this new blog might seem shocking since they seem different from what I used to say, they’ve actually been under development for a long time. What actually happened is

1. My technical opinions changed gradually over the course of 5 or 6 years while I was still at IONA.
2. I finally chose to leave because my preferred technical direction had diverged significantly from where IONA was going.
3. I left when I did because of a wonderful opportunity that came up at a new place where I could put all my latest ideas to work. So far, it’s all I hoped it would be, and more.
4. I couldn’t publicly blog about my new opinions, write about them in my column, or present them at conferences until after I had changed jobs, for obvious reasons.

If you had enough time and patience to read through all my IC columns, you’d certainly find plenty of evidence that my thinking about all this stuff had already been changing for years prior to my departure from IONA. I hope this clears things up for those of you who mistakenly think I’ve just done an abrupt about-face.

Now, on to Sergey’s questions:

1. Do you think client code generation is evil ? If yes, do you expect people to do manual programming on a large scale ?

In days long gone by, code generation clearly helped us get a grip on certain types of distributed systems, and make advances as a result. But since then we’ve learned that you can’t pretend a distributed system is a local one, that RPC is less than desirable, and all kinds of other lessons.

All in all, I am now of the opinion that it’s generally best to avoid distributed system development based on language-first approaches (unless it’s a language like Erlang where distribution is effectively built in, but even then you still can’t ignore distribution issues). Code generation is based on the notion that I want to develop my distributed system using the idioms and practices of my programming language, which I now believe is almost always wrong. If you avoid specialized interfaces and use a uniform interface instead, it significantly reduces the need for code generation. On top of that, using standard MIME types for exchanged data rather than making up your own data types in WSDL or Java pretty much eliminates the need for code generation in many cases, IMO.

Now, I have to say that your question seems to imply that code generation is required for large-scale development. I don’t see why that would be the case. “Manual” programming isn’t much of a chore when you choose the right language.

People seem to get really upset when I say that the static typing benefits of popular imperative languages are greatly exaggerated, and when I say that developing real, working systems in dynamic languages is not only possible, it’s preferable. I usually find that, coincidentally, those are the same people who’ve never honestly tried the alternatives. Some who actually do try the alternatives do so by trying to use the new language exactly the same way they use their favorite language, and when they fail for what should be obvious reasons, they blame the new language.

2. If code generation is acceptable, would you welcome WADL? If yes, what to do with generated client types with respect to the change management ?

I’ve personally never encountered a need for WADL, but that’s just my opinion and experience. Marc Hadley and others obviously find it useful, and the RESTful Web Services book promotes it, and they’re all smart folks, so maybe there’s something to it.

Either way, no interface definition language is ever going to keep you or some other real live person from having to figure out what the service actually does and how to actually use it, and then coding your client accordingly. Normally you figure out that sort of thing by reading some sort of human-readable document, not just by looking at WSDL, WADL, or any other IDL. So if you have to read a document, and you’re avoiding code generation thanks to the uniform interface and the use of standard data formats, why bother with any IDL?

3. Do you think the idea of interfaces is broken ? Do you see any point in creating minimalistic yet not generic interfaces with encouraging users to focus on data ?

I believe that it’s important to avoid specialized interfaces whenever possible and prefer a uniform interface instead. Interface specialization, even of the minimal variety you mention, inhibits reuse. My Jan/Feb ’08 IC column, which is not yet published but is already written, covers this in detail.

4. Would you expect in the future even software modules interacting with each other through a common generic interfaces ?

I assume you’re referring to modules residing together within a single address space. To some extent that’s already been happening for years, thanks to frameworks, which of course are also based on interface uniformity. But will we ever get to the point in the foreseeable future where all entities residing within a single address all have the same generic interface? No.

Remember, REST is an example of applying well-chosen constraints to achieve desired architectural properties for a broad class of distributed systems, and so that’s what its constraints are all about. It might be an interesting brain game to consider what properties REST’s constraints could induce within a non-distributed system, but I’m not sure the exercise would be of any practical benefit.

5. “WS-* was simply not worth it to any customer or to me” – was it not ?

The customers that I saw benefit from WS-related technology gained those benefits only because of IONA-specific innovations that were not part of WS-*. In general, WS-* didn’t do anything for them that wasn’t already possible with prior technologies.

6. Do you think WS-Policy is a useless technology ?

I personally have no use for it.

7. Do you think AtomPub is the best way to do REST ? Is AtomPub better than SOAP ?

AtomPub fits a particular class of problems, but it targets only a subset of what REST can be applied to, so it doesn’t solve everything (and neither does REST, of course). I don’t believe AtomPub and SOAP are directly comparable, but I personally haven’t found a use for SOAP, other than when I’m forced to talk to systems that offer only SOAP-based interfaces.

In 1999-2000 I thought SOAP was finally going to help glue the CORBA and COM worlds together and make for a happy integrated place. But as I learned more and more about REST, starting at roughly the same time, I saw the light that SOAP was missing the boat, big time, by abusing HTTP.

8. What is a better way to protect investments made into WS-* ? Throw them away and start from scratch or start embracing the WEB while improving on what can be done with WS-* ?

I’m not sure there’s a general answer to this question, as it’s not really a technical question and it depends too much on particular business situations. Personally, if my current job involved WS-*-based systems that I had any say over, I’d throw them away, but I’m in a startup and we could easily do that. Thankfully, though, we have no such systems.

9. Do you think an “integration” problem IONA has been so good at is an “overblown” problem ?

I think that in the whole scheme of things the integration problems that IONA is good at solving are not very common. Nevertheless, those problems are very real; that’s IONA’s niche, and they’re great at solving them. The very bright developers, customer engineers, and sales engineers at IONA regularly come up with wonderful well-engineered and budget-pleasing solutions when nobody else in the industry can. But if you work in that environment day after day, it’s only natural that you might start to think that all integration problems look like those uncommonly difficult ones. In the big picture, that’s just not the case.

10. Can you please, if possible, describe a bit what kind of (software) clients will use your RESTful systems, (Web)UI tools or client software modules pushing the data up the response chain ?

In general, the company I work for is in stealth mode, so there’s virtually nothing I can tell you. But I will say that when you’re building RESTful resources, you tend not to think of the browser and other client software differently — they’re both just clients. If you’re primarily doing Web development and maybe sprinkling in some support for programmatic clients for good measure, that’s different. Either way, anti-REST folks commonly claim that REST’s success is due only to the fact that there’s a human-driven browser in the mix, but that’s one of the dumbest things I’ve ever heard.

11. What is the difference between service factories found in Corba and RESTful services creating new child resources as part of POST (as far as managing child resources is concerned) ?

As far as managing child resources is concerned, the difference is that CORBA has no uniform interfaces or data representations. The CORBA client therefore has to be specifically coded to be able to manage the newly-created resource.

12. Do you always prefer dealing with fine-grained resources ?

No.

Dare Obasanjo explains that “WS-* is to REST as Theory is to Practice.” Very well stated!

Like Dare, I used to work on WS-* stuff, so I’m going to echo him and tell you why I shifted entirely away from it. In the past I definitely thought WS-* had potential for solving certain vexing enterprise integration problems, especially multi-protocol integration scenarios, but I also knew it had other problems. In addition, I was aware that that particular set of vexing integration problems, in the big picture, wasn’t very large or common. Meanwhile, REST has just appealed to me more and more over the past five years since I first wrote about it. As I learned more about REST, the size of that set of vexing problems just kept shrinking.

Finally, I realized that WS-* was simply not worth it to any customer or to me. My decision to leave WS-* behind and use only REST was based entirely on real-world commercial integration issues, not because I thought REST was cool. REST is not only the better theoretical distributed systems model by far, but also the better development model as well in terms of helping deliver working solutions within budget, so ultimately I chose to completely part ways with WS-*.

Nowadays, all the distributed systems development I do is REST-oriented. I know from significant first-hand experience what both sides of the coin look like, and there’s no question that REST-oriented systems are easier and less expensive to develop, and far less costly to extend and manage. Like Dare said, anyone who thinks otherwise is either so emotionally or monetarily attached to WS-* that they can’t be objective, or they don’t actually write any code or build or maintain any actual systems. It’s no contest, really.

What if you knew a person who clearly knew an awful lot about large-scale software systems? Someone who earned their way to their knowledge and wisdom by spending days and nights analyzing, measuring and debugging when everyone else had already given up and gone home? Someone considered the “go-to guy” who could always fix things if the system were to encounter a mysterious problem so serious that it resulted in very real revenue loss for the customer for every second of downtime? And what if you could rig up some sort of device to easily tap into the way that person thinks, thereby getting a clear understanding of the rules by which he or she designs, builds, deploys, analyzes, measures, debugs, operates, manages, and upgrades those large-scale systems?

Michael Nygard is such a person, and luckily for the rest of us, he’s already created such a device: his book, entitled Release It! Design and Deploy Production-Ready Software.

Success is a problem we’d like our systems to have. So, we focus our energies on building them as best as we can. Since we tend to focus there, books also tend target only that initial development phase. Most books focus on analysis and design issues, or on the methods and processes for doing analysis, design, implementation, and testing. Sometimes you find a book that focuses on debugging, and some even get into performance and scalability concerns.

But I’ve never seen a book like this one. It addresses the truly hard part of software development, which is running a successful large-scale system in production.

The book is a patterns book, but the patterns it presents are concrete. First, there are patterns and anti-patterns for stability and capacity, intermixed with war stories about real-life large-scale systems that failed hard for reasons that wouldn’t ever occur at smaller scales. Nygard’s war stories bring the patterns and anti-patterns into focus, providing very real reasons for their existence, and hard-won proof that they do indeed work.

Following the patterns, the second half of the book discusses general design issues and operations issues. These parts build on the patterns and focus on gotchas, large and small, that keep small systems from growing into large ones. The general design chapters provide numerous suggestions for eliminating seemingly innocuous mistakes that can kill a system as scale increases. Finally, the operations sections put you in the shoes of the people who have to keep the system running in production, describing what you as a developer can do to make their lives easier (thereby making your own life easier too). Along the way, Nygard keeps us grounded by occasionally presenting more stories and scenarios, some of which include actual dollar figures for the choices and trade-offs made. Engineering always comes down to building the best system possible within the allotted budget, but how many books targeting software developers do you know that talk in concrete terms of costs?

Nygard’s writing style is clear and concise. If you read his book and read his blog, you’ll find the styles to be identical. I’m guessing his book’s copy editor didn’t have a lot of work to do to get this book ready for production.

Bottom line: if you care at all about production software systems, you’ll want to read this book.