Choose your battles

Here’s an interesting set of posts from Shaneal Manek, the developer behind (born as  He first makes the point that LISP is a fast language for internet server development, and with some small optimization hints, can be 4x faster than Java, and 15x faster than Python.  I think that qualifies him as a free thinker.

For most projects, I’ve long been a proponent of choosing a language based on how quickly and accurately developers can get their work done and not so much on the basis of how fast the language is.  Yeah, think about rewriting for speed in another language is a fine thing to think about in version 3 or 4 well after you have something that works and delivers the goods.  While someone in a lower level language is still debugging their intermittent crashes (the long tail of development), you’ll have time to both refine your feature set and tweak the parts of the application that are the slowest.

Which brings us to post number two.  He argues that if you’re going to think about optimizing and scaling, think about the huge chunk of time your application is waiting for databases or disks.

Because of the huge disparity in CPU and disk speed, many applications are disk or I/O bound. If rewriting their application in C makes your code run twice as fast, it may only buy you an absolute performance boost of 10%. On the other hand, if you could make your database go twice as fast, it might buy you a net 30% performance boost – which is huge (and still far easier to do than rewriting your app in C).

He makes his case with supporting benchmarks.  Yes, your mileage will vary.

Speaking of which, if your app has a server-based SQL backend and you are looking for a speed boost, I suggest you ask yourself these questions:

  • What is the proportion of time your app spends in database calls?
  • Out of that, for the heaviest used table (or even single query), what is the proportion of time spent waiting on that?
  • What would it take to rewrite just that table or single query access with a No-SQL solution?
  • What would it gain?

One of the user comments to Shaneal’s post says the comparison given is apples to oranges: MySQL is network based, and BDB is an in-process solution.  He suggests comparing MySQL with a network based key-value pair database.  The point is a good one, you don’t really know if the impressive speedups were due to BDB being a key-value vs. SQL, or whether it was due to the speed of having everything work within the same process.  I suspect the reason is some of both.

Database types in a 2 by 2 matrix

If you’re in the upper left corner, you can fully explore the matrix by trying the lower left first: SQLite (the BDB distribution now includes a version of SQLite that is backed by BDB).  For the upper right, you could try a network key value store, like memcachedb. Core BDB belongs in the lower right category.

In any event I think the main point is clear.  When looking for a speedup, understand what is taking time and figure out what the alternatives will buy you.  Beware these words — “I see this is taking a good chunk of our time, but unfortunately we can’t do anything about that”. Because sometimes you can.


About ddanderson

Berkeley DB, Java, C, C , C# consultant and jazz trumpeter
This entry was posted in Uncategorized. Bookmark the permalink.

One Response to Choose your battles

  1. ddanderson says:

    To make the points clearer, I added the matrix (I love 2×2 matrices!) to speak in broad terms about some of the alternatives.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s