fiddling with knobs

Get your mind out the gutter you. I'm talking about control knobs. Also buttons, switches, levers and anything else that allows for interesting behavioural tweaks.

Messing around with settings on software is a great way to pass the time. For those who enjoy such things there are over 100 configuration properties in JBossTS. Which admittedly may be a little too much of a good thing. Fortunatly they mostly have sensible default values.

I recently discussed the performance benefits of using one of those properties to swap in a new ObjectStore implementation for transaction logging in JBossTS. In the last few days I've been adding some more configuration options to allow overriding of the settings in the underlying HornetQ Journal. It turns out that there are also a bunch of interesting knobs to fiddle with in that code. By judicious tweaking of the settings on the Journal and the file system I've got a nice speedup over the last iteration:

60119 tx/second with in-memory logging, no disk I/O

43661 tx/second with the new Journal on SSD

43k is a big step up from 30-36k or so we were seeing previously, but still not saturating the CPU. However, a new test case shows the disk is not to blame:

89221 logs writes / second.

That's taking most of the transaction manger code out of the picture and writing an equivalent amount of data direct to the Journal based ObjectStore. So, plenty of disk write capacity to spare now.

So why can't we use it all?

The log write done by an individual transaction is very small. To get an efficient batch size you need to aggregate the writes for a significant number of transactions. Whilst waiting for that batch to be written, the transactions are blocked. So, to keep the CPU busy you need to throw more threads / concurrent transactions at it. Infact the disk is so quick you need a LOT more threads. At which point lock contention elsewhere in the transaction manager code become a problem, despite the significant improvements we made to it anyhow for pure in-memory transaction cases.

Naturally it's possible we can get some additional benefit from another iteration of lock refactoring. We've already picked off most of the reasonably tractable cases though and what's left is going to be complex to change.

Another possibility is to actually run the Journal out of process and use IPC so several JVMs can share it. The additional communication overhead may just be worthwhile. It's a similar model to what we do with the SQL db based objectstore, but without the overhead of SQL. Although naturally this multi process architecture is attractive only where the application workload is amenable to partitioning between JVMs.

It may have to wait a while though. We're entering the final weeks of 4.15 development and it's time to start cleaning up all the loose ends and getting things in shape for use by JBossAS 7 GA. Onward.