Done (for now).

JUDCon Boston 2011 is well under way. After many interesting technical sessions, including presentations on Byteman and on Transactions in the Cloud, we're onto the beer and pizza part of the day. If you're in Boston get yourself down here tomorrow for the second day, it's a great event.

Unfortunately I'm going to miss it as I'm off on vacation at oh-my-god-early in the morning. Please try not to break the shiny new JBossTS release in my absence. Normal services will resume, from the not so normal and sadly (for me) temporary location of Brisbane, Australia, in a couple of week's time.

almost done

JBossTS 4.15.0.Final is done. It's the fastest, most robust release yet. It's also the last feature release of its line.

As well as working standalone, 4.15 is the feature release for integration with JBossAS 7 and thence JBossEAP 6. As AS 7 is still a moving target we've got a bit of tweaking to do for the integration over the next few weeks, particularly on JTS and XTS. After that it's maintenance only on the 4.15.x branch and the community focus moves on to the next major version.

As well as new features, the next major release cycle will have a new project name (complete with snazzy new logo, naturally) and a new project lead.

That change should hopefully leave me with some time to do new things too, particularly looking at the implications of the cloud computing model and NoSQL storage solutions on transaction systems.

I'm starting out on that road with a presentation next week at JUDCon so if you're in the Boston area come along and join in. Don't worry, I'll have written the slides by then, I promise. I'm just not quite done yet :-)

Messaging/Database race conditions

Programmers are divided between those who only write single threaded code and those who will, at the slightest hint of provocation, tell interminably long war stories about debugging race conditions. Although I'm firmly in the second category, this is not such a story. It is however a lesson in what happens when spec authors spend insufficient time listening to them.

A common pattern in enterprise apps involves a business logic method writing a database update, then sending a message containing a key to that data via a queue to a second business method, which then reads the database and does further processing. Naturally this involves transactions, as it is necessary to ensure the database update(s) and message handling occur atomically.

methodA {
  beginTransaction();
  updateDatabase(key, values);
  sendMessage(queueName, key);
  commitTransaction();
}

methodB {
  beginTransaction();
  key = receiveMessage(queueName);
  values = readDatabase(key);
  doMoreStuff(values);
  commitTransaction();
}

So, like all good race condition war stories, this one starts out with a seemingly innocuous chunk of code. The problem is waiting to ambush us from deep within the transaction commit called by methodA.

Inside the transaction are two XAResources, one for the database and one for the messaging system. The transaction manager calls prepare on both, get affirmative responses and writes its log. Then it calls commit on both, at which point the resource managers can end the transaction isolation and make the data/message visible to other processes.

Spot the problem yet?

Nothing in the XA or JTA specifications defines the order in which XAResources are processed. The commit calls to the database and message queue may be issued in either order, or even in parallel.

Therefore it is inevitable that every once in a while, probably on the least convenient occasions, the database read in methodB will fail as the data has not yet been released from the previous transaction.

Oops.

So, having been let down by shortcomings in the specifications, what can the intrepid business logic programmer do to save the day?

Polling: Throw away the messaging piece and construct a loop that will look for new data in the db periodically. yuck.

Messaging from afterCompletion: The transaction lifecycle provides an afterCompletion notification that is guaranteed to be ordered after all the XAResource commits complete, so we can send the message from there to avoid the race. Also icky, since it is then no longer atomic with the db write in the case of failures, which means we need polling as a backup anyhow.

Let it fail: An uncaught exception in the second tx, e.g. an NPE from the database read, should cause the container to fail the transaction and the messaging system to do message redelivery. Hopefully by that time the db is ready. Inefficient and prone to writing a lot of failure messages into your system logs, which is to say: yuck.

Force resource ordering: Many transaction systems offer some non-spec compliant mechanism for XAResource ordering. Due to the levels of abstraction the container places between the transaction manager and the programmer this is not always easy to use from Java EE. It also ties your code to a specific vendor's container. You can do this with JBoss if you really want to, but it's not recommended.

Delay the message delivery: Similar to the above, some messaging systems go above and beyond the JMS spec to offer the ability to mark a queue or specific message as having a built-in delay. The messaging system will hold the message for a specified interval before making it available to consumers. Again you can do with with JBoss (see 'Scheduled Messages' in HornetQ) but whilst it's easy to code it's not a runtime efficient solution as tuning the delay interval is tricky.

Backoff/Retry in the consumer: Instead of letting the consuming transaction fail and be retried by message redelivery, catch the exception and handle it with a backoff/retry loop:

methodB {
  beginTransaction();
  key = receiveMessage(queueName);

  do {
    values = readDatabase(key);
    if(values == null) {
      sleep(x);
    }
  } while(values == null);

  doMoreStuff(values);
  commitTransaction();
}

This has the benefit of being portable across containers and, with a bit of added complexity, pretty robust. Yes it is a horrible kludge, but it works. Sometimes when you are stuck working with an environment built on flawed specifications that's really the best you can hope for.

World domination

It's nearing that time of year again... The transactions team are packing their bags and preparing to spend a couple of days holed up in a top secret underground lair, plotting the next steps towards world domination.

I normally spend a few days beforehand checking up on the state of the world, particularly new developments in transactions, the list of open issues in our bug tracker, the last few month's worth of support cases and other such sources of interesting information. I'm starting to think the reason that no Evil Genius has succeeded in taking over the world yet has very little to do with being repeatedly thwarted by James Bond and an awful lot to do with the sheer volume of paperwork involved. Perhaps if they spent less on underground lairs and more on secretarial support...

Anyhow, as per usual we have way too many ideas on the list, so some prioritization is in order. One of the key factors is the number of users a given chunk of work may benefit. Improvements to the crash recovery system benefit pretty much everyone. I'd like to say the same about documentation, but it's apparent only a tiny proportion of users actually read it. Somewhere in between lie improvements to optional modules or functionality that only some users take advantage of. The tricky part is gauging how many there are.

Users occasionally get in touch and tell us how they are using the code. Unfortunately this usually happens only when they need help. The rest of the time we mostly just make educated guesses about the use cases. I was mulling over this problem today when a tweet caught my eye:

"Today's royalty check for "Advanced CORBA Programming": $24.51. Last year each check was ~$400. I guess CORBA is officially dead now :-)" - Steve Vinoski

This got me thinking about the future of JTS. In Java EE environments, JTS exists pretty much only to allow transactional calls between EJBs using IIOP. Now that we have @WebService on EJBs, how much traffic still uses IIOP anyhow? What are the odds of RMI/IIOP (and hence JTS) being deprecated out of Java EE in the next revision or two? How much time should we be sinking into improvements of JTS vs. web services transactions?

That's just one of many topics on the agenda for our team meeting. I suppose we could also debate the relative merits of man-portable freeze rays vs. orbital laser cannons, but I don't think the corporate health and safety folks will let us have either. Spoilsports. Guess we'll just have to continue progressing towards world domination the hard way.

The Competitors

I don't normally spend much time worrying about our competitors. I figure keeping the customers happy is a better use of my time. But no set of blog posts about performance would be complete without a final showdown...

There aren't all that many standalone JTA transaction managers out there. Most function only as part of an application server and are not a viable option for embedding in other environments. Others are incomplete or largely abandoned. In my opinion there are currently only two serious challengers to JBossTS for use outside a full Java EE app server. Let's see how they do.

results in transactions per second, using quad core i7, jdk6, in-memory logging i.e. we're testing the CPU not the disk.

threads  JBossTS       Competitor A.
1         93192           9033
2        164744          20242
3        224114          22649
4        249781          21148
100      239865          19978

umm, guys, "synchronized ( Foo.class )" is really not a good idea, ok? Come on, put up a bit more of a fight please, this is no fun.

Next contender, please....

threads  JBossTS        Competitor B.
1         93192           102270
2        164744           145011
3        224114           185528
4        249781           193199
100      239865            29528

Ahh, that's more like it. Good to have a worthy opponent to keep us on our toes. Some kind of scaling problem there though. Looks like contention on the data structures used to queue transaction timeouts. Been there, done that. Feel free to browse the source code for inspiration guys :-)

Since nobody in their right mind actually runs a transaction manager with logging turned off, this is really all just a bit of fun. Well, for some of us anyhow. So where is the competitive comparison for the real world scenario?

Well, I do have the comparison numbers for all three systems with logging enabled, but I'd hesitate to describe them as competitive. You'll have to try it for yourself or you just won't believe the difference. Suffice it to say there is only one Clebert, and we've got him. Sorry guys.

Now if this had been an ease of use contest I'd be a bit more worried. Sure we have a ton of detailed and comprehensive reference documentation, but the learning curve is near vertical. Time to work on some intro level material: quick starts, config examples etc. Not as much fun as performance tuning, but probably more useful at this point.

fast enough?

Remember the performance tuning we've been doing? Well, the powers that be sat up and took notice too. I think the phrase was 'more than enough to satisfy requirements' with a non-too-subtle subtext of 'so now put down that profiler and do some real work'. Fortunately I'm not too good at taking hints...

After some more tweaking today (finalizers are evil and must die), I thought I'd see how we were doing. Here is trunk, soon to become JBossTS 4.15 and be used in EAP6, against JBossTS 4.6.1.CP11, used in the current EAP5.1 platform release. 4.6.1 originally dates from September 2009, but we backported some performance improvements into it in mid 2010.

results in transactions per second, using quad core i7, jdk6, in-memory objectstore i.e. we're testing the CPU not the disk.

threads  4.6.1.CP12    4.15.SNAPSHOT   improvement
1        37285          93192          2.50x
2        55309         164744          2.97x
3        65070         224114          3.44x
4        66172         249781          3.77x
100      29027         239865          8.26x

hmm, ok, so I guess maybe we did exceed the requirements a bit.

Concurrency control

We managed to gloss over a few important points during the discussion on Isolation. Let's dig into them here.

How can you ensure isolation? Well typically this is covered by what's referred to as concurrency control for the resources within the transaction. A very simple and widely used approach is to regard all operations on resources (objects) to be of type ‘read’ or ‘write’, which follow the synchronization rule permitting ‘concurrent reads' but exclusive ‘writes’. This rule is imposed by requiring that any computation intending to perform an operation that is of type read (write) on an object, first acquire a ‘read lock’ (‘write lock’) associated with that object. A read lock on an object can be held concurrently by many computations provided no computation is holding a write lock on that object. A write lock on an object, on the other hand, can only be held by a computation provided no other computation is holding a read or a write lock. (Note, although we'll talk in terms of locks, this should not be used to infer a specific implementation. Timestamp-based concurrency control could just as easily be used, for example.)

In order to ensure the atomicity property, all computations must follow a ‘two–phase’ locking policy. During the first phase, termed the growing phase, a computation can acquire locks, but not release them. The tail end of the computation constitutes the shrinking phase, during which time held locks can be released but no locks can be acquired. Now suppose that a computation in its shrinking phase is to be rolled back, and that some objects with write locks have already been released. If some of these objects have been locked by other computations, then abortion of the computation will require these computations to be aborted as well. To avoid this cascade roll back problem, it is necessary to make the shrinking phase ‘instantaneous’.

Most transaction systems utilize what is commonly referred to as pessimistic concurrency control mechanisms: in essence, whenever a data structure or other transactional resource is accessed, a lock is obtained on it as described earlier. This lock will remain held on that resource for the duration of the transaction and the benefit of this is that other users will not be able to modify (and possibly not even observe) the resource until the holding transaction has terminated. There are a number of disadvantages of this style: (i) the overhead of acquiring and maintaining concurrency control information in an environment where conflict or data sharing is not high, (ii) deadlocks may occur, where one user waits for another to release a lock not realizing that that user is waiting for the release of a lock held by the first.

Therefore, optimistic concurrency control assumes that conflicts are not high and tries to ensure locks are held only for brief periods of time: essentially locks are only acquired at the end of the transaction when it is about to terminate. This kind of concurrency control requires a means to detect if an update to a resource does conflict with any updates that may have occurred in the interim and how to recover from such conflicts. Typically detection will happen using timestamps, whereby the system takes a snapshot of the timestamps associated with resources it is about to use or modify and compares them with the timestamps available when the transaction commits.

Resolution of conflicts is a different problem entirely, since in order to do so requires semantic information about the resources concerned. Therefore, most transaction systems that offer optimistic schemes will typically cause the detecting transaction to roll back and the application must retry, this time with new data. Obviously this may result in a lot of work being lost, especially if the transaction that rolls back has been running for some time.

Assuming both optimistic and pessimistic concurrency control are available to you (and they may not be), then which one to use is up to you. A close examination of the environment in which the application and transactional resources reside is necessary to determine whether a) shared access to resources occurs and b) the relative probability that sharing will cause a transaction to roll back. This might very well not be a black or white choice and may change over the lifetime of your objects or application. Certainly the use of different concurrency control schemes can be important when trying to improve the throughput of user requests and committed transactions, so it’s well worth considering and understanding the issues involved.
Type specific concurrency control

Another possible enhancement is to introduce type specific concurrency control, which is a particularly attractive means of increasing the concurrency in a system (and yes, it's supported in JBossTS). Concurrent read/write or write/write operations are permitted on an object from different transactions provided these operations can be shown to be non-interfering (for example, for a directory object, reading and deleting different entries can be permitted to take place simultaneously). Object-oriented systems are well suited to this approach, since semantic knowledge about the operations of objects can be exploited to control permissible concurrency within objects. Additional work may be needed when working with procedural systems.

Finally, what about deadlocks? When multiple transactions compete for the same resources in conflicting modes (locks), it is likely that some of them will fail to acquire those resources. If a transaction that cannot acquire a lock on a resource waits for it to be released, then that transaction is blocked – no forward progress can be made until the lock has been acquired. In some environments, it is possible for some transactions to be waiting for each other, where each of them is blocked and is also blocking another transaction. In this situation, none of the transactions can proceed and the system is deadlocked.

For example, let’s consider two transactions T1 and T2 that operate on two resources X and Y. Let’s assume that the execution of the operations involved in these transactions is:

T1: read(X); write(Y)
T2: read(Y); write(X)

If the serial execution of these transactions were to result in:

readT1(X); readT2(Y); writeT2(X); readT1(Y)

Note, readT1 means the read operation performed by T1 etc.

Assume that T1 obtained a read lock on X and then T2 gets a read lock on Y – possible because these operations aren’t conflicting and can thus occur in parallel. However, when T2 comes to write to X its attempt to get a write lock on X will block because T1 still holds its read lock. Likewise, T1’s attempt to get a write lock on Y will block because of the read lock that T2 holds. Each transaction is blocked waiting for the release of the others read lock before they can progress: they are deadlocked.
The only way for the deadlock to be resolved is for at least one of the transactions to release its locks that are blocking another transaction. Obviously such a transaction cannot commit (it has not been able to perform all of its work since it was blocked); therefore, it must roll back.

Deadlock detection and prevention is complicated enough in a non-distributed environment without then including the extra complexity of distribution. In general, most transaction systems allow deadlocks to occur, simply because to do otherwise can be too restrictive for applications. There are several techniques for deadlock detection, but the two most popular are:

• Timeout-based: if a transaction has been waiting longer than a specified period of time, the transaction system will automatically roll back the transaction on the assumption it is deadlocked. The main advantage of this approach is that it is easy to implement in a distributed environment; the main disadvantage is that some transactions may execute for longer than expected and be rolled back when they are not in fact deadlocked.


• Graph-based: this explicitly tracks waiting transaction dependencies by constructing a waits-for graph: nodes are waiting transactions and edges are waiting situations. The main advantage of this approach is that it is guaranteed to detect all deadlocks, whereas the main disadvantage is that in a distributed environment is can be costly to execute.

A slight variation on the timeout-based approach exists in some transaction systems, where timeouts can be associated with lock acquisition, such that the system will only block for the specified period of time. If the lock has not been acquired by the time this period elapses, it returns control to the application indicating that the lock has not been obtained. It is then up to the application to determine what to do; for example, it may be possible to acquire the required data elsewhere or to ask for a lock in a different mode. The advantage of this approach is that a transaction is not automatically rolled back if it cannot acquire a lock, possibly saving the application lots of valuable time; the disadvantage is that it requires additional effort on behalf of the application to resolve lock acquisition failures. However, for many objects and applications, this is precisely the best place to resolve lock conflicts because this is the only place where the semantic information exists to know what (if anything) can be done to resolve such conflicts.