Thursday, March 31, 2011

The Competitors

I don't normally spend much time worrying about our competitors. I figure keeping the customers happy is a better use of my time. But no set of blog posts about performance would be complete without a final showdown...

There aren't all that many standalone JTA transaction managers out there. Most function only as part of an application server and are not a viable option for embedding in other environments. Others are incomplete or largely abandoned. In my opinion there are currently only two serious challengers to JBossTS for use outside a full Java EE app server. Let's see how they do.

results in transactions per second, using quad core i7, jdk6, in-memory logging i.e. we're testing the CPU not the disk.


threads JBossTS Competitor A.
1 93192 9033
2 164744 20242
3 224114 22649
4 249781 21148
100 239865 19978


umm, guys, "synchronized ( Foo.class )" is really not a good idea, ok? Come on, put up a bit more of a fight please, this is no fun.

Next contender, please....


threads JBossTS Competitor B.
1 93192 102270
2 164744 145011
3 224114 185528
4 249781 193199
100 239865 29528


Ahh, that's more like it. Good to have a worthy opponent to keep us on our toes. Some kind of scaling problem there though. Looks like contention on the data structures used to queue transaction timeouts. Been there, done that. Feel free to browse the source code for inspiration guys :-)

Since nobody in their right mind actually runs a transaction manager with logging turned off, this is really all just a bit of fun. Well, for some of us anyhow. So where is the competitive comparison for the real world scenario?

Well, I do have the comparison numbers for all three systems with logging enabled, but I'd hesitate to describe them as competitive. You'll have to try it for yourself or you just won't believe the difference. Suffice it to say there is only one Clebert, and we've got him. Sorry guys.

Now if this had been an ease of use contest I'd be a bit more worried. Sure we have a ton of detailed and comprehensive reference documentation, but the learning curve is near vertical. Time to work on some intro level material: quick starts, config examples etc. Not as much fun as performance tuning, but probably more useful at this point.

Monday, March 28, 2011

fast enough?

Remember the performance tuning we've been doing? Well, the powers that be sat up and took notice too. I think the phrase was 'more than enough to satisfy requirements' with a non-too-subtle subtext of 'so now put down that profiler and do some real work'. Fortunately I'm not too good at taking hints...

After some more tweaking today (finalizers are evil and must die), I thought I'd see how we were doing. Here is trunk, soon to become JBossTS 4.15 and be used in EAP6, against JBossTS 4.6.1.CP11, used in the current EAP5.1 platform release. 4.6.1 originally dates from September 2009, but we backported some performance improvements into it in mid 2010.

results in transactions per second, using quad core i7, jdk6, in-memory objectstore i.e. we're testing the CPU not the disk.


threads 4.6.1.CP12 4.15.SNAPSHOT improvement
1 37285 93192 2.50x
2 55309 164744 2.97x
3 65070 224114 3.44x
4 66172 249781 3.77x
100 29027 239865 8.26x


hmm, ok, so I guess maybe we did exceed the requirements a bit.

Saturday, March 19, 2011

Concurrency control

We managed to gloss over a few important points during the discussion on Isolation. Let's dig into them here.

How can you ensure isolation? Well typically this is covered by what's referred to as concurrency control for the resources within the transaction. A very simple and widely used approach is to regard all operations on resources (objects) to be of type ‘read’ or ‘write’, which follow the synchronization rule permitting ‘concurrent reads' but exclusive ‘writes’. This rule is imposed by requiring that any computation intending to perform an operation that is of type read (write) on an object, first acquire a ‘read lock’ (‘write lock’) associated with that object. A read lock on an object can be held concurrently by many computations provided no computation is holding a write lock on that object. A write lock on an object, on the other hand, can only be held by a computation provided no other computation is holding a read or a write lock. (Note, although we'll talk in terms of locks, this should not be used to infer a specific implementation. Timestamp-based concurrency control could just as easily be used, for example.)

In order to ensure the atomicity property, all computations must follow a ‘two–phase’ locking policy. During the first phase, termed the growing phase, a computation can acquire locks, but not release them. The tail end of the computation constitutes the shrinking phase, during which time held locks can be released but no locks can be acquired. Now suppose that a computation in its shrinking phase is to be rolled back, and that some objects with write locks have already been released. If some of these objects have been locked by other computations, then abortion of the computation will require these computations to be aborted as well. To avoid this cascade roll back problem, it is necessary to make the shrinking phase ‘instantaneous’.

Most transaction systems utilize what is commonly referred to as pessimistic concurrency control mechanisms: in essence, whenever a data structure or other transactional resource is accessed, a lock is obtained on it as described earlier. This lock will remain held on that resource for the duration of the transaction and the benefit of this is that other users will not be able to modify (and possibly not even observe) the resource until the holding transaction has terminated. There are a number of disadvantages of this style: (i) the overhead of acquiring and maintaining concurrency control information in an environment where conflict or data sharing is not high, (ii) deadlocks may occur, where one user waits for another to release a lock not realizing that that user is waiting for the release of a lock held by the first.

Therefore, optimistic concurrency control assumes that conflicts are not high and tries to ensure locks are held only for brief periods of time: essentially locks are only acquired at the end of the transaction when it is about to terminate. This kind of concurrency control requires a means to detect if an update to a resource does conflict with any updates that may have occurred in the interim and how to recover from such conflicts. Typically detection will happen using timestamps, whereby the system takes a snapshot of the timestamps associated with resources it is about to use or modify and compares them with the timestamps available when the transaction commits.

Resolution of conflicts is a different problem entirely, since in order to do so requires semantic information about the resources concerned. Therefore, most transaction systems that offer optimistic schemes will typically cause the detecting transaction to roll back and the application must retry, this time with new data. Obviously this may result in a lot of work being lost, especially if the transaction that rolls back has been running for some time.

Assuming both optimistic and pessimistic concurrency control are available to you (and they may not be), then which one to use is up to you. A close examination of the environment in which the application and transactional resources reside is necessary to determine whether a) shared access to resources occurs and b) the relative probability that sharing will cause a transaction to roll back. This might very well not be a black or white choice and may change over the lifetime of your objects or application. Certainly the use of different concurrency control schemes can be important when trying to improve the throughput of user requests and committed transactions, so it’s well worth considering and understanding the issues involved.
Type specific concurrency control

Another possible enhancement is to introduce type specific concurrency control, which is a particularly attractive means of increasing the concurrency in a system (and yes, it's supported in JBossTS). Concurrent read/write or write/write operations are permitted on an object from different transactions provided these operations can be shown to be non-interfering (for example, for a directory object, reading and deleting different entries can be permitted to take place simultaneously). Object-oriented systems are well suited to this approach, since semantic knowledge about the operations of objects can be exploited to control permissible concurrency within objects. Additional work may be needed when working with procedural systems.

Finally, what about deadlocks? When multiple transactions compete for the same resources in conflicting modes (locks), it is likely that some of them will fail to acquire those resources. If a transaction that cannot acquire a lock on a resource waits for it to be released, then that transaction is blocked – no forward progress can be made until the lock has been acquired. In some environments, it is possible for some transactions to be waiting for each other, where each of them is blocked and is also blocking another transaction. In this situation, none of the transactions can proceed and the system is deadlocked.

For example, let’s consider two transactions T1 and T2 that operate on two resources X and Y. Let’s assume that the execution of the operations involved in these transactions is:

T1: read(X); write(Y)
T2: read(Y); write(X)

If the serial execution of these transactions were to result in:

readT1(X); readT2(Y); writeT2(X); readT1(Y)

Note, readT1 means the read operation performed by T1 etc.

Assume that T1 obtained a read lock on X and then T2 gets a read lock on Y – possible because these operations aren’t conflicting and can thus occur in parallel. However, when T2 comes to write to X its attempt to get a write lock on X will block because T1 still holds its read lock. Likewise, T1’s attempt to get a write lock on Y will block because of the read lock that T2 holds. Each transaction is blocked waiting for the release of the others read lock before they can progress: they are deadlocked.
The only way for the deadlock to be resolved is for at least one of the transactions to release its locks that are blocking another transaction. Obviously such a transaction cannot commit (it has not been able to perform all of its work since it was blocked); therefore, it must roll back.

Deadlock detection and prevention is complicated enough in a non-distributed environment without then including the extra complexity of distribution. In general, most transaction systems allow deadlocks to occur, simply because to do otherwise can be too restrictive for applications. There are several techniques for deadlock detection, but the two most popular are:

  • Timeout-based: if a transaction has been waiting longer than a specified period of time, the transaction system will automatically roll back the transaction on the assumption it is deadlocked. The main advantage of this approach is that it is easy to implement in a distributed environment; the main disadvantage is that some transactions may execute for longer than expected and be rolled back when they are not in fact deadlocked.

  • Graph-based: this explicitly tracks waiting transaction dependencies by constructing a waits-for graph: nodes are waiting transactions and edges are waiting situations. The main advantage of this approach is that it is guaranteed to detect all deadlocks, whereas the main disadvantage is that in a distributed environment is can be costly to execute.


A slight variation on the timeout-based approach exists in some transaction systems, where timeouts can be associated with lock acquisition, such that the system will only block for the specified period of time. If the lock has not been acquired by the time this period elapses, it returns control to the application indicating that the lock has not been obtained. It is then up to the application to determine what to do; for example, it may be possible to acquire the required data elsewhere or to ask for a lock in a different mode. The advantage of this approach is that a transaction is not automatically rolled back if it cannot acquire a lock, possibly saving the application lots of valuable time; the disadvantage is that it requires additional effort on behalf of the application to resolve lock acquisition failures. However, for many objects and applications, this is precisely the best place to resolve lock conflicts because this is the only place where the semantic information exists to know what (if anything) can be done to resolve such conflicts.

Durability

D is for Dog, Design and Durability, i.e., the effects of a committed transaction are never lost (except by a catastrophic failure).

The durability (or persistence) property means that any state changes that occur during the transaction must be saved in a manner such that a subsequent failure will not cause them to be lost. How these state changes are made persistent is typically dependant on the implementation of the transaction system and the resources that are ultimately used to commit the work done by the transactional objects. For example, the database will typically maintain its state on hard disk in order to ensure that a machine failure (e.g., loss of power) does not result in loss of data.

Although most users of transactions will see durability from their application’s point-of-view, there is also an aspect within the transaction system implementation itself. In order to guarantee atomicity in the presence of failures (both transaction coordinator and participant), it is necessary for the transaction service itself to maintain state. For example, in some implementations the coordinator must remember the point in the protocol it has reached (i.e., whether it is committing or aborting), the identity of all participants that are registered with the transaction and where they have reached in the protocol (e.g., whether they have received the prepare message). This is typically referred to as the transaction log, though this should not be interpreted as implying a specific implementation. Some implementations may maintain a separate log (file) per transaction, with this information recorded within it and removed when it is no longer needed. Another possible implementation has a single log for all transactions and the transaction information is appended to the end of the log and pruned from the log when the respective transaction completes.

Let’s look at what happens at the participant in terms of durability, when it’s driven through the two-phase commit protocol. When the participant receives a prepare message from the coordinator it must decide whether it can commit or roll back. If it decides to roll back then it must undo any state changes that it may control and inform the coordinator; there is no requirement for durable access at this point. If the participant can commit, it must write its intentions to a durable store (participant log) along with sufficient information to either commit or roll back the state changes it controls. The format of this information is typically dependant on the type of participant, but may include the entire original and new states.

Once successfully recorded, the participant informs the coordinator that it can commit and awaits the coordinator’s decision. When this second phase message arrives, the participant will either cancel all state changes (the coordinator wants to roll back), or if the coordinator wants to commit, make those state changes the current state (e.g., overwrite the original state with the new state). It can then delete the participant log and inform the coordinator.

Isolation

I is for Iguana, Igloo and Isolation, i.e., intermediate states produced while a transaction is executing are not visible to others. Furthermore transactions appear to execute serially, even if they are actually executed concurrently.

This property is often referred to as serializability. If we assume that objects and services can be shared between various programs then it is necessary to ensure that concurrent executions of programs are free from interference, i.e., concurrent executions should be equivalent to some serial order of execution. We have already seen a fairly trivial example of this through the online bank account, but it is worth formalizing this requirement. Consider the following two programs (where w, x, y and z are distinct state variables):

P1 : z := 10; x := x+1; y := y+1
P2 : w := 7: x := x * 2; y := y * 2

Assume that x=y=2 initially. Then, a serial execution order ’P1;P2 ’ will produce the result z=10, w=7, x=y=6, and execution ’P2 ;P1 ’ will produce the results z=10, w=7, x=y=5. The partly concurrent execution order given below will be termed interference free or serializable, since it is equivalent to the serial order ’P1 ;P2 ’:

(z := 10 || w := 7); x := x+1; y := y+1; x := x * 2; y := y * 2

However, an execution order such as the one below is not free from interference since it cannot be shown to be equivalent to any serial order.

(z := 10 || w := 7); x := x+1; x := x * 2; y := y * 2; y := y+1

Programs that possess the above-mentioned serializable property are said to be atomic with respect to concurrency. The serializability property is extremely important, especially in an environment where multiple concurrent users may be attempting to use the same resources consistently.

Consistency

C is for Crayon, Cup and Consistency, i.e., transactions produce consistent results and preserve application specific invariants.

A transactional application should maintain the consistency of the resources (e.g., databases, file-systems, etc.) that it uses. In essence, transactional applications should move from one consistent state to another. However, unlike the other transactional properties (A, I and D) this is something that the transaction system cannot achieve by itself since it does not possess any semantic information about the resources it manipulates: it would be impossible for a transaction processing system to assert that the resources are moving to (or from) consistent states. All a transaction system can ensure is that any state changes that do occur are performed in a manner that is guaranteed despite failures. It is the application programmers’ responsibility to ensure consistency (in whatever way makes sense for the resources concerned.)

Atomicity

We've already covered some of this in previous postings, but in the next four entries let's explicitly look at all ACID properties. And of course that means we start here with ... A is for Apple, Aardvark and Atomic, i.e., the transaction completes successfully (commits) or if it fails (aborts) all of its effects are undone (rolled back).

In order to ensure that a transaction has an atomic outcome, the two-phase commit protocol that we discussed earlier, is typically used. This protocol is used to guarantee consensus between participating members of the transaction. When each participant receives the coordinator’s phase 1 message, they record sufficient information on stable storage to either commit or abort changes made during the action. After returning the phase 1 response, each participant that returned a commit response must remain blocked until it has received the coordinator’s phase 2 message. Until they receive this message, these resources are unavailable for use by other actions. If the coordinator fails before delivery of this message, these resources remain blocked. However, if failed machines eventually recover, crash recovery mechanisms can be employed to unblock the protocol and terminate the transaction.