Workshop on the Theory of Transactional Memory

I'm just back from the WTTM in Rome. It was good to go to the workshop and listen to presentations from the likes of Maurice Herlihy on advances in both hardware and software transactional memory. I was there representing the Cloud-TM work that we're doing in collaboration with others and specifically around Infinispan and JBossTS.

One of my favourite presentations of the event was on the use of pessimistic concurrency control in STM. Most STM approaches use an optimistic approach, which obviously works well in cases where contention is limited. The problem with pessimistic is retaining locks for the duration even when updates are infrequent. However, as the presenter showed, with suitable a priori ordering, pessimistic can be more efficient even in low contention environments. This was good because we tend to use pessimistic concurrency control in JBossTS a lot, and as was shown in the presentation, it's also a lot easier for developers to understand and reason about.

Unfortunately I had to leave the workshop before the last session, so I didn't get to listen to Torvald who represents Red Hat on the transactional C++ working group. That's a shame, because I've been following this effort for several years and know that we may be seeing this in gcc sometime soon. And who knows, maybe EE8 will see some STM eventually.

RESTful Transactions in the Cloud: Now they're for real

A few months back Mark posted on the subject of REST, transactions and the cloud. Well now you can test the validity of his claims for yourselves. RedHat recently announced their OpenShift initiative for deploying applications into the cloud and we have put together a demonstration to show how easy it is to perform end to end transactional requests whilst remaining RESTful/RESTlike throughout and using little more than a JAX-RS container, an HTTP client library and a transaction coordinator.

As our starting point we use an implementation of the draft specification for a RESTful interface to 2-Phase-Commit transactions which we refer to as REST-AT.

The demonstration starts out by showing local clients (written in various languages such as javascript and Ruby) and local (JAX-RS) web services interacting transactionally using HTTP for all communications. The transaction coordinator that implements REST-AT is written using JAX-RS and also runs locally.

The push to cloud based deployments demands that providers support piecemeal migration of applications into the cloud infrastructure. REST-AT together with OpenShift makes the realisation of this goal, at least for transactional services, a relatively painless exercise. The demonstration code and accompanying video shows how to migrate the transaction coordinator component leaving clients and services outside of the infrastructure. We achieve this by using the OpenShift JBoss AS7 cartridge and deploy REST-AT as a war into the application server that this cartridge provides. In a later post we will show how to migrate Java and Ruby services into the OpenShift infrastructure using the AS7 and Ruby cartridges, respectively.

To begin using OpenShift a good starting point would be to watch the various getting started videos (look for the Zero to Cloud in 30 minutes track or go directly to the demo video). To start using REST-AT check out the quickstarts and the sources for the demo.

Using JBossTS in OpenShift Express

Hi,

I just wanted to let you know that you can use transactions in Red Hat's new OpenShift Express tool. Having had some hand's on time with the system I can safely say its one of the most exciting new projects I have seen in a while.

It makes deploy applications into the cloud so easy, plus as it supports deploying applications onto AS7 you can benefit from all the cool features that have been added there.

As AS7 features transactions (naturally!) I decided to kick the tyres of Openshift Express a little harder by not just using JPA, EJB, JTA and JSF, but also utilise one of the JBoss Transactions extension features - TXOJ. You can watch my video of this here, and the source code is available here.

Safe to say it was extremely simple to get this up and running, if you follow the video you can too!

For those who don't like videos though, here's a little cheat sheet I put together to get it running:

1. You need an openshift express domain, go to: http://openshift.redhat.com to get one
2. rhc-create-app -a txapp -t jbossas-7.0
3. cd txapp
4. svn export https://anonsvn.jboss.org/repos/labs/labs/jbosstm/trunk/ArjunaJTA/quickstarts/jee_transactional_app
5. mv jee_transactional_app/* .
6. rmdir jee_transactional_app
7. Add an openshift profile to your pom.xml:

   <profile>
    <id>openshift</id>
    <build>
       <plugins>
         <plugin>
           <artifactId>maven-war-plugin</artifactId>
           <configuration>
             <outputDirectory>deployments</outputDirectory>
             <warName>ROOT</warName>
           </configuration>
         </plugin>
       </plugins>
     </build>
   </profile>

8. git add src pom.xml
9. git commit -a -m “jee_transactional_app” && git push
10. Open up firefox and go to http://txapp-<YOUR_DOMAIN>.openshift.redhat.com/jee_transactional_app/

I hope you enjoy using Openshift Express and also can add a few Transactional Objects to your applications!

Tom

norecoveryxa

"Infamy! Infamy! They've all got it in for me!"

Of all the error and warning messages in JBossTS, none is more infamous than norecoveryxa. Pretty much every JBossTS user has an error log containing this little gem. Usually more than once. A lot more. But not for much longer. It has incurred my wrath and its days are numbered. I've got it in for that annoying little beast.


[com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa]
Could not find new XAResource to use for recovering non-serializable XAResource


Transaction logs preserve information needed to complete interrupted transactions after a crash. Boiled down to pure essence, a transaction log is a list of the Xids involved in a transaction. An Xid is simply a bunch of bytes, beginning with a unique tx id and ending with a branch id that is (usually) different for each resource manager in the transaction. During recovery, these Xids are used to tell the resource managers to commit the in-doubt transactions. All of which is fine in theory, but an utter pain in practice.

The core problem is that an Xid is not a lot of use unless you can actually reconnect to the resource manager and get an XAResource object from its driver, as it is the methods on that object to which you pass the Xid. So you need some way of storing connection information also. In rare cases the RM's driver provides a Serializable XAResource implementation, in which case the simplest (although not necessarily fastest) thing to do is serialize it into the tx log file. Recovery is then easy - deserialize the XAResource and invoke commit on it. Well, except for the whole multiple classloaders thing, but that's another story. Besides, hardly any drivers are actually so accommodating. So, we need another approach.

To deal with non-serializable XAResources, the recovery system allows registration of plugins, one per resource manager, that provide a new XAResource instance on demand. In olden times configuring these was a manual task, frequently overlooked. As a result the recovery system found itself unable to get a new XAResource instance to handle the Xid from the transaction logs. Hence norecoveryxa. That problem was solved, at least for xa-datasource use within JBossAS, by having the datasource deployment handler automatically register a plugin with the recovery system. Bye-bye AppServerJDBCXARecovery, you will not be missed.

That leaves cases where the resource manager is something other than an XA aware JDBC datasource. To help diagnose such cases it would be helpful to have something more than a hex representation of the Xid. Whilst the standard XAResource API does not allow for such metadata, we now have an extension that provides the transaction manager with RM type, version and JNDI binding information. This gets written to the tx logs and, in the case of the JNDI name, encoded into the Xid.

As a side benefit, this information allows for much more helpful debug messages and hopefully means I spend less time on support cases. Didn't really think I was doing all this just for your benefit did you?

before:

TRACE: XAResourceRecord.topLevelPrepare for < formatId=131076,
gtrid_length=29, bqual_length=28, tx_uid=0:ffff7f000001:ccd6:4e0da81f:2,
node_name=1, branch_uid=0:ffff7f000001:ccd6:4e0da81f:3>


after:
TRACE: XAResourceRecord.topLevelPrepare for < formatId=131076,
gtrid_length=29, bqual_length=28, tx_uid=0:ffff7f000001:ccd6:4e0da81f:2,
node_name=1, branch_uid=0:ffff7f000001:ccd6:4e0da81f:3>,
product: OracleDB/11g, jndiName: java:/jdbc/TestDB >


The main reason we want the RM metadata though is so that when recovery tries and fails to find an XAResource instance to match the Xid, it can now report the details of the RM to which that Xid belongs. This makes it more obvious which RM is unavailable e.g. because it's not registered for recovery or it is temporarily down.


WARN: ARJUNA16037: Could not find new XAResource to use for recovering
non-serializable XAResource XAResourceRecord < resource:null,
txid:< formatId=131076, gtrid_length=29, bqual_length=41,
tx_uid=0:ffff7f000001:e437:4e294bb6:6, node_name=1,
branch_uid=0:ffff7f000001:e437:4e294bb6:7, eis_name=java:/jdbc/DummyDB >,
heuristic: TwoPhaseOutcome.FINISH_OK,
product: DummyProductName/DummyProductVersion,
jndiName: java:/jdbc/DummyDB>


Despite the name change (part of the new i18n framework), that's still our old nemesis norecoveryxa. It's made of stern stuff and, although now a little more useful, is not so easily vanquished entirely.

Even with all the resource managers correctly registered, it's still possible to be unable to find the Xid during a recovery scan. That's because of a timing window in the XA protocol, where a crash occurs after the RM has committed and forgotten the branch but before the TM has disposed of its no longer needed log file. In such cases no RM will claim ownership of the branch during replay, leaving the TM to assume that the owning RM is unavailable for some reason. With the name of the owning RM to hand, we now have the possibility to match against the name of the scanned RMs, conclude the branch belongs to one of the scanned RMs even though it pleaded ignorance, and hence dispose of it as a safely committed branch.

All of which should ensure that norecoveryxa's days in the spotlight are severely numbered.

rethinking log storage

Transaction coordinators like JBossTS must write log files to fault tolerant storage in order to be able to guarantee correct completion of transactions in the event of a crash. Historically this has meant using files hosted on RAIDed hard disks.

The workload is characterised by near 100% writes, reads for recovery only, a large number of small (sub 1 KB) operations and, critically, the need to guarantee that the data has been forced through assorted cache layers to non-volatile media before continuing.

Unfortunately this combination of requirements has a tendency to negate many of the performance optimizations provided in storage systems, making log writing a common bottleneck for high performance transaction systems. Addressing this can often force users into unwelcome architectural decisions, particularly in cloud environments.

So naturally transaction system developers like the JBossTS team spend a lot of time thinking about how to mange log I/O better. Those who have read or watched my JUDCon presentation will already have some hints of the ideas we're currently looking at, including using cluster based in-memory replication instead of disk storage.

More recently I've been playing with SSD based storage some more, following up on earlier work to better understand some of the issues that arise as users transition to a new generation of disk technology with new performance characteristics.

As you'll recall, we recently took the high speed journalling code from HornetQ and used it as the basis for a transaction log. Relatively speaking, the results were a pretty spectacular improvement over our classic logging code. In absolute terms however we still weren't getting the best out the hardware.

A few weeks back I got my grubby paws on one of the latest SSDs. On paper its next generation controller and faster interface should have provided a substantial improvement over its predecessor. Not so for my transaction logging tests though - in keeping with tradition it utterly failed to outperform the older generation of technology. On further investigation the reasons for this become clear.

Traditional journalling solutions are based on a) aggregating a large number of small I/Os into a much smaller number of larger I/Os so that the drive can keep up with the load and b) serializing these writes to a single append-only file in order to avoid expensive head seeks.

With SSDs the first of those ideas is still sound, but the number of I/O events the drive can deal with is substantially higher. This requires re-tuning the journal parameters. For some usage it even becomes undesirable to batch the I/Os - until the drive is saturated with requests it's just pointless overhead that delays threads unnecessarily. As those threads (transactions) may have locks on other data any additional latency is undesirable. Also, unlike some other systems, a transaction manager has one thread per tx, as it's the one executing the business logic. It can't proceed until the log write completes, so write batching solutions involve significant thread management and scheduling overhead and often have a large number of threads parked waiting on I/O.

There is another snag though: even where the journalling code can use async I/O to dispatch multiple writes to the O/S in parallel, the filesystem still runs them sequentially as they contend on the inode semaphore for the log file. Thus writes for unrelated transactions must wait unnecessarily, much like the situation that arises in business applications which uses too coarse-grained locking. Also, the nice ncq for the drive remains largely unused, limiting the ability of the drive controller to execute writes in parallel.

The serialization of writes to the hardware, whilst useful for head positioning on HDDs, is a painful mistake for SSDs. These devices, like a modern CPU, require high concurrency in the workload to perform at their best. So, just as we go looking for parallelization opportunities in our apps, so we must look for them in the design of the I/O logging.

The most obvious solution is to load balance the logging over several journal files when running on an SSD. It's not quite that simple though - care must be taken to avoid a filesystem journal write as that will trigger a buffer flush for the entire filesystem, not just the individual file. Not to mention contention on the filesystem journal. For optimal performance it may pay to put each log file on its own small filesystem. But I'm getting ahead of myself. First there is the minor matter of actually writing a prototype load balancer to test the ideas. Any volunteers?

STM Arjuna

I'd forgotten how long ago I'd promised to talk about some of the STM work we've been doing. Well I haven't been able to do much more on it for a while, but I do have time to at least outline what's possible at the moment. So let's just remember a bit about the current way in which you can use JBossTS to build transactional POJOs without the need for a database; we'll use an example to illustrate:

  public class AtomicObject extends LockManager
 {
 public AtomicObject()
 {
     super();

     state = 0;

     AtomicAction A = new AtomicAction();

     A.begin();

     if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
     {
         if (A.commit() == ActionStatus.COMMITTED)
             System.out.println("Created persistent object " + get_uid());
         else
             System.out.println("Action.commit error.");
     }
     else
     {
         A.abort();

         System.out.println("setlock error.");
     }
 }

 public void incr (int value) throws Exception
 {
     AtomicAction A = new AtomicAction();

     A.begin();

     if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
     {
         state += value;

         if (A.commit() != ActionStatus.COMMITTED)
             throw new Exception("Action commit error.");
         else
             return;
     }

     A.abort();

     throw new Exception("Write lock error.");
 }

 public void set (int value) throws Exception
 {
     AtomicAction A = new AtomicAction();

     A.begin();

     if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
     {
         state = value;

         if (A.commit() != ActionStatus.COMMITTED)
             throw new Exception("Action commit error.");
         else
             return;
     }

     A.abort();

     throw new Exception("Write lock error.");
 }

 public int get () throws Exception
 {
     AtomicAction A = new AtomicAction();
     int value = -1;

     A.begin();

     if (setlock(new Lock(LockMode.READ), 0) == LockResult.GRANTED)
     {
         value = state;

         if (A.commit() == ActionStatus.COMMITTED)
             return value;
         else
             throw new Exception("Action commit error.");
     }

     A.abort();

     throw new Exception("Read lock error.");
 }

 public boolean save_state (OutputObjectState os, int ot)
 {
     boolean result = super.save_state(os, ot);

     if (!result)
         return false;

     try
     {
         os.packInt(state);
     }
     catch (IOException e)
     {
         result = false;
     }

     return result;
 }

 public boolean restore_state (InputObjectState os, int ot)
 {
     boolean result = super.restore_state(os, ot);

     if (!result)
         return false;

     try
     {
         state = os.unpackInt();
     }
     catch (IOException e)
     {
         result = false;
     }

     return result;
 }

 public String type ()
 {
     return "/StateManager/LockManager/AtomicObject";
 }

 private int state;
}

Yes, quite a bit of code for a transactional integer. But if you consider what's going on here and why, such as setting locks and creating potentially nested transactions within each method, it starts to make sense. So how would we use this class? We'll let's just take a look at a unit test to see:

     AtomicObject obj = new AtomicObject();
    AtomicAction a = new AtomicAction();

    a.begin();

    obj.set(1234);

    a.commit();

    assertEquals(obj.get(), 1234);

    a = new AtomicAction();

    a.begin();

    obj.incr(1);

    a.abort();

    assertEquals(obj.get(), 1234);

But we can do a lot better in terms of ease of use, especially if you consider what's behind STM: where objects are volatile (don't survive machine crashes). And this is where our approach comes in. Let's look at the example above and create an interface for the AtomicObject:

public interface Atomic
{
   public void incr (int value) throws Exception;

   public void set (int value) throws Exception;

   public int get () throws Exception;
}

And now we can create an implementation of this:

@Transactional
public class ExampleSTM implements Atomic
{
   @ReadLock
   public int get () throws Exception
   {
       return state;
   }

   @WriteLock
   public void set (int value) throws Exception
   {
       state = value;
   }

   @WriteLock
   public void incr (int value) throws Exception
   {
       state += value;
   }

   @TransactionalState
   private int state;
}

Here we now simply use annotations to specify what we want. The @Transactional is needed to indicate that this will be a transactional object. Then we use @ReadLock or @WriteLock to indicate the types of locks that we need on a per method basis. (If you don't define these then the default is to assume @WriteLock). And we use @TransactionalState to indicate to the STM implementation which state we want to operate on and have the isolation and atomicity properties (remember, with STM there's no requirement for the D in ACID). If there's more state in the implementation than we want to recover (e.g., it can be recomputed on each method invocation) then we don't have to annotate it.

But that's it: the AtomicObject and ExampleSTM classes are identical. So let's take a look at another unit test:

      RecoverableContainer theContainer = new RecoverableContainer();
     ExampleSTM basic = new ExampleSTM();
     Atomic obj = obj = theContainer.enlist(basic);    
     AtomicAction a = new AtomicAction();
  
     a.begin();
  
     obj.set(1234);
  
     a.commit();

     assertEquals(obj.get(), 1234);
  
     a = new AtomicAction();

     a.begin();

     obj.incr(1);
  
     a.abort();

     assertEquals(obj.get(), 1234);

Think of the RecoverableContainer as the unit of software transactional memory, within which we can place objects that it will manage. In this case ExampleSTM. Once placed, we get back a reference to the instance within the unit of memory, and as long as we use this reference from that point forward we will have the A, C and I properties that we want.

Ever wondered about transactions and threads?

No? Well you really should! When transaction systems were first developed they were single-threaded (where a thread is defined to be an entity which performs work, e.g., a lightweight process, or an operating-system process.) Executing multiple threads within a single process was a novelty! In such an environment the thread terminating the transaction is, by definition, the thread that performed the work. Therefore, the termination of a transaction is implicitly synchronized with the completion of the transactional work: there can be no outstanding work still going on when the transaction starts to finish.

With the increased availability of both software and hardware multi-threading, transaction services are now being required to allow multiple threads to be active within a transaction (though it’s still not mandated anywhere, so if this is something you want then you may still have to look around the various implementations). In such systems it is important to guarantee that all of these threads have completed when a transaction is terminated, otherwise some work may not be performed transactionally.

Although protocols exist for enforcing thread and transaction synchronization in local and distributed environments (commonly referred to as checked transactions), they assume that communication between threads is synchronous (e.g., via remote procedure call). A thread making a synchronous call will block until the call returns, signifying that any threads created have terminated. However, a range of distributed applications exists (and yours may be one of them) which require extensive use of concurrency in order to meet real-time performance requirements and utilize asynchronous message passing for communication. In such environments it is difficult to guarantee synchronization between threads, since the application may not communicate the completion of work to a sender, as is done implicitly with synchronous invocations.

As we’ve just seen, applications that do not create new threads and only use synchronous invocations within transactions implicitly exhibit checked behavior. That is, it is guaranteed that whenever the transaction ends there can be no thread active within the transaction which has not completed its processing. This is illustrated below, in which vertical lines indicate the execution of object methods, horizontal lines message exchange, and the boxes represent objects.

The figure illustrates a client who starts a transaction by invoking a synchronous ‘begin’ upon a transaction manager. The client later performs a synchronous invocation upon object a that in turn invokes object b. Each of these objects is registered as being involved in the transaction with the manager. Whenever the client invokes the transaction ‘end’ upon the manager, the manager is then able to enter into the commit protocol (of which only the final phase is shown here) with the registered objects before returning control to the client.

However, when asynchronous invocation is allowed, explicit synchronization is required between threads and transactions in order to guarantee checked (safe) behavior. The next figure illustrates the possible consequences of using asynchronous invocation without such synchronization. In this example a client starts a transaction and then invokes an asynchronous operation upon object a that registers itself within the transaction as before. a then invokes an asynchronous operation upon object b. Now, depending upon the order in which the threads are scheduled, it’s possible that the client might call for the transaction to terminate. At this point the transaction coordinator knows only of a’s involvement within the transaction so enters into the commit protocol, with a committing as a consequence. Then b attempts to register itself within the transaction, and is unable to do so. If the application intended the work performed by the invocations upon a and b to be performed within the same transaction, this may result in application-level inconsistencies. This is what checked transactions are supposed to prevent.

Some transaction service implementations will enforce checked behavior for the transactions they support, to provide an extra level of transaction integrity. The purpose of the checks is to ensure that all transactional requests made by the application have completed their processing before the transaction is committed. A checked transaction service guarantees that commit will not succeed unless all transactional objects involved in the transaction have completed the processing of their transactional requests. If the transaction is rolled back then a check is not required, since all outstanding transactional activities will eventually rollback if they are not told to commit.

As a result, most (though not all) modern transaction systems provide automatic mechanisms for imposing checked transactions on both synchronous and asynchronous invocations. In essence, transactions must keep track of the threads and invocations (both synchronous and asynchronous) that are executing within them and whenever a transaction is terminated, the system must ensure that all active invocations return before the termination can occur and that all active threads are informed of the termination. This may sound simple, but believe us when we say that it isn’t!

Unfortunately this is another aspect of transaction processing that many implementations ignore. As with things like interposition (for performance) and failure recovery, it is an essential aspect that you really cannot do without. Not providing checked transactions is different from allowing checking to be disabled, which most commercial implementations support. In this case you typically have the ability to turn checked transactions off when you know it is safe, to help improve performance. If you think there is the slightest possibility you’ll be using multiple threads within the scope of a single transaction or may make asynchronous transactional requests, then you’d better find out whether your transaction implementation it up to the job. I'm not even going to bother about the almost obligatory plug for JBossTS here ;-)