Tuesday, October 18, 2011

nested transactions 101

You wait ages for an interesting technical problem, then you get the same one twice in as many weeks. If you are a programmer, you now write a script so that you don't have to do any manual work the third time you encounter the problem. If you are a good programmer, you just use the script you wrote the previous week.

When applied to technical support questions, this approach results in the incremental creation of a FAQ. My last such update was way back in April, on the topic of handling XA commit race conditions between db updates and JMS messages. Like I said, you wait ages for an interesting problem, but another has finally come along. So, this FAQ update is all about nested transactions.

Mark has posted about nested transaction before, back in 2010 and 2009. They have of course been around even longer than that and JBossTS/ArjunaTS has support for them that actually pre-dates Java - it was ported over from C++. So you'd think people would have gotten the hang of it by now, but not so. Nested transactions are still barely understood and even less used.

Let's deal with the understanding bit first. Many people use the term 'nested transactions' to mean different things. A true nested transaction is used mainly for fault isolation of specific tasks within a wider transaction.

tm.begin();
doStuffWithOuterTransaction();
tm.begin();
try {
doStuffWithInnerTransaction();
tm.commit();
} catch(Exception e) {
handleFailureOfInnerTransaction();
}
doMoreStuffWithOuterTransaction();
tm.commit();

This construct is useful where we have some alternative way to achieve the work done by the inner transaction and can call it from the exception handler. Let's try a concrete example:

tm.begin(); // outer tx
bookTheatreTickets();
tm.begin(); // inner tx
try {
bookWithFavoriteTaxiCompany();
tm.commit(); // inner tx
} catch(Exception e) {
tm.begin(); // inner tx
bookWithRivalTaxiFirm();
tm.commit(); // inner tx
}
bookRestaurantTable();
tm.commit(); // outer tx

So, when everything goes smoothly you have behaviour equivalent to a normal flat transaction. But when there is minor trouble in a non essential part of the process, you can shrug it off and make forward progress without having to start over and risk losing your precious theatre seats.

As it turns out there are a number of reasons this a not widely used.

Firstly, it's not all that common to have a viable alternative method available for the inner update in system level transactions. It's more common for business process type long running transactions, where ACID is frequently less attractive than an extended tx model such as WS-BA anyhow. What about the case where you have no alternative method, don't care if the inner tx fails, but must not commit its work unless the outer transaction succeeds? That's what afterCompletion() is for.

Secondly, but often of greater practical importance, nested transactions are not supported by any of the widely deployed databases, message queuing products or other resource managers. That severely limits what you can do in the inner transaction. You're basically limited to using the TxOJ resource manager bundled with JBossTS, as described in Mark's posts. Give up any thought of updating your database conditionally - it just won't work. JDBC savepoints provide somewhat nested transaction like behaviour for non-XA situations, but they don't work in XA situations. Nor does the XA protocol, foundation of the interoperability between transaction managers and resource managers, provide any alternative. That said, it's theoretically possible to fudge things a bit. Let's look at that example again in XA terms.

tm.begin(); // outer tx
bookTheatreTickets(); // enlist db-A.
tm.begin(); // inner tx
try {
bookWithFavoriteTaxiCompany(); // enlist db-B.
tm.commit(); // inner tx - prepare db-B. Don't commit it though. Don't touch db-A.
} catch(Exception e) {
// oh dear, the prepare on db-B failed. roll it back. Don't rollback db-A though.
tm.begin(); // inner tx
bookWithRivalTaxiFirm(); // enlist db-C
tm.commit(); // inner tx - prepare db-C but don't commit it or touch db-A
}
bookRestaurantTable(); // enlist db-D
tm.commit(); // outer tx - prepare db-A and db-D. Commit db-A, db-C and db-D.

This essentially fakes a nested transaction by manipulating the list of resource managers in a single flat transaction - we cheated a bit by removing db-B part way through, so the tx is not true ACID across all the four participants, only three. JBossTS does not support this, because it's written by purists who think you should use an extended transaction model instead. Also, we don't want to deal with irate users whose database throughput has plummeted because of the length of time that locks are being held on db-B and db-C.

Fortunately, you may not actually need true nested transactions anyhow. There is another sort of nested transaction, properly known as nested top-level, which not only works with pretty much any environment, but is also handy for many common use cases.

The distinction is founded on the asymmetry of the relationship between the outer and inner transactions. For true nested transactions, failure of the inner tx need not impact the outcome of the outer tx, whilst failure of the outer tx will ensure the inner tx rolls back. For nested top-level, the situation is reversed: failure of the outer transaction won't undo the inner tx, but failure of the inner tx may prevent the outer one from committing. Sound familiar? The most widely deployed use case for nested top-level is ensuring that an audit log entry of the processing attempt is made, regardless of the outcome of the business activity.

tm.begin();
doUnauditedStuff();
writeAuditLogForProcessingAttempt();
doSecureBusinessSystemUpdate();
tm.commit();

The ACID properties of the flat tx don't achieve what we want here - the audit log entry must be created regardless of the success or failure of the business system update, whereas we have it being committed only if the business system update also commits. Let's try that again:

tm.begin(); // tx-A
doUnauditedStuff();
Transaction txA = tm.suspend();
tm.begin(); // new top level tx-B
try {
writeAuditLogForProcessingAttempt();
tm.commit(); // tx-B
} catch(Exception e) {
tm.resume(txA);
tm.rollback(); // tx-A
return;
}
tm.resume(txA);
doSecureBusinessSystemUpdate();
tm.commit(); // tx-A

Well, that's a little better - we'll not attempt the business logic processing unless we have first successfully written the audit log, so we're guaranteed to always have a log of any update that does take place. But there is a snag: the audit log will only show the attempt, not the success/failure outcome of it. What if that's not good enough? Let's steal a leaf from the transaction optimization handbook: presumed abort.

tm.begin(); // tx-A
doUnauditedStuff();
Transaction txA = tm.suspend();
tm.begin(); // new top level tx-B
try {
writeAuditLogForProcessingAttempt("attempting update, assume it failed");
tm.commit(); // tx-B
} catch(Exception e) {
tm.resume(txA);
tm.rollback(); // tx-A
return;
}
tm.resume(txA);
doSecureBusinessSystemUpdate();
writeAuditLogForProcessingAttempt("processing attempt completed successfully");
tm.commit(); // tx-A

So now we have an audit log will always show an entry and always show if it succeeded or not. Also, I'll hopefully never have to answer another nested transaction question from scratch. Success all round I'd say.

memristor based logs

Long time readers will recall that I've been tinkering with shiny toys in the form of SSDs, trying to assess how changes in storage technology cause changes in the way transaction logging should be designed. SSDs are here now, getting cheaper all the time and therefore becoming more 'meh' by the minute. So, I need something even newer and shinier to drool over...

Enter memristors, arguably the coolest tech to emerge from HP since the last Total-e-Server release. Initially intended to complete with flash, memristor technology also has the longer term potential to give us persistent RAM. A server with a power-loss tolerant storage mechamism that runs at approximately main memory speed will fundamentally change the way we think about storage hierarchies, process state and fault tolerance.

Until now the on-chip cache hierarchy and off-chip RAM have both come under the heading of 'volatile', whilst disk has been considered persistent, give or take a bit of RAID controller caching.

Volatile storage is managed by the memory subsystem, with the cache hierarchy within that tier largely transparent to the O/S much less the apps. Yes, performance gurus will take issue with that - understanding cache coherency models is vital to getting the best out of multi-core chips and multi-socket servers. But by and large we don't control it directly - MESI is hardcoded in the CPU and we only influence it with simple primitives - memory fencing, thread to core pinning and such.

Persistent storage meanwhile is managed by the file system stack - the O/S block cache, disk drivers, RAID controllers, on-device and on-controller caches etc. As more of it is in software we have a little more control over the cache model, by O_DIRECT, fsync, firmware config tweaking and such. Most critically, we can divide the persistent storage into different pools with different properties. The best known example is the age old configuration suggestion for high performance transaction processing: put the log storage on a dedicated device.

So what will the programming model look like when we have hardware that offers persistent RAM, either for all the main memory or, more likely in the medium term, for some subset of it? Will the entire process state survive a power loss at all cache tiers from the on-CPU registers to the disk platters, or will we need fine grained cache control to say 'synchronously flush from volatile RAM to persistent RAM', much as we currently force a sync to disk? How will we resume execution after a power loss? Will we need to explicitly reattach to our persistent RAM and rebuild our transient data structures from its contents, or will the O/S magically handle it all for us? Do we explicitly serialize data moving between volatile and non-volatile RAM, as we currently do with RAM to disk transfers, or is it automatic as with cache to RAM movements? What does this mean for us in terms of new language constructs, libraries and design patterns?

Many, many interesting questions, the answers to which will dictate the programming landscape for a new generation of middleware and the applications that use it. The shift from HDD to SSD may seem minor in comparison. Will everything we know about the arcane niche of logging and crash recovery become obsolete, or become even more fundamental and mainstream? Job prospects aside, on this occasion I'm leaning rather in favour of obsolete.

Sunday, September 25, 2011

Workshop on the Theory of Transactional Memory

I'm just back from the WTTM in Rome. It was good to go to the workshop and listen to presentations from the likes of Maurice Herlihy on advances in both hardware and software transactional memory. I was there representing the Cloud-TM work that we're doing in collaboration with others and specifically around Infinispan and JBossTS.

One of my favourite presentations of the event was on the use of pessimistic concurrency control in STM. Most STM approaches use an optimistic approach, which obviously works well in cases where contention is limited. The problem with pessimistic is retaining locks for the duration even when updates are infrequent. However, as the presenter showed, with suitable a priori ordering, pessimistic can be more efficient even in low contention environments. This was good because we tend to use pessimistic concurrency control in JBossTS a lot, and as was shown in the presentation, it's also a lot easier for developers to understand and reason about.

Unfortunately I had to leave the workshop before the last session, so I didn't get to listen to Torvald who represents Red Hat on the transactional C++ working group. That's a shame, because I've been following this effort for several years and know that we may be seeing this in gcc sometime soon. And who knows, maybe EE8 will see some STM eventually.

Friday, August 12, 2011

RESTful Transactions in the Cloud: Now they're for real

A few months back Mark posted on the subject of REST, transactions and the cloud. Well now you can test the validity of his claims for yourselves. RedHat recently announced their OpenShift initiative for deploying applications into the cloud and we have put together a demonstration to show how easy it is to perform end to end transactional requests whilst remaining RESTful/RESTlike throughout and using little more than a JAX-RS container, an HTTP client library and a transaction coordinator.

As our starting point we use an implementation of the draft specification for a RESTful interface to 2-Phase-Commit transactions which we refer to as REST-AT.

The demonstration starts out by showing local clients (written in various languages such as javascript and Ruby) and local (JAX-RS) web services interacting transactionally using HTTP for all communications. The transaction coordinator that implements REST-AT is written using JAX-RS and also runs locally.

The push to cloud based deployments demands that providers support piecemeal migration of applications into the cloud infrastructure. REST-AT together with OpenShift makes the realisation of this goal, at least for transactional services, a relatively painless exercise. The demonstration code and accompanying video shows how to migrate the transaction coordinator component leaving clients and services outside of the infrastructure. We achieve this by using the OpenShift JBoss AS7 cartridge and deploy REST-AT as a war into the application server that this cartridge provides. In a later post we will show how to migrate Java and Ruby services into the OpenShift infrastructure using the AS7 and Ruby cartridges, respectively.

To begin using OpenShift a good starting point would be to watch the various getting started videos (look for the Zero to Cloud in 30 minutes track or go directly to the demo video). To start using REST-AT check out the quickstarts and the sources for the demo.

Using JBossTS in OpenShift Express

Hi,

I just wanted to let you know that you can use transactions in Red Hat's new OpenShift Express tool. Having had some hand's on time with the system I can safely say its one of the most exciting new projects I have seen in a while.

It makes deploy applications into the cloud so easy, plus as it supports deploying applications onto AS7 you can benefit from all the cool features that have been added there.

As AS7 features transactions (naturally!) I decided to kick the tyres of Openshift Express a little harder by not just using JPA, EJB, JTA and JSF, but also utilise one of the JBoss Transactions extension features - TXOJ. You can watch my video of this here, and the source code is available here.

Safe to say it was extremely simple to get this up and running, if you follow the video you can too!

For those who don't like videos though, here's a little cheat sheet I put together to get it running:

1. You need an openshift express domain, go to: http://openshift.redhat.com to get one
2. rhc-create-app -a txapp -t jbossas-7.0
3. cd txapp
4. svn export https://anonsvn.jboss.org/repos/labs/labs/jbosstm/trunk/ArjunaJTA/quickstarts/jee_transactional_app
5. mv jee_transactional_app/* .
6. rmdir jee_transactional_app
7. Add an openshift profile to your pom.xml:
   <profile>
<id>openshift</id>
<build>
<plugins>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<outputDirectory>deployments</outputDirectory>
<warName>ROOT</warName>
</configuration>
</plugin>
</plugins>
</build>
</profile>
8. git add src pom.xml
9. git commit -a -m “jee_transactional_app” && git push
10. Open up firefox and go to http://txapp-<YOUR_DOMAIN>.openshift.redhat.com/jee_transactional_app/

I hope you enjoy using Openshift Express and also can add a few Transactional Objects to your applications!

Tom

Friday, July 22, 2011

norecoveryxa

"Infamy! Infamy! They've all got it in for me!"

Of all the error and warning messages in JBossTS, none is more infamous than norecoveryxa. Pretty much every JBossTS user has an error log containing this little gem. Usually more than once. A lot more. But not for much longer. It has incurred my wrath and its days are numbered. I've got it in for that annoying little beast.


[com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa]
Could not find new XAResource to use for recovering non-serializable XAResource


Transaction logs preserve information needed to complete interrupted transactions after a crash. Boiled down to pure essence, a transaction log is a list of the Xids involved in a transaction. An Xid is simply a bunch of bytes, beginning with a unique tx id and ending with a branch id that is (usually) different for each resource manager in the transaction. During recovery, these Xids are used to tell the resource managers to commit the in-doubt transactions. All of which is fine in theory, but an utter pain in practice.

The core problem is that an Xid is not a lot of use unless you can actually reconnect to the resource manager and get an XAResource object from its driver, as it is the methods on that object to which you pass the Xid. So you need some way of storing connection information also. In rare cases the RM's driver provides a Serializable XAResource implementation, in which case the simplest (although not necessarily fastest) thing to do is serialize it into the tx log file. Recovery is then easy - deserialize the XAResource and invoke commit on it. Well, except for the whole multiple classloaders thing, but that's another story. Besides, hardly any drivers are actually so accommodating. So, we need another approach.

To deal with non-serializable XAResources, the recovery system allows registration of plugins, one per resource manager, that provide a new XAResource instance on demand. In olden times configuring these was a manual task, frequently overlooked. As a result the recovery system found itself unable to get a new XAResource instance to handle the Xid from the transaction logs. Hence norecoveryxa. That problem was solved, at least for xa-datasource use within JBossAS, by having the datasource deployment handler automatically register a plugin with the recovery system. Bye-bye AppServerJDBCXARecovery, you will not be missed.

That leaves cases where the resource manager is something other than an XA aware JDBC datasource. To help diagnose such cases it would be helpful to have something more than a hex representation of the Xid. Whilst the standard XAResource API does not allow for such metadata, we now have an extension that provides the transaction manager with RM type, version and JNDI binding information. This gets written to the tx logs and, in the case of the JNDI name, encoded into the Xid.

As a side benefit, this information allows for much more helpful debug messages and hopefully means I spend less time on support cases. Didn't really think I was doing all this just for your benefit did you?

before:

TRACE: XAResourceRecord.topLevelPrepare for < formatId=131076,
gtrid_length=29, bqual_length=28, tx_uid=0:ffff7f000001:ccd6:4e0da81f:2,
node_name=1, branch_uid=0:ffff7f000001:ccd6:4e0da81f:3>


after:
TRACE: XAResourceRecord.topLevelPrepare for < formatId=131076,
gtrid_length=29, bqual_length=28, tx_uid=0:ffff7f000001:ccd6:4e0da81f:2,
node_name=1, branch_uid=0:ffff7f000001:ccd6:4e0da81f:3>,
product: OracleDB/11g, jndiName: java:/jdbc/TestDB >


The main reason we want the RM metadata though is so that when recovery tries and fails to find an XAResource instance to match the Xid, it can now report the details of the RM to which that Xid belongs. This makes it more obvious which RM is unavailable e.g. because it's not registered for recovery or it is temporarily down.


WARN: ARJUNA16037: Could not find new XAResource to use for recovering
non-serializable XAResource XAResourceRecord < resource:null,
txid:< formatId=131076, gtrid_length=29, bqual_length=41,
tx_uid=0:ffff7f000001:e437:4e294bb6:6, node_name=1,
branch_uid=0:ffff7f000001:e437:4e294bb6:7, eis_name=java:/jdbc/DummyDB >,
heuristic: TwoPhaseOutcome.FINISH_OK,
product: DummyProductName/DummyProductVersion,
jndiName: java:/jdbc/DummyDB>


Despite the name change (part of the new i18n framework), that's still our old nemesis norecoveryxa. It's made of stern stuff and, although now a little more useful, is not so easily vanquished entirely.

Even with all the resource managers correctly registered, it's still possible to be unable to find the Xid during a recovery scan. That's because of a timing window in the XA protocol, where a crash occurs after the RM has committed and forgotten the branch but before the TM has disposed of its no longer needed log file. In such cases no RM will claim ownership of the branch during replay, leaving the TM to assume that the owning RM is unavailable for some reason. With the name of the owning RM to hand, we now have the possibility to match against the name of the scanned RMs, conclude the branch belongs to one of the scanned RMs even though it pleaded ignorance, and hence dispose of it as a safely committed branch.

All of which should ensure that norecoveryxa's days in the spotlight are severely numbered.

Friday, July 8, 2011

rethinking log storage

Transaction coordinators like JBossTS must write log files to fault tolerant storage in order to be able to guarantee correct completion of transactions in the event of a crash. Historically this has meant using files hosted on RAIDed hard disks.

The workload is characterised by near 100% writes, reads for recovery only, a large number of small (sub 1 KB) operations and, critically, the need to guarantee that the data has been forced through assorted cache layers to non-volatile media before continuing.

Unfortunately this combination of requirements has a tendency to negate many of the performance optimizations provided in storage systems, making log writing a common bottleneck for high performance transaction systems. Addressing this can often force users into unwelcome architectural decisions, particularly in cloud environments.

So naturally transaction system developers like the JBossTS team spend a lot of time thinking about how to mange log I/O better. Those who have read or watched my JUDCon presentation will already have some hints of the ideas we're currently looking at, including using cluster based in-memory replication instead of disk storage.

More recently I've been playing with SSD based storage some more, following up on earlier work to better understand some of the issues that arise as users transition to a new generation of disk technology with new performance characteristics.

As you'll recall, we recently took the high speed journalling code from HornetQ and used it as the basis for a transaction log. Relatively speaking, the results were a pretty spectacular improvement over our classic logging code. In absolute terms however we still weren't getting the best out the hardware.

A few weeks back I got my grubby paws on one of the latest SSDs. On paper its next generation controller and faster interface should have provided a substantial improvement over its predecessor. Not so for my transaction logging tests though - in keeping with tradition it utterly failed to outperform the older generation of technology. On further investigation the reasons for this become clear.

Traditional journalling solutions are based on a) aggregating a large number of small I/Os into a much smaller number of larger I/Os so that the drive can keep up with the load and b) serializing these writes to a single append-only file in order to avoid expensive head seeks.

With SSDs the first of those ideas is still sound, but the number of I/O events the drive can deal with is substantially higher. This requires re-tuning the journal parameters. For some usage it even becomes undesirable to batch the I/Os - until the drive is saturated with requests it's just pointless overhead that delays threads unnecessarily. As those threads (transactions) may have locks on other data any additional latency is undesirable. Also, unlike some other systems, a transaction manager has one thread per tx, as it's the one executing the business logic. It can't proceed until the log write completes, so write batching solutions involve significant thread management and scheduling overhead and often have a large number of threads parked waiting on I/O.

There is another snag though: even where the journalling code can use async I/O to dispatch multiple writes to the O/S in parallel, the filesystem still runs them sequentially as they contend on the inode semaphore for the log file. Thus writes for unrelated transactions must wait unnecessarily, much like the situation that arises in business applications which uses too coarse-grained locking. Also, the nice ncq for the drive remains largely unused, limiting the ability of the drive controller to execute writes in parallel.

The serialization of writes to the hardware, whilst useful for head positioning on HDDs, is a painful mistake for SSDs. These devices, like a modern CPU, require high concurrency in the workload to perform at their best. So, just as we go looking for parallelization opportunities in our apps, so we must look for them in the design of the I/O logging.

The most obvious solution is to load balance the logging over several journal files when running on an SSD. It's not quite that simple though - care must be taken to avoid a filesystem journal write as that will trigger a buffer flush for the entire filesystem, not just the individual file. Not to mention contention on the filesystem journal. For optimal performance it may pay to put each log file on its own small filesystem. But I'm getting ahead of myself. First there is the minor matter of actually writing a prototype load balancer to test the ideas. Any volunteers?