JBossTS team blog

connecting the dots

2012-01-12T10:36:00.002Z

Hmm, looks like it's 2012 already, which means we're overdue for another rant. Today's lecture topic is the perils of trying to replace or do without bits of code that you don't understand.

Programming in the large is all about abstraction, loose coupling and separation of concerns. Java EE promotes the ability to use complex functionality without understanding how it is implemented. As long as you follow the instructions this works surprisingly well. With an app server at your disposal you require only the sketchiest understanding of distributed transactions to write code that consumes a message and updates a database in an ACID fashion. Which on the whole is a good thing, since not all that many people actually want to understand all the plumbing that makes it work. There is a snag though: this apparent simplicity can cause some users to think the Java EE container is not doing all that much work and can easily be replaced or done without. Likewise it makes it difficult to tell the difference between a full fledged container that will support the functionality you need and a cut down framework or lightweight container that may not.

Let's debunk a few myths...

Spring is not a JTA.

It's an abstraction layer on top of a transaction manager, not an implementation of one. For transactions involving only a single resource, it simply delegates the hard bits to that resource manager - so called 'native transactions'. Simple, fast and mostly adequate if you need only a single resource. You don't get transaction lifecycle events, but you're probably using an ORM that provides the equivalent of the beforeCompletion hook for its cache anyhow.

For transactions with more than one resource, you need to wire in a real transaction manager to Spring. You can do this with bitronix, atomikos or JBossTS, usually just by specifying the right TransactionManager and UserTransaction implementation bean classes. But your problems don't end there, because...

Spring is not a JCA.

The roles, responsibilities and relationship between the JTA and JCA components of an app server are critical considerations when you're trying to do without one. The JTA manages transaction lifecycle - begin/commit/rollback. Most importantly, it make appropriate calls on any enlisted XAResources as the transaction progresses. But here is the bit that most users don't pay attention to: A JTA does not magically know what resources you want to participate in the transaction. Telling it that is the JCA's job.

In a full on app server, the JCA manages connections to resource managers such as databases and message queues. If you deploy those drivers/connectors in a manner that identifies them as XA enabled, the JCA ensures that they are correctly associated with the transaction. Application code simply e.g. looks up the JNDI name for a connection pool and calls getConnection(). The JCA intercepts the call, get the XAResource for the connection and passes it to the transaction manager.

In some cases you don't need a full JCA. You can often make do with a transaction manager aware XA connection pool, which is essentially a subset of the JCA functionality. But you can't get away with only an XA aware driver or a non-XA connection pool. Trying to do that leads to some interesting behaviour: your app will deploy and run, but you have a transaction and a connection that know nothing about one another. Committing or rolling back the transaction won't commit or rollback the work in the database. oops.

So, you also need to wire in a JCA or suitable connection pooling implementation. Most 'standalone' JTA implementations ship with a simple connection management solution that is suitable for light use. The one in JBossTS is called the TransactionalDriver. For serious deployments you want IronJacamar or some other JCA that has robust and fast connection management.

So now you have wired up your JTA and JCA in Spring, but you are still not done because...

The standard contract between JTA and JCA does not include recovery management setup. Wiring up resources for crash recovery requires a proprietary solution that differs for each transaction manager. The connection manager that ships with the transaction manager may do this more or less automatically, but third party JCAs or XA aware connection pools probably won't. So, go read the transaction manager documentation and write a few test cases.

phew, that was a lot of work, wasn't it? Spring is good at what it does, but its not an out of box replacement for all the transactional plumbing in a Java EE app server. Nor is tomcat - you'll have much the same steps to perform there. In both cases it is possible, but not as simple as unzipping and starting JBossAS. Do you really want to make your life harder than it has to be?

Transactional Android coming soon!

2012-01-01T18:14:00.003Z

I spent some time this Christmas porting JBossTS to run on Android. It's pretty much done, with the exception of a few workarounds that I need to fix properly over the next few weeks, when I find the time. But once I check this code into the repository, you'll be able to write your own transactional Android applications. Unfortunately I can't guarantee which version this will be in at the moment, but if it is going to take me too long to do the merges then I may create a branch in svn that interested people can just pull from directly, with the usual caveats. If time allows, I may say something about this at JUDCon India too!

Transactions making a comeback? They were never away!

2011-12-23T19:06:00.003Z

Over the years that I've been involved with transaction processing theory and practice (way too many years for me to admit!) I've seen transactions used, abused and ignored in a wide variety of applications and situations. I've discussed many of these scars in articles here and elsewhere for decades (ouch! that pains me to even use such a timeframe!) But every new software wave, whether it's Web Services, object-orientation, and even the Web itself, has come to the inevitable conclusion that transactions are needed in some way and in one form or another. Maybe not ACID transactions; maybe it's compensation based, or some other variation on the extended transaction theme.

So it is nice to see that being repeated in the past couple of years with Cloud and NoSQL. If you read this blog enough you'll have heard from myself and others on the fact that although some believe that you need to ditch transactions in order to achieve scalability, others aren't quite convinced that it is always necessary to do so - and I count us amongst the latter group. Complete reliance on transactions is sometimes too much. However, completely ignoring them and pushing consistency and recovery up to the application is sometimes too much as well. This is something that appears to be dawning on a number of transaction-less implementations and hopefully it's a trend that will continue.

Anyone who knows me or has followed things I've written or presented over the years will also know that I think transactions are a great structuring technique for concurrent programming. Of course in distributed systems we tend to take this for granted: you've inherently got multiple clients/users manipulating your data, typically at the same time, so transactions with their ACID properties allow developers to concentrate on the functional aspects of the application or service, whilst letting the transaction system do the hard work on ensuring isolation and consistency in the presence of these concurrent users (and possibly failures).

But distribution isn't the only place where you get concurrency, and particularly these days with multi-core processors. Back in the last century (ouch!) some of us in academia and industry, had access to multi-processor machines which weren't as common as you might expect (they were expensive!) But when you had them it quickly became apparent that transactions could help with developing applications that had no distribution in them but were (massively) parallel. From this early work techniques such as software transactional memory were born. Back then it was more of an edge case and people found it hard to understand why you'd need transactions without distribution or even a database involved. Well obviously the advances in hardware have silenced most of those critics and we're seeing more and more vendors, open source projects, languages etc. looking at transactions are a fundamental component and not just an add-on for some scenarios.

So what does all of this mean? Well first of all I think it's great to see transactions continuing to have an impact in these new waves. Fundamental requirements like fault tolerance, concurrency control, security etc. are fundamental for a reason. Secondly I think it's fair to say that as with previous waves, we'll see transaction theory and implementations adapt and change to better address some of the new concerns and requirements that are bound to arise. I'm excited by all of this, as I am whenever there's something new to apply transactions. I'm also excited by the fact that JBossTS (implementation and team) is well placed to help drive some of this as we've done for ... well ... let's just say for quite a long time and leave it at that!

my last commit

2011-12-19T14:00:00.002Z

Today I created the tag for the JBossTS 4.16.0.Final release, thereby making my last commit to the JBossTS repository as team lead.

It is the end of an era in more ways than one, as 4.16 is planned to be the last feature release for the 4.x line and the last for the JBossTS brand.

Starting in the new year, the project gets a new name, Narayana, a new major version, 5.0, and a new development team lead, Tom Jenkinson. So, exciting times ahead for the project in 2012 and beyond, with fresh blood and fresh ideas.

As for myself, I start 2012 with a shiny new JBoss role looking at Big Data and noSQL. Who knows, there may even be a new blog for you to subscribe to. But first there is the small matter of a seasonal vacation...

Merry Christmas

Jonathan.

HPTS 2011

2011-11-11T15:53:00.002Z

I've uploaded most of the presentations and posters sessions for HPTS 2011 to the website now. If you interested in transactions, NoSQL, eventual consistency, data provenance and a whole raft of topical subjects, then you really should check them out!

unnecessary

2011-11-10T13:44:00.003Z

Hot on the heals of thinking about information quality, I came across this little gem regarding Spring and JPA configuration:

"Using multiple session factories in a logical transaction imposes challenging issues concerning transaction management... We quickly abandoned [JTA transaction management] due to the potential cost and complexity of an XA protocol with two-phase commit. Since most of the modules of an application share the same data source, the imposed cost and complexity are definitively unnecessary."

The authors then go on to describe how they built a custom solution that causes the same database connection to be used by the modules, removing the need for transaction coordination across multiple connections.

"Extending Spring’s transaction management with SessionFactory swapping and the Shared Transaction Resource pattern was a very challenging task."

That last bit should probably have read "challenging and unnecessary task".

A good JCA will automatically track connections enlisted with a JTA transaction and will reuse the already enlisted connection to satisfy a further dataSource.getConnection() request. Further, even if it does enlist multiple connections, the JTA will use isSameRM to detect that they relate to the same resource manager and thus still maintain the one phase commit optimisation. All of these challenging tasks are taken care of for you by the application server.

You probably should not bother to invent a better mousetrap until you've determined that current mousetraps don't catch your mice. The imposed cost and complexity are definitively unnecessary.

musings on peer review

2011-11-10T12:36:00.004Z

I've been reading up in a new field of study recently and as a result have been thinking about information provenance, reputation and collaboration.

Once upon a time, getting up to speed on a new topic in computing science required finding a good library and wading through a stack of conference proceedings and journals. Now it requires an online search and, critically, the ability to evaluate the quality of the massive quantity of material that comes back.

Formal peer review of publications is gone, unable to scale to online use. Meanwhile online systems for review, comment and reputation management are still in their infancy. The best we have right now is an ad-hoc stew of brands, social graphs and distributed conversations.

Those with enough time to invest can build a mental model of a new field that, after the initial investment in learning the landscape, allows them to maintain an ongoing overview of developments with relatively little effort. Those who's work only tangentially involves a deeply technical topic don't have this luxury. They typically perform searches not to learn about the new field in general, but to get specific solutions to a problem outside of their core field of expertise, after which they move on. Such users vastly outnumber the domain experts for niche topics like transactions.

What implications does this new model of communication and information dissemination have for the behaviour of professed experts in technical fields? Clearly our obligation to ensure that information we provide is accurate remains unchanged, and is in any case in our own best interest. Should we consider there to be an implied professional obligation to publish information only in a context that allows feedback to be attached directly to it e.g. blog with comments enabled? Or even allow collaborative editing e.g. wiki? How about taking time out to correct misinformation where we find it - is that now part of our social contract?

Question for the audience: Where do you get your information on transactions, and how do you assess its quality?

pen or pencil for writing logs?

2011-10-20T10:15:00.004+01:00

Those old enough to remember the days of manned space flight in the western world will recall that NASA expended considerable resources to come up with a pen that would work in space. Meanwhile, half a world away, the Russians just used a bunch of pencils.

I can't help feeling that story is going to have a new counterpart in the history books. Whilst HP are working on memristor based persistent RAM, someone else just grafted a bunch of flash and a super-capacitor onto a regular DIMM instead. Now I just need the linux guys to come up with a nice API...

New RAM shunts data into flash in power cuts

nested transactions 101

2011-10-18T17:52:00.004+01:00

You wait ages for an interesting technical problem, then you get the same one twice in as many weeks. If you are a programmer, you now write a script so that you don't have to do any manual work the third time you encounter the problem. If you are a good programmer, you just use the script you wrote the previous week.

When applied to technical support questions, this approach results in the incremental creation of a FAQ. My last such update was way back in April, on the topic of handling XA commit race conditions between db updates and JMS messages. Like I said, you wait ages for an interesting problem, but another has finally come along. So, this FAQ update is all about nested transactions.

Mark has posted about nested transaction before, back in 2010 and 2009. They have of course been around even longer than that and JBossTS/ArjunaTS has support for them that actually pre-dates Java - it was ported over from C++. So you'd think people would have gotten the hang of it by now, but not so. Nested transactions are still barely understood and even less used.

Let's deal with the understanding bit first. Many people use the term 'nested transactions' to mean different things. A true nested transaction is used mainly for fault isolation of specific tasks within a wider transaction.


tm.begin();
doStuffWithOuterTransaction();
tm.begin();
try {
  doStuffWithInnerTransaction();
  tm.commit();
} catch(Exception e) {
  handleFailureOfInnerTransaction();
}
doMoreStuffWithOuterTransaction();
tm.commit();

This construct is useful where we have some alternative way to achieve the work done by the inner transaction and can call it from the exception handler. Let's try a concrete example:


tm.begin(); // outer tx
bookTheatreTickets();
tm.begin(); // inner tx
try {
  bookWithFavoriteTaxiCompany();
  tm.commit(); // inner tx
} catch(Exception e) {
  tm.begin(); // inner tx
  bookWithRivalTaxiFirm();
  tm.commit(); // inner tx
}
bookRestaurantTable();
tm.commit(); // outer tx

So, when everything goes smoothly you have behaviour equivalent to a normal flat transaction. But when there is minor trouble in a non essential part of the process, you can shrug it off and make forward progress without having to start over and risk losing your precious theatre seats.

As it turns out there are a number of reasons this a not widely used.

Firstly, it's not all that common to have a viable alternative method available for the inner update in system level transactions. It's more common for business process type long running transactions, where ACID is frequently less attractive than an extended tx model such as WS-BA anyhow. What about the case where you have no alternative method, don't care if the inner tx fails, but must not commit its work unless the outer transaction succeeds? That's what afterCompletion() is for.

Secondly, but often of greater practical importance, nested transactions are not supported by any of the widely deployed databases, message queuing products or other resource managers. That severely limits what you can do in the inner transaction. You're basically limited to using the TxOJ resource manager bundled with JBossTS, as described in Mark's posts. Give up any thought of updating your database conditionally - it just won't work. JDBC savepoints provide somewhat nested transaction like behaviour for non-XA situations, but they don't work in XA situations. Nor does the XA protocol, foundation of the interoperability between transaction managers and resource managers, provide any alternative. That said, it's theoretically possible to fudge things a bit. Let's look at that example again in XA terms.


tm.begin(); // outer tx
bookTheatreTickets(); // enlist db-A.
tm.begin(); // inner tx
try {
  bookWithFavoriteTaxiCompany(); // enlist db-B.
  tm.commit(); // inner tx - prepare db-B. Don't commit it though. Don't touch db-A.
} catch(Exception e) {
  // oh dear, the prepare on db-B failed. roll it back. Don't rollback db-A though.
  tm.begin(); // inner tx
  bookWithRivalTaxiFirm(); // enlist db-C
  tm.commit(); // inner tx - prepare db-C but don't commit it or touch db-A
}
bookRestaurantTable(); // enlist db-D
tm.commit(); // outer tx - prepare db-A and db-D. Commit db-A, db-C and db-D.

This essentially fakes a nested transaction by manipulating the list of resource managers in a single flat transaction - we cheated a bit by removing db-B part way through, so the tx is not true ACID across all the four participants, only three. JBossTS does not support this, because it's written by purists who think you should use an extended transaction model instead. Also, we don't want to deal with irate users whose database throughput has plummeted because of the length of time that locks are being held on db-B and db-C.

Fortunately, you may not actually need true nested transactions anyhow. There is another sort of nested transaction, properly known as nested top-level, which not only works with pretty much any environment, but is also handy for many common use cases.

The distinction is founded on the asymmetry of the relationship between the outer and inner transactions. For true nested transactions, failure of the inner tx need not impact the outcome of the outer tx, whilst failure of the outer tx will ensure the inner tx rolls back. For nested top-level, the situation is reversed: failure of the outer transaction won't undo the inner tx, but failure of the inner tx may prevent the outer one from committing. Sound familiar? The most widely deployed use case for nested top-level is ensuring that an audit log entry of the processing attempt is made, regardless of the outcome of the business activity.


tm.begin();
doUnauditedStuff();
writeAuditLogForProcessingAttempt();
doSecureBusinessSystemUpdate();
tm.commit();

The ACID properties of the flat tx don't achieve what we want here - the audit log entry must be created regardless of the success or failure of the business system update, whereas we have it being committed only if the business system update also commits. Let's try that again:


tm.begin(); // tx-A
doUnauditedStuff();
Transaction txA = tm.suspend();
tm.begin(); // new top level tx-B
try {
  writeAuditLogForProcessingAttempt();
  tm.commit(); //  tx-B
} catch(Exception e) {
  tm.resume(txA);
  tm.rollback(); // tx-A
  return;
}
tm.resume(txA);
doSecureBusinessSystemUpdate();
tm.commit(); // tx-A

Well, that's a little better - we'll not attempt the business logic processing unless we have first successfully written the audit log, so we're guaranteed to always have a log of any update that does take place. But there is a snag: the audit log will only show the attempt, not the success/failure outcome of it. What if that's not good enough? Let's steal a leaf from the transaction optimization handbook: presumed abort.


tm.begin(); // tx-A
doUnauditedStuff();
Transaction txA = tm.suspend();
tm.begin(); // new top level tx-B
try {
  writeAuditLogForProcessingAttempt("attempting update, assume it failed");
  tm.commit(); //  tx-B
} catch(Exception e) {
  tm.resume(txA);
  tm.rollback(); // tx-A
  return;
}
tm.resume(txA);
doSecureBusinessSystemUpdate();
writeAuditLogForProcessingAttempt("processing attempt completed successfully");
tm.commit(); // tx-A

So now we have an audit log will always show an entry and always show if it succeeded or not. Also, I'll hopefully never have to answer another nested transaction question from scratch. Success all round I'd say.

memristor based logs

2011-10-18T16:49:00.002+01:00

Long time readers will recall that I've been tinkering with shiny toys in the form of SSDs, trying to assess how changes in storage technology cause changes in the way transaction logging should be designed. SSDs are here now, getting cheaper all the time and therefore becoming more 'meh' by the minute. So, I need something even newer and shinier to drool over...

Enter memristors, arguably the coolest tech to emerge from HP since the last Total-e-Server release. Initially intended to complete with flash, memristor technology also has the longer term potential to give us persistent RAM. A server with a power-loss tolerant storage mechamism that runs at approximately main memory speed will fundamentally change the way we think about storage hierarchies, process state and fault tolerance.

Until now the on-chip cache hierarchy and off-chip RAM have both come under the heading of 'volatile', whilst disk has been considered persistent, give or take a bit of RAID controller caching.

Volatile storage is managed by the memory subsystem, with the cache hierarchy within that tier largely transparent to the O/S much less the apps. Yes, performance gurus will take issue with that - understanding cache coherency models is vital to getting the best out of multi-core chips and multi-socket servers. But by and large we don't control it directly - MESI is hardcoded in the CPU and we only influence it with simple primitives - memory fencing, thread to core pinning and such.

Persistent storage meanwhile is managed by the file system stack - the O/S block cache, disk drivers, RAID controllers, on-device and on-controller caches etc. As more of it is in software we have a little more control over the cache model, by O_DIRECT, fsync, firmware config tweaking and such. Most critically, we can divide the persistent storage into different pools with different properties. The best known example is the age old configuration suggestion for high performance transaction processing: put the log storage on a dedicated device.

So what will the programming model look like when we have hardware that offers persistent RAM, either for all the main memory or, more likely in the medium term, for some subset of it? Will the entire process state survive a power loss at all cache tiers from the on-CPU registers to the disk platters, or will we need fine grained cache control to say 'synchronously flush from volatile RAM to persistent RAM', much as we currently force a sync to disk? How will we resume execution after a power loss? Will we need to explicitly reattach to our persistent RAM and rebuild our transient data structures from its contents, or will the O/S magically handle it all for us? Do we explicitly serialize data moving between volatile and non-volatile RAM, as we currently do with RAM to disk transfers, or is it automatic as with cache to RAM movements? What does this mean for us in terms of new language constructs, libraries and design patterns?

Many, many interesting questions, the answers to which will dictate the programming landscape for a new generation of middleware and the applications that use it. The shift from HDD to SSD may seem minor in comparison. Will everything we know about the arcane niche of logging and crash recovery become obsolete, or become even more fundamental and mainstream? Job prospects aside, on this occasion I'm leaning rather in favour of obsolete.

Workshop on the Theory of Transactional Memory

2011-09-25T20:58:00.003+01:00

I'm just back from the WTTM in Rome. It was good to go to the workshop and listen to presentations from the likes of Maurice Herlihy on advances in both hardware and software transactional memory. I was there representing the Cloud-TM work that we're doing in collaboration with others and specifically around Infinispan and JBossTS.

One of my favourite presentations of the event was on the use of pessimistic concurrency control in STM. Most STM approaches use an optimistic approach, which obviously works well in cases where contention is limited. The problem with pessimistic is retaining locks for the duration even when updates are infrequent. However, as the presenter showed, with suitable a priori ordering, pessimistic can be more efficient even in low contention environments. This was good because we tend to use pessimistic concurrency control in JBossTS a lot, and as was shown in the presentation, it's also a lot easier for developers to understand and reason about.

Unfortunately I had to leave the workshop before the last session, so I didn't get to listen to Torvald who represents Red Hat on the transactional C++ working group. That's a shame, because I've been following this effort for several years and know that we may be seeing this in gcc sometime soon. And who knows, maybe EE8 will see some STM eventually.

RESTful Transactions in the Cloud: Now they're for real

2011-08-12T16:28:00.000+01:00

A few months back Mark posted on the subject of REST, transactions and the cloud. Well now you can test the validity of his claims for yourselves. RedHat recently announced their OpenShift initiative for deploying applications into the cloud and we have put together a demonstration to show how easy it is to perform end to end transactional requests whilst remaining RESTful/RESTlike throughout and using little more than a JAX-RS container, an HTTP client library and a transaction coordinator.

As our starting point we use an implementation of the draft specification for a RESTful interface to 2-Phase-Commit transactions which we refer to as REST-AT.

The demonstration starts out by showing local clients (written in various languages such as javascript and Ruby) and local (JAX-RS) web services interacting transactionally using HTTP for all communications. The transaction coordinator that implements REST-AT is written using JAX-RS and also runs locally.

The push to cloud based deployments demands that providers support piecemeal migration of applications into the cloud infrastructure. REST-AT together with OpenShift makes the realisation of this goal, at least for transactional services, a relatively painless exercise. The demonstration code and accompanying video shows how to migrate the transaction coordinator component leaving clients and services outside of the infrastructure. We achieve this by using the OpenShift JBoss AS7 cartridge and deploy REST-AT as a war into the application server that this cartridge provides. In a later post we will show how to migrate Java and Ruby services into the OpenShift infrastructure using the AS7 and Ruby cartridges, respectively.

To begin using OpenShift a good starting point would be to watch the various getting started videos (look for the Zero to Cloud in 30 minutes track or go directly to the demo video). To start using REST-AT check out the quickstarts and the sources for the demo.

Using JBossTS in OpenShift Express

2011-08-12T16:26:00.002+01:00

Hi,

I just wanted to let you know that you can use transactions in Red Hat's new OpenShift Express tool. Having had some hand's on time with the system I can safely say its one of the most exciting new projects I have seen in a while.

It makes deploy applications into the cloud so easy, plus as it supports deploying applications onto AS7 you can benefit from all the cool features that have been added there.

As AS7 features transactions (naturally!) I decided to kick the tyres of Openshift Express a little harder by not just using JPA, EJB, JTA and JSF, but also utilise one of the JBoss Transactions extension features - TXOJ. You can watch my video of this here, and the source code is available here.

Safe to say it was extremely simple to get this up and running, if you follow the video you can too!

For those who don't like videos though, here's a little cheat sheet I put together to get it running:

1. You need an openshift express domain, go to: http://openshift.redhat.com to get one
2. rhc-create-app -a txapp -t jbossas-7.0
3. cd txapp
4. svn export https://anonsvn.jboss.org/repos/labs/labs/jbosstm/trunk/ArjunaJTA/quickstarts/jee_transactional_app
5. mv jee_transactional_app/* .
6. rmdir jee_transactional_app
7. Add an openshift profile to your pom.xml:

   <profile>
    <id>openshift</id>
    <build>
       <plugins>
         <plugin>
           <artifactId>maven-war-plugin</artifactId>
           <configuration>
             <outputDirectory>deployments</outputDirectory>
             <warName>ROOT</warName>
           </configuration>
         </plugin>
       </plugins>
     </build>
   </profile>

8. git add src pom.xml
9. git commit -a -m “jee_transactional_app” && git push
10. Open up firefox and go to http://txapp-<YOUR_DOMAIN>.openshift.redhat.com/jee_transactional_app/

I hope you enjoy using Openshift Express and also can add a few Transactional Objects to your applications!

Tom

norecoveryxa

2011-07-22T13:36:00.000+01:00

"Infamy! Infamy! They've all got it in for me!"

Of all the error and warning messages in JBossTS, none is more infamous than norecoveryxa. Pretty much every JBossTS user has an error log containing this little gem. Usually more than once. A lot more. But not for much longer. It has incurred my wrath and its days are numbered. I've got it in for that annoying little beast.


[com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa]
Could not find new XAResource to use for recovering non-serializable XAResource

Transaction logs preserve information needed to complete interrupted transactions after a crash. Boiled down to pure essence, a transaction log is a list of the Xids involved in a transaction. An Xid is simply a bunch of bytes, beginning with a unique tx id and ending with a branch id that is (usually) different for each resource manager in the transaction. During recovery, these Xids are used to tell the resource managers to commit the in-doubt transactions. All of which is fine in theory, but an utter pain in practice.

The core problem is that an Xid is not a lot of use unless you can actually reconnect to the resource manager and get an XAResource object from its driver, as it is the methods on that object to which you pass the Xid. So you need some way of storing connection information also. In rare cases the RM's driver provides a Serializable XAResource implementation, in which case the simplest (although not necessarily fastest) thing to do is serialize it into the tx log file. Recovery is then easy - deserialize the XAResource and invoke commit on it. Well, except for the whole multiple classloaders thing, but that's another story. Besides, hardly any drivers are actually so accommodating. So, we need another approach.

To deal with non-serializable XAResources, the recovery system allows registration of plugins, one per resource manager, that provide a new XAResource instance on demand. In olden times configuring these was a manual task, frequently overlooked. As a result the recovery system found itself unable to get a new XAResource instance to handle the Xid from the transaction logs. Hence norecoveryxa. That problem was solved, at least for xa-datasource use within JBossAS, by having the datasource deployment handler automatically register a plugin with the recovery system. Bye-bye AppServerJDBCXARecovery, you will not be missed.

That leaves cases where the resource manager is something other than an XA aware JDBC datasource. To help diagnose such cases it would be helpful to have something more than a hex representation of the Xid. Whilst the standard XAResource API does not allow for such metadata, we now have an extension that provides the transaction manager with RM type, version and JNDI binding information. This gets written to the tx logs and, in the case of the JNDI name, encoded into the Xid.

As a side benefit, this information allows for much more helpful debug messages and hopefully means I spend less time on support cases. Didn't really think I was doing all this just for your benefit did you?

before:


TRACE: XAResourceRecord.topLevelPrepare for < formatId=131076,
gtrid_length=29, bqual_length=28, tx_uid=0:ffff7f000001:ccd6:4e0da81f:2,
node_name=1, branch_uid=0:ffff7f000001:ccd6:4e0da81f:3>

after:

TRACE: XAResourceRecord.topLevelPrepare for < formatId=131076,
gtrid_length=29, bqual_length=28, tx_uid=0:ffff7f000001:ccd6:4e0da81f:2,
node_name=1, branch_uid=0:ffff7f000001:ccd6:4e0da81f:3>,
product: OracleDB/11g, jndiName: java:/jdbc/TestDB >

The main reason we want the RM metadata though is so that when recovery tries and fails to find an XAResource instance to match the Xid, it can now report the details of the RM to which that Xid belongs. This makes it more obvious which RM is unavailable e.g. because it's not registered for recovery or it is temporarily down.


WARN: ARJUNA16037: Could not find new XAResource to use for recovering
non-serializable XAResource XAResourceRecord < resource:null,
txid:< formatId=131076, gtrid_length=29, bqual_length=41,
tx_uid=0:ffff7f000001:e437:4e294bb6:6, node_name=1,
branch_uid=0:ffff7f000001:e437:4e294bb6:7, eis_name=java:/jdbc/DummyDB >,
heuristic: TwoPhaseOutcome.FINISH_OK,
product: DummyProductName/DummyProductVersion,
jndiName: java:/jdbc/DummyDB>

Despite the name change (part of the new i18n framework), that's still our old nemesis norecoveryxa. It's made of stern stuff and, although now a little more useful, is not so easily vanquished entirely.

Even with all the resource managers correctly registered, it's still possible to be unable to find the Xid during a recovery scan. That's because of a timing window in the XA protocol, where a crash occurs after the RM has committed and forgotten the branch but before the TM has disposed of its no longer needed log file. In such cases no RM will claim ownership of the branch during replay, leaving the TM to assume that the owning RM is unavailable for some reason. With the name of the owning RM to hand, we now have the possibility to match against the name of the scanned RMs, conclude the branch belongs to one of the scanned RMs even though it pleaded ignorance, and hence dispose of it as a safely committed branch.

All of which should ensure that norecoveryxa's days in the spotlight are severely numbered.

rethinking log storage

2011-07-08T14:00:00.000+01:00

Transaction coordinators like JBossTS must write log files to fault tolerant storage in order to be able to guarantee correct completion of transactions in the event of a crash. Historically this has meant using files hosted on RAIDed hard disks.

The workload is characterised by near 100% writes, reads for recovery only, a large number of small (sub 1 KB) operations and, critically, the need to guarantee that the data has been forced through assorted cache layers to non-volatile media before continuing.

Unfortunately this combination of requirements has a tendency to negate many of the performance optimizations provided in storage systems, making log writing a common bottleneck for high performance transaction systems. Addressing this can often force users into unwelcome architectural decisions, particularly in cloud environments.

So naturally transaction system developers like the JBossTS team spend a lot of time thinking about how to mange log I/O better. Those who have read or watched my JUDCon presentation will already have some hints of the ideas we're currently looking at, including using cluster based in-memory replication instead of disk storage.

More recently I've been playing with SSD based storage some more, following up on earlier work to better understand some of the issues that arise as users transition to a new generation of disk technology with new performance characteristics.

As you'll recall, we recently took the high speed journalling code from HornetQ and used it as the basis for a transaction log. Relatively speaking, the results were a pretty spectacular improvement over our classic logging code. In absolute terms however we still weren't getting the best out the hardware.

A few weeks back I got my grubby paws on one of the latest SSDs. On paper its next generation controller and faster interface should have provided a substantial improvement over its predecessor. Not so for my transaction logging tests though - in keeping with tradition it utterly failed to outperform the older generation of technology. On further investigation the reasons for this become clear.

Traditional journalling solutions are based on a) aggregating a large number of small I/Os into a much smaller number of larger I/Os so that the drive can keep up with the load and b) serializing these writes to a single append-only file in order to avoid expensive head seeks.

With SSDs the first of those ideas is still sound, but the number of I/O events the drive can deal with is substantially higher. This requires re-tuning the journal parameters. For some usage it even becomes undesirable to batch the I/Os - until the drive is saturated with requests it's just pointless overhead that delays threads unnecessarily. As those threads (transactions) may have locks on other data any additional latency is undesirable. Also, unlike some other systems, a transaction manager has one thread per tx, as it's the one executing the business logic. It can't proceed until the log write completes, so write batching solutions involve significant thread management and scheduling overhead and often have a large number of threads parked waiting on I/O.

There is another snag though: even where the journalling code can use async I/O to dispatch multiple writes to the O/S in parallel, the filesystem still runs them sequentially as they contend on the inode semaphore for the log file. Thus writes for unrelated transactions must wait unnecessarily, much like the situation that arises in business applications which uses too coarse-grained locking. Also, the nice ncq for the drive remains largely unused, limiting the ability of the drive controller to execute writes in parallel.

The serialization of writes to the hardware, whilst useful for head positioning on HDDs, is a painful mistake for SSDs. These devices, like a modern CPU, require high concurrency in the workload to perform at their best. So, just as we go looking for parallelization opportunities in our apps, so we must look for them in the design of the I/O logging.

The most obvious solution is to load balance the logging over several journal files when running on an SSD. It's not quite that simple though - care must be taken to avoid a filesystem journal write as that will trigger a buffer flush for the entire filesystem, not just the individual file. Not to mention contention on the filesystem journal. For optimal performance it may pay to put each log file on its own small filesystem. But I'm getting ahead of myself. First there is the minor matter of actually writing a prototype load balancer to test the ideas. Any volunteers?

STM Arjuna

2011-06-26T23:04:00.000+01:00

I'd forgotten how long ago I'd promised to talk about some of the STM work we've been doing. Well I haven't been able to do much more on it for a while, but I do have time to at least outline what's possible at the moment. So let's just remember a bit about the current way in which you can use JBossTS to build transactional POJOs without the need for a database; we'll use an example to illustrate:

  public class AtomicObject extends LockManager
 {
 public AtomicObject()
 {
     super();

     state = 0;

     AtomicAction A = new AtomicAction();

     A.begin();

     if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
     {
         if (A.commit() == ActionStatus.COMMITTED)
             System.out.println("Created persistent object " + get_uid());
         else
             System.out.println("Action.commit error.");
     }
     else
     {
         A.abort();

         System.out.println("setlock error.");
     }
 }

 public void incr (int value) throws Exception
 {
     AtomicAction A = new AtomicAction();

     A.begin();

     if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
     {
         state += value;

         if (A.commit() != ActionStatus.COMMITTED)
             throw new Exception("Action commit error.");
         else
             return;
     }

     A.abort();

     throw new Exception("Write lock error.");
 }

 public void set (int value) throws Exception
 {
     AtomicAction A = new AtomicAction();

     A.begin();

     if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
     {
         state = value;

         if (A.commit() != ActionStatus.COMMITTED)
             throw new Exception("Action commit error.");
         else
             return;
     }

     A.abort();

     throw new Exception("Write lock error.");
 }

 public int get () throws Exception
 {
     AtomicAction A = new AtomicAction();
     int value = -1;

     A.begin();

     if (setlock(new Lock(LockMode.READ), 0) == LockResult.GRANTED)
     {
         value = state;

         if (A.commit() == ActionStatus.COMMITTED)
             return value;
         else
             throw new Exception("Action commit error.");
     }

     A.abort();

     throw new Exception("Read lock error.");
 }

 public boolean save_state (OutputObjectState os, int ot)
 {
     boolean result = super.save_state(os, ot);

     if (!result)
         return false;

     try
     {
         os.packInt(state);
     }
     catch (IOException e)
     {
         result = false;
     }

     return result;
 }

 public boolean restore_state (InputObjectState os, int ot)
 {
     boolean result = super.restore_state(os, ot);

     if (!result)
         return false;

     try
     {
         state = os.unpackInt();
     }
     catch (IOException e)
     {
         result = false;
     }

     return result;
 }

 public String type ()
 {
     return "/StateManager/LockManager/AtomicObject";
 }

 private int state;
}

Yes, quite a bit of code for a transactional integer. But if you consider what's going on here and why, such as setting locks and creating potentially nested transactions within each method, it starts to make sense. So how would we use this class? We'll let's just take a look at a unit test to see:

     AtomicObject obj = new AtomicObject();
    AtomicAction a = new AtomicAction();

    a.begin();

    obj.set(1234);

    a.commit();

    assertEquals(obj.get(), 1234);

    a = new AtomicAction();

    a.begin();

    obj.incr(1);

    a.abort();

    assertEquals(obj.get(), 1234);

But we can do a lot better in terms of ease of use, especially if you consider what's behind STM: where objects are volatile (don't survive machine crashes). And this is where our approach comes in. Let's look at the example above and create an interface for the AtomicObject:

public interface Atomic
{
   public void incr (int value) throws Exception;

   public void set (int value) throws Exception;

   public int get () throws Exception;
}

And now we can create an implementation of this:

@Transactional
public class ExampleSTM implements Atomic
{
   @ReadLock
   public int get () throws Exception
   {
       return state;
   }

   @WriteLock
   public void set (int value) throws Exception
   {
       state = value;
   }

   @WriteLock
   public void incr (int value) throws Exception
   {
       state += value;
   }

   @TransactionalState
   private int state;
}

Here we now simply use annotations to specify what we want. The @Transactional is needed to indicate that this will be a transactional object. Then we use @ReadLock or @WriteLock to indicate the types of locks that we need on a per method basis. (If you don't define these then the default is to assume @WriteLock). And we use @TransactionalState to indicate to the STM implementation which state we want to operate on and have the isolation and atomicity properties (remember, with STM there's no requirement for the D in ACID). If there's more state in the implementation than we want to recover (e.g., it can be recomputed on each method invocation) then we don't have to annotate it.

But that's it: the AtomicObject and ExampleSTM classes are identical. So let's take a look at another unit test:

      RecoverableContainer theContainer = new RecoverableContainer();
     ExampleSTM basic = new ExampleSTM();
     Atomic obj = obj = theContainer.enlist(basic);    
     AtomicAction a = new AtomicAction();
  
     a.begin();
  
     obj.set(1234);
  
     a.commit();

     assertEquals(obj.get(), 1234);
  
     a = new AtomicAction();

     a.begin();

     obj.incr(1);
  
     a.abort();

     assertEquals(obj.get(), 1234);

Think of the RecoverableContainer as the unit of software transactional memory, within which we can place objects that it will manage. In this case ExampleSTM. Once placed, we get back a reference to the instance within the unit of memory, and as long as we use this reference from that point forward we will have the A, C and I properties that we want.

Ever wondered about transactions and threads?

2011-06-14T15:25:00.000+01:00

No? Well you really should! When transaction systems were first developed they were single-threaded (where a thread is defined to be an entity which performs work, e.g., a lightweight process, or an operating-system process.) Executing multiple threads within a single process was a novelty! In such an environment the thread terminating the transaction is, by definition, the thread that performed the work. Therefore, the termination of a transaction is implicitly synchronized with the completion of the transactional work: there can be no outstanding work still going on when the transaction starts to finish.

With the increased availability of both software and hardware multi-threading, transaction services are now being required to allow multiple threads to be active within a transaction (though it’s still not mandated anywhere, so if this is something you want then you may still have to look around the various implementations). In such systems it is important to guarantee that all of these threads have completed when a transaction is terminated, otherwise some work may not be performed transactionally.

Although protocols exist for enforcing thread and transaction synchronization in local and distributed environments (commonly referred to as checked transactions), they assume that communication between threads is synchronous (e.g., via remote procedure call). A thread making a synchronous call will block until the call returns, signifying that any threads created have terminated. However, a range of distributed applications exists (and yours may be one of them) which require extensive use of concurrency in order to meet real-time performance requirements and utilize asynchronous message passing for communication. In such environments it is difficult to guarantee synchronization between threads, since the application may not communicate the completion of work to a sender, as is done implicitly with synchronous invocations.

As we’ve just seen, applications that do not create new threads and only use synchronous invocations within transactions implicitly exhibit checked behavior. That is, it is guaranteed that whenever the transaction ends there can be no thread active within the transaction which has not completed its processing. This is illustrated below, in which vertical lines indicate the execution of object methods, horizontal lines message exchange, and the boxes represent objects.

The figure illustrates a client who starts a transaction by invoking a synchronous ‘begin’ upon a transaction manager. The client later performs a synchronous invocation upon object a that in turn invokes object b. Each of these objects is registered as being involved in the transaction with the manager. Whenever the client invokes the transaction ‘end’ upon the manager, the manager is then able to enter into the commit protocol (of which only the final phase is shown here) with the registered objects before returning control to the client.

However, when asynchronous invocation is allowed, explicit synchronization is required between threads and transactions in order to guarantee checked (safe) behavior. The next figure illustrates the possible consequences of using asynchronous invocation without such synchronization. In this example a client starts a transaction and then invokes an asynchronous operation upon object a that registers itself within the transaction as before. a then invokes an asynchronous operation upon object b. Now, depending upon the order in which the threads are scheduled, it’s possible that the client might call for the transaction to terminate. At this point the transaction coordinator knows only of a’s involvement within the transaction so enters into the commit protocol, with a committing as a consequence. Then b attempts to register itself within the transaction, and is unable to do so. If the application intended the work performed by the invocations upon a and b to be performed within the same transaction, this may result in application-level inconsistencies. This is what checked transactions are supposed to prevent.

Some transaction service implementations will enforce checked behavior for the transactions they support, to provide an extra level of transaction integrity. The purpose of the checks is to ensure that all transactional requests made by the application have completed their processing before the transaction is committed. A checked transaction service guarantees that commit will not succeed unless all transactional objects involved in the transaction have completed the processing of their transactional requests. If the transaction is rolled back then a check is not required, since all outstanding transactional activities will eventually rollback if they are not told to commit.

As a result, most (though not all) modern transaction systems provide automatic mechanisms for imposing checked transactions on both synchronous and asynchronous invocations. In essence, transactions must keep track of the threads and invocations (both synchronous and asynchronous) that are executing within them and whenever a transaction is terminated, the system must ensure that all active invocations return before the termination can occur and that all active threads are informed of the termination. This may sound simple, but believe us when we say that it isn’t!

Unfortunately this is another aspect of transaction processing that many implementations ignore. As with things like interposition (for performance) and failure recovery, it is an essential aspect that you really cannot do without. Not providing checked transactions is different from allowing checking to be disabled, which most commercial implementations support. In this case you typically have the ability to turn checked transactions off when you know it is safe, to help improve performance. If you think there is the slightest possibility you’ll be using multiple threads within the scope of a single transaction or may make asynchronous transactional requests, then you’d better find out whether your transaction implementation it up to the job. I'm not even going to bother about the almost obligatory plug for JBossTS here ;-)

WTF is Interposition?

2011-06-02T18:44:00.000+01:00

Consider the situation depicted below, where there is a transaction coordinator and three participants. For this sake of this example, let us assume that each of these participants is on a different machine to the transaction coordinator and each other. Therefore, each of the lines not only represents participation within the transaction, but also remote invocations from the transaction coordinator to the participants and vice versa.

In a distributed system there’s always an overhead incurred when making remote invocations compared to making a purely local (within the same VM) invocation. Now the overhead involved in making these distributed invocations will depend upon a number of factors, including how congested the network is, the load on the respective machines, the number of transactions being executed etc. Some applications may be able to tolerate this overhead, whereas others may not. As the number of participants increase, so does the overhead for fairly obvious reasons.

A common approach to reduce this overhead is to realize that as far as a coordinator is concerned, it does not matter what the participant implementation does. For example, although one participant may interact with a database to commit the transaction, another may just as readily be responsible for interacting with a number of databases: essentially acting as a coordinator itself, as shown below:

In this case, the participant is acting like a proxy for the transaction coordinator (the root coordinator): it is responsible for interacting with the two participants when it receives an invocation from the coordinator and collating their responses (and it’s own) for the coordinator. As far as the participants are concerned, a coordinator is invoking them, whereas as far as the root coordinator is concerned it only sees participants.

This technique of using proxy coordinators (or subordinate coordinators) is known as interposition. Each domain (machine) that imports a transaction context may create a subordinate coordinator that enrolls with the imported coordinator as though it were a participant. Any participants that are required to enroll in the transaction within this domain actually enroll with the subordinate coordinator. In a large distributed application, a tree of coordinators and participants may be created.

A subordinate coordinator must obviously execute the two-phase commit protocol on its enlisted participants. Thus, it must have its own transaction log and corresponding failure recovery subsystem. It must record sufficient recovery information for any work it may do as a participant and additional recovery information for its role as a coordinator. Therefore, it is impossible for a normal participant to simply be a sub-coordinator because the roles are distinctly different; sub-coordinators are tightly coupled with the transaction system.

So the question then becomes when and why does interposition occur?

Performance: if a number of participants reside on the same node, or are located physically close to one another (e.g., reside in the same LAN domain) then it can improve performance for a remote coordinator to send a single message to a sub-coordinator that is co-located with those participants and for that sub-coordinator to disseminate the message locally, rather than for it to send each participant the same message.
Security and trust: a coordinator may not trust indirect participants and neither may indirect participants trust a remote coordinator. This makes direct registration impossible. Concentrating security and trust at coordinators can make it easier to reason about such issues in a large scale, loosely coupled environment.
Connectivity: some participants may not have direct connectivity with a specific coordinator, requiring a level of indirection.
Separation of concerns: many domains and services may simply not want to export (possibly sensitive) information about their implementations to the outside world.

You'll find interposition used in a number of transaction systems. Within JBossTS it's used by the JTS and XTS components.

When is a transaction not a transaction?

2011-06-02T18:25:00.000+01:00

Most of the time when we talk about transactions we really mean database transactions (local transactions), or ACID transactions. These may also share the common name of Top-level (Flat) Transaction, or maybe Traditional Transactions. Several enhancements to the traditional flat-transaction model have been proposed and in this article I want to give an overview of some of them:

Of course the first has to be nested transactions. But I've mentioned these before several times, so won't go over them again here. But I will remind everyone that you can use them in JBossTS!
Next up are Independent Top-Level transactions which can be used to relax strict serializability. With this mechanism it is possible to invoke a top-level transaction from within another transaction. An independent top-level transaction can be executed from anywhere within another transaction and behaves exactly like a normal top-level transaction, that is, its results are made permanent when it commits and will not be undone if any of the transactions within which it was originally nested roll back. If the invoking transaction rolls back, this does not lead to the automatic rollback of the invoked transaction, which can commit or rollback independently of its invoker, and hence release resources it acquires. Such transactions could be invoked either synchronously or asynchronously. In the event that the invoking transaction rolls back compensation may be required. Guess what? Yup, we support these too!
Now we move on to Concurrent Transactions: just as application programs can execute concurrently, so too can transactions (top-level or nested), i.e., they need not execute sequentially. So, a given application may be running many different transactions concurrently, some of which may be related by parent transactions. Whether transactions are executed concurrently or serially does not affect the isolation rules: the overall affect of executing concurrent transactions must be the same as executing them in some serial order.
Glued Transactions are next in our line up: top-level transactions can be structured as many independent, short-duration top-level transactions, to form a “logical” long-running transaction; the duration between the end of one transaction and the beginning of another is not perceivable and selective resources (e.g., locks on database tables) can be atomically passed from one transaction to the next. This structuring allows an activity to acquire and use resources for only the required duration of this long-running transactional activity. In the event of failures, to obtain transactional semantics for the entire long-running transaction may require compensation transactions that can perform forward or backward recovery. As you might imagine, implementing and supporting glued transactions is not straightforward. In previous work we did around the CORBA Activity Service, we used JBossTS as the core coordinator and supported glued transactions. Although some of that code still exists today, it would definitely need a dusting off before being used again.
Last but by no means least ... distributed transactions. Need I say more?

So there's an outline of some, but not all, of the more exotic extended transaction models out there. The interesting thing is that with the basic transaction engine in JBossTS we can support many of them. Some of the ones I haven't outlined can also be implemented using a technique that another of the original Arjuna developers came up with: Multi-coloured Actions. Well worth a read, and perhaps something we may look at again someday.

Done (for now).

2011-05-02T23:18:00.000+01:00

JUDCon Boston 2011 is well under way. After many interesting technical sessions, including presentations on Byteman and on Transactions in the Cloud, we're onto the beer and pizza part of the day. If you're in Boston get yourself down here tomorrow for the second day, it's a great event.

Unfortunately I'm going to miss it as I'm off on vacation at oh-my-god-early in the morning. Please try not to break the shiny new JBossTS release in my absence. Normal services will resume, from the not so normal and sadly (for me) temporary location of Brisbane, Australia, in a couple of week's time.

almost done

2011-04-26T13:59:00.000+01:00

JBossTS 4.15.0.Final is done. It's the fastest, most robust release yet. It's also the last feature release of its line.

As well as working standalone, 4.15 is the feature release for integration with JBossAS 7 and thence JBossEAP 6. As AS 7 is still a moving target we've got a bit of tweaking to do for the integration over the next few weeks, particularly on JTS and XTS. After that it's maintenance only on the 4.15.x branch and the community focus moves on to the next major version.

As well as new features, the next major release cycle will have a new project name (complete with snazzy new logo, naturally) and a new project lead.

That change should hopefully leave me with some time to do new things too, particularly looking at the implications of the cloud computing model and NoSQL storage solutions on transaction systems.

I'm starting out on that road with a presentation next week at JUDCon so if you're in the Boston area come along and join in. Don't worry, I'll have written the slides by then, I promise. I'm just not quite done yet :-)

Messaging/Database race conditions

2011-04-20T10:34:00.001+01:00

Programmers are divided between those who only write single threaded code and those who will, at the slightest hint of provocation, tell interminably long war stories about debugging race conditions. Although I'm firmly in the second category, this is not such a story. It is however a lesson in what happens when spec authors spend insufficient time listening to them.

A common pattern in enterprise apps involves a business logic method writing a database update, then sending a message containing a key to that data via a queue to a second business method, which then reads the database and does further processing. Naturally this involves transactions, as it is necessary to ensure the database update(s) and message handling occur atomically.


methodA {
  beginTransaction();
  updateDatabase(key, values);
  sendMessage(queueName, key);
  commitTransaction();
}

methodB {
  beginTransaction();
  key = receiveMessage(queueName);
  values = readDatabase(key);
  doMoreStuff(values);
  commitTransaction();
}

So, like all good race condition war stories, this one starts out with a seemingly innocuous chunk of code. The problem is waiting to ambush us from deep within the transaction commit called by methodA.

Inside the transaction are two XAResources, one for the database and one for the messaging system. The transaction manager calls prepare on both, get affirmative responses and writes its log. Then it calls commit on both, at which point the resource managers can end the transaction isolation and make the data/message visible to other processes.

Spot the problem yet?

Nothing in the XA or JTA specifications defines the order in which XAResources are processed. The commit calls to the database and message queue may be issued in either order, or even in parallel.

Therefore it is inevitable that every once in a while, probably on the least convenient occasions, the database read in methodB will fail as the data has not yet been released from the previous transaction.

Oops.

So, having been let down by shortcomings in the specifications, what can the intrepid business logic programmer do to save the day?

Polling: Throw away the messaging piece and construct a loop that will look for new data in the db periodically. yuck.

Messaging from afterCompletion: The transaction lifecycle provides an afterCompletion notification that is guaranteed to be ordered after all the XAResource commits complete, so we can send the message from there to avoid the race. Also icky, since it is then no longer atomic with the db write in the case of failures, which means we need polling as a backup anyhow.

Let it fail: An uncaught exception in the second tx, e.g. an NPE from the database read, should cause the container to fail the transaction and the messaging system to do message redelivery. Hopefully by that time the db is ready. Inefficient and prone to writing a lot of failure messages into your system logs, which is to say: yuck.

Force resource ordering: Many transaction systems offer some non-spec compliant mechanism for XAResource ordering. Due to the levels of abstraction the container places between the transaction manager and the programmer this is not always easy to use from Java EE. It also ties your code to a specific vendor's container. You can do this with JBoss if you really want to, but it's not recommended.

Delay the message delivery: Similar to the above, some messaging systems go above and beyond the JMS spec to offer the ability to mark a queue or specific message as having a built-in delay. The messaging system will hold the message for a specified interval before making it available to consumers. Again you can do with with JBoss (see 'Scheduled Messages' in HornetQ) but whilst it's easy to code it's not a runtime efficient solution as tuning the delay interval is tricky.

Backoff/Retry in the consumer: Instead of letting the consuming transaction fail and be retried by message redelivery, catch the exception and handle it with a backoff/retry loop:


methodB {
  beginTransaction();
  key = receiveMessage(queueName);

  do {
    values = readDatabase(key);
    if(values == null) {
      sleep(x);
    }
  } while(values == null);

  doMoreStuff(values);
  commitTransaction();
}

This has the benefit of being portable across containers and, with a bit of added complexity, pretty robust. Yes it is a horrible kludge, but it works. Sometimes when you are stuck working with an environment built on flawed specifications that's really the best you can hope for.

World domination

2011-04-04T16:19:00.001+01:00

It's nearing that time of year again... The transactions team are packing their bags and preparing to spend a couple of days holed up in a top secret underground lair, plotting the next steps towards world domination.

I normally spend a few days beforehand checking up on the state of the world, particularly new developments in transactions, the list of open issues in our bug tracker, the last few month's worth of support cases and other such sources of interesting information. I'm starting to think the reason that no Evil Genius has succeeded in taking over the world yet has very little to do with being repeatedly thwarted by James Bond and an awful lot to do with the sheer volume of paperwork involved. Perhaps if they spent less on underground lairs and more on secretarial support...

Anyhow, as per usual we have way too many ideas on the list, so some prioritization is in order. One of the key factors is the number of users a given chunk of work may benefit. Improvements to the crash recovery system benefit pretty much everyone. I'd like to say the same about documentation, but it's apparent only a tiny proportion of users actually read it. Somewhere in between lie improvements to optional modules or functionality that only some users take advantage of. The tricky part is gauging how many there are.

Users occasionally get in touch and tell us how they are using the code. Unfortunately this usually happens only when they need help. The rest of the time we mostly just make educated guesses about the use cases. I was mulling over this problem today when a tweet caught my eye:

"Today's royalty check for "Advanced CORBA Programming": $24.51. Last year each check was ~$400. I guess CORBA is officially dead now :-)" - Steve Vinoski

This got me thinking about the future of JTS. In Java EE environments, JTS exists pretty much only to allow transactional calls between EJBs using IIOP. Now that we have @WebService on EJBs, how much traffic still uses IIOP anyhow? What are the odds of RMI/IIOP (and hence JTS) being deprecated out of Java EE in the next revision or two? How much time should we be sinking into improvements of JTS vs. web services transactions?

That's just one of many topics on the agenda for our team meeting. I suppose we could also debate the relative merits of man-portable freeze rays vs. orbital laser cannons, but I don't think the corporate health and safety folks will let us have either. Spoilsports. Guess we'll just have to continue progressing towards world domination the hard way.

The Competitors

2011-03-31T14:52:00.001+01:00

I don't normally spend much time worrying about our competitors. I figure keeping the customers happy is a better use of my time. But no set of blog posts about performance would be complete without a final showdown...

There aren't all that many standalone JTA transaction managers out there. Most function only as part of an application server and are not a viable option for embedding in other environments. Others are incomplete or largely abandoned. In my opinion there are currently only two serious challengers to JBossTS for use outside a full Java EE app server. Let's see how they do.

results in transactions per second, using quad core i7, jdk6, in-memory logging i.e. we're testing the CPU not the disk.


threads  JBossTS       Competitor A.
1         93192           9033
2        164744          20242
3        224114          22649
4        249781          21148
100      239865          19978

umm, guys, "synchronized ( Foo.class )" is really not a good idea, ok? Come on, put up a bit more of a fight please, this is no fun.

Next contender, please....


threads  JBossTS        Competitor B.
1         93192           102270
2        164744           145011
3        224114           185528
4        249781           193199
100      239865            29528

Ahh, that's more like it. Good to have a worthy opponent to keep us on our toes. Some kind of scaling problem there though. Looks like contention on the data structures used to queue transaction timeouts. Been there, done that. Feel free to browse the source code for inspiration guys :-)

Since nobody in their right mind actually runs a transaction manager with logging turned off, this is really all just a bit of fun. Well, for some of us anyhow. So where is the competitive comparison for the real world scenario?

Well, I do have the comparison numbers for all three systems with logging enabled, but I'd hesitate to describe them as competitive. You'll have to try it for yourself or you just won't believe the difference. Suffice it to say there is only one Clebert, and we've got him. Sorry guys.

Now if this had been an ease of use contest I'd be a bit more worried. Sure we have a ton of detailed and comprehensive reference documentation, but the learning curve is near vertical. Time to work on some intro level material: quick starts, config examples etc. Not as much fun as performance tuning, but probably more useful at this point.

fast enough?

2011-03-28T17:03:00.001+01:00

Remember the performance tuning we've been doing? Well, the powers that be sat up and took notice too. I think the phrase was 'more than enough to satisfy requirements' with a non-too-subtle subtext of 'so now put down that profiler and do some real work'. Fortunately I'm not too good at taking hints...

After some more tweaking today (finalizers are evil and must die), I thought I'd see how we were doing. Here is trunk, soon to become JBossTS 4.15 and be used in EAP6, against JBossTS 4.6.1.CP11, used in the current EAP5.1 platform release. 4.6.1 originally dates from September 2009, but we backported some performance improvements into it in mid 2010.

results in transactions per second, using quad core i7, jdk6, in-memory objectstore i.e. we're testing the CPU not the disk.


threads  4.6.1.CP12    4.15.SNAPSHOT   improvement
1        37285          93192          2.50x
2        55309         164744          2.97x
3        65070         224114          3.44x
4        66172         249781          3.77x
100      29027         239865          8.26x

hmm, ok, so I guess maybe we did exceed the requirements a bit.

Concurrency control

2011-03-19T11:21:00.001Z

We managed to gloss over a few important points during the discussion on Isolation. Let's dig into them here.

How can you ensure isolation? Well typically this is covered by what's referred to as concurrency control for the resources within the transaction. A very simple and widely used approach is to regard all operations on resources (objects) to be of type ‘read’ or ‘write’, which follow the synchronization rule permitting ‘concurrent reads' but exclusive ‘writes’. This rule is imposed by requiring that any computation intending to perform an operation that is of type read (write) on an object, first acquire a ‘read lock’ (‘write lock’) associated with that object. A read lock on an object can be held concurrently by many computations provided no computation is holding a write lock on that object. A write lock on an object, on the other hand, can only be held by a computation provided no other computation is holding a read or a write lock. (Note, although we'll talk in terms of locks, this should not be used to infer a specific implementation. Timestamp-based concurrency control could just as easily be used, for example.)

In order to ensure the atomicity property, all computations must follow a ‘two–phase’ locking policy. During the first phase, termed the growing phase, a computation can acquire locks, but not release them. The tail end of the computation constitutes the shrinking phase, during which time held locks can be released but no locks can be acquired. Now suppose that a computation in its shrinking phase is to be rolled back, and that some objects with write locks have already been released. If some of these objects have been locked by other computations, then abortion of the computation will require these computations to be aborted as well. To avoid this cascade roll back problem, it is necessary to make the shrinking phase ‘instantaneous’.

Most transaction systems utilize what is commonly referred to as pessimistic concurrency control mechanisms: in essence, whenever a data structure or other transactional resource is accessed, a lock is obtained on it as described earlier. This lock will remain held on that resource for the duration of the transaction and the benefit of this is that other users will not be able to modify (and possibly not even observe) the resource until the holding transaction has terminated. There are a number of disadvantages of this style: (i) the overhead of acquiring and maintaining concurrency control information in an environment where conflict or data sharing is not high, (ii) deadlocks may occur, where one user waits for another to release a lock not realizing that that user is waiting for the release of a lock held by the first.

Therefore, optimistic concurrency control assumes that conflicts are not high and tries to ensure locks are held only for brief periods of time: essentially locks are only acquired at the end of the transaction when it is about to terminate. This kind of concurrency control requires a means to detect if an update to a resource does conflict with any updates that may have occurred in the interim and how to recover from such conflicts. Typically detection will happen using timestamps, whereby the system takes a snapshot of the timestamps associated with resources it is about to use or modify and compares them with the timestamps available when the transaction commits.

Resolution of conflicts is a different problem entirely, since in order to do so requires semantic information about the resources concerned. Therefore, most transaction systems that offer optimistic schemes will typically cause the detecting transaction to roll back and the application must retry, this time with new data. Obviously this may result in a lot of work being lost, especially if the transaction that rolls back has been running for some time.

Assuming both optimistic and pessimistic concurrency control are available to you (and they may not be), then which one to use is up to you. A close examination of the environment in which the application and transactional resources reside is necessary to determine whether a) shared access to resources occurs and b) the relative probability that sharing will cause a transaction to roll back. This might very well not be a black or white choice and may change over the lifetime of your objects or application. Certainly the use of different concurrency control schemes can be important when trying to improve the throughput of user requests and committed transactions, so it’s well worth considering and understanding the issues involved.
Type specific concurrency control

Another possible enhancement is to introduce type specific concurrency control, which is a particularly attractive means of increasing the concurrency in a system (and yes, it's supported in JBossTS). Concurrent read/write or write/write operations are permitted on an object from different transactions provided these operations can be shown to be non-interfering (for example, for a directory object, reading and deleting different entries can be permitted to take place simultaneously). Object-oriented systems are well suited to this approach, since semantic knowledge about the operations of objects can be exploited to control permissible concurrency within objects. Additional work may be needed when working with procedural systems.

Finally, what about deadlocks? When multiple transactions compete for the same resources in conflicting modes (locks), it is likely that some of them will fail to acquire those resources. If a transaction that cannot acquire a lock on a resource waits for it to be released, then that transaction is blocked – no forward progress can be made until the lock has been acquired. In some environments, it is possible for some transactions to be waiting for each other, where each of them is blocked and is also blocking another transaction. In this situation, none of the transactions can proceed and the system is deadlocked.

For example, let’s consider two transactions T1 and T2 that operate on two resources X and Y. Let’s assume that the execution of the operations involved in these transactions is:

T1: read(X); write(Y)
T2: read(Y); write(X)

If the serial execution of these transactions were to result in:

readT1(X); readT2(Y); writeT2(X); readT1(Y)

Note, readT1 means the read operation performed by T1 etc.

Assume that T1 obtained a read lock on X and then T2 gets a read lock on Y – possible because these operations aren’t conflicting and can thus occur in parallel. However, when T2 comes to write to X its attempt to get a write lock on X will block because T1 still holds its read lock. Likewise, T1’s attempt to get a write lock on Y will block because of the read lock that T2 holds. Each transaction is blocked waiting for the release of the others read lock before they can progress: they are deadlocked.
The only way for the deadlock to be resolved is for at least one of the transactions to release its locks that are blocking another transaction. Obviously such a transaction cannot commit (it has not been able to perform all of its work since it was blocked); therefore, it must roll back.

Deadlock detection and prevention is complicated enough in a non-distributed environment without then including the extra complexity of distribution. In general, most transaction systems allow deadlocks to occur, simply because to do otherwise can be too restrictive for applications. There are several techniques for deadlock detection, but the two most popular are:

Timeout-based: if a transaction has been waiting longer than a specified period of time, the transaction system will automatically roll back the transaction on the assumption it is deadlocked. The main advantage of this approach is that it is easy to implement in a distributed environment; the main disadvantage is that some transactions may execute for longer than expected and be rolled back when they are not in fact deadlocked.

Graph-based: this explicitly tracks waiting transaction dependencies by constructing a waits-for graph: nodes are waiting transactions and edges are waiting situations. The main advantage of this approach is that it is guaranteed to detect all deadlocks, whereas the main disadvantage is that in a distributed environment is can be costly to execute.

A slight variation on the timeout-based approach exists in some transaction systems, where timeouts can be associated with lock acquisition, such that the system will only block for the specified period of time. If the lock has not been acquired by the time this period elapses, it returns control to the application indicating that the lock has not been obtained. It is then up to the application to determine what to do; for example, it may be possible to acquire the required data elsewhere or to ask for a lock in a different mode. The advantage of this approach is that a transaction is not automatically rolled back if it cannot acquire a lock, possibly saving the application lots of valuable time; the disadvantage is that it requires additional effort on behalf of the application to resolve lock acquisition failures. However, for many objects and applications, this is precisely the best place to resolve lock conflicts because this is the only place where the semantic information exists to know what (if anything) can be done to resolve such conflicts.

Durability

2011-03-19T11:16:00.000Z

D is for Dog, Design and Durability, i.e., the effects of a committed transaction are never lost (except by a catastrophic failure).

The durability (or persistence) property means that any state changes that occur during the transaction must be saved in a manner such that a subsequent failure will not cause them to be lost. How these state changes are made persistent is typically dependant on the implementation of the transaction system and the resources that are ultimately used to commit the work done by the transactional objects. For example, the database will typically maintain its state on hard disk in order to ensure that a machine failure (e.g., loss of power) does not result in loss of data.

Although most users of transactions will see durability from their application’s point-of-view, there is also an aspect within the transaction system implementation itself. In order to guarantee atomicity in the presence of failures (both transaction coordinator and participant), it is necessary for the transaction service itself to maintain state. For example, in some implementations the coordinator must remember the point in the protocol it has reached (i.e., whether it is committing or aborting), the identity of all participants that are registered with the transaction and where they have reached in the protocol (e.g., whether they have received the prepare message). This is typically referred to as the transaction log, though this should not be interpreted as implying a specific implementation. Some implementations may maintain a separate log (file) per transaction, with this information recorded within it and removed when it is no longer needed. Another possible implementation has a single log for all transactions and the transaction information is appended to the end of the log and pruned from the log when the respective transaction completes.

Let’s look at what happens at the participant in terms of durability, when it’s driven through the two-phase commit protocol. When the participant receives a prepare message from the coordinator it must decide whether it can commit or roll back. If it decides to roll back then it must undo any state changes that it may control and inform the coordinator; there is no requirement for durable access at this point. If the participant can commit, it must write its intentions to a durable store (participant log) along with sufficient information to either commit or roll back the state changes it controls. The format of this information is typically dependant on the type of participant, but may include the entire original and new states.

Once successfully recorded, the participant informs the coordinator that it can commit and awaits the coordinator’s decision. When this second phase message arrives, the participant will either cancel all state changes (the coordinator wants to roll back), or if the coordinator wants to commit, make those state changes the current state (e.g., overwrite the original state with the new state). It can then delete the participant log and inform the coordinator.

Isolation

2011-03-19T11:14:00.000Z

I is for Iguana, Igloo and Isolation, i.e., intermediate states produced while a transaction is executing are not visible to others. Furthermore transactions appear to execute serially, even if they are actually executed concurrently.

This property is often referred to as serializability. If we assume that objects and services can be shared between various programs then it is necessary to ensure that concurrent executions of programs are free from interference, i.e., concurrent executions should be equivalent to some serial order of execution. We have already seen a fairly trivial example of this through the online bank account, but it is worth formalizing this requirement. Consider the following two programs (where w, x, y and z are distinct state variables):

P1 : z := 10; x := x+1; y := y+1
P2 : w := 7: x := x * 2; y := y * 2

Assume that x=y=2 initially. Then, a serial execution order ’P1;P2 ’ will produce the result z=10, w=7, x=y=6, and execution ’P2 ;P1 ’ will produce the results z=10, w=7, x=y=5. The partly concurrent execution order given below will be termed interference free or serializable, since it is equivalent to the serial order ’P1 ;P2 ’:

(z := 10 || w := 7); x := x+1; y := y+1; x := x * 2; y := y * 2

However, an execution order such as the one below is not free from interference since it cannot be shown to be equivalent to any serial order.

(z := 10 || w := 7); x := x+1; x := x * 2; y := y * 2; y := y+1

Programs that possess the above-mentioned serializable property are said to be atomic with respect to concurrency. The serializability property is extremely important, especially in an environment where multiple concurrent users may be attempting to use the same resources consistently.

Consistency

2011-03-19T11:12:00.000Z

C is for Crayon, Cup and Consistency, i.e., transactions produce consistent results and preserve application specific invariants.

A transactional application should maintain the consistency of the resources (e.g., databases, file-systems, etc.) that it uses. In essence, transactional applications should move from one consistent state to another. However, unlike the other transactional properties (A, I and D) this is something that the transaction system cannot achieve by itself since it does not possess any semantic information about the resources it manipulates: it would be impossible for a transaction processing system to assert that the resources are moving to (or from) consistent states. All a transaction system can ensure is that any state changes that do occur are performed in a manner that is guaranteed despite failures. It is the application programmers’ responsibility to ensure consistency (in whatever way makes sense for the resources concerned.)

Atomicity

2011-03-19T11:07:00.000Z

We've already covered some of this in previous postings, but in the next four entries let's explicitly look at all ACID properties. And of course that means we start here with ... A is for Apple, Aardvark and Atomic, i.e., the transaction completes successfully (commits) or if it fails (aborts) all of its effects are undone (rolled back).

In order to ensure that a transaction has an atomic outcome, the two-phase commit protocol that we discussed earlier, is typically used. This protocol is used to guarantee consensus between participating members of the transaction. When each participant receives the coordinator’s phase 1 message, they record sufficient information on stable storage to either commit or abort changes made during the action. After returning the phase 1 response, each participant that returned a commit response must remain blocked until it has received the coordinator’s phase 2 message. Until they receive this message, these resources are unavailable for use by other actions. If the coordinator fails before delivery of this message, these resources remain blocked. However, if failed machines eventually recover, crash recovery mechanisms can be employed to unblock the protocol and terminate the transaction.

fiddling with knobs

2011-03-15T11:54:00.000Z

Get your mind out the gutter you. I'm talking about control knobs. Also buttons, switches, levers and anything else that allows for interesting behavioural tweaks.

Messing around with settings on software is a great way to pass the time. For those who enjoy such things there are over 100 configuration properties in JBossTS. Which admittedly may be a little too much of a good thing. Fortunatly they mostly have sensible default values.

I recently discussed the performance benefits of using one of those properties to swap in a new ObjectStore implementation for transaction logging in JBossTS. In the last few days I've been adding some more configuration options to allow overriding of the settings in the underlying HornetQ Journal. It turns out that there are also a bunch of interesting knobs to fiddle with in that code. By judicious tweaking of the settings on the Journal and the file system I've got a nice speedup over the last iteration:

60119 tx/second with in-memory logging, no disk I/O

43661 tx/second with the new Journal on SSD

43k is a big step up from 30-36k or so we were seeing previously, but still not saturating the CPU. However, a new test case shows the disk is not to blame:

89221 logs writes / second.

That's taking most of the transaction manger code out of the picture and writing an equivalent amount of data direct to the Journal based ObjectStore. So, plenty of disk write capacity to spare now.

So why can't we use it all?

The log write done by an individual transaction is very small. To get an efficient batch size you need to aggregate the writes for a significant number of transactions. Whilst waiting for that batch to be written, the transactions are blocked. So, to keep the CPU busy you need to throw more threads / concurrent transactions at it. Infact the disk is so quick you need a LOT more threads. At which point lock contention elsewhere in the transaction manager code become a problem, despite the significant improvements we made to it anyhow for pure in-memory transaction cases.

Naturally it's possible we can get some additional benefit from another iteration of lock refactoring. We've already picked off most of the reasonably tractable cases though and what's left is going to be complex to change.

Another possibility is to actually run the Journal out of process and use IPC so several JVMs can share it. The additional communication overhead may just be worthwhile. It's a similar model to what we do with the SQL db based objectstore, but without the overhead of SQL. Although naturally this multi process architecture is attractive only where the application workload is amenable to partitioning between JVMs.

It may have to wait a while though. We're entering the final weeks of 4.15 development and it's time to start cleaning up all the loose ends and getting things in shape for use by JBossAS 7 GA. Onward.

Slightly alkaline transactions if you please ...

2011-03-13T18:39:00.000Z

Given that the traditional ACID transaction model is not appropriate for long running/loosely coupled interactions, let’s pose the question, “what type of model or protocol is appropriate?” The answer to that question is that that no one specific protocol is likely to be sufficient, given the wide range of situations that transactions are likely to be deployed within. In fact trying to shoe-horn ACID transactions into a wide range of situations for which they were never designed is one of the reasons they've gotten such a bad reputation over the years.

There are a number of different extensions to the standard transaction model that have been proposed to address specific application needs, that may not be easily or efficiently addressed through the use of traditional transactions:

• Nested transactions: permits a finer control over recovery and concurrency. The outermost transaction of such a hierarchy is referred to as the top-level transaction. The permanence of effect property is only possessed by the top-level transaction, whereas the commits of nested transactions (subtransactions) are provisional upon the commit/abort of an enclosing transaction. And yes, JBossTS supports nested transactions!

• Type specific concurrency control: concurrent read/write or write/write operations are permitted on an object from different transactions provided these operations can be shown to be non-interfering. Oh and yes, we support this too!

• Independent top-level transactions: with this model it is possible to invoke a top-level transaction from within another (possibly deeply nested) transaction. If the logically enclosing transaction rolls back, this does not lead to the rollback of the independent top-level transaction, which can commit or rollback independently. In the event that the enclosing transaction rolls back, compensation may be required, but this is typically left up to the application. Yes, we've got this one covered as well.

• Structured top-level transactions: long-running top-level transactions can be structured as many independent, short-duration top-level transactions. This allows an activity to acquire and use resources for only the required duration. In the event of failures, to obtain transactional semantics for the entire duration may require compensations for forward or backward recovery. Now although we don't support this directly within JBossTS since it is really workflow, JBossTS has been used to implement workflow systems for many years.

What this range of extended transaction models illustrate is that a single model is not sufficient for all applications. Therefore, is it possible to develop a framework within which all of these models can be supported, and also facilitate the development of other models? This was the question asked by the Object Management Group when it began its work on attempting to standardise extended transaction models. In this paper we shall given an overview of the results of the work we performed with IBM, Iona and others in producing the final Activity Service OMG specification that attempts to answer that question. This also became a Jave specification with JSR 95.

Now I'm not going to go into this standard in any detail, since that could be the subject of several other blog posts, but I will summarise it as this: have a generic coordination infrastructure that allows the intelligence behind the protocol (e.g., two-phase or three-phase) as well as the trigger points for the protocol (e.g., at the end of the transaction or during the transaction) to be defined at runtime and in a pluggable manner. But why did I mention all of this? Because this standard formed the basis of the Web Services transactions standard (and the various competing specifications that came before it), with WS-Coordination as the pluggable infrastructure.

At this point in time there are only two types of transaction protocol supported by the Web Services standard and the REST-based approach that we're working on:

• ACID transactions: Web services are for interoperability in closely coupled environments such as corporate intranets as much as they are for the Web. Interoperability between heterogeneous transaction service implementations is a requirement and yet has been difficult to achieve in practice. This transaction model is designed to support interoperability of existing transaction processing systems via Web services, given such systems already form the backbone of enterprise class applications. Although ACID transactions may not be suitable for all Web services, they are most definitely suitable for some, and particularly high-value interactions such as those involved in the finance sector. For example, in a J2EE environment, JTA-to-JTA interoperability is supported through the JTS specification, but this is neither mandated nor universally implemented.

• Forward compensation based transactions: this model is designed for those business interactions that are long in duration, where traditional ACID transactions are inappropriate. With this model, all work performed within the scope of an application should be able to be compensated such that an application’s work is either performed successfully or undone. However, how individual Web services perform their work and ensure it can be undone if compensation is required, is an implementation choice. The model simply defines the triggers for compensation actions and the conditions under which those triggers are executed.

And the nice thing about our implementations of both of these is that they're based on the exact same transaction engine: ArjunaCore. Because it was designed with extensibility in mind and also with the requirement to be able to relax the various ACID properties, over the last decade we've been able to use it relatively untouched as the basis of pretty much every transaction protocol and implementation that's been out there. So if we do decide that we need to add another extended transaction protocol for other use cases, I'm pretty confident that it won't require us to start from scratch!

When ACID is too strong

2011-03-13T18:30:00.000Z

Transaction processing itself is not by any means a new discipline. Though much innovative and interesting work continues in the field, many of the fundamental techniques and algorithms are well-known and for several decades transactions have been supported on various platforms. However, as Web services have evolved as a means to integrate processes and applications at an inter-enterprise level, traditional transaction semantics and protocols have proven to be inappropriate. Web services or REST-based transactions, colloquially termed Business Transactions, differ from traditional transactions in that they execute over long periods, they require commitments to the transaction to be “negotiated” at runtime, and isolation levels have to be relaxed. Business Transactions require an extended transaction model that builds on existing standards where possible and defines interoperable transaction protocol and message flows that help negotiate transactions guarantees at the inter-enterprise level.

Structuring certain activities from long-running transactions can reduce the amount of concurrency within an application or (in the event of failures) require work to be performed again. For example, there are certain types of application where it is known that resources acquired within a transaction can be released “early”, rather than having to wait until the transaction terminates; in the event of the transaction rolling back, however, certain compensation activities may be necessary to restore the system to a consistent state.

Long-running activities can be structured as many independent, short-duration transactions, to form a “logical” long-running transaction. This structuring allows an activity to acquire and use resources for only the required duration of this long-running activity, as shown.

In the figure, an application activity (shown by the dotted ellipse) has been split into different, coordinated short-duration transactions. Assume that the application activity is concerned with booking a taxi (t1), reserving a table at a restaurant (t2), reserving a seat at the theatre (t3), and then booking a room at a hotel (t4), and so on. If all of these operations were performed as a single transaction then resources acquired during t1 would not be released until the transaction has terminated. If subsequent activities t2, t3 etc. do not require those resources, then they will be needlessly unavailable to other clients.

However, if failures and concurrent access occur during the lifetime of these individual transactional activities then the behavior of the entire “logical long-running transaction” may not possess ACID properties. Therefore, some form of compensation may be required to attempt to return the state of the system to consistency. For example, let us assume that t4 aborts. Further assume that the application can continue to make forward progress, but in order to do so must now undo some state changes made prior to the start of t4 (by t1, t2 or t3). Therefore, new activities are started; tc1 which is a compensation activity that will attempt to undo state changes performed, by say t2, and t3 which will continue the application once tc1 has completed. tc5’ and tc6’ are new activities that continue after compensation, e.g., since it was not possible to reserve the theatre, restaurant and hotel, it is decided to book tickets at the cinema.

Previous transaction processing systems shared a great deal of commonality in terms of the crux of the problem that they address and the abstractions they use to address it. Specifically, transaction processing systems were developed for particular platforms and each system assumes that it is in sole control of the transaction domain and hence does not generally have to interoperate with other transaction processing systems (though interoperability with lower-level components like databases is generally well supported via interfaces like X/Open XA). Early attempts at transaction interoperability (e.g., the Object Transaction Service from the Object Management Group) did not manage to get past the “vendor lock-in” barrier, and attempts at using transactions across enterprise boundaries failed because in such systems transactions are assumed to exhibit ACID properties.

Web services, or REST-based applications, present a different kind of problem: they are specifically about fostering systems interoperability as well as long duration interactions. This presents some interesting problems from a transaction management point of view. What makes Web services so interesting is the fact that the architecture is deliberately not prescriptive about what happens behind service endpoints – Web services are ultimately only concerned with the transfer of structured data between parties, plus any meta-level information to safeguard such transfers (e.g. by encrypting or digitally signing messages) – yet it is behind service endpoints that we find traditional transaction processing architectures supporting business activities.

Thus we are presented with a paradox. The Web services or REST approaches provide a service-oriented, loosely coupled, and potentially asynchronous means of propagating information between parties, whilst in the background we have traditional transaction processing infrastructures whose behavior is neither or mutually interoperable. Furthermore, the fact that transactions in these systems are assumed to exhibit ACID properties potentially leads to problems when exposing resources to third parties, since it presents opportunities to those parties to tie up resources and prevent transactions from making progress. Thus if transactions were to be supported in either of these architectural approaches then it is clear that some re-addressing of the problem is required.

Drum roll ... stay tuned for the next thrilling instalment in the series!

Non-atomic outcome

2011-03-13T10:08:00.000Z

We've already touched on the concept of heuristic outcomes (where atomicity cannot be guaranteed). But lets delve into this in a bit more detail and consider the types of heuristic that may occur.

So far we've seen that in order to guarantee atomicity, the two-phase commit protocol is necessarily blocking. What this means is that as a result of failures, participants may remain blocked for an indefinite period of time even if failure recovery mechanisms exist. Some applications and participants simply cannot tolerate this blocking. To break this blocking nature, participants that have got past the prepare phase are allowed to make autonomous decisions as to whether they commit or rollback: such a participant must record this decision in case it is eventually contacted to complete the original transaction. If the coordinator eventually informs the participant of the transaction outcome and it is the same as the choice the participant made, then there’s no problem. However, if it is contrary, then a non-atomic outcome has obviously happened: a heuristic outcome. How this heuristic outcome is reported to the application and resolved is usually the domain of complex, manually driven system administration tools, since in order to attempt an automatic resolution requires semantic information about the nature of participants involved in the transactions.

Precisely when a participant makes a heuristic decision is obviously implementation dependant. Likewise, the choice the participant makes (to commit or to roll back) will depend upon the implementation and possibly the application/environment in which it finds itself. The possible heuristic outcomes are:

• Heuristic rollback: the commit operation failed because some or all of the participants unilaterally rolled back the transaction.
• Heuristic commit: an attempted rollback operation failed because all of the participants unilaterally committed. This may happen if, for example, the coordinator was able to successfully prepare the transaction but then decided to roll it back (e.g., it could not update its log) but in the meanwhile the participants decided to commit.
• Heuristic mixed: some updates (participants) were committed while others were rolled back.
• Heuristic hazard: the disposition of some of the updates is unknown. For those which are known, they have either all been committed or all rolled back.

Heuristic decisions should be used with care and only in exceptional circumstances since there is the possibility that the decision will differ from that determined by the transaction service and will thus lead to a loss of integrity in the system. Having to perform resolution of heuristics is something you should try to avoid, either by working with services/participants that don’t cause heuristics, or by using a transaction service that provides assistance in the resolution process.

Optimisations to the transaction protocol

2011-03-13T10:03:00.000Z

There are several variants to the standard two-phase commit protocol that are worth knowing about because they can have an impact on performance and failure recovery. We shall briefly describe those that are the most common variants on the protocol:

• Presumed abort: if a transaction is going to roll back then it may simply record this information locally and tell all enlisted participants. Failure to contact a participant has no affect on the transaction outcome; the transaction is effectively informing participants as a courtesy. Once all participants have been contacted the information about the transaction can be removed. If a subsequent request for the status of the transaction occurs there will be no information available and the requestor can assume that the transaction has aborted (rolled back). This optimization has the benefit that no information about participants need be made persistent until the transaction has decided to commit (i.e., progressed to the end of the prepare phase), since any failure prior to this point will be assumed to be an abort of the transaction.

• One-phase: if there is only a single participant involved in the transaction, the coordinator need not drive it through the prepare phase. Thus, the participant will simply be told to commit and the coordinator need not record information about the decision since the outcome of the transaction is solely down to the participant.

• Read-only: when a participant is asked to prepare, it can indicate to the coordinator that no information or data that it controls has been modified during the transaction. Such a participant does not need to be informed about the outcome of the transaction since the fate of the participant has no affect on the transaction. As such, a read-only participant can be omitted from the second phase of the commit protocol.

So these optimisations let us get around some concerns that two-phase commit, particularly in a distributed environment, is overkill. Most modern transaction manager implementations will support all of these and when you consider that many transactional applications use only a single participant, or don't modify state, then you can see how the above considerations can dramatically reduce (or even remove) any overhead that may appear to be there at first glance. We'll go into this a bit more in a subsequent posting. But next up for consideration is what happens when we can't guarantee transactional semantics?

Synchronizations: what are they and why do you need them?

2011-03-13T10:00:00.000Z

As well as the two-phase commit protocol, traditional transaction processing systems employ an additional protocol, often referred to as the synchronization protocol. If you recall the original ACID properties, then you’ll remember that Durability is important in the case where state changes have to be available despite failures. What this means is that applications interact with a persistence store of some kind (e.g., a database) and this can impose a significant overhead – disk access is orders of magnitude slower than access to main computer memory.

One apparently obvious solution to this problem would be to cache the state in main memory and only operate on that for the duration of a transaction. Unfortunately you’d then need some way of being able to flush the state back to the persistent store before the transaction terminates, or risk losing the full ACID properties. This is what the synchronization protocol does, with Synchronization participants.

Synchronizations are informed that a transaction is about to commit, so they can, for example, flush cached state, which may be being used to improve performance of an application, to a durable representation prior to the transaction committing. They are then informed when the transaction has completed and in what state it completed.

Synchronizations essentially turn the two-phase commit protocol into a four-phase protocol:

• Before the transaction starts the two-phase commit, all registered Synchronizations are informed. Any failure at this point will cause the transaction to roll back.
• The coordinator then conducts the normal two-phase commit protocol.
• Once the transaction has terminated, all registered Synchronizations are informed. However, this is a courtesy invocation because any failures at this stage are ignored: the transaction has terminated so there’s nothing to affect.

Unlike the two-phase commit protocol, the synchronization protocol does not have the same failure requirements. For example, Synchronization participants don’t need to make sure they can recover in the event of failures; this is because any failure before the two-phase commit protocol completes means the transaction will roll back, and failures after it has completed can’t affect the data the Synchronization participants were managing.

So we've now looked at ACID and Synchronizations. What about optimisations to the protocol when we don't need two-phase commit? Stay tuned.

Transactions 101

2011-03-13T09:41:00.000Z

When I was at QCon the other day I was asked a number of questions around transactions that made me realise that I really need to take my JavaOne presentation and turn it into some blog entries as I promised last year. So over the next indeterminate amount of time (hey, I'm busy with things like JUDCon too), we'll take a tour through transactions and try to dispel some of the myths that surround them.

So let's start here with some of the basics. And you don't get more basic than defining what we mean by transaction. Put simply, a transaction provides an “all-or-nothing” (atomic) property to work that is conducted within its scope, whilst at the same time ensuring that shared resources are isolated from concurrent users. Importantly application programmers typically only have to start and end a transaction; all of the complex work necessary to provide the transaction’s properties is hidden by the transaction system, leaving the programmer free to concentrate on the more functional aspects of the application at hand.

Let’s take a look at just how a transaction system could help in a real-world application environment. Consider the case of an on-line cinema reservation system. The cinema has many seats that can be reserved individually, and the state of a seat is either RESERVED or UNRESERVED. The cinema service exports two operations, reserveSeat and unreserveSeat (we’ll ignore the other operations that are obviously required to make this service truly usable). Finally we’ll assume that there is a transaction manager service that will be used to manage any transactions that the cinema may require in order to process the user’s requests.

Let’s consider a very simple example: imagine that Mr. Doe wants to reserve a block of seats for his family (1A, 1B and 1C). Now, the service only allows a single seat to be reserved through the reserveSeat operation, so this will require Mr. Doe to call it 3 times, once for each seat. Unfortunately the reservation process may be affected by failures of software or hardware that could affect the overall consistency of the system in a number of ways. For example, if a failure occurs after reserving 1A, then obviously none of the other seats will have been reserved. Mr. Doe can try to complete the reservation when (assuming) the cinema service eventually recovers, but by this time someone else may have reserved the seats.

What Mr. Doe really wants is the ability to reserve multiple seats as an atomic (indivisible) block. This means that despite failures and concurrent access, either all of the seats Mr. Doe requires will be reserved for him, or none will. At first glance this may seem like a fairly straightforward thing to achieve, but it actually requires a lot of effort to ensure that these requirements can be guaranteed. Fortunately atomic transactions possess the following (ACID) properties that make them suitable for this kind of scenario:

• Atomicity: The transaction completes successfully (commits) or if it fails (aborts) all of its effects are undone (rolled back).
• Consistency: Transactions produce consistent results and preserve application specific invariants.
• Isolation: Intermediate states produced while a transaction is executing are not visible to others. Furthermore transactions appear to execute serially, even if they are actually executed concurrently.
• Durability: The effects of a committed transaction are never lost (except by a catastrophic failure).

A transaction can be terminated in two ways: committed or aborted (rolled back). When a transaction is committed, all changes made within it are made durable (forced on to stable storage, e.g., disk). When a transaction is aborted, all of the changes are undone. Atomic transactions can also be nested, and in which case the effects of a nested action are provisional upon the commit/abort of the outermost (top-level) atomic transaction.

Associated with every transaction is a coordinator, which is responsible for governing the outcome of the transaction. The coordinator may be implemented as a separate service or may be co-located with the user for improved performance. It communicates with enlisted participants to inform them of the desired termination requirements, i.e., whether they should accept (commit) or reject (rollback) the work done within the scope of the given transaction. For example, whether to purchase the (provisionally reserved) flight tickets for the user or to release them. A transaction manager factory is typically responsible for managing coordinators for many transactions. The initiator of the transaction (e.g., the client) communicates with a transaction manager and asks it to start a new transaction and associate a coordinator with the transaction.

Traditional transaction systems use a two-phase protocol to achieve atomicity between participants, (a three-phase protocol may also be supported, but it rarely is these days): during the first (preparation) phase, an individual participant must make durable any state changes that occurred during the scope of the transaction, such that these changes can either be rolled back or committed later once the transaction outcome has been determined. Assuming no failures occurred during the first phase, in the second (commitment) phase participants may “overwrite” the original state with the state made durable during the first phase.

In order to guarantee consensus, two-phase commit is necessarily a blocking protocol: after returning the first phase response, each participant who returned a commit response must remain blocked until it has received the coordinator’s phase 2 message. Until they receive this message, any resources used by the participant are unavailable for use by other transactions, since to do so may result in non-ACID behavior. If the coordinator fails before delivery of the second phase message these resources remain blocked until it recovers.

As we’ve mentioned, transactions are required to provide fault tolerance. What this means is that information about running transactions (often referred to as in-flight transactions) and the participants involved must survive failures and be accessible during recovery. This information (the transaction log) is held in some durable state-store. Typically the transaction log is scanned to determine whether there are transactions mentioned in it that require recovery to be performed. If there are, then the information within the log is used to recreate the transaction and the recovery subsystem will then continue to complete the transaction.

Failures aren’t restricted to just the transaction coordinator. Therefore, participants must retain sufficient information in durable store so that they too can be recovered in the event of a failure. What information is recorded will obviously depend upon the participant implementation.

OK, so that's ACID in a nutshell. Next time we'll move on to take a look at the protocol that may run either side of two-phase commit: synchronizations.

Large scale transactions (ACID, BASE and CAP)

2011-03-12T23:34:00.000Z

I've been working with transactions for quite a while and in the area of large-scale (numbers of participants, physical distance) since the original work on the Additional Structuring Mechanisms for the OTS (aka Activity Service). However, it wasn't until Web Services transactions, BTP, WS-CAF and WS-TX that theory started to get put into practice. We first started to talk about relaxing the ACID properties back with the CORBA Activity Service, but it was with the initial submissions to BTP that things started to be made more explicit and directly relevant.

Within the specifications/standards and associated papers or presentations, we made statements along the lines that isolation should be a back-end issue for services or the transaction model (remembering that one-size does not fit all). The notions of global consistency and global atomicity were relaxed by all of the standards. For instance, sometimes it is necessary to commit some participants in a transaction and roll back others (similar to what nested transactions would give us). Likewise, globally consistent updates and a globally consistent view of the transaction outcome have to be relaxed as you scale up and out.

Now I didn't find this as much of a leap of faith as some others, but I think that's because when I was doing my PhD I spent a lot of time working with weak consistency replication protocols. There's always been a close relationship between transactions and replication. Traditional replica consistency protocols are strongly consistent: all of the replicas are kept identical and this is fine for closely coupled groups, but it doesn't scale. Therefore, weak consistency replication protocols evolved in the 1980's and 1990s, where the states of replicas are allowed to diverge, either forever or for a defined period of time (see gossip protocols for some background). You trade of consistency for performance and availability. For many kinds of applications, this works really well.

It turns out that the same is true for transactions: in fact, it's necessary in Web Services if you want to glue together disparate services and domains, some of which may not be using the same transaction implementation behind the service boundary. I still think the best specification to illustrate this relaxation of the various properties is WS-BusinessProcess, part of WS-TransactionManager (OASIS WS-CAF). Although Eric and I came up with the original concept, we were never able to sell it to our co-authors on WS-TX (so far). I think one of our failings was to not write enough papers, articles or blogs about the benefits it offered and the practicalities it fit. However, every time I explained it to people in the field it was such an easy sell for them to understand how it fit into the Web Services world so much better than other approaches. (The original idea behind WS-BP came from some of the RESTful transactions work we did in HP, where it was code-named the JFDI-transaction implementation.)

I still find it a pleasant surprise that although our co-authors from Microsoft on WS-TX didn't get the reasons behind WS-BP, other friends and colleagues such as Pat Helland started to write about the necessity to relax transactionality. I like Pat's use of relativity to explain some of the problems. However, when I had to come and talk about what we'd been doing in the world of transactions for the past decade I thought Heisenberg's Uncertainty Principle was perhaps slightly better: you can either know the state that all participants will have, but not when; or vice versa.

Transactions and TorqueBox

2011-03-12T23:32:00.000Z

About a year ago Bob McWhirter and I were talking about TorqueBox and the subject of transactions came up. Bob and team are doing a great job of exposing the enterprise capabilities that Java developers take for granted to the Ruby community, so once they'd added messaging I think transactions were the next obvious step. Since I was looking for an opportunity to improve my Ruby skills and Bob was looking to add transactions, I think it was definitely a Casablanca moment.

We scoped out the requirements for transactions, both short term and long, using JBossTS within JBossAS. Short term was pretty simple: just adding the ability to start and end transactions, with the assumption that the underlying components written in Java, such as a messaging system, would use any transaction context associated with the thread to "do the right thing" as far as transactional resources are concerned. So really this is no more than adding a Ruby client interface to something like UserTransaction in the JTA. Useful, but limiting. That work was done and should hopefully be coming to a TorqueBox release near you soon.

Longer term we want to give the Ruby developer full access to a toolkit to write transactional applications from the ground up. That means writing transactions objects and participants, not just being able to start or stop transactions. And because the Ruby community isn't as tied to the legacy of XA, we don't need to mandate XAResources, top-level transactions only or two-phase commit. Now this is where it really gets interesting. I'm hoping to use a lot of the underlying ArjunaCore implementation (and possibly the STM implementation too, which reminds me that I need to finish the blog entries around that soon). I want to provide something that is incredibly simple and yet also incredibly powerful if you want to really get into the details.

I like the way that the TorqueBox team have architected their system and I like the opportunities that it and Ruby in general provides. Transaction support should be something that fits into that ethos and I think that having a relatively clean slate, which we don't really have with Java, is a good thing. It's almost back to the old days when we were writing Arjuna in C++, which at that time hadn't even been released officially! Happy days ahead!

Heuristics and why you need to know they can happen!

2011-03-12T23:09:00.000Z

Imagine you walk into a bank and want to perform a transaction (banks are very useful things in transaction examples). That transaction involves you transferring money from one account (savings) to another (current. You obviously want this to happen with some kind of guarantee, so for the sake of this example let's assume we use an ACID transaction.

To ensure atomicity between multiple participants, a two phase commit mechanism is required: during the first (preparation) phase, an individual participant must make durable any state changes that occurred during the scope of the atomic transaction, such that these changes can either be rolled back (undone) or committed later once consensus to the transaction outcome has been determined amongst all participants, i.e., any original state must not be lost at this point as the atomic transaction could still roll back. Assuming no failures occurred during the first phase (in which case all participants will be forced to undo their changes), in the second (commitment) phase participants may “overwrite” the original state with the state made durable during the first phase.

In order to guarantee atomicity, the two-phase protocol is necessarily blocking. If the coordinator fails, for example, any prepared participants must remain in their prepared state until they hear the transaction outcome from the coordinator. All commercial transaction systems incorporate a failure recovery component that ensures transactions that were active when a failure occurred are eventually completed. However, in order for recovery to occur, machines and processes obviously need to recover! In addition, even if recovery does happen, the time it takes can be arbitrarily long.

So, in our bank example, despite the fact that we're using transactions and assuming that the transaction system is reliable, certain failures will always occur, given enough time and probabilities. The kinds of failure were interested in for this example are those that occur after the participants in the two-phase commit transaction have said they will do the work requested of them (transfer the money) i.e., during the second (commit) phase. So, the money has been moved out of the current account (it's really gone) and is being added to the savings account, when the disk hosting the savings account dies, as shown in the diagram. Usually what this means is that we have a non-atomic outcome, or a heuristic outcome: the transaction coordinator has said commit, one participant (savings account) has said “done”, but the second one (current account) has said “oops!”. There's no going back with the work the current account participant has done, so this transaction isn't going to be atomic (all or nothing).

Imagine that this error happens and you don't know about it! Or at least don't know about it until the next time you check your account. Not good. Personally I'd like to know if there's been an error as soon as possible. In our bank scenario, I can go and talk to someone in the branch. If I was doing this via the internet there's usually a number I can call to talk to someone.

But fortunately most enterprise transaction specifications, such as the OMG’s Object Transaction Service and the X/Open XA specification, and implementations such as JBossTS allow for heuristics to occur. This basically means that the transaction system can be informed (and hence can inform) that such an error has happened. There's not a lot that can be done automatically to fix these types of error. They often require semantic information about the application in order to restore consistency, so have to be handled by a system administrator. However, the important thing is that someone knows there's been a problem.

REST, Cloud and transactions

2011-03-12T22:46:00.000Z

REST has grown in popularity recently for a variety of reasons. Developers are attracted to the simplicity of the interfaces created. Since HTTP is such a ubiquitous protocol, developers get lightweight interoperability out of the box because most languages and platforms support both client and server interactions with their built-in HTTP support. REST also provides developers with a strong set of architectural guidelines and constraints. As developers explore these techniques, they are finding that their distributed interfaces become more decoupled, usable, and maintainable over time.

It is true that the Web and REST have progressed well without transactions. However, we believe that there are circumstances and particular applications where the use of transactions, or at least atomicity, would be beneficial. As we have evangelized REST, we have found that a frequent question is: how can application developers leverage transactions? As the movement to the Cloud (public or private) gathers momentum, these same questions arise more and more, either when cloud-based applications need to interact with clients or services outside of the Cloud or within the Cloud, where HTTP is often the only guaranteed means of communication.

These questions are often the result of having tried to do without transactions initially and found the resulting systems inadequate. Sometimes those users have come from backgrounds such as Java Enterprise Edition, where they expect such capabilities and have architected for them. Of course it could be that some of these applications were designed inappropriately and the apparent need for transactions would disappear through a careful redesign. However, this cannot account for all of these use cases.

However, you might ask: Why not use Web Services transactions? After all, WS-Transactions defines atomic and compensation based models and has demonstrated interoperability between all of the major transactions vendors. So the obvious question is why not simply use WS-Transactions? There are several reasons for this:

• The typical Web Services stack is often too large and complex. By leveraging HTTP as both a rich protocol and message format we can reduce the footprint at both the client and the server.
• The HTTP protocol already has a rich vocabulary. For instance, we use Links to convey to clients different ways in which they can interact with the transaction manager.

So how do we go about creating a RESTful transaction protocol? Understanding state and how it relates to transactions has influenced our approach to the REST transaction protocol. We have tried to ensure that the protocol embraces HATEOAS principles rather than just using HTTP as a means of conveying message protocols. For instance, if we consider the two-phase commit protocol, one way of instructing a participant to prepare and commit would be through the use of multiple URIs, such as /participant/prepare and /participant/commit, where the root of the URI (/participant) is the actual participant resource on which the protocol is ultimately operating and whose state is ultimately being changed as a result. A POST request on these URIs could then be used to trigger the relevant operation.

However, we took a different approach; one which is intimately tied to state management and which we believe is more in the HATEOAS approach. Rather than define a URI per operation, our protocol requires a single URI for each participant (as well as coordinator) and the invoker (e.g., the coordinator) requests that the participant change its state to the relevant value via PUT, e.g., to prepare a participant the coordinator would PUT the status Prepare to the URI.

As mentioned previously, working with HTTP gives us a lot of flexibility to address transactional and distributed system faults without having to resort to ad hoc approaches. For instance, attempting to DELETE any transaction will return a 403 response code. Requesting that a transaction commit may return either 200 or 202 HTTP codes (OK and Accepted, respectively). In the latter case the Location header should contain a URI upon which a GET may be performed to obtain the asynchronous transaction outcome later. If the operation fails, e.g., because a participant cannot be committed, then the protocol requires that implementations return the 409 code, i.e., Conflict. If the participant is not in the correct state for the requested operation, e.g., it receives a Prepare when it has been already been prepared, then HTTP gives us another convenient code to return: 412, i.e., Precondition Failed.

But just as with WS-Transactions, we believe that ACID semantics and specifically atomic outcomes, are not appropriate for all applications. The use of ACID transactions within a REST application can break REST principles. Fortunately we also believe that there is a solution in what are commonly referred to as extended transactions. If the transaction needs to undo then it instructs the services to perform some compensation work. This means that in a loosely coupled environment there is no retaining of locks or provisional state change for long durations. We have been working on a compensation transaction model for REST based on Sagas. At this stage the compensation protocol is still under development but the goal is to provide something that is not only a good REST citizen but also does not turn a RESTful application that uses it into one that cannot claim to be RESTful.

Uncaging Cassandra

2011-03-11T12:44:00.000Z

The ongoing debate between proponents of locking and multi-version concurrency control (MVCC) is almost as heated as that between supporters of vi and emacs. There is nothing fundamentally wrong with doing concurrency control in your data store using multiple reader, single writer locking. Except that multi-version concurrency control is often better.

In an MVCC system with ongoing transactions there may be several copies of a given item of data, representing its state at different points in time. A consistent read over multiple data items can be obtained by using the version of each item that most recently pre-dates the start of the transaction. In other words, the read simply ignores updates from others that occur after its transaction began. Using this system, readers never block each other, nor do they block writers.

Compared to locking, an MVCC approach places very little extra latency on reads. Instead of adding distributed communication to manage a global lock, there is just some extra data managed in the read - multiple copies of the value, from which one is selected for use according to the versioning information. Writes are a bit more involved, as they must preserve the old value and timestamp before updating the new one. So, the reduced read cost comes at the price of more storage space and slightly more activity on writes. Which, as it happens, is exactly the design philosophy on which Cassandra is built...

In most systems built using MVCC, the versioning is entirely server side and not exposed to the client. We don't currently have that luxury with Cassandra, but it does have some features we can use to layer MVCC on top of the existing storage engine. Since it's a schemaless environment, we can simply inline the versions as additional Columns in the existing row, rather than requiring distinct storage. A get becomes a get_slice and the client uses the timestamps to decide which value to use. Of course we may need to consider a more involved approach to storage if we're sharing the store with clients that are not MVCC aware, but that kind of problem is true of external locking solutions too. With the new expiring columns support in 0.7 we can also avoid doing explicit garbage collection of older values - just set the expiry to an interval longer than we expect any transaction to take.

Of course we can't use MVCC as a complete replacement for locking on Cassandra, as there is no CAS support. We're stuck with locking on write to avoid lost updates. Nevertheless, the mixed approach has got some interesting possibilities to explore. Now I just need a few extra hours in each day so that I have time to actually try it...

NoSQL != NoTx

2011-03-10T15:29:00.000Z

The extent of support for transaction concepts in NoSQL systems is almost as diverse as the systems themselves. Some have not only full support for 'native' i.e. local transactions, but also support distributed transactions using XA. Others barely support CAS or durable write, much less any higher level constructs. This is not necessarily a bad thing, but does make life 'interesting' for those of us trying to build consistent abstractions over various systems, or to use them as components in the composition of large, reliable systems.

My current preference in NoSQL is Cassandra, which tends towards the 'no support for transactions' end of the spectrum and therefore presents a more interesting challenge than systems that come with XA already baked in.

Cassandra is architected as a BASE rather than an ACID system. In CAP terms it favours availability and partition tolerance over consistency. That's not to say there is no hope of consistency, just that you have to make some compromises if you want it.

There is durability (but don't forget to switch CommitLogSync from periodic to batch), and atomicity in so far as operations on a single row key are atomic. Batch mutates are non-atomic and there is no locking primitive so you can't do much in the way of consistency over multiple API calls.

Non of which is necessarily a problem if you're aware of it and writing your app from the ground up with those constraints in mind. In which case, lucky you. The rest of us get to endure some degree of pain, particularly when migrating, either mentally or physically, from a SQL environment.

There are several ongoing efforts to put a JPA like abstraction on top of Cassandra, thus making it readily accessible to the hordes of JPA trained Java EE programmers out there. Thus far these focus mainly on the data mapping and indexing considerations for query support, whilst avoiding or deferring the tricky transactional bits. That's not to say there is no appreciation of the looming problem, just that it seems nobody has stepped up and tackled it seriously yet.

In order to build an even semi-credible OGM façade over Cassandra or some other approximately key-value oriented store, we need support for certain transactional characteristics, of which repeatable read is perhaps the most fundamental.

The expectation of repeatable read is that if I retrieve the value for a given key twice in the same transaction, I expect it to have the same value, regardless of other users manipulating the database in the meanwhile. If I perform some form of search or index based query multiple times I expect the result set to be consistent within the transaction. If I update a value and then retrieve it, I expect to see the result of the update within the transaction even before I make a commit.

In an RDBMS it's the database engine's job to deal with this and all of them do, using mechanisms from locking to MVCC. All the ORM layer has to do is delegate the problem to the store. In a NoSQL world where the store has no such support and no plans to add any, the client has to do some additional work.

For simple key lookups the client can cache the result locally and use that copy to serve subsequent reads in the same transaction, effectively doing its own MVCC. Most ORMs do this already for performance, although they'll flush modifications and use the relational db engine to perform queries rather than compute the result over the cached copies, which avoids reimplementing large chunks of a db in the ORM but is not feasible when there is no support for it in the store layer.

The client cache approach is fine for small tx, but becomes a problem when the read size is large. Although they are arguably an anti-pattern, tx on large sets can be supported by using a backing store, perhaps the NoSQL engine itself, to store the cache copies when they overspill the available client RAM. However, the client needs some way of isolating these private copies and for rewriting queries to use them.

Alternatively, it can use an external lock manager to perform read/write locking on keys to prevent them being changed for the duration of the transaction. However, this only works where all access to the stores is going via clients that are lockmanager aware. In siloed deployments where a single app exclusively owns the data that is not unreasonable. In enterprise deployments where a single store is shared by multiple apps, each perhaps using a different client library, potentially in a different language, it gets a bit hairy.

The Cassandra community's best effort in this direction thus far is Cages, written by Dominic Williams. Cages is a Java client solution that uses ZooKeeper as a distributed lock manager. It's not a general purpose solution though, as it assumes the complete list of locks required in known in advance. That's fine for apps written to Cassandra's native usage model, where the assumption is that you've laid out your data based on knowing exactly what queries you're going to be performing.

Where you are trying to use Cassandra to support arbitrary JPA-QL style queries it's going to fall short though. The transaction systems in traditional databases allow for gradual acquisition of locks on demand over the course of multiple operations in the transaction and JPA or any other interface that supports ad-hoc querying is going to assume that support. The locking model used by Cages also assumes relatively low contention and is likely to scale poorly compared to an MVCC based solution. Fair enough - it's written by an end user to scratch their own itch, so it's simpler and less powerful than middleware written for the general case.

Whilst a good locking system allows for the provision of several transactional characteristics, it's not the only piece we need to provide full ACID behaviour. In particular, it's not going to help us with durable atomic updates over multiple keys. Nor does it address the issues of making updates of the NoSQL store atomic with respect to actions in other systems the app may be using, such as a message queue or relational databases. There is an expectation that such components support XA, allowing them to participate in distributed transactions coordinated by a JTA.

We're still a long way from being able to use Cassandra as a full member of the transactional resource ecosystem and it's going to be an interesting journey to get there, even assuming we should try. Sounds like it could be a fun trip for those who like adventures though.

SQL != NoSQL

2011-03-10T11:32:00.000Z

The new menagerie of NoSQL databases do not have the same characteristics as traditional SQL databases, much less each other. Which is kind of the point of using them in the first place.

So why are so many otherwise sane and smart people doing their best to ignore this fact? And why is the middleware community pandering to their delusions?

There is a dangerously seductive attraction in the idea that you can improve an existing system just by swapping out some component implementation for another using the same interface. A java.util.Map may abstract a Hashtable, HashMap, ConcurrentHashMap or distributed, replicated in-memory data grid like Infinispan. Choose wisely and your system works better. Choose poorly and things unravel pretty fast. It is necessary to look beyond the interface to the underlying implementation and understand the details in order to know how to drive the interface in optimal, or even correct, manner.

The JPA may abstract not only one of several ORMs, but one of an even larger number of relational database engines behind them. It may even be implemented using an OGM backed by a key-value store such as Infinispan or Cassandra. The ability to reuse existing JPA code or programming skills when migrating from relational to non-relational storage is attractive to both developers and management. The middleware community responds to this user demand with solutions like Kundera and Hibernate OGM, which developers lap up in ever increasing numbers. Unfortunately they often do this with an inadequate understanding of the underlying implementation details.

As middleware developers we are guilty of doing too good a job of abstracting away the underlying implementation detail. Many users willingly buy into this delusion, being all too keen to believe we can magically shield them from having to understand those details.

There are two approaches to dealing with this problem: Improve the abstraction so it becomes less important to understand the implementation details, and provide material to help the users understand those details in cases where they must.

These tasks are going to occupy a big chunk of my time in the future, as I shift attention towards providing transaction management capability for the new generation of cloud environments, where data is managed and manipulated in both SQL and NoSQL stores. Interesting times ahead I think.

Is it turned On?

2011-03-09T15:27:00.000Z

Question #0 on any tech troubleshooting checklist is, as you well know, 'Is it plugged in?'. This is followed closely by question #1, 'Is it turned On?'. Sometimes I have to relearn this the hard way.

Those joining the party late will have missed yesterday's exciting episode, at the end of which the intrepid hero is left scratching his head at the failure of his Shiny New SSD to outperform his clunky HDD. Let's move the incredibly riveting plot forward a bit with some hot command line action scenes:

$ dd if=/dev/zero of=/tmp/ssd/foo bs=4k count=1000000 oflag=direct

32.9 MB/s

This sucks. Let's swap in the noop scheduler, downgrade the journalled ext3 to ext2 and change the mount to noatime:

$ dd if=/dev/zero of=/tmp/ssd/foo bs=4k count=1000000 oflag=direct

35.0 MB/s

Better, but still sucking.

$ dd if=/dev/zero of=/tmp/ssd/foo bs=8k count=500000 oflag=direct

58.2 MB/s

$ dd if=/dev/zero of=/tmp/ssd/foo bs=16k count=250000 oflag=direct

87.6 MB/s

ok, so the performance is a direct factor of the block size. Ramp the block size up high enough and we saturate the drive at over 200MB/s. But what is limiting the number of blocks we can throw at the device? The hardware spec rates it at 50000 x 4k IOPS, which would be 195MB/s . Let's throw a few more processor cores at the problem just for the hell of it:

$ dd if=/dev/zero of=/tmp/ssd/foo1 bs=4k count=1000000 oflag=direct &

$ dd if=/dev/zero of=/tmp/ssd/foo2 bs=4k count=1000000 oflag=direct &

$ dd if=/dev/zero of=/tmp/ssd/foo3 bs=4k count=1000000 oflag=direct &

$ dd if=/dev/zero of=/tmp/ssd/foo4 bs=4k count=1000000 oflag=direct &

10.5 MB/s

10.5 MB/s

10.5 MB/s

10.5 MB/s

Well, 42 > 35, but nowhere near a linear speedup. Something is fishy here. SATA should do NCQ, which would allow all four of those processes (actually up to 32) to have outstanding requests, so we should be soaking up a lot more of that lovely bandwidth.

Unless...

$ lsmod | grep libata
libata 209361 1 piix

umm, oops.

The Intel ICH10R on our P6X58D-E is running in running in legacy IDE mode, because someone didn't check the BIOS settings carefully enough when building the machine. Not that I have any clue who that may have been. No, Sir, not at all.

Ahem. Let's reboot shall we...

$ lsmod | grep libata
libata 209361 1 ahci

Right, that's better. Off we go again:

$ dd if=/dev/zero of=/tmp/ssd/foo bs=4k count=1000000 oflag=direct
76.8 MB/s

Double the speed. Not too bad for five minutes work, even if it did require walking all the way down the hall to the machine room.

$ dd if=/dev/zero of=/tmp/ssd/foo1 bs=4k count=1000000 oflag=direct &

$ dd if=/dev/zero of=/tmp/ssd/foo2 bs=4k count=1000000 oflag=direct &

34.5 MB/s

34.5 MB/s

Huh?

With libata now correctly driving the SSD with all its features supported, those concurrent processes should be getting 70+MB/s each, not sharing it. Grrr.

Oh well, let's see how the transaction system is doing shall we. It's writing a single file at a time anyhow. Since we were already running at over 36k tx/s against a theoretical max of 55k we can't expect a 2x speedup the raw dd numbers would suggest, but we should see some improvement...

30116 tx/second.

Some days it's just not worth getting out of bed.

Change request to Clebert: make the block size in the Journal a config option.

hardware p0rn

2011-03-08T16:49:00.000Z

Like most geeks I adore shiny new tech toys. I recently built two machines to host additional virtualized slave nodes for our hudson cluster. Thanks to the drop in RAM prices I had enough budget to stuff them full (i7 based workstations, so triple channel RAM on a 6 slot board topping out at 24GB). There was even enough left over for an SSD. yay. So today I fired one up and dumped my infamous XA transaction microbenchmark onto it.

The SSD totally failed to outperform the HDD.

So I checked the disk mount points, ran some diagnostics with dd and was still left scratching my head. The test code is basically write only, that being the nature of the beast when working with transaction recovery logs. Fortunately I'd opted for an SSD with the Sandforce controller, which provides for much more symmetric read/write performance compared to some of the competition. The manufacturer specs are 285MB/s read and 275MB/s write. Let's give it a try:

$dd if=/dev/zero of=/tmp/ssd/foo bs=1M count=4000 oflag=direct
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 19.0998 seconds, 220 MB/s

ok, close enough. The HDD does not do so well:

$dd if=/dev/zero of=/tmp/hdd/foo bs=1M count=4000 oflag=direct
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 30.5214 seconds, 137 MB/s

So if I can push around 2x the data onto it, I should be able to log 2x the number of transactions per second, right? cool.

Umm, No.

25106 tx/second to HDD log.

24996 tx/second to SSD log.

hmmm.

A little bit of fiddling around with the ObjectStore provides another useful data point: a single transaction log record for this test case is a little over 600 bytes - basically two Xids (one for each XAResource) plus transaction id and overhead.
So, 25k tx/s * 600 bytes = 15MB/s give or take. Less that 1/10th of what the SSD should handle.

We know the benchmark will run at over 50k tx/s with an in-memory store, so we're definitely bottlenecked on the I/O somewhere, but it's not write bandwidth. With a conventional HDD I'd put my money on the number of physical writes (syncs/forces) a drive head can handle. High performance log stores are designed to do contiguous append to a single file to eliminate head seek latency, so it's something to do with the number of events the device can handle in series. Let's use the filesystem's native block size:

dd if=/dev/zero of=/tmp/hdd/foo bs=4k count=1000000 oflag=direct
1000000+0 records in
1000000+0 records out
4096000000 bytes (4.1 GB) copied, 139.289 seconds, 29.4 MB/s

dd if=/dev/zero of=/tmp/sdd/foo bs=4k count=1000000 oflag=direct
1000000+0 records in
1000000+0 records out
4096000000 bytes (4.1 GB) copied, 158.286 seconds, 25.9 MB/s

and there we have it - the block management is killing us.

Looks like I need to spend some time down in the native code and kernel to understand the I/O paths at work here. My feeling is that SSD technology just invalidated a lot of design assumptions that were probably a bit outdated already. SSDs don't do head seeks. The unit of parallelism is the memory bank, not the drive head. A log store designed for SSDs should probably stripe writes over multiple files for lock concurrency and not care about contiguous blocks. Unless the SSD drive firmware is written to assume and optimise for legacy HDD usage patterns. Not that a modern HDD actually does so much in the way of truly contiguous writes anyhow - there is an awful lot of abstraction between the filesystem blocks and the physical layout these days.

Even setting aside the redesign for SSD optimization it's clear there is still room for improvement here. On the HDD we should be able to trade transaction latency against batch size to reduce the number of write events and drive up the overall throughput until we saturate the drive bandwidth. On the SSD I'm not so sure we can actually saturate a properly tuned drive on this machine - the CPU can run 50k tx/s but that's only ~30MB/s of log data. Naturally a server class machine is going to have more CPU power without necessarily increasing the number of drives, but if we can retain a ratio of say one SSD to between 8-16 cores then we should be firmly in CPU bound territory. Welcome to the new era. Your design assumptions are now invalid and your transaction manager needs redesigning to be more CPU efficient. Have a nice day.

More Speed!

2011-03-08T15:41:00.000Z

"You don't have to outrun the bear, you just have to outrun the guy next to you."

Nevertheless it's always a good idea to run as fast as possible - there is no telling when the guy next to you may put on a sudden burst of acceleration.

So we did some more performance improvements on JBossTS.

For transactions that involve multiple XAResources, the protocol overhead is divided between network round trips to communicate between the TM and RMs, and disks writes to create a recovery log. There are a limited number of tricks that can be done with the network I/O (issuing the prepares in parallel rather than series springs to mind), but the disk I/O is another matter.

JBossTS writes its log through an abstract storage API called the ObjectStore. There are several ObjectStore implementations, including one that uses a relational database. Most however are filesystem based. The default store is an old, reliable piece of code that has worked well for years, but recent hardware evolution has led us to question some of the design decisions.

For years now multicore chips meant the number of concurrent threads in a typical production app server deployment rising and the processor capability in general outstripping the disk I/O capability. Relatively recently SSD have rebalanced things a bit, but we still see a large number of threads (transactions) contending for relatively limited I/O capability.

Time to break out the profiler again...

For total reliability a transaction log write must be forced to disk before the transaction can proceed. No in-memory fs block buffering by the O/S, thank you. This limits the ability of the O/S and disk to batch and reorder writes for efficiency. Essentially there are a fixed number of syncs() a drive can perform per second and for small data like tx logs it's not bottlenecked on the I/O bandwidth.

The design of the current default store is such that it syncs once for each transaction. This becomes a problem when the number of transactions your CPUs and RMs can handle exceeds the number of syncs your disk array can handle. Which is pretty much what's happened. So, back to the drawing board.

Clearly what we need is some way for multiple transaction log writes to be batched into a single sync. Which in turn means putting the log records in a single file, with all the packing and garbage collection problems that entails. Then you have the thread management problems, making sure individual writes blocks until the shared sync is done. That's a lot of fairly complex code and testing. Nightmare.

So we found someone who has already done it and borrowed their implementation. Got to love this open source thing :-)

The new ObjectStore implementation is based on Clebert's brilliant Journal code from HornetQ. The same bit of code that makes persistent messaging in HornetQ so staggeringly quick.

The Journal API is not a perfect for for what we want, but nothing that can't be fixed with another layer of abstraction. One adaptor layer later and we're ready to run another microbenchmark.

First let us get a baseline by using an in-memory ObjectStore (basically a ConcurrentHashMap). Useless for production use, but helpful to establish the runtime needed to execute a basic transaction with two dummy resources and all the log serialization overhead but no actual disk write.

53843 tx/second using 400 threads.

ok, that will do for starters - thanks to lock reduction and other performance optimizations made earlier we're pretty much saturating the quad core i7 CPU. We could probably improve matters a bit by tweaking the ConcurrentHashMap lock striping, but let's move on and swap in the default ObjectStore:

1650 tx/second using 100 threads.

iowait through the roof, CPU mostly idle. dear oh dear. Look on the bright side: that means we've got a lot of scope for improvement.

Adding more threads just causes more scheduling and locking overhead - we're bottlenecked on the number of disk syncs the 2x HDD RAID-1 can handle.

Let's wheel out the secret weapon and plug in it...

20306 tx/second at 400 threads.

yup, you read that right. Don't get too excited though - you won't see that kind of performance improvement in production. We're running an empty transaction with dummy resources - no business logic and no RM communication overhead. Still, pretty sweet huh? Better buy Clebert a beer next time you see him. And one for me too of course.

And if you thought that was good...

Linux has a nifty little library for doing efficient asynchronous I/O operations. If you are running the One True Operating System and you don't mind polluting your Pure Java with a little bit of native code you can drop a .so file into LD_LIBRARY_PATH and leave the other guy to be eaten by that bear:

36491 tx/second at 400 threads.

Coming Soon to a JBossTS release near you. Enjoy.

Must Go Faster

2011-03-08T14:17:00.000Z

Transaction management systems, like much other middleware, embody a very simple tradeoff: they make life easier for developers at the cost of a runtime overhead. Maximising the ease of use and minimising the overhead are things we spend a fair bit of time thinking about.

Conventional wisdom in the transaction community is that the runtime cost of a classic ACID transaction is dominated by the disk writing needed to create a log for crash recovery purposes. As with much folklore there is some truth in this, but it's not the whole story. For starters, there are some use cases where we don't need to write a log at all.

In transactions that have only one resource it's easy to optimise the 2PC protocol to a 1PC and avoid much of the overhead. But here is the snag: the design of the transaction API in Java EE does not allow for developers to communicate metadata about number of resource managers expected in the transaction ahead of time. In some cases it's not actually known, as it may depend on runtime data values. However, it's disappointing that you still need some parts of the XA protocol overhead even where the transaction is known at design time to be local (native) to a single RM.

xaresource1.start();
xaresource1.end();
xaresource1.commit();

is still three network rounds trips and one disk sync. That's a huge improvement on the eight trips and three syncs needed for a 2PC, but it's nevertheless more than the single trip and sync in the native case with

connection.commit();

Fortunately there is a workaround: define both XA and non-XA ('local-tx' in JBossAS -ds.xml terminology) datasources in the app server and use the local one wherever you know the transaction won't involve other RMs.

Perhaps one day we'll have a less clunky solution - maybe being able to define a single datasource as supporting both XA and non-XA cases, then annotating transactional methods with e.g. @OneResource or @MultiResource to tell the JCA and TM how to manage the connection. Or even being able to escalate an RM local tx to an XA one on demand rather than having to chose in advance, although that would need RM support as well as changes to the XA protocol and Java APIs. Dream On.

Even where it's running with 1PC optimization for a single RM, the transaction manager still provides benefits over the native connection based transaction management. The most critical is the ability to handle certain lifecycle events, in particular beforeCompletion(), a notification phase that allows in-memory caches such as an ORM session to be flushed to stable store and its companion afterCompletion() which allows for resource cleanup. The TM's ability to manage transaction timeouts is also important to prevent poorly written operations from locking up resources for a protracted period.

As with writing logs for 2PC recovery, the management of timeouts is one of those activities we have to do every time, even though it turns out to be required in only a tiny minority of cases. Efficiently managing the rollback of transactions that have exceeded their allotted lifetime is a seemingly trivial overhead compared to the log write and as a result the code for it received little attention until fairly recently. This is where the folklore came to bite us: conventional wisdom dictated that we should focus the performance tuning effort on the I/O paths and not worry too much about functions that just operated on in-memory structures.

WRONG.

For the reasons outlines above, in a typical app server workload there are an awful lot of transactions containing just a single resource and hence not doing a log write. D'Oh. For those use cases the overhead of the in-memory activity in the TM is actually significant. So we sat down, wrote a highly concurrent 1PC microbenchmark test scenario, put it though a profiler, shuddered and went down the pub.

When we'd recovered a bit we tuned the transaction reaper, a background process responsible for timing out transactions. By deferring much of the work as long as possible it turns out to be possible to skip it entirely for many transactions. A short lived tx does not always need to be inserted into the time ordered reaper queue - it may terminate normally long before the reaper needs to take any action. By being lazy we saved a lot of list sorting and the associated locking overhead.

As a result the recent JBossTS releases substantially outperform their predecessors in the single resource case, particularly when scaling to a large number of threads. Upgrade and Enjoy.

Next time: More Speed! Very fast I/O for 2PC logging.

Hello World

2011-03-08T13:47:00.000Z

So apparently there is this thing called 'blogging', which seems to consist of equal parts narcissism and self-publicity. As far as I can tell the aim of the geek's version of this game is to make yourself insanely attractive to hiring managers who need to fill obscure technical vacancies. Oddly enough this activity not only counts as work, but is actively encouraged by management. Sign me up!

What's that? You want actual content? Oh, umm, right then. Well let's start with something simple shall we? Maybe I should introduce myself. I'm Jonathan Halliday, dev team lead for JBossTS. I spend my days working on transactions code, trying to remove more bugs than I add. After ten years or so I think I'm gradually getting the hang of it, which probably means it's time to find a new challenge. Meanwhile I'll try to spend a little more time communicating about the cool stuff I'm doing instead of just getting on with it. Starting, in the next exciting episode, with some impressive performance improvements. Because hey, who doesn't like faster code, right?

Jonathan on Memory Resident Transactional Objects

2011-02-17T12:30:00.001Z

I came across this gem from the JBossTS project lead. Well worth a read. It also reminds me that I need to write up some of the stuff I did over Christmas. Now where did I put that spare hour or so?

HAL FUD rebuttal

2010-11-22T15:50:00.000Z

A while ago I got rather annoyed at the shoddy work that IBM did at FUD around JBossTS. I later said that we were going to produce a response (as if one was really needed after 25 years of work, but hey, this is the Youtube generation!) Well it's taken a little longer than I had expected, but it's now available. And unlike the original IBM mess, this demo includes a reference to the source code so you can download it and try it for yourself. I'd recommend to IBM that they give it a go and perhaps re-examine their own tests as a result, but I doubt they're interested in facts and the truth.

Thanks the the entire team for pulling this together! It was a great JBoss effort.

JavaOne presentations

2010-11-10T12:18:00.001Z

Catch various ones here, including a one of transactions.

Standalone XTS and lightweight clients

2010-10-29T10:00:00.000+01:00

I've recently been working with a community user to help restore the ability to deploy the Web Services Transactions (XTS) code outside of JBoss AS. XTS is a layer built on top of the JBoss Transactions core which allows you to implement transactional web services and transactional web service clients. Those long enough in the tooth to know about the previous incarnation of the XTS code when it was part of the Arjuna codebase will recall that in the good old, bad old days XTS had to be deployed bundled in with the web service client and/or the web service itself. When we re-implemented XTS to conform to the WSTX 1.1 standard we rebuilt it so that it deployed as a service inside JBoss AS. We also relaid the internal message transport on new foundations. Communications between the client or web service participants and the transaction coordinator were rebuilt over the JaxWS implementation provided by JBossWS-Native. More recently we upgraded XTS so that it also works with JBossWS-CXF.

Although this decision appears to tie XTS tightly into the JBoss AS product it was actually motivated by the desire to render the code independent of a specific transport layer. In theory JaxWS provides an abstraction that is available in many implementation flavours and, in theory, this means that it should be possible to deploy the 1.1 implementation in any container that

provides support for deployment of JaxWS endpoints on the server side and
allows for creation of JaxWS service proxies on the client side

That's pretty much all that is needed to allow XTS to run in any web service container, except ... it ignores a significant lacuna in the JaxWS specifications.

XTS requires the use of Web Service Addressing (WSA) headers on the protocol messages sent between the client and transaction coordinator or between the participant web service and the coordinator. The original draft JaxWS-WSA specification defined an API which enabled the two sides of a JaxWS exchange to access and manipulate WSA headers. However, this feature got shunted off into the long grass and the final version only provided a minimal WSA capability which made it impossible to implement the WSTX spec using only JaxWS standard functionality.

Luckily for me, JBossWS Native and the CXF project have both always provided an implementation of the original draft WSA feature and, modulo a few annoyingly variations (that's the problem with not standardising), this means that both these WS stacks include all that is needed to deploy the XTS service. JBossWS now includes a small library containing a simple abstraction layer which hides the minor differences in the WSA implementations. That is all that is needed for XTS to deploy inside JBoss AS over either JBossWS-CXF or JBossWS-Native. However, this also re-opens the possibility of bundling the XTS code with the JBossTS code and CXF as a standalone deployment.

Obviously, there is not a great deal of benefit in deploying a transactional web service outside of an application server. It's very likely that you will want such a web service to run inside a container located in a managed environment with 24/7 monitoring, support and so on. A web service is likely to need to employ a database and require configuration of data backups, crash recovery, centralised logging, IT staff on hand to deal with catastrophic failures etc.

Deployment of a lightweight transactional web service client is a much more interesting proposition. If you can bundle the transactional web service client software in a simple desktop Java application then this provides an easy management path for exposing reliable distributed service applications to users. The desktop client negotiate with a networked coordinator service when it creates a transaction. It can use normal web service invocations to request service from transactional web services located in the intranet or even on the internet. The web services negotiate with the same coordinator to ensure that they either all commit their changes or all revert to their original state. The more heavy-weight apparatus needed on the coordinator or web service hosts to ensure reliability, availability, security etc are not actually needed on the client node.

In theory all the transactional client needs to be able to do in order to participate in a Web Service Transaction is talk to the coordinator and the web services using JaxWS client capabilities built over an HTTP client library. There is no reason for the client to require all the capabilities of a full web service container and if definitely does not need to impose the management overheads that running a full container would mandate. The client only requires minimal installation, configuration and maintenance to operate as an HTTP client. It does not need to be involved in automatic or manual crash recovery procedures. It does not need to take part in centralised logging in order to allow monitoring/supervision by IT support staff. Nor does the client need to expose service endpoints on a public network interface and, in consequence, open holes in the client machine's firewall. Bundling the CXF client libraries with the XTS client code in a standalone Java app seems like a very attractive proposition.

In practice things are not quite so simple as that which is why I have been busy for the last month or two. The practical problems were twofold. The first problem was that XTS was built to be deployed as a monolith. When you install the XTS service inside JBoss AS you actually install the code required by all 3 of the agents involved in a transaction: the client, the participant web service and the transaction coordinator. The XTS code actually comprises about half a dozen main software services with their associated web services. These software services are layered so as to support the functions defined by the WSTX specifications. Unfortunately, the needs of the three agents, client, web service, coordinator, cut across these layers making it hard to decouple them.

So, when it comes to deploying a transactional application you might decide that you want to deploy your client application to one JBoss AS instance on Host1 and your web services on two other JBoss AS instances running on Host2 and Host3. For better reliability you might want to the coordinator services to operate in a separate JBoss AS instance on Host4. Unfortunately, up until a few weeks ago you would have to deploy, load and initialise the XTS services for all three types of agent on all of these hosts.

Well, not any more. There is now a simple configuration mechanism which allows you to enable or disable the client, participant web service or coordinator capabilities in a given XTS deployment. There is still a single XTS service archive to drop into the JBoss AS deploy directory. However, you can easily configure the deployment using either dependency injection or file-based property configuration to ensure that only the code and services you want get loaded and initialised. So, you can configure Host1 only to initialise the client capability, and so on.

If you want to deploy to a non-JBoss web container it is now also straightforward to isolate the configuration elements which control deployment of the JaxWS endpoints used internally by XTS. The XTS service archive contains several web archives which do nothing more than map endpoints exposed by the 3 XTS agents. These can easily be unbundled and the mapping can be redefined using an alternative deployment framework like, say, Spring. All you need to do to deploy XTS outside JBoss AS is to bundle CXF, the XTS and JBossTS libraries and the JBossWS WSA compatibility library with your deployment, configure which XTS agents you want to initialise then call the JBossTS and XTS start and stop methods during deployment and undeployment of your application.

This is all you need if you want to deploy to a web container. Unfortunately there is still a second barrier to deploying a simple, lightweight transactional client. This is a harder problem because it lies within the WSTX standard itself. Unfortunately, the WSTX specification, or at least the WSAT component (which specifies the Atomic Transaction protocol), requires a transactional client to expose a service endpoint. WSTX bypasses this issue as far as the WSBA (Business Activity) protocol specification is concerned, but only because it leaves the corresponding part of the WSBA specification undefined! In practice the same requirement applies for WSBA.

This may seem a bit bizarre but it's a rather unfortunate artefact of how WSTX was defined. I guess that at the time WSTX was defined it was felt that anyone implementing a transactional client would only want to deploy it inside an application server. So, this was not considered to be an issue. In order to explain why the spec is like this and what is required to fix it I'll need to go into some of the details of what the spec actually requires a client implementation to do.

The problem does not arise when an application client starts a transaction, either by calling UserTransaction.begin() or UserBusinessActivity.begin(). Under the hood these two calls use JaxWS to talk to an XTS service called the Activation Coordinator. This service employs a request-response message exchange pattern (MEP). The client opens an HTTP connection and sends a create request. The service creates a transaction and uses the HTTP back channel to hand back a transaction context containing a unique identifier for the transaction and the address of another coordinator service called the Registration Coordinator. This is the same information which gets smuggled into client requests to the transactional web services, allowing them to talk to the Registration Coordinator and register as a participant in the WSAT or WSBA transaction (although, confusingly, the standard term used for this operation is enlist).

The application client does not just stash away the context returned by the Activation Coordinator and then return fro the begin() call. It also registers (or enlists) as a participant in the transaction. Although it uses the same operation as the web services it does so for a different purpose. A transactional web service has to enlist so that it can be told when to prepare and commit (or, possibly, abort) any changes made during the transaction. The client has to enlist in order to be able to complete the transaction. In a WSAT transaction the client sends a message to the Registration Coordinator to enlist with the WSAT Completion Coordinator service. This allows it to send a termination request, either Commit or Rollback, later on. Once again the enlist operation uses a request-response MEP but this time the response on the back channel includes the address of the Completion Coordinator service. Likewise, a WSBA client enlists with the Termination Coordinator service and gets back that service's address.

The problem arises when the application client code calls UserTransaction.commit() or UserBusinessActivity.close() to terminate the transaction. The XTS code uses JaxWS to send, respectively, a Commit or Close request to the Completion or Termination Coordinator service. Alternatively, if the application client code calls UserTransaction.rollback() or UserBusinessActivity.cancel() the XTS code uses JaxWS to send, respectively, an Abort or Cancel request. Unfortunately, in all of these cases the WSAT specification mandates that this request empoloys one-way MEP. That means that the client sends the request then closes the connection without waiting for a response. So, how is it supposed to find out when the request has been handled and what happened?

Well, that is why the client needs to expose a JaxWS endpoint. The WSAT client has to implement a Completion Initiator service which will accept incoming messages using a one-way MEP, either a Committed or an Aborted message. After posting , say, a Commit request the XTS code places the calling thread in a wait until one or other messages is received by the Completion Initiator service. This service wakes up the waiting thread and notifies it of the outcome. So, what is essentially a request-response exchange is achieved using two independent messages on two independent HTTP connections one initiated from the application client to the coordinator host and one initiated from the coordinator host back to the client. The WSBA specification does not actually specify a termination protocol but the old Arjuna implementation adopted a similar model based on two one-way messages thus requiring a WSBA client also to expose an end point, in this case for the Termination Participant service.

So, how does the Completion or Termination Coordinator get hold of the address for the client endpoint? Well, the answer to that question really explains how and why this problem comes about. The enlist request supported by the Registration Service expects to be passed three values, a unique identifier for the enlisting party derived from the transaction id, the name of the service the caller wants to enlist for and, finally, a URL providing the missing address. Once the registration request has completed both sides have each others addresses. The protocol expects them to conduct any further communications using a sequence of one way messages labelled with the unique identifier supplied in the enlist call.

One way messages are not a bad idea for communications between the coordinator and the participant web services. Using a request-response MEP requires holding open a connection while waiting for the response. For, say, a Prepare-Prepared exchange this might take a long time, especially if a web service has made a lot of state changes which have to be logged to disk. Holding open lots of connections in a busy web server might cause a resource bottleneck. Also, request-response is only useful if the exchange is always a simple two-way exchange. WSBA sometimes requires longer message sequences not easily mapped onto the request-reponse model. For example, if a WSBA web service is told to complete its changes and finds that it cannot do so the 3-way message exchange, Complete, CannotComplete, NotCompleted is required.

At the client the first of these concerns is less important (there is normally only one connection per client) and the second does not apply. If the completion requests could be implemented using a simple request-response exchange then this would remove the need to expose service endpoints and, hence, allow deployment of a really lightweight client. So, the solution we have adopted is to augment the standard and provide an alternative version of the Completion and Termination Coordinator services which support this type of MEP. Our coordinator service implementations exposes two new endpoints alongside the original completion services, the Completion Coordinator RPC service and the Termination Coordinator RPC service. These services have no corresponding Initiator service. Our client implementation can be configured to use either the normal service or the RPC vwersion. So, if you deploy a client to a web container you can configure it to use the standard one-way MEP requests to the standard services. If you deploy a client as a standalone Java app you can configure it to use request-response MEPs to the extended completion services.

Of course, you can only use this lightweight client if you point it at a JBoss XTS coordinator. If you tried to use another vendor's implementation of WSTX then you would get an error when you begin your transaction because the 3rd party coordinator service would not recognise enlist requests for our extended completion services. This is not quite such a leap into the unknown as it appears. WSBA does not specify completion protocol. So, if you try to use another vendor's coordinator service with our WSBA client library it will fail whichever type of client you deploy. That's a very good reason why the WSTX specification should be extended to include a WSBA completion protocol and to add the request-response MEP variants of the completion protocols so they sit alongside the one-way MEP versions.

Transactions and volatile state

2010-06-06T17:03:00.000+01:00

So far we've looked at some of the basic of writing transactional applications using just JBossTS and TXOJ, as well as the importance of nested transactions. In this entry we'll look at another way of relaxing some of the traditional ACID properties, namely that of durability (persistence).

Hopefully you'll realize that normally when using transactions the Durability aspect kicks in with the transaction log, which maintains a record of the participants and the state of the transaction, as well as transactional participants (e.g., business objects), which have to record any state changes that are made in the context of the transaction. Both of these are needed to ensure atomicity in the presence of failures. But although disk speeds have improved significantly over the years, compared to improvements in processor speeds the disk is still the bottleneck. But it needn't be if all you want is ACI for your objects and applications. Such objects are often referred to as volatile or recoverable in that they have state that cannot survive machine crashes but which is still subject to atomic updates, i.e., previous states can be recovered if the transaction rolls back as long as there are no crashes.

But of course it's no good to simply stop your business objects from writing state, assuming you are even aware that they are doing so within a transactional application; you must have similar recognition by the transaction manager so that it doesn't write the log on behalf of volatile objects. Obviously if there are only volatile objects in the transaction then there's no need for a lot at all and performance can improve significantly. But of course you need to understand the trade-offs that using volatile objects give, which can essentially be summarized as performance vs crash recovery.

Fortunately JBossTS and TXOJ support volatile objects, including recognition by the coordinator. You don't need to do anything special when creating or starting your transactions: the coordinator will only create a log during commit if it detects more than one non-volatile object within its scope. And for your TXOJ objects you have to do very little, as illustrated by the modified AtomicObject code below from one of the previous articles:


public class AtomicObject extends LockManager
{
    public AtomicObject ()
    {
        state = 0;
    }
    
    public void incr (int value) throws TestException
    {
        AtomicAction A = new AtomicAction();

        A.begin();

        if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
        {
            state += value;

            if (A.commit() != ActionStatus.COMMITTED)
                throw new TestException("Action commit error.");
            else
                  return;
        }

        A.abort();

        throw new TestException("Write lock error.");
    }

    public void set (int value) throws TestException
    {
        AtomicAction A = new AtomicAction();

        A.begin();

        if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
        {
            state = value;

            if (A.commit() != ActionStatus.COMMITTED)
                throw new TestException("Action commit error.");
            else
                return;
        }

        A.abort();

        throw new TestException("Write lock error.");
    }

    public int get () throws TestException
    {
        AtomicAction A = new AtomicAction();
        int value = -1;

        A.begin();

        if (setlock(new Lock(LockMode.READ), retry) == LockResult.GRANTED)
        {
            value = state;

            if (A.commit() == ActionStatus.COMMITTED)
                return value;
            else
                throw new TestException("Action commit error.");
        }

        A.abort();

        throw new TestException("Read lock error.");
    }

    public boolean save_state (OutputObjectState os, int ot)
    {
        boolean result = super.save_state(os, ot);

        if (!result)
            return false;

        try
        {
            os.packInt(state);
        }
        catch (IOException e)
        {
            result = false;
        }

        return result;
    }

    public boolean restore_state (InputObjectState os, int ot)
    {
        boolean result = super.restore_state(os, ot);

        if (!result)
            return false;

        try
        {
            state = os.unpackInt();
        }
        catch (IOException e)
        {
            result = false;
        }

        return result;
    }

    public String type ()
    {
        return "/StateManager/LockManager/AtomicObject";
    }

    private int state;
}

The only real differences here are that we don't need the constructor call to LockManager with the ANDPERSISTENT flag (the default for all TXOJ objects is that they are recoverable) and it doesn't make sense to have a Uid constructor because there will never be any persistent state for this object, so it can never be recreated later. If you know that the object is only ever going to be recoverable then you may also be able to play tricks with state manipulation during save_state and restore_state calls, which is why there's a second parameter to them, which is either ANDPERSISTENT or RECOVERABLE.

But that's it. With these very minor changes to the code you can create atomic and recoverable objects that have a lot of the benefit of being used within a transaction, including transactional locking, but lacking the overhead of logging. And of course it's possible to mix recoverable and persistent objects in the same transaction seamlessly. But again, there's a price to be paid for turning off persistence. However, for some types of object and application this is a price worth paying, and one of those areas is STM. But it's not alone and you may find the idea of volatile transactional objects appealing for your own needs. In which case give it a go.

Nested transactions

2010-05-23T21:03:00.000+01:00

Here's another periodic entry on our road to discussing the STM work we're doing around JBossTS. We're going to build on the Transactional Objects for Java toolkit that I described before. But before doing so it's worth stressing that everything I'm talking about here is based on the ArjunaCore module, so it doesn't need an ORB or SOAP stack. In fact it doesn't actually need anything from an application server, whether or not that is J(2)EE based. So yes, you could take that module and embed it within your application, framework or whatever and gain full ACID transaction capabilities.

Hopefully if you're reading these entries then you already understand the concept of traditional transactions, which we'll refer to as top-level transactions here. If not then there are enough articles, presentations and books to get you there, and the JBossTS documentation will help too. But suffice it to say that with a top-level transaction, you get the all-or-nothing (Atomic) affect: no matter how many resources are updated within the scope of that transaction, if one of them commits the work then all of them will and likewise, if one of them rolls back the work then all of the others will also roll back their work.

Now this is a great capability to have, especially if you have to do very little as a user to get there. However, for some applications and objects it can be restrictive. For instance, if your transactions last for a long period of time (say minutes), updating different resources throughout, and just at the very end there's a failure, then everything has to be done again. This and other limitations with top-level transactions lead us and others to look at various extended transaction models, and one of the earliest approaches was nested transactions from 1981, also known as subtransactions (not to be confused with subordinate coordinators).

Transactions which are contained within the other transaction are said to be nested (or subtransactions), and the resulting transaction is referred to as the enclosing transaction. The enclosing transaction is sometimes referred to as the parent of a nested (or child) transaction. A hierarchical transaction structure can thus result, with the root of the hierarchy being referred to as the top-level transaction. An important difference exists between nested and top-level transactions: the effect of a nested transaction is provisional upon the commit/roll back of its enclosing transaction(s), i.e., the effects will be recovered if the enclosing transaction aborts, even if the nested transaction has committed.

Subtransactions are a useful mechanism for two reasons:

fault-isolation: if subtransaction rolls back (e.g., because an object it was using fails) then this does not require the enclosing transaction to rollback, thus undoing all of the work performed so far.

modularity: if there is already a transaction associated with a call when a new transaction is begun, then the transaction will be nested within it. Therefore, a programmer who knows that an object require transactions can use them within the object: if the object’s methods are invoked without a client transaction, then the object’s transactions will simply be top-level; otherwise, they will be nested within the scope of the client’s transactions. Likewise, a client need not know that the object is transactional, and can begin its own transaction.

Imagine what you could do with nested transactions, particularly if you get to control them programmatically. You can start transactions within your application or objects without having to be concerned about whether or not there are transactions already in scope. If you do know and care, then you can utilize them for application needs. For instance, nested transactions give you the ability to try alternate paths in a decision tree without having to worry about the impact on transactional resources; you try each option in a separate nested transaction and roll back the paths you don't chose, relying on the transaction system to undo that work automatically, and then you simply commit the selected path (nested transaction).

So how do you use them? Well in JBossTS there are no special constructs for nesting of transactions: if a transaction is begun while another is running on the thread then it is automatically nested. This allows for a modular structure to applications, whereby objects can be implemented using transactions within their operations without the application programmer having to worry about the applications which use them, i.e., whether or not the applications will use transactions as well. Thus, in some applications those transactions may be top-level, whereas in others they may be nested. Objects written in this way can then be shared between applications and JBossTS will guarantee their consistency. If a nested transaction is rolled back then all of its work will be undone, although strict two-phase locking means that any locks it may have obtained will be retained until the top-level transaction commits or rolls back. As mentioned before, if a nested transaction commits then the work it has performed will only be committed by the system if the top-level transaction commits; if the top-level transaction aborts then all of the work will be undone.

What all of the above means is that if you want to use nested transactions in your applications then all you need to is start them programmatically as we saw earlier. So for example, if we wanted to use the AtomicObject we defined earlier within a top-level transaction, the code could look like:


AtomicAction A = new AtomicAction();
AtomicObject obj = new AtomicObject();

A.begin();

obj.incr(1);

A.commit();

The fact that the AtomicObject incr method uses a transaction internally is hidden from the invoker. In this case it'll be nested. If the code looked like ...


AtomicObject obj = new AtomicObject();

obj.incr(1);

... then the transaction would be top-level. Hopefully the differences in behaviour are pretty obvious by now, but just in case: in the former case we get a chance to undo the increment if we were to roll back the enclosing transaction, whereas in the latter, assuming the incr method didn't fail, then there is no opportunity to roll back; we would have to do an application driven undo which isn't guaranteed to be equivalent from an ACID perspective (e.g., another user could have seen the state between the increment and the decrement). And of course that leads us towards compensating transactions, but they are definitely out of scope for this article.

So I hope I've given you a flavour to nested transactions and some of the power they could bring to applications and objects. Why not give them a go?

Thank you HAL Corporation for FUD spreading

2010-05-18T14:03:00.001+01:00

I mentioned a few months back how a certain group of people at a certain company that was conjectured to be the basis for HAL 9000 in 2001: A Space Odyssey were spreading FUD about JBossTS. Now not doing your homework is bad enough, but then building a whole marketing exercise around it is even worse! To say it's like building a house of cards is an understatement. It only takes someone to start questioning, probing and pushing for the whole thing to start to come tumbling down and the sheer stupidity of the original exercise comes to the fore.

Now comparing and contrasting projects, products or pretty much anything, should really be done on a scientific basis. (OK there are some things in life where that might be overkill or inappropriate, but not here.) It should be testable and reproduceable, ideally by anyone independent. If you're going to make statements about the pros and cons of this or that, then in some ways your reputation is being put on the line. Of course if you don't value your reputation then this is all moot. However, for JBoss and Red Hat, reputation and integrity are very important, so we're not going to say something bad or potentially maligning unless we have all of the facts, and even then we'll typically do it in a very scientific manner once again. This means that if we're wrong and it's because we made a mistake, e.g., had overlooked something, then we'll admit it.

Another thing we won't do is go around prospective customers spreading FUD, demonstrating FUD and making statements about "fundamental issues" with a competitor's product/project that mean it can never do X, Y or Z, unless there's proof, and in which case it's not really FUD. But of course our principles aren't shared by everyone. But when I heard that that was precisely what was happening it made their previous FUD attempts pale into relative insignificance. Despite my previous blog post, lots of real world deployments showing that recovery works, a history dating back to the 1980's which included being part of HP's foray into the middleware space, it wasn't enough for some who had been bedazzled by the IBM Roadshow (and to a degree you can't really blame them, since this is Big Blue after all.)

So we decided that enough was enough. Our honour had been besmirched (insert smiley). Fortunately recovery is so critical to a transaction system that we have lots of demonstrators and QA tests that cover pretty much every situation you can consider. And most of them are publicly available in our repository. Wow: who knew?! (Insert tongue firmly in cheek). So our teams (TS and QA) got together and in less than a day had produced something that duplicated the scenario, showed that recovery works, and therefore showed what a load of rubbish was being spouted.

Now of course it can be argued that we knew all of this to start with so we just wasted time and effort proving it yet again. I agree in part, and when this impacts on my holidays, as it did, it does make me a little bit mad. But I don't blame anyone other than the original mud slingers. Hopefully they'll learn something from this exercise. Hopefully they'll learn what open source is about and how to use it as well as contribute towards it. Unfortunately I have my doubts. So until the next time I'll leave you with this thought: what's that they say about glass houses?

Building transactional applications with JBossTS

2010-05-02T21:47:00.000+01:00

Towards the end of last year I mentioned that we've been doing some work on STM around JBossTS. I said I'd give some hints at what we've been doing, but in order to do so we need to cover some background on how you can construct transactional applications with the existing code using Transactional Objects for Java (TXOJ). You'll find a lot more information about what I'm going to summarize in the documentation and various papers.

In order for your applications to possess all of the necessary ACID properties when using transactions, the objects manipulated by those transactions need to be both durable and isolated from conflicts. If your application objects aren't backed by a database, then you have the headache of providing these capabilities yourself, unless you have your friendly neighborhood transactional framework like TXOJ. Without going into all of the details (which really are in the documentation) probably the most important class is StateManager which provides all of the basic support mechanisms required by an object for state management purposes. As with concurrency control, which we'll come on to later, durability is obtained through inheritance.

Objects in TXOJ are assumed to be of three possible basic flavours. They may simply be recoverable, in which case StateManager will attempt to generate and maintain appropriate recovery information for the object. Such objects have lifetimes that do not exceed the application program that creates them. Objects may be recoverable and persistent, in which case the lifetime of the object is assumed to be greater than that of the creating or accessing application so that in addition to maintaining recovery information StateManager will attempt to automatically load (unload) any existing persistent state for the object at appropriate times. Finally, objects may possess none of these capabilities in which case no recovery information is ever kept nor is object activation/deactivation ever automatically attempted.

There are quite a few StateManager methods that your application could utilize directly, but for now the only three that are of immediate importance are save_state, restore_state and type. The first two methods are responsible for saving (or restoring) the state of the object, whereas the last is used when positioning the state in the transactional object store. Now although the store may also be the location of the transaction log, the application object states aren't necessarily maintained within the log. (Yes, I'm glossing over subjects such as activating and passivating objects, but we may come back to those in a later posting.)

With these three StateManager methods defined you could manage the state of your object's within the scope of a transaction. But typically you'll also want isolation, so let's move on to LockManager, which inherits from StateManager. So your application objects can become isolated and durable by having their corresponding classes inherit from LockManager. Although you have to be concerned about the three StateManager methods mentioned earlier, there's nothing else here that you need worry about in terms of overriding or defining. But of course you do need to tell the system when and how to isolate your objects. The primary programmer interface to the concurrency controller is via the setlock operation. By default the runtime enforces strict two-phase locking following a multiple reader, single writer policy on a per object basis. Although lock acquisition is under programmer control, lock release is normally under control of the system and requires no further intervention by the programmer. This ensures that the two-phase property can be correctly maintained. (Again, I'll ignore subjects like type specific concurrency control for this posting.)

But sometimes a picture paints a thousand words, so here's an example:


public class AtomicObject extends LockManager
{
    public AtomicObject ()
    {
        super(ObjectType.ANDPERSISTENT);

        state = 0;

        AtomicAction A = new AtomicAction();

        A.begin();

        if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
        {
            if (A.commit() == ActionStatus.COMMITTED)
                System.out.println("Created persistent object " + get_uid());
            else
                System.out.println("Action.commit error.");
        }
        else
        {
            A.abort();

            System.out.println("setlock error.");
        }
    }

    public AtomicObject (Uid u)
    {
        super(u, ObjectType.ANDPERSISTENT);

        state = -1;

        AtomicAction A = new AtomicAction();

        A.begin();

        if (setlock(new Lock(LockMode.READ), 0) == LockResult.GRANTED)
        {
            System.out.println("Recreated object " + u);
            A.commit();
        }
        else
        {
            System.out.println("Error recreating object " + u);
            A.abort();
        }
    }
    
    public void incr (int value) throws TestException
    {
        AtomicAction A = new AtomicAction();

        A.begin();

        if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
        {
            state += value;

            if (A.commit() != ActionStatus.COMMITTED)
                throw new TestException("Action commit error.");
            else
                  return;
        }

        A.abort();

        throw new TestException("Write lock error.");
    }

    public void set (int value) throws TestException
    {
        AtomicAction A = new AtomicAction();

        A.begin();

        if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
        {
            state = value;

            if (A.commit() != ActionStatus.COMMITTED)
                throw new TestException("Action commit error.");
            else
                return;
        }

        A.abort();

        throw new TestException("Write lock error.");
    }

    public int get () throws TestException
    {
        AtomicAction A = new AtomicAction();
        int value = -1;

        A.begin();

        if (setlock(new Lock(LockMode.READ), retry) == LockResult.GRANTED)
        {
            value = state;

            if (A.commit() == ActionStatus.COMMITTED)
                return value;
            else
                throw new TestException("Action commit error.");
        }

        A.abort();

        throw new TestException("Read lock error.");
    }

    public boolean save_state (OutputObjectState os, int ot)
    {
        boolean result = super.save_state(os, ot);

        if (!result)
            return false;

        try
        {
            os.packInt(state);
        }
        catch (IOException e)
        {
            result = false;
        }

        return result;
    }

    public boolean restore_state (InputObjectState os, int ot)
    {
        boolean result = super.restore_state(os, ot);

        if (!result)
            return false;

        try
        {
            state = os.unpackInt();
        }
        catch (IOException e)
        {
            result = false;
        }

        return result;
    }

    public String type ()
    {
        return "/StateManager/LockManager/AtomicObject";
    }

    private int state;
}

What we have here is a durable and isolatable integer. It may look like quite a bit of code, but there's a lot of defensive programming here to illustrate explicitly some of the problems that could occur. It's left as an exercise for the reader to optimize the code.

However, what you can see here is the use of AtomicAction for creating transactions (which will be automatically nested if there's already a transaction associated with the thread of control) and then the acquiring of the right types of Lock within the application object code (READ or WRITE to signify whether or not the state of the object may be modified). If you consider what's needed to ensure that the object is transactional, including full recovery, there isn't a lot for the developer to do. OK, it's a bit more than you might expect if you're used to EJB3s, for example, but then there's a bit more going on here, and some of it could well be automated (future posting hint). But when you consider that an entire database system was built on this basis back in the early 1990's (OK, the C++ version of the transaction system and not Java, but very little difference), it helps to illustrate the power of the abstractions as well as the implementation.

OK, that's enough for now. In a later post we may go into more detail on some of the concepts mentioned here (or glossed over so far) before heading off into STM land. If there's anything specific you'd like concentrating on then just suggest via the comments.

Transactions: Over used or just misunderstood?

2010-03-15T14:55:00.000Z

I gave a presentation at QCon London last week on the above subject. I was part of the Solutions Track and QCon have made the slides available. The room was packed out, which was good though always a little surprising: although transactions are dear to my heart it's not something that lights everyone's fire! It seemed to go down well and there were plenty of questions during and afterwards to convince me that I probably have enough material to do 4 or 5 deep dives into the issues raised by the current presentation. So maybe at the next JUDCon event you'll be able to hear me present on the duality between successful marriages and two-phase commit!

REST-Atomic Transaction

2010-01-07T20:34:00.000Z

We've been working a lot on updating the REST transactions work for Bill's REST-* initiative. The latest drafts aren't up there at the moment but are accessible through the mail archives. The current draft 3 of the atomic model is still called REST-ACID due to some WS-ACID history, but the next draft will see a name change to REST-Atomic Transaction to more clearly call out what the model defines. Then of course there's the compensation based model that will see a new draft going up soon.

On a slight side note it's good to see others thinking about the same problem. Though I think this model has some issues it's good that others have recognized that the problem remains undefined within REST. Of course Eric and Phil wrote about some of this in their updated book (worth a read).

Someone at IBM needs to do their homework!

2010-01-05T19:11:00.001Z

I have a lot of respect for IBM, with many good friends and colleagues who work there, or have worked there over the years. I've collaborated with them for a long time on various transaction standards, articles and books. They know a thing or two about transactions, being the home of CICS.

However, sometimes they slip up as do we all. Case in point is this quality article (yes, I'm being sarcastic) from some anonymous author(s) over there in IBM Land. It's a good example of FUD and a really bad example of scientific investigation. I'm not going to help them improve their act on getting those two things to work together, but I will point out a few areas that quickly made me consign the document to the trash.

For a start, making statements like "JBoss AS has a relatively new integration with a third-party transaction manager." and "Public forums seem to indicate that this transaction manager is not proven" could easily be fixed by just reading the community pages. But then I get the impression that the author(s) didn't go near the community pages much except to find "facts" to fit their aims, carefully ignoring anything that didn't fit (not good scientific method!)

Then they concentrate on a specific AS issue which is about configuring AS to automatically recover datasources. Manual editing of files is required at the moment. The author(s) didn't do their homework (or even bother posting on the free forums as far as I can tell) to locate the information they needed. Now maybe they're relatively new to open source and didn't realise you don't have to buy a support contract or buy the software in order to talk with the developers! Maybe even reading the documentation might have helped, although perhaps there was just far too much of it for the author(s).

But does this mean that there are fundamental problems with JBossAS or JBossTS around data integrity? Not that we're aware of and there are enough QA tests and samples to justify that position. The code for any tests that would suggest otherwise is conveniently absent from the author(s) document. If anyone has it or something similar then be a good community member and send it our way. Of course there may be things we've missed (even CICS has issues to this day). Until then I suggest doing your bit for the environment and not waste paper, ink or energy on this article. Your Carbon Footprint will thank you.

STM for Arjuna

2009-12-19T19:50:00.000Z

For many years we've discussed the benefits of object-oriented programming for developing transactional applications, with Transactional Objects for Java (TXOJ) as the implementation of those ideas. As we've discussed before, this ties in with other advanced techniques such as nested transactions.

However, in order for developers to use TXOJ you have to use inheritance (remember: object-oriented principles) and modify your classes, whether C++ or Java. These days POJOs (such an overloaded term as well) are the preferred way to develop applications, acquiring enterprise capabilities through the likes of annotations and AOP. So a few months back we started to look at other approaches to TXOJ with some of the researchers at Newcastle University (the historical home of the Arjuna Project). We also looked at Software Transactional Memory and have managed to tie the two requirements together. It's too early to report on everything we've been doing (it's far from complete just yet anyway), but I hope to give some hints in some future postings.

Monitoring your JVM using Byteman 1.1.1

2009-10-21T11:47:00.000+01:00

Byteman is a bytecode injection tool developed to support testing of Java code using a technique called fault injection. However, it is also very useful for debugging and tracing Java program execution, including execution of the JVM itself. In this post I walk through an example Byteman script which monitors creation, start and exit of threads inside the JBoss Application Server.

The latest version of Byteman (1.1.1) has just been released and it includes the first batch in a series of sample Byteman scripts, all of which provide examples of how to monitor different aspects of JVM operation. The scripts inject tracing code into different JVM classes so that you can watch the JVM as it starts or stops threads, opens sockets, closes files, finalizes instances etc.

Seeing these scripts at work can be very educational -- for example, runnning up the JBoss App Server with script ThreadMonitor.txt installed I could see exactly which threads are created, started and stopped during AS bootstrap.

The script only needs 3 simple rules:


########################################################################
#
# Rule to trace thread creation
#

RULE ThreadMonitor trace create
CLASS java.lang.Thread
METHOD <init>
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceCreate($0)
ENDRULE

########################################################################
#
# Rule to trace thread start
#

RULE ThreadMonitor trace start
CLASS java.lang.Thread
METHOD start
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceStart($0)
ENDRULE

########################################################################
#
# Rule to trace thread exit
#

RULE ThreadMonitor trace exit
CLASS java.lang.Thread
METHOD exit
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT ENTRY
IF TRUE
DO traceExit($0)
ENDRULE

The first rule is injected into the constructor for java.lang.Thread, the second into Thread.start() and the third into Thread.exit(). The conditions are all TRUE because we want to see every call in the trace output. The rule actions call built-in methods traceCreate(Thread), traceStart(Thread) and traceExit(Thread),respectively. The argument to these calls, $0, is the thread object which is the target of the method invocation which triggers the rule, i.e. the one being created, started or exited. The built-in methods access this object to identify the thread's name and it's class name.

The implementation of the three trace methods is provided by a custom HELPER class, ThreadMonitorHelper, which specialises the library class StackTraceHelper provided in the sample lib jar. There is nothing special about this class -- it does not need to use Byteman library code to do its job. It is just a POJO which wraps up the code needed to print meaningful trace with a neat, simple API. All three methods print a line of text identifying the thread event (create, start or exit), thread name and thread class. The first two also use methods inherited from StackTraceHelper to look down the stack for the frame from which new Thread() or Thread.start() was called and print the details of the calling method.

Here's a sample of the output when running JBoss AS with these rules installed:


starting server default
=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT

JAVA: /usr/java/jdk1.6.0_14/bin/java

JAVA_OPTS: -Dprogram.name=run.sh -server -Xms600M -Xmx600M -XX:MaxPermSize=128m -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005  -Dorg.jboss.byteman.quodlibet -javaagent:/home/adinn/byteman/install/lib/byteman.jar=listener:true,boot:/home/adinn/byteman/install/lib/byteman.jar,boot:/home/adinn/byteman/install/sample/lib/byteman-sample.jar,script:/home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt -Djava.net.preferIPv4Stack=true

CLASSPATH: /home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/bin/run.jar:/usr/java/jdk1.6.0_14/lib/tools.jar

=========================================================================

*** Thread create JDWP Transport Listener: dt_socket java.lang.Thread
from VM runtime
Listening for transport dt_socket at address: 5005
*** Thread create JDWP Event Helper Thread java.lang.Thread
from VM runtime
*** Thread create CompilerThread0 java.lang.Thread
from VM runtime
*** Thread create CompilerThread1 java.lang.Thread
from VM runtime
*** Thread create Low Memory Detector java.lang.Thread
from VM runtime
*** Thread create main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread start main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread exit main java.lang.Thread
*** Thread create DestroyJavaVM java.lang.Thread
from VM runtime
*** Thread create Thread-1 java.util.logging.LogManager.Cleaner
from java.util.logging.LogManager$Cleaner.<init> at LogManager.java:200
*** Thread create Thread-2 org.jboss.Main.ShutdownHook
from org.jboss.Main$ShutdownHook.<init> at Main.java:883
12:38:16,588 INFO  [JBossASServerImpl] Server Configuration:

Bootstrap URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/conf/bootstrap.xml
JBOSS_HOME URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/
Common Base URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/common/
Common Library URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/common/lib/
Server Name: default
Server Base URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/
Server Library URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/lib/
Server Config URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/conf/
Server Home URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/
Server Data URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/data/
Server Log URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/log/
Server Temp URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/tmp/

*** Thread create Thread-3 java.lang.Thread
from org.jboss.bootstrap.impl.base.server.AbstractServer.start at AbstractServer.java:350
*** Thread start Thread-3 java.lang.Thread
from org.jboss.bootstrap.impl.base.server.AbstractServer.start at AbstractServer.java:352
12:38:16,590 INFO  [AbstractServer] Starting: JBoss Server[6.0.0.SNAPSHOT (build: SVNTag=JBoss_6.0.0-SNAPSHOT date=r93729)]
12:38:17,344 INFO  [AbstractMCServerBase] Starting Microcontainer, Main bootstrapURL=file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/conf/bootstrap.xml
*** Thread create Thread-4 java.util.TimerThread
from java.util.TimerThread.<init> at Timer.java:456
*** Thread start Timer-0 java.util.TimerThread
from java.util.Timer.<init> at Timer.java:154
12:38:18,259 INFO  [VFSCacheFactory] Initializing VFSCache [org.jboss.virtual.plugins.cache.CombinedVFSCache]
12:38:18,262 INFO  [VFSCacheFactory] Using VFSCache [CombinedVFSCache[real-cache: null]]
12:38:18,670 INFO  [CopyMechanism] VFS temp dir: /home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/tmp
12:38:18,675 INFO  [ZipEntryContext] VFS force nested jars copy-mode is enabled.
*** Thread create Thread-5 java.util.TimerThread
from java.util.TimerThread.<init> at Timer.java:456
*** Thread start ZipFile Lock Reaper java.util.TimerThread
from java.util.Timer<init> at Timer.java:154

During this initial stage of bootstrap most of the threads are created by the VM itself. The first one actually created and started by the AS is identified by the following lines:

*** Thread create main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread start main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread exit main java.lang.Thread

The trace shows that the name of the newly created thread is main and it is an instance of class java.lang.Thread. It is created and immediately started by a call to Thread.start() in method org.jboss.Main.main at line 719 of file Main.java. Here's the actual code:


701       Runnable worker = new Runnable() {
702            public void run()
703            {
. . .
716            };
717
718       ThreadGroup threads = new ThreadGroup("jboss");
719       new Thread(threads, worker, "main").start();

The third trace line is a little confusing. The thread which has exited is not actually the JBoss main thread just created. It is in fact the parent thread, created by the VM and also called main. The JBoss main thread goes on to create a whole lot of other threads, connection handlers, pool threads and so on.

Clearly, there are lots of possibilities for developing alternative scripts which provide different, more interesting tracing behaviour. The other scripts in the samples directory provide further simple examples of how you might instrument other JVM classes to provide runtime trace. By coding more selective rule conditions or more complex rule sets it is possible to make the tracing much more fine-grained and specific to particular circumstances. Adding more actions, possibly using other custom helper classes, would allow a variety of different statistics to be gathered and displayed.

I will explain next how you set up your program to inject these rules. The trace above shows Byteman injecting the scripted rules from program start. However, when it comes to tracing execution it would be nice if it were possible to start up your program without any scripts installed and then switch on tracing by loading and unloading rules as needed. Well, of course, it is possible to do just and I'll go on to explain what you need to change to achieve that too.

First off let's run through how JBossAS was configured to start up the byteman agent and execute this script from JVM bootstrap. All that is needed is to supply the java command which starts the JBossAS Main class with a -javaagent argument. Passing this argument to JBossAS is most easily achieved by inserting it into environment variable JBOSS_OPTS ans the running the normal JBossAS startup script.

The actual setting required was displayed in one of the very long lines of trace output:

JAVA_OPTS: -Dprogram.name=run.sh -server -Xms600M -Xmx600M -XX:MaxPermSize=128m -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005  -Dorg.jboss.byteman.quodlibet -javaagent:/home//byteman/install/lib/byteman.jar=listener:true,boot:/home/adinn/byteman/install/lib/byteman.jar,boot:/home/adinn/byteman/install/sample/lib/byteman-sample.jar,script:/home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt -Djava.net.preferIPv4Stack=true

Let's look at the actual -javaagent argument to understand what is needed to point the agent at the script and the relevant libraries. To make it a bit easier to see what is going on I have added line breaks after the equal sign and commas which separate the different options. In reality the argument needs to be supplied as one long string with no spaces or line breaks.

-javaagent:/home/adinn/byteman/install/lib/byteman.jar=
listener:true,
boot:/home/adinn/byteman/install/lib/byteman.jar,
boot:/home/adinn/byteman/install/sample/lib/byteman-sample.jar,
script:/home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt

Before we look at these options in detail I'll also point out that the java command line included the following property setting

-Dorg.jboss.byteman.quodlibet

This property enables injection into java.lang classes. Normally the agent ignores rules which apply to this package on the grounds that by modifying these classes it will likely shoot itself in the foot changing the JVM functionality it relies on to execute rules. Setting this flag tells it to remove this restriction. There is no great harm in trying this but beware! Get your rules wrong and you will probably see a JVM error such as a stack overflow.

Ok, warnings aside let's return to the -javaagent argument. The first value following the -javaagent: prefix is the location of the byteman jar. If BYTEMAN_HOME identifies the directory into which the release has been installed then the library is located in $BYTEMAN_HOME/lib/byteman.jar, in this case /home/adinn/byteman/install/lib/byteman.jar.

The = sign following the jar file name is used to separate it from the agent options which configure various aspects of the agent's behaviour. Where more than one option is supplied then they must be separated using commas. The options provided in this case include a script: option. This points the agent at the script file containing rules which trace thread activity. You can provide multiple script: options if you want to inject several sets of rules. For example, you might also want to use the ClassLoadMonitor script to monitor class loading. The agent options list also contains two boot: options which supply jars to be added to the bootstrap classpath (we'll come to this later) and a listener: option which is used to start the agent listener.

The first option is listener:true. This causes the agent to start a listener thread which can be communicated with using the submit.sh script located in the Byteman bin directory. The submit script allows you to manage the rule base at runtime. You can check which rules have been injected, compiled and executed. You can also load and unload rules while the program is running, including loading updated versions of rules to fix errors or patch the rule behaviour. The listener will not be started if listener:false is provided or if the listener option is omitted.

So, if you want to be able to switch tracing on and off you must provide option listener:true. If not then you cannot add or remove any rules and you will be stuck with whatever rules were loaded at boot time. You can, of course, omit the script: option. It's ok to start with a vanilla JVM. If you do this then you can load the rules later when you want to turn tracing on.

Let's take a closer look at the submit script. Running submit.sh with no arguments lists all the currently installed rules:

[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh
# File /home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt line 66
RULE ThreadMonitor trace start
CLASS java.lang.Thread
METHOD start
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceStart($0)
ENDRULE
Transformed in:
loader: sun.misc.Launcher$AppClassLoader@480457
trigger class: java.lang.Thread
compiled successfully
# File /home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt line 52
RULE ThreadMonitor trace create
CLASS java.lang.Thread
METHOD <init>
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceCreate($0)
ENDRULE
Transformed in:
loader: sun.misc.Launcher$AppClassLoader@480457
trigger class: java.lang.Thread
compiled successfully
# File /home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt line 80
RULE ThreadMonitor trace exit
CLASS java.lang.Thread
METHOD exit
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT ENTRY
IF TRUE
DO traceExit($0)
ENDRULE
Transformed in:
loader: sun.misc.Launcher$AppClassLoader@480457
trigger class: java.lang.Thread
compiled successfully

The output shows each of the 3 rules which were in the original script and shows that they have been injected into class java.lang.Thread loaded by the bootstrap class loader (it actually says sun.misc.Launcher$AppClassLoader@480457 but no other classloader will load this class). The last line of output for each rule indicates that the rule was parsed and type-checked successfully.

Running the submit script with the -u flag causes the rules to be unloaded:

[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh -u \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt
uninstall RULE ThreadMonitor trace create
uninstall RULE ThreadMonitor trace start
uninstall RULE ThreadMonitor trace exit
[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh
no rules installed

The agent responds to the uninstall request by removing any rules defined in the supplied scripts (n.b. if submit.sh -u is run with no script arguments it removes all rules). The agent recompiles the methods of any affected classes to remove the injected trigger code. In this case the rules only apply to class Thread so only one class needs to be recompiled. When submit completes the Thread methods will have reverted to operating as they would do normally and no more trace will be generated. So, tracing has been switched off.

The rules can be reinstated by resubmitting the script:

[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt
install rule ThreadMonitor trace create
install rule ThreadMonitor trace start
install rule ThreadMonitor trace exit

The agent handles an install request by identifying all classes to which the rules apply and recompiling them, injecting the newly uploaded trigger code. In this case the only class affected is Thread so its methods are recompiled inserting trigger code for the tracing rules once again.

If an uploaded rule has the same name as one that is currently loaded then the old trigger code is removed and new code injected. So, the submit script can be used to modify injected rules on the fly. If your rule shows something interesting and you want to see more details then just redefine and upload your trace rules to print out more details.

In other cases uploaded rules may not match any currently loaded classes. However, they are still added to the agent's rule base. The rule code will be injected when a class is loaded to which the rule applies.

Ok, finally, let's return to the boot: options provided as part of the -javaagent argument. The options are used to ensure that the two jar files are available when the trigger/rule code are type-checked and compiled.

The first jar to consider is the sample library /home/adinn/byteman/install/sample/lib/byteman-sample.jar which contains the implementation of the helper classes StackTraceHelper and ThreadMonitorHelper. When the rules are injected these classes need to be available in the JVM classpath or else the rule code will fail to typecheck and compile.

Normally, it would be sufficient to place the samples library jar in the runtime classpath provided on the java command line. Once this is done it does not usually matter what classloader the trigger class is loaded by since the system classloader is normally a parent for it. So, when type checking the injected rule the helper class will be found by delegation. As an alternative, the agent option sys: can be used to append the jar file to the system classpath. This also ensures the helper classes will be found by delegation.

The only problem with this is that the rules in the thread monitor script need to be injected into JVM classes. These classes are loaded by the bootstrap loader. Unfortunately, the system classloader is not a parent loader for the bootstrap loader -- in fact, the parent relationship is the other way round. Adding the sample library jar to the bootstrap classpath means that it is visible when injecting into JVM classes. Hence, we need to supply the boot: option for this jar.

If you didn't install your helper jar at boot time then you could always use submit.sh to install it at runtime. As long as you have started the listener you can call submit -s my.jar or submit -b my.jar to add a jar to, respectively, the system classpath or bootstrap classpath. Obviously, you need to do this before you load any rules which refer to the helper classes. Unfortunately, there is no way to remove jar files once they have been added in this way.

The other jar supplied using the boot: option is the byteman jar itself. The reasoning here is much the same -- the agent classes need to be visible to bootstrap classes. Normally, when the agent jar is installed its classes get loaded by the system classloader. However, in this case we want to inject a trigger call into a JVM class. If the agent classes such as Rule etc are loaded by the system classloader then they will not be visible from classes such as Thread. So, we also need to supply the boot: option for the byteman jar.

Note that it is not possible to add the byteman jar to the bootstrap classpath using submit. By the time the listener is up and running some agent classes will already have been loaded into the system classloader. Trying to install the byteman jar into the bootstrap classpath at this point will send you straight to classloader hell.

So, let's recap. If you want to start the agent so you can load and unload scripts at runtime and inject into any class, including JVM classes, you need to set

export BJAR=$BYTEMAN_HOME/lib/byteman.jar

and then run your java command with arguments

-javaagent:$BJAR=listener:true,boot:$BJAR

-Dorg.jboss.byteman.quodlibet

When you want to start tracing (firsttime only) you install your helper jar using

$BYTEMAN_HOME/bin/submit.sh \
-b $BYTEMAN_HOME/sample/lib/byteman-sample.jar

You can then load your script using command

$BYTEMAN_HOME/bin/submit.sh \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt

To check that your rules have been injected and compiled use command

$BYTEMAN_HOME/bin/submit.sh

To stop tracing you unload the script using

$BYTEMAN_HOME/bin/submit.sh -u \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt

Native transactions - cross post

2009-09-29T21:58:00.001+01:00

I had a few things to say about the latest presentation from Spring on transactions. Worth a look if you're thinking about whether or not you need to use a transaction manager (from anyone, not just us) since the presentation isn't always as accurate as it could (should) be.

Byteman 1.1.0 now available

2009-09-18T13:08:00.000+01:00

Byteman 1.1.0 has now been tagged, released and made available for download.

This release includes one major new feature:

support for dynamic upload of rules into running programs

Note that this includes the ability to retransform already loaded classes, injecting rules into code while the program is executing it. This feature allows Byteman to complement conventional debugging tools, injecting tracing code or fixes into programs on the fly. In order to support this feature Byteman now only runs on JDK 6 upwards.

The release also provides several bug fixes, most notably:

full support for injection into bootstrap loader classes (only java.lang.* is exempt -- here be dragons ;-)
retrospective injection of rules into bootstrap classes loaded prior to Byteman agent installation
fixing of several problems with array types/instances and throw action

New (old) Byteman release now available, New (new) Byteman release in the works

2009-09-04T17:09:00.000+01:00

Yesterday I published the latest release of the 1.0 Byteman series, version 1.0.3. That's the new old release in the title. This is a maintenance release which fixes about 15 bugs/features discovered while using Byteman to fix timing issues in various of the JBossTS unit tests. This version has also been used for testing on the MicroContainer project and plans are afoot to extend its use to other JBoss projects.

New features in 1.0.3 include: the ability to reference method parameters by name in rule conditions and actions; option to omit empty BIND clauses; and new helper built-ins to enforce waits on thread exits and to provide better management of rendezvous. Bug fixes include: better parser error detection; and correct reporting of line numbers in parser/type checker errors. Binaries, sources and full release notes are included on the download site.

The new new Byteman release is 1.1.0 which I have been developing in parallel with the maintenance release. It will only run on JDK 6 upwards, allowing it to profit from the retransform capability introduced into the JDK 6 instrumentation package. This new new release will support dynamic loading/reloading of rules into a running program much as is possible using the btrace package. The byteman agent can be configured to open a listener which, currently, can be used either to list all currently applied bytecode transformations or to upload new rules or redefinitions of existing rules. The rule upload capability includes applying rule injection retrospectively to already loaded classes.

This dynamic upload capability should simplify development of rule sets. It should also make it a lot easier to use Byteman as a debugging aid. Hopefully, it will also pave the way for automation of rule injection via IDE plug-ins. Developing tool support is definitely on the agenda for the project so anyone who wants to contribute is more than welcome to respond.

I have just checked in a first draft of the dynamic upload functionality into the byteman repository trunk and I already have a community user lined up to try out the redefine capability. Feedback and suggestions for improvements will be graciously (and, quite possibly, gratefully :-) received.

Byteman project goes live on jboss.org

2009-06-23T22:30:00.000+01:00

Just in time for my talk at Jazoon I have set up a JBoss project for Byteman -- that's the new, official name for TOAST the fault injection tool we have been using to test Web Services Transactions crash recovery. The project page is pretty minimal at the moment, containing only a welcome and downloads page. However, we will very soon have a user forum and JIRA for the project and will be moving the code from its current temporary location in the JBossTS repository.

I want to to make it as easy as possible for people to try out Byteman and provide feedback on problems encountered, contributions to the code base or new ideas for how to improve or extend its functionality. If you have a Java program you want to test, or even just trace its execution, then take a look at Byteman and see how it can help you.

RESTful transactions brings out the crowds

2009-06-16T12:29:00.000+01:00

We gave a session on the RESTful transactions work at JavaOne. It was well attended and there was a lot of good discussion afterwards. Then a separate discussion kicked off on the REST-discuss mailing list, followed by some InfoQ article comments. At that point I got a serious case of deja vu (or maybe there's a glitch in the Matrix?)

I've spent many years working in various groups, companies, books and articles on explaining this. I really don't want to have to do it all again! It doesn't matter if you're living in the Land of WS-*, or REST World, or the SOA Nation, we all know that ACID transactions aren't suitable for loosely coupled interactions. We all know that there are problems with forcing atomicity, consistency, isolation and maybe even durability. As they say, we've been there done that and bought the T-shirt! Move along. This is not the argument you're looking for. Move along.

Another REST+transactions protocol

2009-06-10T19:54:00.000+01:00

During discussions on REST and transactions, Alexadros pointed us at some work they've been doing. Very interesting. Once we get a chance to look through each others approaches I intend to start a discussion on our design forums. If you're interested then join in.

JBossTS offsite

2009-03-29T20:24:00.001+01:00

We had another good offsite meeting earlier this week. As usual Jonathan was spot-on with the selection of venue (the Black Swan Hotel): good food and a great atmosphere for discussions and planning. We had the JBossTS and Blacktie teams together in one place, so were able to cover a lot of things from TOAST through RESTful transactions and OSGi. Jonathan was able to plan out the next 6 months or so for the team and it's a good mix of definite current needs and beliefs on future requirements. I can't wait for the next meeting!

Compiled Configurable TOAST

2009-03-25T16:43:00.000Z

In between completing fully functional crash recovery for the Web Services Transactions (XTS) code and writing a comprehensive test suite for it I've been able to squeeze in a couple of nice extensions to the functionality of TOAST, the code injection tool I have been using to automate XTS crash recovery testing.

Attentive readers may recall that TOAST is a nifty, easy to use tool for injecting side-effects into Java code. I have been using TOAST to test XTS recovery by injecting code which crashes the JVM in the middle of a transaction commit. I also use TOAST to inject further faults during recovery in order to delay or drop messages exchanged between the transaction coordinator and the web service participants it is trying to corral into a successful commit. Finally, I inject trace statements at suitable locations in the code which track the progress of the test and allow the test client to identify successful or failed execution.

None of this requires any modifications to or recompilation of the deliverable application being tested. TOAST just needs to be supplied with a script identifying the trigger points, locations in the application code which need to be tweaked, and the Java expressions which are to be (conditionally) executed at those trigger points. Point your JVM at the TOAST jar and the script when you start your Java application and TOAST makes sure all the required side effects get executed.

For those who have not yet had a look at TOAST I'll recap a little. Scripts comprise a sequence of Event Condition Action (ECA) rules. The Event part is the specification of the trigger point, a class name, a method located in that class and some location in the method code. Locations identify a point in the source code through reference to the code structure. Examples might be: the start of the method (AT ENTRY) when a return occurs (AT EXIT), before the first read of an instance field (AT READ myField), after the 2nd call to a method (AFTER CALL getMyField() 2) and so on. The event specifies WHEN the side-effects coded in the rule are are considered for execution by describing a trigger point. Whenever a thread reaches this trigger point it enters the rule engine to decide whether to fire the associated rule.

The Condition part is a Java expression specifying WHETHER to fire the rule. If this evaluates to true then the rule fires and executes its Action. If the condition is false then control returns to the trigger method and execution continues as normal.

The Action part specifies WHAT side-effects need to occur. It is a sequence of Java expressions to be evaluated one after another. The last expression may be a return or a throw action. These cause an immediate abnormal return of control from the triggering method bypassing normal execution of the rest of the method code. If the method is non-void then the return action must include an expression evaluated to compute the return value. A throw action must include an exception type and arguments for the exception constructor.

In order to simplify testing of multi-threaded code TOAST's rule language also includes built-in methods for use in conditions and actions. These are a variety of operations which simplify the development of multi-threaded tests. For example, one rule might include a call to waitFor() in its action list, suspending any thread which fires the rule. Another rule might include call to signalWake() in its action. If a second thread fires this rule then the call to signalWake() wakes up the suspended thread. This can be used to ensure that the first thread is only allowed to progress once the second thread has reached a suitable point during its execution.

So, what are the two updates which I have sneaked in? Well, the first one is to compile the rule code to bytecode. Up until now the rule execution engine has worked by interpreting the rule parse tree. Method and builtin calls were executed using reflection. Java operations were executed by evaluating their arguments and then combining them based on the operation type. Well, the latest version of TOAST now compiles the rule code into Java bytecode attached to a helper class associated with the rule. This bytecode is handed to the JIT compiler which means it can be compiled to native machine code and optimized like any other Java method. This will not make a lot of difference for rules which are only executed once or twice but it will make a big difference to the usability of TOAST in cases where rules are triggered many times.

The second update is a very nice generalization of the built-in functionality. The rule parser, type-checker and compiler don't actually care much about the presence of builtin methods. As far as they are concerned a call is a call is a call. The type checker, in particular, attempts to resolve calls by looking at the method name and signature i.e. the types of the call arguments. Any calls whose name and signature matches a public instance method of a class called Helper are treated as built-in calls. When the rule is triggered an instance of Helper is created to store the list of event bindings. A built-in call can be executed by calling the matching method with this instance as the method target. The compiler actually generates a subclass of Helper whose execute() method comprises the compiled rule bytecode. So, built-in calls are actually just invocations of methods on this.

Now, this mechanism is completely independent of the details of how class Helper is defined. Any public POJO class could actually be used to define the rule language built-ins. So, a simple declaration in the rule source specifying the name of an alternative class will allow the rules to use a different helper more appropriate to the application in question. A domain-specific helper class makes scripting of side-effects simpler and tidier by encoding complex, application-specific actions as single operations. If you like you can keep the current built-ins and add some more of your own by extending class Helper and adding extra public methods. If you don't like how one of the default built-in operations works you can override the version Helper provides to re-implement it.

These upgrades are described more fully in the TOAST documentation. Please download TOAST, take a look and give it a test run.

Nested transaction support

2009-03-16T07:36:00.001Z

For well over 20 years we've been saying that JBossTS (aka ArjunaTS aka Arjuna) supports nested transactions. If you check out most of the papers, documents or presentations that have been written on the subject over the years they'll all tell you that nested transactions (subtransactions) are good because:

fault-isolation: if subtransaction rolls back (e.g., because an object it was using fails) then this does not require the enclosing transaction to rollback, thus undoing all of the work performed so far.

modularity: if there is already a transaction associated with a call when a new transaction is begun, then the transaction will be nested within it. Therefore, a programmer who knows that an object require transactions can use them within the object: if the object’s methods are invoked without a client transaction, then the object’s transactions will simply be top-level; otherwise, they will be nested within the scope of the client’s transactions. Likewise, a client need not know that the object is transactional, and can begin its own transaction.

Now nested transactions have been around for a while (since the early 80's) but they have had limited adoption outside of academia. We did manage to get them adopted within the OTS, even if in a fairly brain-dead/limited approach. But other standards either ignored them (e.g., XA) or made such vague statements about them as to be useless (e.g., JTA). The reason for this is mainly because database vendors have not traditionally supported nested transactions and is primarily why I'm writing this blog entry: I've noticed a few new transaction manager implementations that say they support nested transactions. But what does that actually mean?

If you look at either the JBossTS architecture or even the OTS interfaces, there's not a lot you have to do to the coordinator in order for it to support a parent-child relationship. That's the easy part! (In fact it's easier for the coordinator to support nested transactions because there's no requirement for durability.) But as with a top-level transaction implementation, you need participants (e.g., databases) to be able to do anything useful. The semantics of nested transactions (relaxing ACID properties) mean that a top-level aware participant can't be used for a nested transactions (e.g., your database is going to write the state changes to disk when the XAResource is driven through 2PC and what about lock inheritance?)

Thus whenever we said "JBossTS supports nested transactions" we meant that the coordinator had the parent-child relationship, but we also provided nested transaction-aware resources. Rather than doing very little to the coordinator and pushing most of the real effort on to the application developer or resource implementer, we provide a complete toolkit for developing transactional applications. As a developer you don't have to worry about whether or not a transaction is nested and whether or not you've got nested transaction-aware resources in the mix somehow. All of that is taken care of by JBossTS. So be careful if you come across a transaction manager that says it "supports" nested transactions: it's a meaningless statement unless this covers the coordinator and participants (state and concurrency control). And of course you want to do this in a distributed environment as well as in a flexible manner! (Yes, we do that too!)

RESTful transactions reborn

2009-03-11T19:47:00.000Z

I've mentioned a few times about work we did almost 10 years ago on transactional REST services. This was when JBossTS was part of the HP Netaction suite, we were working on BTP, as well as the precursor to WS-TX/WS-CAF and the world's first Web Services transactions product HP-WST.

Without reminiscing too much, in those days I was participating in the W3C Web Services Architecture working group and the topic of REST came up there quite regularly. So we decided to give it a try as far as transactions were concerned and 'hey presto' RESTful transactions were born. This was way before JAX-RS so we were hand crafting the interactions ourselves, but we demonstrated it could work with the involvement of several companies. And there is remained once HP divested themselves of their middleware division.

But having transactional REST-based services has been a pet project for a long time, so it was bound to rear its head. Especially as some people (rightly) push the Web as an integration platform (shouldn't really be news to people since the WWW has always been about integrating data). Of course there are those who say you don't need transactions in the Web or can (somehow) program around them, but they miss the point. It's a bit like saying you don't need an airbag in your car but can use a pillow (and quick reflexes) yourself to save money. Yes, transactions aren't right for all situations and some bespoke approaches may be more efficient, but if you think about what you get from a transaction system and what you'd have to do in order to obtain the same guarantees, it's more often than not worth using them.

So we originally created a student project to dust off the old REST transactions protocol(s) and implement them today. This started life before JAX-RS (again) but by the time something really happened JAX-RS (and RESTeasy) was around. Michael from the JBossTS team took this work on and did a great job. It's not finished (only the atomic outcome protocol has been implemented so far, for example). Plus the protocols haven't been changed at all since they were created back in 2000/2001. So there are still some things we need to look at and more interesting work ahead.

Is this something you should be looking at for your projects? Should you believe those people who say transactions and REST don't go together? Well as I'm fond of saying "One size does not fit all." Transactions have played a critical part in all distributed systems for over four decades and for very good reasons. Of course we have XTS and Web Services Transactions support if you're using SOAP. But maybe SOAP isn't "your thing" and HTTP is all you can use. Well in that case take a look at this. In just the same way that you wouldn't want to be using a tank in the ocean and a submarine on land, use the right software tool for the right job!

Lessons learned and magic numbers.

2009-02-15T08:58:00.000Z

One of the things you do when you start developing software, especially in a new or cutting-edge area, is believe you know what's best for your users and try to hide any complexities from them. That's one of the reasons we pushed RPC as the best way in which to develop distributed applications in the 1980's. Well back when we started developing Arjuna we took hiding complexity to heart, since simplifying the development of transactional applications was core to all of our PhDs. And I think we did a very good job with the initial releases.

However, once we gave the system to people (the source was made freely available by FTP in 1991, but various industry sponsors were using it before then) the feedback we got was very interesting: users will always do things you didn't expect and demand flexibility in what you produce.

In Arjuna this didn't impact the interfaces users saw but many of the internal development parameters that we had used, thinking that they would never need to be configurable (statically or dynamically). For example, when making a remote invocation on an object failures can occur (the network could partition, the machine hosting the object could crash, etc.) In the absence of a perfect failure detector you use failure suspicion and timeouts. Basically if a response does not come back from the object in time T then you assume the object has failed and act accordingly. But if you get the timeout value wrong then you can make the wrong decision with associated consequences. The timeout value really needs to take into account how long work might take to execute, how overloaded a machine may be, network congestion etc. So one value is rarely right for every application. But we didn't take that into account initially and hard-coded a magic number into the system.

There were other examples of magic numbers, including: the number of RPC retransmissions to use before assuming that a request (or response) cannot get through to the endpoint (why should 5 be any better than 2 or worse than 10?); the number of clustered name server instances; the object store location; the orphan detection period; the lock-conflict detection timeout etc. These were all things we believed had the best (sensible) values, but with hindsight it was clear that it was all based on limited deployment knowledge.

What this all lead to quite quickly was a methodology of expect the unexpected and develop accordingly as far as your users are concerned. We made the system extremely flexible and configurable, where many of the old magic numbers could be overridden either as the system ran or during deployment, with sensible defaults (trying to hit Pareto's 80/20 principle). Those magic numbers we didn't make configurable were clearly documented (both so we could remember as much as users could determine why something was behaving the way it was).

From the feedback we received over the past 20 years I think we managed to come close to the right set of compromises. It's true that most users are happy with the default values we set (which were/are revised based on feedback). But those smaller users who really do need the ability to change things now (or since 20 years ago) have the ability to modify them without rebuilding the system. This has been important in the systems adoption and it's a lesson that we continue to apply. So if you're developing software, beware of using too many magic numbers and if you don't make them configurable you need to understand (and believe) why that's the case and, importantly, document them just in case!

JBossTS and Blacktie: the only combination you'll need!

2009-02-09T18:07:00.001Z

[Cue advertizing jingle.]

Are you a major transaction user, or do you know someone who is? Are you worried about your legacy transaction infrastructure because the vendor may not be reliable? Do you have sleepless nights wondering how to drag your transactional applications into the modern era? Or are just concerned about the ever escalating costs for the transaction system you're currently tied into? Well worry no more: what you need is Blacktie from JBoss, the company that brought you those other useful gadgets like JBossAS, the JBoss T-shirt, the JBoss thong (picture no longer available!), and the JBoss World Virtual Conference.

Unlike offerings from other vendors, Blacktie is entirely open source. It comes in a convenient source or binary bundle for easy deployment to tackle those stubborn legacy applications. Blacktie also builds upon other mature products from your favourite JBoss vendor so it fits nicely into your existing investments from them. And if this is your first entry into open source and JBoss, then it's a great way to start. Blacktie will be the only legacy transaction implementation you'll ever need to solve those annoying problems at a price you'll love!

And coming soon iBlacktie: because everything needs a little i magic!

[Cue advertizing jingle.]

Go on. Check it out. You know you want to!

Stay away from pseudo-transactions!

2009-02-03T21:41:00.000Z

I tried to stay clear of commenting on this article, but while participating in tonight's WS-RA meeting I let my defenses down! In short, the article can be summarized by: "How to use non-transactional resources in a transaction when you can't be bothered to do it right in the first place." Or "How I learned to break transactional semantics and put my data consistency on the line."

I'm fairly sure the authors are trying to help their audience, but they really aren't. They ignore critical problems with their proposal, either deliberately or because they simply haven't given the problem space enough thought. What they are trying to do is emulate the last-resource commit optimization within the application, but whereas a transaction manager will do this and provide support for crash failures, the article ignores that completely. There is no reference to failure recovery at all.

The article also appears to assume that because a datasource is managed by an XAResource it will always honor the business logic agreement when the transaction commits, e.g., if the table update succeeded through the session instance then prepare/commit will work later. I hate to break it to the authors, but that isn't always the case. This works in a transaction manager because we manage the resource ordering and durability very carefully to cope with failures, i.e., there are good reasons we don't require the application programmer to do this!

Oh and the way in which multiple non-transactional resources are managed in the transaction just scares the %$&* out of me! Look, these are resources that do their work when told to and you can't undo that (if you could, then wrap the &%*& things in an XAResource!) So if you crash part way through the normal flow of execution, or part way through the "rollback", what is the state of the application? How do you find out what happened to whom and when? Where's the log?! (I would hate to be a systems administrator in this situation when the sh*t hits the fan!)

The authors say that "Although this is not as robust and comprehensive as a truly transactional interface, it can provide excellent coverage at a fraction of the development cost of a JTA compliant interface." Unfortunately it does not. That's a bit like saying a car with worn brakes is roadworthy in all situations, when it clearly isn't! Ask yourself this: what happens to my data in the cases that aren't covered by this approach and can I really afford to lose it or spend the new hours/days/weeks repairing it manually? If the answer is yes, then you probably don't want to use transactions at all. If the answer is no, then stay away from this approach and go with a transaction system and transactional resources.

In conclusion: use transactions correctly and if you can't make your data items transactional then try using compensating transactions. At least you get logging and recovery!

Building transactional applications

2009-02-03T12:53:00.000Z

I've been thinking about how we develop transactional applications for a very long time. That thought process got interrupted for a bit, but I came back to it over Christmas and recent events and articles have pushed this further. Plus we're working on transactional aspects for Drools 5 which has covered writing XAResources. Back in HP we had a tutorial on how to do this, which I think we're going to dust off and maybe Jonathan and the team will write some articles around that topic. (I know it's the subject of an update to the book, whenever that happens!)

But it got me thinking that as soon as people start asking about how they can write XAResources to make their data transactional then we really haven't progressed as an industry. XA is a good standard. But it's not perfect. (No standard is!) Plus it was designed with very specific database use cases in mind, which were fine 30 years ago but aren't always a perfect fit for the 21st century. The question should be "how do I make my data transactional?" and leave the implementation specifics to the engine or container.

We spent a great deal of time over the last 20 years making the development of transactional applications easier (HP was using Arjuna 20 years ago and liking it). Some PhD students wrote an entire database system based on it too. Simplicity, flexibility and power were keys to its success (nested transactions rule!) Back then there was little differentiation between the transaction engine and the way in which you developed applications. Subsequently we emphasized that they were different (Arjuna became known as the engine, while Transactional Objects for Java was the framework for developing applications).

However, we've not concentrated on the latter in recent years, which is a shame and something that we're going to remedy, because it's a proven way of developing flexible transactional applications. I think we'll throw in annotations and try to make it slightly less invasive than it was (back in the 1980's there was no such thing as Java let alone annotations). Hopefully we'll then be able to offer this to developers building on JBossTS and they can move away from asking "How do I write an XAResource?" to "What do my users want from my application/framework/service/..?" Leave the transaction complexity to JBossTS.

Side-note: while writing this I was reminded that we have a student project on some of these ideas, though not necessarily tied to TOJ. So if you are a student and looking for something to do, now may be the time to give it a go.

The Blacktie project goes live

2009-01-09T11:12:00.000Z

At last we can announce the first milestone release for the Blacktie project. If you're interested in XATMI then take a look and get involved!

Web Services Test Forum

2008-12-09T13:41:00.000Z

We've been working with IBM and others on the formation of the WSTF. Thanks to Andrew Dinn for the Web Services Transactions scenario work and Alessio Soldano for our WS-Addressing participation. Interoperability is something that I've been pushing for many years and we've always participated in as many of the official standards-based events as we possibly can. We're also members of WS-I and Apache Stonehenge, so you can be assured that Web Service interoperability will remain high on our agenda.

Java TOAST - A New Java Multi-threaded Test Automation Tool

2008-12-01T13:04:00.000Z

I've been working on a Java test scripting tool which I have been using to help automate running of crash recovery tests for the JBossTS Web Service Transactions code. The tool is provisionally called Java TOAST (Thread Orchestration Automation Scripting Tool) and I'm planning to submit a presentation on it to Java One. I have also made the code available under the LGPL.

The tool uses the Java agent package (java.lang.instrumentation) and the ASM bytecode manipulation library to insert side effects or synthetic throws/returns at chosen 'trigger' points in application or JVM method code. Side effects are scripted using a simple event condition action language and the conditions and actions can include arbitrary invocations of Java code including reference to objects in context at the trigger point. Rules can be used to force 'wrong' or 'unexpected' behaviours to occur and to trace subsequent execution and propagate further side effects which ensure that it follows the desired path.

The tool is particularly oriented towards dealing with the problems involved in convincing non-deterministic, multi-threaded code to run in a reliable and repeatable manner. So, the rule language provides a suite of builtins which can be invoked from conditions or actions to orchestrate thread execution. These builtins implement operations such as delays, waits and notifies, countdowns, rendezvous, etc and can be used to make sure that events in the thread's life cycle happen with the desired timings or orderings. However, the tool is more general than this. So, even if you are just interested in running a simple single-threaded test it provides a very flexible means of scripting errant and aberrant behaviours in arbitrary application and/or JVM code and managing and tracing subsequent execution.

If anyone wants to take a look at the tool and or give it a road test the sources are in the JBossTS repository. There is a README which explains the basics plus how to compile and run the tool and a more complete user guide .
.

JTS and WS-TX implementations go LGPL

2008-06-26T20:39:00.000+01:00

Back when ArjunaTS was acquired by JBoss the decision was made to licence the core transaction engine (ArjunaCore) and the local JTA under LGPL, but use GPL for the OTS (JTS) implementation as well as for XTS, our Web Services Transactions component. There were some strategic decisions at the time (a dual-licence model) that made sense for JBoss, but with the Red Hat acquisition we've been reconsidering. As a result, the next release of JBossTS will be entirely based on LGPL, i.e., there will no longer be any GPL component. Hopefully this will satisfy those users and customers out there who wanted to use JTS or XTS but found that they were uncomfortable with GPL. From a community perspective, let's also hope that this helps encourage everyone to get behind JBossTS and continue to ensure it is the premier open source transaction engine out there. Next step: Project Blacktie.

JavaOne

2008-05-05T16:00:00.000+01:00

Some of the team are at JavaOne. Come and say hi.

DTF open sourced

2008-04-15T17:50:00.000+01:00

Back in 2001 we had tens of hundreds of tests for the HP Transaction Service and a team of dedicated QA engineers. It took a week to run all of the tests across the combination of OS, hardware, JDK and database combinations. When we span out we didn't have those resources so had to develop the Distributed Test Framework, which is many ways is a Grid infrastructure: it supports the automatic deployment of services and software across arbitrary nodes, VM, OS etc. to execute tests and collate the results. It is truly distributed in nature and runs distributed tests (where the client(s) and service(s) run across different combinations of VM, hardware, OS, db etc.) We managed to have the same QA coverage with a fraction of the people and in a fraction of the time (and cost). I'm happy to say that we've finally open sourced it as well!

Shout it from the roof tops!

2008-03-02T00:21:00.001Z

Don't you just hate it when people don't research things sufficiently? How many times do I have to point out over the past 10 years that we support Web Services transactions and have been the only open source implementation to participate in interoperability workshops with the other main transaction manager offerings out there? So let me say it one more time: we support Web Services transactions and specifically WS-TX.

Sometimes it's easier to ask

2007-12-23T01:21:00.000Z

While waiting for something to compile at 1am on the night before the night before Christmas, I came across Andrew's brush with JBossTS. There are some interesting quotes, like "JBossTS hasn't been built or developed using eclipse as far as I can tell" (JBossTS began life years before there was such a thing as Eclipse), and "Documentation surrounding great software tools is sometimes just laughable" (I like to think our docs are pretty good, given they've been updated for the past 20 years; but nothing's perfect) but overall I understand some of his concerns. The blog trail hasn't been updated since July, but if you're still struggling with this, get in touch here or ask questions on the Design Forum: I'd be more than happy to help fill in the blanks and maybe in the process improve JBossTS based on your feedback.

Transaction Processing in a Service Oriented Architecture

2007-12-07T23:11:00.003Z

This should be an interesting cross post about the evolution of transaction processing and how it has been shaped by Web Services and SOA.

A minor rant around stand-alone transaction management and standards

2007-11-17T11:47:00.000Z

Over the past few years there's been a backlash against application servers (both commercial and open source), with some people saying they're too heavy-weight. I'm not going to get into that argument here, but what does surprise me is the number of open source transaction managers that are trying to differentiate themselves on this very fact, as if it was something new and wonderful. Well guys I hate to break it to you but we, IBM, BEA and a few others have been doing that for years. In the open source arena, JBossTS was the first to run outside of an application server simply because it began life before there was such a thing! We've continued that over the intervening years, but maybe I just need to make it more explicit: JBossTS does not require any application server within which to run. I'm pretty sure I mentioned this before, but it's also highly embeddable, which we have proved time and again (yes, footprint size is very small, but the architecture is designed so that it doesn't require anything from the underlying environment).

Then there are some transaction service implementations (typically open source, but not exclusively) that seem to need to try to differentiate themselves by pouring scorn on transaction standards like OTS/JTS or WS-T: guys, if you have problems with these things then get involved in the process, but don't cry foul because the rest of the industry (and users) have got together to develop them. Oh, and yes while it's true that JBossTS has been at the forefront of these (and other) standards since the start, the architecture is not built on any of them since, once again, with the exception of XA they all came after JBossTS (aka Arjuna) started life.

Now maybe we don't go around "blowing our own trumpet" enough, but I think that's got a lot to do with our background: in academia there's a heavy peer review process that permeates throughout everything you do and unsubstantiated statements quickly become something you avoid like the plague. We've had 20+ years to document what we've done in this way, so everything has been verified and screened for accuracy. Maybe we assume too much that people will check that backlog of information, particularly in this day and age with so many good internet search engines. However, maybe we need to change that and start revisiting our previous publications and discussions more to refresh people's memories.

So if you're a user looking for some or all of the following: power, configurability, performance, openness, pedigree, flexibility, excellent documentation, standards (where necessary), extensibility, embeddability, stand-alone or application server deployment options, continuous vision, knowledge base, 24x7 support, etc. etc., then there really is only one option: JBossTS. TIBCO, webMethods and many others over the past 20+ years have made the JBossTS choice and found it to be the right one.

Heuristics, one-phase commit and compensations

2007-08-30T13:04:00.001+01:00

Another cross post that may be of interest.

WS-AT and SOA

2007-07-17T12:46:00.001+01:00

Just a cross post.

XTS update

2007-07-17T12:06:00.000+01:00

It's nice to see that there are other commercial and OSS vendors looking to provide new implementations of WS-TX. However, it is worth stressing the points I made earlier: not only was XTS the world's first Web Services transactions implementation, but it is the only open source implementation that has been tested for interoperability with IBM, Microsoft and others. So if interoperability with heterogeneous implementations is important to you, come talk to us.

Trying to put JBossTS into perspective: XTS

2007-07-13T17:21:00.000+01:00

Up to this point we've covered the core engine component, the local JTA, and then the JTS and remote JTA implementations. That leaves XTS: the Web Services transactions component.

There has been a Web Services transactions component in JBossTS since it's HP days, when it was called HP Web Services Transactions (HP-WST). Once again it was the world's first Web Services transactions product. At that time it was based on the OASIS Business Transactions Protocol (BTP), which didn't get the support of major vendors. Over the intervening years we tracked, authored and influenced BTP, WS-CAF, WS-T and then WS-TX. Each time, we used XTS as the platform for testing these protocols.

Just like the local JTA and JTS implementations, XTS leverages the core engine. It has no dependency on CORBA. It has no dependency on JTA. As well as webMethods, others have used XTS successfully in the past too. Because all of the various Web Services transactions specifications/standards have included support for extended (non-ACID) transactions, XTS is able to exploit yet more of the power of the underlying core engine: the ability to relax all of the ACID properties in a controlled manner. So unlike other vendors, we have the same core engine providing the support for all of the transaction models!

Then of course there's end-to-end transactions (E2EX as we were calling it in HP). We've been talking about transaction bridging for many years and working on one prototype or implementation after another. But the recent work that Jonathan et al have been doing has really pulled everything together into a more coherent approach, at least as far as JEE is concerned.

I could say more about the architecture of all of our components. I could also wax lyrically about recovery or extended transactions. But most of what I'd say is covered by the various hyperlinks I've embedded throughout these entries. So please follow them and read up. If you've got any questions or comments, put them against the relevant entry or post in our forums. But overall, enjoy using JBossTS as much as we've enjoyed developing it over the past 20 years.

Trying to put JBossTS into perspective: JTS

2007-07-13T14:56:00.000+01:00

Well if you're still with me you're either really interested in transactions and JBossTS, or lost.

So far I've covered the core transaction engine and the local JTA components of JBossTS and shown how they are related. In this entry I'll go over the JTS implementation as well as the remote JTA.

JBossTS began life in C++. In the early 1990's when the CORBA OTS was being developed within the OMG, we were involved and came up with the world's first complete OTS implementation (supporting all of the optional components). We did some pretty cool things with end-to-end transactions back then too (but more on that in a separate entry). When Java came along and we created the world's first 100% pure Java transaction service (predating JTA and JTS), we decided to do our own Java language mapping of the OTS, JTSArjuna.

JTSArjuna, which became Total-e-Transactions, HP Transaction Service, ArjunaTS and now JBossTS, is a full implementation of the OTS (now JTS). It builds on the core transaction engine and, because OTS is distributed in nature, obtains all of the distribution support from a variety of CORBA ORBs. Quite early on in its evolution we developed an ORB Portability Layer, to isolate ourselves from the differences in ORB implementations. So over the years, in C++ and Java, we've probably ported TS to most of the ORBs that have ever seen the light of day.

Anyway, the JTS component leverages the core, but doesn't touch the local JTA: it doesn't need to. As I said earlier, when we first did out JTA support it was built on top of the JTS. So we had distributed JTA from the start: local JTA was an after thought. The relationship between the remote JTA and the local JTA is the core engine.

Being a JTS implementation, it supports distributed transactions and hence distributed failure recovery. We have both top-down (coordinator driven) and bottom-up (participant driven) recovery. As I said above, we also support all of the optional capabilities within the OTS (e.g., nested transactions and interoposition), with XA interoperability and an XA TX look-alike veneer API. Plus we provide quite a few enhancements to allow the power of the underlying core engine to come out. For example, neither the OTS/JTS nor JTA support ordering of participants with the log. So when you call commit (or rollback) there is no guarantee on the order in which participants will be driven. The core engine allows you to order participants and therefore so does the JTS.

That's about all I have to say about the JTS component. There's a lot more in the shipped documentation that also goes into the details around the OTS. If you want to have distributed JTA, then this is your only route (at the moment). You don't have to use JTS via the JTA of course: you could go straight to the JTS. Plus, you can continue to mix-and-match your participants and transactions from the core with the JTS if you need to.

Trying to put JBossTS into perspective: local JTA

2007-07-13T12:17:00.000+01:00

I covered the core component of JBossTS earlier, so now it's the time of local JTA. Although we have two JTA implementations now, early on in the evolution of JBossTS there was only one and that was built on our JTS/OTS implementation. However, with the development of ArjunaCore and the fact that some of HP's customers didn't want to worry about configuring/maintaining an ORB, we looked at creating a purely local JTA implementation.

ArjunaCore has its own set of APIs that are not based on XA or any other standard. In fact, many of them pre-date the current standards in these areas. But as we saw in the earlier entry, there's a lot of flexibility in the core engine: if anything, it essentially presents a superset of capabilities of the various transaction specifications. So producing a purely local JTA implementation based on it was relatively straightforward. JTA is a narrowing of those capabilities, because it is closely tied to XA: the participants can only be XA participants and even with the UserTransaction/TransactionManager split there's not a lot of scope for doing "interesting" things like nested transactions or nested top-level transactions.

Both of our JTA implementations do support nested transactions. But the fact that XA doesn't support nesting makes it harder than it should be. In order to have support for nested (or sub) transactions, you need the cooperation of a nested transaction-aware coordinator and nested transaction-aware participants. Our coordinator is nested transaction-aware, because ArjunaCore has always supported sub-transactions. However, XA participants don't. But luckily for those people who want to try out this capability, you can mix-and-match your APIs with JBossTS. So you can create a JTA transaction and then dive into the core API if necessary to register a nested transaction-aware participant. Plus, ArjunaCore comes with a suite of such participants, so you don't have to write them to start with.

So there you have it. The local JTA (which ships by default in JBossAS 4.2) is a layer on top of ArjunaCore. It provides full non-distributed JTA capabilities including automatic failure recovery. Plus, if you're willing to try, it's still possible to get at the power of the underlying engine.

Trying to put JBossTS into perspective: the core

2007-07-13T11:45:00.000+01:00

There have been a few comments on the forums recently asking how the various components within the JBossTS suite relate to one another. So I thought I'd try to put everything into context here so we can at least refer back to it if these questions come up again in the future. However, there's potentially a lot to say, so I'll split it across a few entries.

Note that these entry won't go into transaction basics: this is purely about the product and it's architecture.

You'll find a lot of this information in the various collatoral documents and presentations we've got. Plus, the documentation that ships with the product is very extensive and does cover this to one degree or another. So don't think this entry is a substitute to doing your homework.

Where to start? Well if you're interested in seeing the overall architecture we have a good description, but I think it's best to start at the beginning: ArjunaCore. As it's name suggests, this is the core of JBossTS: everything that needs transactional capabilities builds on this in one way or another. Using ArjunaCore you can get full transactional semantics for your applications in a local environment. You also get:

Multi-threaded: transactions can have many threads operating within them concurrently.

Nested transactions: transactions may be nested within one another. This allows for improved fault-tolerance the failure of a nested transaction does not automatically undo the work of the parent transaction) and modularity (nesting is transparent to applications, such that an object that requires transaction support can be constructed without a requirement that it must be invoked within a transaction: if it is then it's own transactions will simply be nested within the invokers and if it is not then the transactions will be top-level).

Highly configurable: one of the key components to the performance of any transaction system is its log: the transaction log is used at many times during the lifetime of a transaction and the information it stores must be durable. Thus, disk I/O is a typical bottleneck to any transaction system. However, the usage patterns for transactions can also affect the log such that different log implementations perform better for different applications. Therefore, rather than provide a single log implementation, ArjunaCore allows different implementations to be plugged in at runtime to best suite the application requirements.

Relaxation of properties across the ACID matrix: all of the ACID properties can be relaxed in a controlled manner. So, for example, the atomic nature of a traditional transaction can be modified to support OASIS BTP cohesive transactions. Furthermore, this means that the same transaction coordinator can be used to support many different transaction models. ArjunaCore provides the necessary support for BTP, but can also be used to support the various WS-CAF (or WS-T) transaction models.

Optimised and performant: ArjunaCore has been developed in one form or another for over 15 years. In that time, it has been tuned and optimised to improve its performance.

Adaptable failure recovery: arbitrary transactional resources can be enlisted with ArjunaCore transactions and recovery mechanisms can be provided that will be automatically driven by the failure recovery sub-system.

No dependency on XA participants: it defines a neutral two-phase aware interface for transactional participants that does not mandate XA semantics.

Note, it is important to realize that ArjunaCore does not require any other software in order to run. It is a complete 100% pure Java implementation of a transaction system and does not require an ORB or application server in order to provide any of its functionality.

As a result of its flexibility, our core engine has been used within mobile transactions (it has a very small footprint and still runs on an HP Jornada 720), JMS implementations, JTS, JTA and Web Services transactions (supporting more than just ACID transactions).

Hopefully that's enough on ArjunaCore. Next entry I'll move "up" to our local JTA implementation.

TIBCO BusinessWorks and JBossTS

2007-07-03T08:00:00.000+01:00

I can't remember why I let this slip by last year, but it's still worth mentioning: TIBCO BusinessWorks uses a version of JBoss Transactions for its transactional capability. It's a long story, going back to my Arjuna days, but we worked closely with those guys for a long time before finally signing them to use ArjunaTS, the old name for JBossTS. The rest, as they say, is history.

JBossTS, Hibernate and JBossCache

2007-04-27T14:43:00.000+01:00

The JBossCache and Hibernate folks have been doing some great integration work recently that has called for JBossTS. Galder has written a nice summary. Check it out.

Transactions and cars

2007-04-21T20:41:00.000+01:00

I was on vacation a couple of weeks ago, SCUBA diving in Lanzarote. Had a great time and didn't think about work once. However, on the way home we were shuttled to the airport in a mini-bus and my mind started to wander back to work. JBossTS 4.2.3 will be the default transaction manager in JBossAS 4.2 and as a result we've been having some interesting discussions with clients about the impact this will have on them. I've written before about why transactions are needed and why this necessitates recovery and durability. However, there are still times when I feel that an everyday analogy may make things easier to explain to people, particularly those who aren't as immersed in transaction principles.

So there I was in the mini-bus and I began to think: cars are something almost everyone knows about and probably interacts with on a daily basis in one form or another. Therefore, is there something in the architecture of a car that plays a similar role to transactions in that you take it for granted, probably don't even think about it, maybe don't use as often as you really should, reckon you can do without because you rarely (if ever) see the need, and yet if it wasn't there you'd regret it and in all likelihood never go without it again? These things are often easier to write about in retrospect, because the list I just gave took me about 20 minutes to forumlate during that trip back to the airport.

There may be more than one answer, but the one I came up with seems fairly obvious: the seat belt! Unless you live in a country that mandates wearing of seatbelts, you may not wear them often, if at all. Even if you do wear them, how many times have you really tested that they work the way they should? You just expect them to. (Most of the people in our mini-bus were not wearing their seatbelts.)

Of course in many countries there are laws that define the safety characteristics of seatbelts, so they should work well enough. But what if there were no such laws, or you lived somewhere they didn't exist, or the manufacturer decided to cut corners? (Can you say: transactions without recovery?) Because accidents (failures) don't happen all that often, the vast majority of the time you (and the vast majority of people) wouldn't care about defective or sub-standard implementations. But when the time comes for them to work and work well enough to protect your life (or critical data), that's when you would complain if they didn't (assuming you survive). You wouldn't complain at having to pay for more saftey features within your car (or mini-bus) or any slight overhead such things may incur. So why complain about adding transactions and recovery?

I haven't tried this analogy yet with any of our customers, but it'll be interesting to see if it makes it easier to explain the differences between the old and new transaction implementations provided by JBossAS.

Welcome

2007-02-19T22:01:00.000Z

Welcome to the JBossTS team blog site.