WS-BA Participant Completion Race Condition: Part Two

The Details

In a previous post, I described a benign race condition that in unusual circumstances can cause some Business Activities to be cancelled that would have otherwise been able to close. In this post I'll go into the details of how this happens.

First consider the following client code:

UserBusinessActivity uba = UserBusinessActivityFactory.userBusinessActivity();
uba.begin();
myWebServiceClient.invoke();
uba.close();

The client code is very simple, it just begins a business activity, invokes a Web service and then closes the business activity. The Web service uses the Participant-Completion protocol and so notifies the coordinator of completion just before returning control to the client.


Here's a diagram showing the pertinent message exchanges that occur under a normal situation.

The messages are numbered to indicate the order in which they are sent.

• 1. request. This represents the application request made by the client.
• 2. completed. After the participant has completed its work, it notifies the coordinator that it has completed.
• 3. response. This represents the response to the client's application request.
• 4. close. The client notifies the coordinator that it wishes to close the activity. It then waits for a 'closed' or failure response from the coordinator.
• 5a. close/5b. closed. The coordinator has processed the '2.completed' message so can close the activity. It starts by sending the 'close' message to the participant and waits for the 'closed' response as confirmation. These two messages are asynchronous.
• 6. closed. The coordinator now has all 'closed' acknowledgements so notifies the client that the activity successfully closed.

Messages '2.completed' and '4.close' are asynchronous (or 'one way' in Web services parlance) so effectively, we have a race condition with the following competing parties:

• Party 1. The completed message '2.completed'.
• Party 2. The response '3.response' followed by '4.close'.

When running in the same VM, or on a low latency network, '3.response' will be sent very quickly. This is because it is simply travelling on the HTTP response over an already open socket. This just leaves messages '2.completed' and '4.close' which will take much longer relative to '3.response'. To understand this, lets take a look at what happens when an asynchronous Web service call is made:

1. The client sends the message to the Web service
2. The server-side SOAP stack uses an existing thread from a pool dedicated to receiving SOAP messages.
3. As the service is asynchronous, the message will be passed to another thread to be processed.
4. The receiving thread will now return the HTTP response.

The race condition occurs because steps 1-3 can happen relatively quickly in a single VM, and thus it's likely that both messages 2 and 4, will be waiting to be processed at the same time. The order in which they are processed is dependent on the implementation of the thread pool and is also at the mercy of thread scheduling in the VM, so it's possible that either could be processed first.

This race condition is much less likely to happen in a distributed environment as the network costs will be significantly higher. As a result message '3.response' will take long enough to send, so as to give message '2.completed' enough of a head start. But it is still possible so the client application must be coded defensively to catch and handle a TransactionRollbackException. Your code ought to be doing this anyway to deal with server crashes.

Here's a diagram showing what messages are exchanged when the race condition occurs. You will see that the activity ends in a consistent state.



I've omitted messages 1-3 from the following explanation as they are the same as in the success case.

• 4. close. This message is processed by the coordinator before message '2.completed'
• 5a. cancel. The coordinator has not yet processed the '2.completed' message so cannot close the activity. The coordinator then sends a 'cancel' message to the participant as it thinks it has not yet completed. This message and subsequent retires, are dropped by the participant as they are not valid for a completed participant.
• 5b. compensate/5c. compensated. After one or more unacknowledged 'cancel' messages, the coordinator switches to sending 'compensate' messages which will cause the participant to compensate the work. The participant acknowledges with a 'compensated' reply.
• 6. Transaction rolledback exception. The coordinator notifies the client that the activity failed to close.

As you can see from the steps above, when this race condition arises, any work done by participants is compensated and the client is notified of the outcome. Thus a consistent outcome is achieved.

WS-BA Participant Completion Race Condition: Part One

Overview

The WS-BA participant-completion protocol has a benign race condition that, in unusual circumstances, can cause some Business Activities to be cancelled that would have otherwise been able to close. This is safe as no inconsistency arises, but it can be annoying for users. This blog post explains why this can happen, under what conditions, and what you can do to tolerate it. This post gives you an overview of the issue and should provide enough details for most developers. A follow up post will get into the nitty-gritty details of what's really going on.

What's happening, in a nutshell

Imagine a scenario where the client begins a business activity and then invokes a Web service. If the Web service uses participant completion, it will notify the coordinator when it has completed its work and then return control to the client. This notification is asynchronous, so it's possible that the client will then ask the coordinator to close the activity before the coordinator processes (or even receives) the completed notification from the participant. In this situation the coordinator will cancel the activity as not all participants (from its perspective) have completed their work. As a result all completed participants are compensated (including, eventually, the participant with the late 'completed' notification) and the client receives a "TransactionRolledBackException".

When is it most likely to happen?

Typically this happens when the client, coordinator and participant are running inside the same VM. This scenario is unlikely to happen in production, but can happen regularly during development where a single VM is used to keep things simple.

How do I know if this is affecting my application?

If the client is occasionally receiving a TransactionRolledbackException when calling UserBusinessActivity#close(), but none of the machines involved in running the transaction have crashed, you could be affected by this. Especially if you are running the client, coordinator and participant(s) in the same server.

We've now added a log message to help you identify this. However, to see this, you will either need to be building transactions from the current source (4.17 or master branches in GitHub) or wait for the JBossTS 4.17.4 or Narayana 5.0.0.M2 release. This is the log message to look out for:

WARN  [com.arjuna.mw.wstx] (TaskWorker-2) ARJUNA045062: Coordinator cancelled the activity

This is only an indication that you are seeing this issue as the coordinator can elect to cancel the activity for other reasons. For example, network problems might mean the coordinator cannot tell the web service to close the activity.

Why can't this be avoided?

The short answer is that for the protocol to avoid this it would need to make the complete message synchronous, throttling throughput by slowing down both the participant and coordinator and holding sockets open for longer.

What can the application do to tolerate this?

A real, distributed deployment will rarely see this problem because communication latency between client, participant and coordinator will dominate the race condition. Even if it does happen your application should tolerate it. Transaction rollbacks and activity cancellations are inevitable in a distributed environment and can happen for many reasons. When handling TransactionRolledBack exceptions you can either retry the Transaction/Activity or notify the caller of the failure. What you choose to do will depend on the requirements of your application.

In part two, I'll get into the details of what's happening.

Raspberry flavoured transactions

I got asked by a few different people lately about how to build Narayana on a Raspberry Pi. Well it's pretty easy and you don't even need one of the upgraded Model Bs with 512Meg of memory. Here are the steps you need to follow:

• I'm going to assume you know how to set up your Pi. I'm also going to assume that you're using the default pi account.
• You'll need to install OpenJDK (sudo apt-get install openjdk-6-jre openjdk-6-jdk), maven and git (maven/git sudo apt-get install maven2 git). Other optionals include emacs (of course!) and your favourite databases, but we'll leave those for the curious.
• Then let's clone the source tree ... git clone https://github.com/jbosstm/narayana

• Then it's the usual steps to build, although due to some issues with idl on the Pi, we can't run JTS just yet using the ORB within the JDK, so let's go with the JTA implementation only ... ./build.sh clean install -P jta
• After that, sit back and wait ... for a long time, so be patient.

• It should take about 2 hours to complete, which is significantly slower than the 30 minutes or so that it can take on your laptop to build everything. (If you try to build everything, then it'll be closer to 5 hours!) But hey, the Pi is underpowered compared to that hardware and if you're looking to use a Pi then hopefully you've already taken that into consideration! And yes, you will see some warnings and maybe a timing error or two during the build execution of some tests, but nothing to be concerned about. We'll look into these and see if there's anything we can do to tidy-up the build in the future.

And that's it! Congratulations, you've just built and tested Narayana on your Raspberry Pi. And apart from the time it takes to build, there really wasn't much to it, now was there?

Note, if you want to build the JTS then you do have the option to use JacORB. However, at the moment this requires you to edit a number of the pom.xml files to remove the dependency on idlj.

JBoss Transactions 4.17.2.Final released

Hi,

I wanted to share with you some fantastic news for those who have either been working with our project for a while or those who might have just recently discovered us. The first version of JBoss Transactions created using our new development environment of GitHub and Maven is available for download right now from:
https://www.jboss.org/jbosstm/downloads/4_17_2_Final

Even better, this version (4.17.2.Final) has made it into the latest branch of AS7! So if you checkout the current version of JBoss AS you will be able to build a version of the app server containing this release!

Here are a few links to help you get started:
1. The projects source code: https://github.com/jbosstm/narayana/
2. The projects documentation repository: https://github.com/jbosstm/documentation/
3. Some quickstarts to get you up and running: https://github.com/jbosstm/quickstarts/

You can checkout these projects using Git (http://git-scm.com/downloads). The following steps show how to obtain the projects source code once you have Git installed (replace the path for documentation or quickstarts):
1. git clone https://github.com/jbosstm/narayana.git
2. git checkout 4.17 (the default branch contains "Narayana" - more on that in later posts...)

To build the project itself, you simply type: ./build.sh install [-DskipTests]
Or, to build either the documentation or quickstarts projects, just type: mvn install

Obtaining the sourcecode for a version of AS7 that contains the new JBoss Transactions is as simple as visiting here:
https://github.com/jbossas/jboss-as/ (the master branch uses our latest release)

As you can tell, the version number has rev'd a little from 4.16.x while we ironed out some of the teething wrinkles with releasing this new version, so for this release we have created a custom release notes link for you that aggregates a couple of version numbers:
https://issues.jboss.org/secure/IssueNavigator.jspa?mode=hide&requestId=12317999
We have also made some considerable enhancements to our documentation pack and with the new quickstarts repo, plus many changes to the tests to make them more robust. The release notes link above has a filter to omit these so you can easily see the new functionality rather than being inundated with numerous related Jiras.

Another major feature is that we have re-introduced for the JDK orb as well as JacORB when using JTS.

The primary theme of this release is collaboration though. We believe that by moving to GitHub we should be much more geared up to accept patches from you. We have a dedicated build server which will check out any "pull request" that you make, build it, and give you feedback as to whether your changes pass our rigorous testing process!

That's about all for now, if you do have any questions I can answer them in the comments or feel free to ask them over on our user or dev forums here: https://community.jboss.org/en/jbosstm?view=discussions and here: https://community.jboss.org/en/jbosstm/dev?view=discussions

Happy developing!
Tom

Transactions and NoSQL

Just a quick cross-posting in case people are interested.

XTS Whirlwind Tour

Recently the Transactions Team delivered a training course to the Red Hat support staff to help them support JBossTS. As part of the course I delivered a demo configuring XTS (our Web service Transactions implementation) and debugging common issues. I created myself a script to help make the demo run smoothly. As I was preparing the demo I realized that this could make a great community resource, so I put a little bit more effort in and added some commentary.

Here's the result: http://community.jboss.org/wiki/XTSWhirlwindTour

The XTS Whirlwind Tour covers:

• A simple single server scenario
• A simple distributed scenario
• Running an XTS server behind a gateway (Apache)
• Recovery
• Bridging WS-AT to JTA
• Many common issues and how to debug them.

I hope you find this useful. Remember, this is a community document so please get involved, comments are strongly encouraged!

If you find this type of document useful, please let us know as we may be able to do more in the future.

Enjoy!

faking it

Every now and then you find yourself unable to use XA in a situation where you need full ACID guarantees across updates to multiple systems. Once upon a time this occurred mainly in situations where one of your relational databases did not support XA. These days it's more likely to be some newfangled NoSQL store that's not playing nice in the XA ecosystem. When this happens you will more often than not try to convince yourself you can fake it. You probably can't.

XA is a consensus protocol for guaranteeing ACID on transactions spanning multiple resource managers. The two phase commit protocol it uses is a means to that end. For everything to work correctly, it requires that the resource managers make some promises: that they keep changes hidden from other users until the transaction commits and that after the prepare stage they remain able to commit (or rollback) even if they crash before a decision is reached. That's Isolation (usually with a side order of Atomicity) and Durability in a nutshell.

The cheap and cheerful fake version of XA is LRCO. Last Resource Commit Optimization, sometimes called Last Resource Gambit, allows for one non-XA RM in an otherwise entirely XA transaction. By ordering the non-XA resource last in the 2PC processing order you can achieve something that behaves like XA for most scenarios. But close inspection reveals the poor quality of the knockoff goods: LRCO breaks horribly if a crash occurs at certain points in the execution. In such cases inconsistencies between the RMs can arise and reconciling them requires human intervention or complex code. So the savings may not be as good as they first appear.

The more convincing version of LRCO is known as LLRO (Logging Last Resource Optimization) or 1.5 phase commit. In this model you write the transaction manager's log entry to the same database that is being used as the last resource. By making the log write atomic with the commit, you close the timing window during which crashes can cause problems. Which is all well and good if your last resource happens to be a database that can take the logging workload. In cases where it is e.g. a mail server, you're still in trouble.

For the more general case providing the ACID properties yourself in a situation where the resource manager doesn't is hard. Like academic research effort hard. Indeed there are several such research projects in progress right now. Some focus on implementing a custom driver for the resource manager that applies changes against a tx local cache during the transaction, logs them to local disk at prepare and flushes the modifications to the real RM at commit time. This is hard where the resource manager implements complex features, as you wind up needing to reimplement them in the driver or avoid using them in the application code. It's also vulnerable to write conflicts in highly concurrent environments, especially where other clients are accessing the resource manager directly rather than through the custom driver.

The inverse of that model is compensation based transactions. These apply the state changes to the resource manager immediately or at prepare time and then apply additional changes to undo the effects if the transaction needs to roll back. This approach does not offer isolation though - the changes are visible to other users for a time even if the tx is not committed. It's also difficult to generate correct compensation logic for many operations.

All the alternatives to XA have limitations or drawbacks. If your circumstances are such that you can live with those, you may be able to use a different protocol to good effect. But beware the corner cases. You can't fool all the people all the time.

references:
http://stackoverflow.com/questions/8280970/using-xa-with-databases-that-dont-support-it-natively  (No I didn't steal the answer, I wrote it.)
http://www.edbt.org/Proceedings/2010-Lausanne/edbt/papers/N128C2.html
http://www.globule.org/?page_id=25