Sunday, March 31, 2013

STM, vert.x and the Pi Part 1

Given other priorities (aka 'my day job'), I'm slowly working my way through another pet project I've had for a while: adding transactions, and specifically STM, to vert.x. Of course at the moment I also try to add an influence from my other pet project, the Raspberry Pi, so I'm doing it on the Pi as much as possible. OK, it may not be the fastest machine on the block, but it's a lot of fun!

We've had STM in Narayana for a while, so I won't cover the basic again. I'm also not going to cover any Pi-related set up: by this point you should know enough to do this yourself. So let's start with cloning the most recent version of Narayana:


Make sure your path and JAVA_HOME are set so we're using JDK 7:

 
Next let's build the components we need for STM:


Don't forget that this is going to take a while, so maybe time to go grab a coffee, read a book, play with your kids, or watch an episode of Lost.


With that build complete, we've already run a lot of STM examples in the form of unit tests. However, let's take a simple example:

import java.io.IOException;

import org.jboss.stm.annotations.Transactional;
import org.jboss.stm.annotations.ReadLock;
import org.jboss.stm.annotations.WriteLock;
import org.jboss.stm.internal.RecoverableContainer;

import com.arjuna.ats.arjuna.AtomicAction;

/**
 * @author Mark Little
 */


public class Example
{  
    @Transactional
    public interface Atomic
     {
     public void change (int value) throws Exception;
       
     public void set (int value) throws Exception;
       
     public int get () throws Exception;
    }
   
    @Transactional
    public class ExampleSTM implements Atomic
     {  
        @ReadLock
     public int get () throws Exception
     {
         return state;
     }

        @WriteLock
     public void set (int value) throws Exception
     {
         state = value;
     }
       
        @WriteLock
     public void change (int value) throws Exception
     {
         state += value;
     }

     private int state;
    }
   
    public void testExample () throws Exception
    {
        RecoverableContainer theContainer = new RecoverableContainer();
        ExampleSTM basic = new ExampleSTM();
        Atomic obj = null;
       
        try
        {
        obj = theContainer.enlist(basic);
        }
        catch (final Throwable ex)
        {
        ex.printStackTrace();
           
        return;
        }
       
        AtomicAction a = new AtomicAction();
       
        a.begin();
       
        obj.set(1234);
       
        a.commit();

    System.out.println("Should get() 1234 after commit: "+obj.get());
       
        a = new AtomicAction();

        a.begin();

        obj.change(1);
       
        a.abort();

    System.out.println("Should get() 1234 after abort: "+obj.get());
    }

    public static void main (String[] args)
    {
    Example ex = new Example();

    try
    {
        ex.testExample();
    }
    catch (final Exception e)
    {
        e.printStackTrace();
    }
    }
}


Stick this at the same level of Narayana if you want to use the same CLASSPATH as here:

narayana/ext/jboss-logging.jar:narayana/ArjunaCore/arjuna/target/test-classes:narayana/common/target/common-5.0.0.M3-SNAPSHOT.jar:narayana/STM/target/core-5.0.0.M3-SNAPSHOT.jar:narayana/ArjunaCore/txoj/target/txoj-5.0.0.M3-SNAPSHOT.jar:narayana/ArjunaCore/arjuna/target/arjuna-5.0.0.M3-SNAPSHOT.jar:.

After compiling, run the example and you should see the following:


Despite what I said at the start of this entry, there's a distinct lack of vert.x involved here at the moment. That's because there are a few changes we need to make to the STM implementation to make it easier to share transactional objects between address spaces. This is all stuff that's available "natively" in TXOJ, so developers could use that API immediately. However, the STM approach simplifies a lot so once those additional features from TXOJ are integrated there, we'll have a follow up article to discuss what, how and why.

Monday, March 25, 2013

API Improvements for WS-AT and REST-AT

In this post I’ll introduce some API changes we are doing to improve the way developers use WS-AT and REST-AT.

I’m a strong believer that the internal architecture of your application, and the means by which client’s communicate with it, are two orthogonal issues. You wouldn’t want to have to change the internals of your application, just because you need to change the way clients invoke your service.

We think you should be able to develop your applications using a common Transactions API. You should then be able to invoke remote services over whatever transport is appropriate and automatically have the transaction distributed. Furthermore, you shouldn’t be restricted to having all participants use the same transport.

There really is only one option for a Java “common Transactions API” - it’s JTA. JTA is well supported in Java EE application servers and it is well understood by developers. Furthermore, there are many applications out there already using JTA.

This is what we are moving towards with each milestone release of Narayana 5.0.0. With milestone two we made it very simple for a pure JTA client to invoke a JTA application over Web Services in a distributed transaction. Take a look at the following example:

Client


@Stateless
public class OrderClient {

  @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
  public void orderItem(String item, String address, double amount) {

    AccountService as = //Lookup AccountService WS client
    WarehouseService ws = //Lookup WarehouseService WS client

    ws.shipItem(item, address);
    as.invoiceCustomer(address, amount);
  }
}


Here the client is using a regular Stateless Session Bean. The method ‘orderItem’ is invoked in a JTA transaction that is committed if the method succeeds and rolled back if it fails.

The calls to the Warehouse and Account services automatically use WS-AT to distribute the JTA transaction. The middleware knows to do this because a JTA transaction was present when the Web Service call was invoked.

Transaction propagation can be configured, but that’s the subject of another blog post (stay tuned).

Service


@Transactional //Currently required, will be removed soon https://issues.jboss.org/browse/JBTM-1468
@Stateless
//JAX-WS annotations omitted for brevity
public class WarehouseServiceImpl implements WarehouseService {

    @PersistenceContext
    protected EntityManager em;

    @WebMethod
    public void shipItem(String item, String address) {
        // Use the Entity Manager to 
        // add an item and shipping address to DB
    }
}

Again the service is implemented as a simple Stateless Session bean that just happens to be offered as a Web Service. Here the application is using JTA to make a database update. The middleware automatically handles the mapping between the incoming WS-AT transaction and the JTA transaction.

What About Other Transports?

With milestone two we already support transparent propagation over Web Services (via WS-AT), Corba (via JTS) and JBoss Remoting. We hope to have REST-AT support in a subsequent milestone release (see JBTM-1468).

Getting Started

Hopefully you are now eager to try this out. The best place to look is at this quickstart, which demonstrates the new API in action.


Acknowledgements

I'd like to say a big thank you to Alessio Soldano and the JBossWS team. They provided a lot of advice and also added new features to the JBossWS SPI to support this feature.

Thursday, March 21, 2013

Narayana and BlackTie 5.0.0.M2 Released!

I am very proud to be able to announce the availability of version 5.0.0.M2 of the Narayana and BlackTie projects which has been a significant release for us on many levels.

One of the greatest achievements for this release is that it is the first version of Narayana (JBoss Transactions 5) that has been consumed by the AS8 project. This means you can start using it straight away by downloading the latest nightly build of AS8!


Narayana's integration with the app server contains the usual bits and pieces from JBoss Transactions (JTA, JTS, XTS) alongside our new APIs for using XTS. It also provides a suitable base line to deploy the BlackTie and REST-AT server-side components. With future milestone releases of Narayana we will add full out-of-the-box support for these extra components.

Some other features that have been added in this version (release notes here and here) which I would like to pick out are:
  • Added a significantly improved API for using XTS in the app server (see separate blog posts for more details)
  • Early release of a new API for compensation-based transactions
  • JDBC Object Store
  • Simplified BlackTie client library (without TAO dependency)
  • Windows 2008 Server support for BlackTie
  • The usual set of enhancements for our documentation and quickstarts

So, where can you download the software from? For Narayana standalone, you can download the software here:
https://www.jboss.org/jbosstm/
For BlackTie, it is available here (a Linux 64 bit distribution and a Windows 32 bit one are provided):
https://www.jboss.org/blacktie/
For a build of AS8 containing Narayana:
https://ci.jboss.org/hudson/job/JBoss-AS-latest-master/lastSuccessfulBuild/artifact/build/target/jboss-as-8.x.zip

As I mentioned on a previous blog, we are going to be merging the BlackTie and Narayana repos/Jira instances/blogs etc in the M3 timeframe, so in future everything will be available from the existing Narayana locations:

Source:
https://github.com/jbosstm/narayana (plus documentation and quickstart )

Issue management:


https://jira.jboss.org/browse/JBTM (use the blacktie component if raising a blacktie issue)

Please do let us know what you think of the release either in the comments on our blog (http://jbossts.blogspot.co.uk/) or forum (https://community.jboss.org/en/jbosstm/?view=discussions).

Happy evaluating!

Friday, March 1, 2013

Merging BlackTie and Narayana

Guys,

We would like to merge the BlackTie and Narayana projects together. As you will know, BlackTie is a C++ API onto the Narayana transaction manager and as such features are being developed which require changes in both components. Furthermore, Narayana already provides support for several other APIs within the main Narayana project so collecting them all together seems to make the most sense.

The changes would be two phase:

Phase 1:
1. Merge the blacktie github repo into the narayana github repo
2. Use a single blog: http://jbossts.blogspot.co.uk/
3. Use a single jira instance: https://jira.jboss.org/browse/JBTM
4. Use a single chatroom: jbossts@irc.freenode.net
5. Use the single community space on jboss.org: https://community.jboss.org/en/jbosstm/

Phase 2:
Merge our web site https://www.jboss.org/blacktie/ into the existing https://www.jboss.org/jbosstm pages

Please do let us know if you have any objections or can see a problem with this,

Tom

Wednesday, January 30, 2013

WS-BA Participant Completion Race Condition: Part Two


The Details

In a previous post, I described a benign race condition that in unusual circumstances can cause some Business Activities to be cancelled that would have otherwise been able to close. In this post I'll go into the details of how this happens.

First consider the following client code:

UserBusinessActivity uba = UserBusinessActivityFactory.userBusinessActivity();
uba.begin();
myWebServiceClient.invoke();
uba.close();


The client code is very simple, it just begins a business activity, invokes a Web service and then closes the business activity. The Web service uses the Participant-Completion protocol and so notifies the coordinator of completion just before returning control to the client.


Here's a diagram showing the pertinent message exchanges that occur under a normal situation.




The messages are numbered to indicate the order in which they are sent.
  • 1. request. This represents the application request made by the client.
  • 2. completed. After the participant has completed its work, it notifies the coordinator that it has completed.
  • 3. response. This represents the response to the client's application request.
  • 4. close. The client notifies the coordinator that it wishes to close the activity. It then waits for a 'closed' or failure response from the coordinator.
  • 5a. close/5b. closed. The coordinator has processed the '2.completed' message so can close the activity. It starts by sending the 'close' message to the participant and waits for the 'closed' response as confirmation. These two messages are asynchronous.
  • 6. closed. The coordinator now has all 'closed' acknowledgements so notifies the client that the activity successfully closed.

Messages '2.completed' and '4.close' are asynchronous (or 'one way' in Web services parlance) so effectively, we have a race condition with the following competing parties:
  • Party 1. The completed message '2.completed'.
  • Party 2. The response '3.response' followed by '4.close'.

When running in the same VM, or on a low latency network, '3.response' will be sent very quickly. This is because it is simply travelling on the HTTP response over an already open socket. This just leaves messages '2.completed' and '4.close' which will take much longer relative to '3.response'. To understand this, lets take a look at what happens when an asynchronous Web service call is made:

  1. The client sends the message to the Web service
  2. The server-side SOAP stack uses an existing thread from a pool dedicated to receiving SOAP messages.
  3. As the service is asynchronous, the message will be passed to another thread to be processed.
  4. The receiving thread will now return the HTTP response.
The race condition occurs because steps 1-3 can happen relatively quickly in a single VM, and thus it's likely that both messages 2 and 4, will be waiting to be processed at the same time. The order in which they are processed is dependent on the implementation of the thread pool and is also at the mercy of thread scheduling in the VM, so it's possible that either could be processed first.

This race condition is much less likely to happen in a distributed environment as the network costs will be significantly higher. As a result message '3.response' will take long enough to send, so as to give message '2.completed' enough of a head start. But it is still possible so the client application must be coded defensively to catch and handle a TransactionRollbackException. Your code ought to be doing this anyway to deal with server crashes.

Here's a diagram showing what messages are exchanged when the race condition occurs. You will see that the activity ends in a consistent state.



I've omitted messages 1-3 from the following explanation as they are the same as in the success case.
  • 4. close. This message is processed by the coordinator before message '2.completed'
  • 5a. cancel. The coordinator has not yet processed the '2.completed' message so cannot close the activity. The coordinator then sends a 'cancel' message to the participant as it thinks it has not yet completed. This message and subsequent retires, are dropped by the participant as they are not valid for a completed participant.
  • 5b. compensate/5c. compensated. After one or more unacknowledged 'cancel' messages, the coordinator switches to sending 'compensate' messages which will cause the participant to compensate the work. The participant acknowledges with a 'compensated' reply.
  • 6. Transaction rolledback exception. The coordinator notifies the client that the activity failed to close.

As you can see from the steps above, when this race condition arises, any work done by participants is compensated and the client is notified of the outcome. Thus a consistent outcome is achieved.

Thursday, January 24, 2013

WS-BA Participant Completion Race Condition: Part One

Overview

The WS-BA participant-completion protocol has a benign race condition that, in unusual circumstances, can cause some Business Activities to be cancelled that would have otherwise been able to close. This is safe as no inconsistency arises, but it can be annoying for users. This blog post explains why this can happen, under what conditions, and what you can do to tolerate it. This post gives you an overview of the issue and should provide enough details for most developers. A follow up post will get into the nitty-gritty details of what's really going on.

What's happening, in a nutshell

Imagine a scenario where the client begins a business activity and then invokes a Web service. If the Web service uses participant completion, it will notify the coordinator when it has completed its work and then return control to the client. This notification is asynchronous, so it's possible that the client will then ask the coordinator to close the activity before the coordinator processes (or even receives) the completed notification from the participant. In this situation the coordinator will cancel the activity as not all participants (from its perspective) have completed their work. As a result all completed participants are compensated (including, eventually, the participant with the late 'completed' notification) and the client receives a "TransactionRolledBackException".

When is it most likely to happen?

Typically this happens when the client, coordinator and participant are running inside the same VM. This scenario is unlikely to happen in production, but can happen regularly during development where a single VM is used to keep things simple.

How do I know if this is affecting my application?

If the client is occasionally receiving a TransactionRolledbackException when calling UserBusinessActivity#close(), but none of the machines involved in running the transaction have crashed, you could be affected by this. Especially if you are running the client, coordinator and participant(s) in the same server.

We've now added a log message to help you identify this. However, to see this, you will either need to be building transactions from the current source (4.17 or master branches in GitHub) or wait for the JBossTS 4.17.4 or Narayana 5.0.0.M2 release. This is the log message to look out for:

WARN  [com.arjuna.mw.wstx] (TaskWorker-2) ARJUNA045062: Coordinator cancelled the activity

This is only an indication that you are seeing this issue as the coordinator can elect to cancel the activity for other reasons. For example, network problems might mean the coordinator cannot tell the web service to close the activity.

Why can't this be avoided?

The short answer is that for the protocol to avoid this it would need to make the complete message synchronous, throttling throughput by slowing down both the participant and coordinator and holding sockets open for longer.

What can the application do to tolerate this?

A real, distributed deployment will rarely see this problem because communication latency between client, participant and coordinator will dominate the race condition. Even if it does happen your application should tolerate it. Transaction rollbacks and activity cancellations are inevitable in a distributed environment and can happen for many reasons. When handling TransactionRolledBack exceptions you can either retry the Transaction/Activity or notify the caller of the failure. What you choose to do will depend on the requirements of your application.

In part two, I'll get into the details of what's happening.

Sunday, December 23, 2012

Raspberry flavoured transactions

I got asked by a few different people lately about how to build Narayana on a Raspberry Pi. Well it's pretty easy and you don't even need one of the upgraded Model Bs with 512Meg of memory. Here are the steps you need to follow:


  • I'm going to assume you know how to set up your Pi. I'm also going to assume that you're using the default pi account.
  • You'll need to install OpenJDK (sudo apt-get install openjdk-6-jre openjdk-6-jdk), maven and git (maven/git sudo apt-get install maven2 git). Other optionals include emacs (of course!) and your favourite databases, but we'll leave those for the curious.
  • Then let's clone the source tree ... git clone https://github.com/jbosstm/narayana

  • Then it's the usual steps to build, although due to some issues with idl on the Pi, we can't run JTS just yet using the ORB within the JDK, so let's go with the JTA implementation only ... ./build.sh clean install -P jta
  • After that, sit back and wait ... for a long time, so be patient.
  • It should take about 2 hours to complete, which is significantly slower than the 30 minutes or so that it can take on your laptop to build everything. (If you try to build everything, then it'll be closer to 5 hours!) But hey, the Pi is underpowered compared to that hardware and if you're looking to use a Pi then hopefully you've already taken that into consideration! And yes, you will see some warnings and maybe a timing error or two during the build execution of some tests, but nothing to be concerned about. We'll look into these and see if there's anything we can do to tidy-up the build in the future.

And that's it! Congratulations, you've just built and tested Narayana on your Raspberry Pi. And apart from the time it takes to build, there really wasn't much to it, now was there?

Note, if you want to build the JTS then you do have the option to use JacORB. However, at the moment this requires you to edit a number of the pom.xml files to remove the dependency on idlj.