Some experiments in migrating transaction logs

Transaction stores

Some time ago I prototyped a Redis based implementation of the SlotStore backend suitable for installations where nodes hosting the storage can come and go making it well suited for cloud based deployments of the Recovery Manager.

In the context of the CAP theorem of distributed computing, the recovery store needs to behave as a CP system, ie it needs to be able to tolerate network partitions and yet continue to provide Strong Consistency. Redis can provide the strong consistency guarantee if the RedisRaft module is used with Redis running as a cluster. RedisRaft achieves consistency and partition tolerance by ensuring that:

• acknowledged writes are guaranteed to be committed and never lost,
• reads will always return the most up-to-date committed write,
• the cluster is sized correctly: a RedisRaft cluster of 3 nodes can tolerate a single node failure and a cluster of 5 can tolerate 2 node failures, … ie if the cluster is to tolerate losing N nodes then the cluster size must be at least 2*N+1, thus the minimum cluster size is 3 and the reason for having an odd number of nodes in the cluster is to avoid “split brain” scenarios during network partitions; an odd number guarantees that one side of the split will be in the majority.

During network splits the cluster will become unavailable for a while, ie the cluster is designed to survive failures of a few nodes in the cluster, but it is not a suitable solution for applications that require availability in the event of large net splits, however transaction systems favour Consistency over Availability.

A key motivator for this new SlotStore backend is to address a common problem with using the Narayana transaction stores on cloud platforms when scaling down a node that has in doubt transactions which can leave them unmanaged. Most cloud platforms can detect crashed nodes and restart them but this must be carefully managed to ensure that the restarted node is identically configured (same node identifier, same transaction store and same resource adapters). The current cloud solution, when running on Openshift, is to use a ReplicaSet and to veto scale down until all transactions are completed which can take an indeterminate amount of time, but if we can ask another member of the deployment to finish these in doubt transactions then all but the last node can be safely shutdown even with in doubt transactions. The resulting increase in availability in the presence of node or network failures is a significant benefit for transactional applications which, after all, is key reason why businesses are embracing cloud based deployments.

Remark: Redis is offered as a managed service on a majority of cloud platforms which can help customers to get started with this solution. But note that standard Redis excludes the RedisRaft module which is a requirment for use as a transaction store.

A Redis backed store

Redis is a key value store. Keys are stored in hash slots and hash slots are shared evenly amongst the shards (keys -> hash slots -> shards), the redis cluster specication contains the details. Re-sharding involves moving hash slots to other nodes, impacting performance. Thus, if we can control which hash slots the [slot store] keys map onto then we can improve performance under both normal and failure conditions. This periodic rebalancing of the cluster can be optimised if keys belonging to the same recovery manager are stored in the same hash slot, additionally having the keys, for a particular recovery node, colocated on a single cluster node is good for the general performance of the transaction store.

Also noteworthy is that the keys mapped to a particular hash slot can operated upon transactionally, which is not the case for keys in different slots, meaning that no inter-node hand-shaking is required. This feature opens up the possibility, perhaps, of allowing concurrent access to recovery logs by different recovery managers - but that’s something for a future iteration of the design, but if the logs in a store are shared then be aware that some Narayana recovery modules cache records so those implementations would need to re-evaluated, noting in particular that Redis has support for optimistic concurrency using the watch API which clients can use to observe updates to key values by other recovery managers.

Key space design

A recovery manager has a unique node identifier. We’d like to be able to form “recovery groups” such that any recovery manager in the group can manage transactions created by the others, but not at the same time. To this end we assign a “failoverGroupId” to each recovery manager and use that as the Redis key prefix. This will force all keys created by members of the failover group into the same hash slot, a cloud example of this idea is that the pods in a deployment would all share the same failoverGroupId so any pod in the deployement can take over when the deployment is scaled down.

Failover

Failover involves detecting when a member of the “recovery group” is removed from the cluster and to then migrate the keys to another member of the group. I added an example to the LRA recovery coordinator and used the jedis redis API rename command to “migrate” the keys which is an atomic operation.

Issues

The performance of Redis Raft in this implementation of the SlotStore backend is poor (more than 4 times slower than the default store); I have not invested any effort on improving it but may follow up with another post to discuss throughput since it is a general issue that needs to be solved for any cloud based object store, examples of topics to investigate include pipelining redis commands (similar to how we batch writes to the Journal Store), using Virtual Threads, etc.

We are therefore investigating other alternatives, including an Infinispan based slot store backend - Infinispan now supports partition tolerance so it has become a suitable candidate for a transaction store. Such a store will produce many of the benefits of a Redis store although its performance will be a key implementation constraint.

This design for the key space may not be suitable for transactions with subordinates or nested transactions or for ones that require participant logs to be distinct from the transaction log, such as JTS or XTS. I say may since a modification to the design should accomodate these models.

Assumptions

The cloud platform administrator is responsible for:

• detecting and restarting failed nodes;
• issuing the migrate command on one of the remaining nodes
• for detecting when the deployment is scaled down to zero with pending transactions (including orphans) and emitting a warning accordingly

Example of how to migrate logs

The demo presents the use case of migrating logs between LRA coordinators. To run the demo you will need to:

1. Clone and build a narayana git branch with support for a Redis backed store.
2. Start a 3-node cluster of Redis nodes running RedisRaft.
3. Build and start two LRA coordinators with distinct node id’s, node1 and node2.
4. Start an LRA on the first coordinator and then halt it to simulate a failure.
5. View the redis keys using the Redis CLI noticing that the keys embed the node id of the owning coordinator.
6. Ask the second coordinator to migrate the keys from node1 to node2.
7. Coordinators maintain a cache of LRAs, but since this is just a PoC I haven’t implemented refreshing internal caches so you will need to simulate that by restarting the second coordinator.
8. When the first periodic recovery cycle runs (the default is every 2 minutes) the migrated LRAs will be detected which you can verify (curl http://localhost:50001/lra-coordinator/|jq).

Please refer to the demonstrator instructions for the full details.

Notes

The JIRA issue and branch is JBTM-3762.

• the implementation is is in the slot store directory
• the tests can be ran using mvn test -Predis-store -Dtest=RedisStoreTest#test1 -f ArjunaCore/arjuna/pom.xml and assume that a redis cluster is running on the test machine
• the demo is work in progress and is strictly a PoC
• redis raft implementation: git clone https://github.com/RedisLabs/redisraft.git and build it with: cmake and make

Software Transactional Memory for the Cloud at µCon London 2017

I have just returned from µCon London 2017: The Microservices Conference. I was down there talking about our Software Transactional Memory implementation which our regular readers may recall was first introduced by Mark back in 2013. In particular I wanted to show how it can be used with the actor model features of Vert.x and the nice scaling features of OpenShift.

I have put the presentation slides in the narayana git repo so feel free to go and take a look to see what you missed out on. You can also access the source code I used for the practical demonstration of the technology from the same repository. The abstract for the talk gives a good overview of the topics discussed:

"Vert.x is the leading JVM-based stack for developing asynchronous, event-driven applications. Traditional ACID transactions, especially distributed transactions, are typically difficult to use in such an environment due to their blocking nature. However, the transactional actor model, which pre-dates Java, has been successful in a number of areas over the years. In this talk you will learn how they have been integrating this model using Narayana transactions, Software Transactional Memory and Vert.x. Michael will go through an example and show how Vert.x developers can now utilise volatile, persistent and nested transactions in their applications, as well as what this might mean for enterprise deployments."

And the demo that this abstract is referring to is pretty trivial but it does serve to draw out some of the impressive benefits of combining actors with STM. Here's how I introduced the technology:

1. Firstly I showed how to write a simple Vert.x flight booking application that maintains an integer count of the number of bookings.
2. Run the application using multiple Vert.x verticle instances.
3. Demonstrate concurrency issues using parallel workloads.
4. Fix the concurrency issue by showing how to add volatile STM support to the application.

I put the without STM and the with STM versions of the demo code in the same repository as the presentation slides.

I then showed how to deploy the application to the cloud using a single-node OpenShift cluster running on my laptop using Red Hat's minishift solution. For this I used "persistent" STM which allows state to be shared between JVMs and this gave me the opportunity to show some of the elastic scaling features of STM, Vert.x and OpenShift.

Transactions and Microservices on the rise!

Our team spoke about how transactions and microservices can be used together over 3 years ago. Since then there have been a few people continuing to suggest there's no role for transactions. However, we're also starting to see more people talking about the same things such as at DevoxxUK and elsewhere. That's great! We'll keep pushing Narayana in this space and if you're interested then get involved with us too!

Achieving Consistency in a Microservices Architecture

Microservices are loosely coupled independently deployable services. Although a well designed service will not directly operate on shared data it may still need to ensure that that data will ultimately remain consistent. For example, the requirement to debit an account to pay for an on-line purchase creates a dependency between the customer and supplier account balances and the stock database. Historically a distributed transaction has been used to maintain this consistency which in turn will employ some flavour of distributed locking during the data update phase. This dependency introduces tight coupling, higher latencies and greater lock contention, especially when failures occur where the locks cannot be released until all services involved in the transaction become available again. Whilst the user of the system may be satisfied with this state of affairs it should not be the only possible interaction pattern. A more common approach will employ the notion of eventual consistency where the data may sometimes be in an inconsistent state but will eventually come back into the desired state: in our example the stock level will be reduced, the payment has been processed and the item delivered.

I have, from time to time, seen blogs and articles that recognise this problem and suggest solutions but they seem to mandate that either service calls naturally map on to a single data update or that the service writer picks one of the services to do the coordination taking on the responsibility of ensuring that all services involved in the interaction will eventually reach their target consistent state (see for example a quote from the article Distributed Transactions: The Icebergs of Microservices: "you have to pick one of the services to be the primary handler for the event. It will handle the original event with a single commit, and then take responsibility for asynchronously communicating the secondary effects to other services"). This sounds feasible but now you have to start thinking about how to provide the reliability guarantees in the presence of failures, how to orchestrate services, storing extra state with every persistent update so that the activity coordinator can continue the interaction after failures have been resolved. In other words, whilst this is a workable approach it hides much of the complexity involved in reliably recovering from system and network failures which at scale will surely happen. A more robust design for microservice architectures is to delegate the coordination component of the workflow to a specialised service explicitly designed for this kind of task.

We have been working in this area for many years and one set of ideas and protocols that we believe are particularly suited to microservices architectures is the use of compensatable units of work to achieve eventual consistency guarantees in this kind of loosely coupled service based environment. I produced a write up of the approach and accompanying protocol for use in REST based systems back in 2009 (Compensating RESTful Transactions) based on earlier work done by Mark Little et al. Mark also wrote some interesting blogs in 2011 (When ACID is too strong and Slightly alkaline transactions if you please ...) about alternatives to ACID when various constraints are loosened and his summary is relevant to the problems facing microservice architects.

The use of compensations, coordinated by a dedicated service, will give all the benefits suggested in Graham Lea's article referred to earlier, but with the additional guarantees of consistency, reliability, manageability, reduced complexity etc in the presence of failures. The essence of the idea is that the prepare step is skipped and instead the services involved in the interaction register compensation actions with a dedicated coordinator:

1. The client creates a coordination resource (identified via a resource url)
2. The client makes service invocations passing the coordinator url by some (unspecified) mechanism
3. The service registers its compensate logic with the coordinator and performs the service request as normal
4. When the client is done it tells the coordinator to complete or cancel the interaction

• in the complete case the coordinator has nothing to do (except clean up actions)
• in the cancel case the coordinator initiates the undo logic. Services are not allowed to fail this step. If they are not available or cannot compensate for the activity immediately the coordinator will keep on trying until all services have compensated (and only then will it clean up)

We do not have an implementation of this (JDI) protocol but we do have an implementation of an ACID variant of it (called RTS) which has had extensive exposure in the field (and this can/will serve as the basis for the implementation of the JDI protocol). The documentation for RTS is available at our project web site. The nice thing about this work is that it can integrate seamlessly into Java EE environments and additionally is available as a WildFly subsystem. This latter feature means that it can be packaged as a WildFly Swarm microservice using the WildFly Swarm Project Generator. In this way if your microservices are using REST for API calls then they can make immediate use of this feature.

We also have a working prototype framework for how to do compensations in a Java SE environment. The API is available at github where we also provide a number of quickstarts showing how to use it.

Finally, we have a solution where we allow the compensation data to be stored at the same time as the data updates in a single (one phase) transaction thus ensuring that the coordinator will have access to the compensation data. This technique works particularly well with document oriented databases such as MongoDB

XA and microservices

I had a very interesting series of discussions this week with a few friends/colleagues about whether or not transactions are applicable in a microservices architecture. I've already covered these points in an earlier entry so I won't repeat them here. However, several things did come up that are worthy of trying to address, including the assumption that they're too hard to use and too hard to test your application for correctness when using them. Now as I've mentioned in several presentations there are valid concerns like these around using or not using transactions in general, i.e., irrespective of microservices.

These points that came up recently are not just to be dismissed out of hand. Having worked in transactions for almost 30 years I've spent a long time hearing about these and other concerns and trying to address them within our industry with many others from a range of companies. For instance, if you look at what we've achieved within Java EE with CDI annotations for transactions, or within Narayana for similar approaches towards compensations, I think we've come a long way to simplifying how developers can use transactions. The STM work that we've done is also another example of putting transactions into the hands of developers and not requiring them to have to worry even about demarcation or state management.

Testing applications and their use of transactions is obviously important, in just the same way as when adding any component or capability, such as Hibernate, Zookeeper or Map/Reduce. However, once again we've made a lot of progress over the years. Within Narayana we have 1000s of unit and QA tests which not only help to prove the correctness of the underlying implementation but also feed into best practices etc. for developers. Then there's transaction extensions to Arquillian which also help to make testing much easier. Now in terms of proving the correctness of transaction systems we're also pretty safe there (no pun intended). Since transactions have been used in mission critical environments for over five decades there's a large body of associated work to prove the various aspects are correct for implementations and applications. Overall I'm fairly sure that these concerns have been addressed sufficiently well to not be a meaningful reason to avoid transactions, in general or with microservices. As an industry could we do better? Of course. We should always be striving to improve the overall developer experience and how transactions fit in.

However, one other thing which did come up during the conversations I referred to initially is that there's often a concern that transactions equals XA, or put another way, that the problems perceived to be present with transactions are the result of design issues inherent within the original XA protocol and associated implementations. As I keep saying, 2PC and transactions existed before XA and many transaction system implementations aren't tied to only supporting the XA protocol. All of the transaction standards have their quirks, just as most standards do no matter their domain. But XA has a few more than its fair share, often due to the fact it's been around for so long that the ways in which we tend to develop applications have changed radically since it was defined. Yet we're tied to it due to the fact that relational databases still represent the majority way in which we store data. Of course we've developed new transaction standards which don't rely upon XA, such as OTS or WS-AT, but most developers still see and use XA and hence why there may be concerns about transactions which are really more to do with XA.

Back to the topic: I'm not suggesting that XA doesn't have a role to play within microservices and between microservices. At least in the short term it most definitely does for some (small) set of applications. But really when I'm suggesting transactions have a role I'm looking well beyond XA. I'm still considering ACID as well as compensation transactions though because as I've said, some implementations aren't tied to XA and yet provide the optimisations you expect from XA (one-phase commit, read-only optimisation) as well as going beyond (interposition, nested transactions). So whilst I definitely understand concerns about transactions, I still believe they either have been addressed and we need to do a better job of developer education, or we can address them; the benefits they bring are worth it.

Narayana is now JTA 1.2 Compliant

I'm very proud to announce that with Narayana 5.0.0.M3 we are now JTA 1.2 compliant! What's more, with the release of JCA 1.7 in WildFly 8.0.0.Beta1 (WFLY-510), the new application server features are now available.

I'd like to say a big "thank you" to the community members who provided feedback during the design and implementation of this specification. Like most of our larger features, this was discussed over at the Narayana developer forums, giving the community the opportunity to shape the development of these features.

So what's new in JTA 1.2?

JTA 1.2 is considered a 'maintenance release', so the change list is quite small. However, that's not to say that the changes aren't exciting. Here's an overview of the three improvements:

@Transactional. This is the headline feature and brings the ability to place transaction annotations on any CDI managed bean. Prior to this feature, you had to make your class an EJB in order to use transactional annotations.
@TransactionScoped. This feature allows you to associate CDI beans with the scope of a transaction.
The third change clarifies when the container should call 'delistResource' on the transactional resource. This is a minor change and is of less interest to an application developer, so I'm not going to discuss it further here.

Tell me more about @Transactional and @TransactionScoped

The remainder of this post will tell you what you need to know about these new features and provide some examples, showing how they can be used.

@Transactional

This new annotation provides an alternative to javax.ejb.TransactionAttribute that can be placed on CDI managed bean classes and its methods. Prior to this feature, many developers were using EJBs just so that they could use declarative transactions. Now you can make an architectural decision of wether EJB or CDI is the right approach for your application, knowing that you will be able to use declarative transactions with either approach. The remainder of this section will show an example using @Transactional and highlight the differences between @Transactional and the EJB @TransactionAttribute.


@Transactional(Transactional.TxType.MANDATORY)
public class MyCDIBean {

    @Transactional(Transactional.TxType.NEVER)
    public void doSomethingWithoutATransaction() throws Exception
    {

        //Do something that must be done outside of a transaction
    }

    @Transactional(
         dontRollbackOn=MyNonCriticalRuntimeException.class,
         rollbackOn=TestException.class)
    public void doSomething() throws Exception {
//Do something that must be done inside a transaction
    }
}

The "@Transactional(Transactional.TxType.MANDATORY)" annotation on the class states that, by default, all methods must be invoked within the scope of a JTA transaction. This can be overridden on a per-method basis.

The 'doSomethingWithoutATransaction' method overrides the 'MANDATORY' type with "NEVER". Therefore, this method will fail if it is invoked in the scope of a JTA transaction.

Specifying what Exceptions should cause the transaction to rollback is specified differently with @Transactional. With EJB's @TransactionalAttribute, the developer had to declare on the actual Exception class whether an exception of that type, thrown from a method annotated with @TransactionalAttribute, should cause the active transaction to be marked for 'rollback only'. The problem with this approach was that it could not (easily) be applied to third-party exception implementations and it also applied for all usages of that exception. @Transactional fixes these issues by allowing the developer to specify on a per-method basis, which Exception types (and sub-classes of) should cause a rollback. This is done through the 'rollbackOn' and 'dontRollbackOn' attributes. By default, RuntimeException and its subclasses will cause a rollback.

Another difference to watch out for is the default behaviour when no transaction annotation is provided. With an EJB, the default value of TransactionalAttribute is TRANSACTION_REQUIRED. This means that a new JTA transaction is begun (when calling the method) if one doesn't already exist. This works for EJB as the developer has already opted-in through the use of an EJB annotation (@Stateless, @Statefull, etc). With CDI, a managed bean can have no annotations, thus it is difficult to differentiate between it and a regular POJO. Therefore, the default value for @Transactional is TxType.SUPPORTED. This means that the method will run within a transaction if one exists; otherwise it will run without one. This is essentially how transactions are handled with java POJOs.

@TransactionScoped

With @TransactionScoped brings an additional CDI context to accompany @SessionScoped, @RequestScoped and @ApplicationScoped. Annotating a bean with @SessionScoped ensures that the same instance of the bean is made available for all usages of it within the scope of a http session. This allows the developer to easily share state between multiple requests within a session, whilst also isolating it from other requests in a different session.

@TransactionScoped allows data to be shared between all usages of the bean within a particular transaction, whilst isolating the instance from other accesses of the bean within a different transaction. The TransactionScoped bean instance's lifecycle matches that of the transaction.

The following code shows an example of this in use:

@TransactionScoped
public class MyCDITransactionScopeBean implements Serializable {

    private int value = 0;
    public int getValue() ... 
    public void setValue(int value) ...
}

MyCDITransactionScopeBean represents the data to be associated with the active transaction. It is simple POJO annotated with @TransactionScoped and also marked as Serializable.

public class SomeClass {

    @Inject
    MyCDITransactionScopeBean myBean;
...
}

MyCDITransactionScopeBean can then be injected into other classes that need to use it.

Go and give it a try!

Just download the latest version of WildFly and start deploying code. We don't yet have any quickstarts for these features, but our Arquillian tests provide a complete example of how to use the functionality.

Compensating Transactions: When ACID is too much (Part 4: Long Lived Transactions)

In part one, I explained how ACID transactions can have a negative impact on applications whose transactions can take a relatively long time to run. In addition, another potential issue with ACID transactions is that the failure of one unit can cause the entire transaction to be rolled back. This is less of an issue for short running transactions, as the previously successful work can be retried quickly. However, for long running transactions, this previously completed work may be significant, leading to a lot of waste should it need to be rolled back. In this post, I'll show how a compensation-based transaction could be a better fit for long lived transactions.

A compensation-based transaction can be composed of multiple short-lived ACID transactions. When each transaction completes, it releases the locks on the resources it held, allowing other transactions, requiring those resources, to proceed. A compensation action is registered for each ACID transaction and can be used to undo any work completed, should the entire compensation-based transaction need to be aborted. Furthermore, should one of these short-lived ACID transactions fail, it could be possible to find an alternative, preventing the entire transaction from failing. This allows forward progress to be achieved. By composing the compensation-based transaction as several units of work, you also gain the opportunity to selectively abort (compensate) particular units as the compensation-based transaction progresses. A simple example should help to clarify these ideas...

For example, take a travel booking scenario. We begin by booking a flight. We then try to book a taxi, but that fails. At this point we don’t want to compensate the flight as it may be fully-booked next time we try. Therefore we try to find an alternative Taxi, which in this example succeeds. Later, in the compensation-based transaction, we may find a cheaper flight, in which case we want to cancel the original flight whilst keeping the taxi and the cheaper flight. In this case we notify our intentions to the transaction manager who ensures that the more expensive flight is compensated when the compensation-based transaction completes.

Code Example

In this example, I expand on the Travel Agent example from part 3 in this series. Here I will show how a failure to complete one unit of work does not have to result in the whole compensation-based transaction being aborted.

public class Agent {
 
 @Inject
 HotelService hotelService;
 
 @Inject
 Taxi1Service taxi1Service;
 
 @Inject
 Taxi2Service taxi2Service;
 
 @Compensatable
 public void makeBooking(String emailAddress, String roomType, String destination) throws BookingException {
 
  hotelService.makeBooking(roomType, emailAddress);
  
  try {
   taxi1Service.makeBooking(destination, emailAddress);
  } catch (BookingException e) {
    /**
     * Taxi1 rolled back, but we still have the hotel booked. We don't want to lose it, so we now try Taxi2
     */
     taxi2Service.makeBooking(destination, emailAddress);
  }
 }
}

For this example, you can imagine that the Hotel and Taxi services are implemented similarly to the HotelService in part 3.
The makeBooking method is annotated with @Compensatable, which ensures that the method is invoked within a compensation-based transaction. The method begins by making a Hotel reservation. If this fails, we don't handle the BookingException, which causes the compensation-based transaction to be canceled. We then move onto booking a taxi. If this particular booking fails, we catch the BookingException and try an alternative Taxi company. Because the Taxi service failed immediately, we know that it should (it's a requirement of using this API) have undone any of it's work. We can therefore chose to fail the compensation-based transaction or, in this case try an alternative Taxi company. The important thing to note here is that we still have the Hotel booked and we don't really want to lose this booking as the hotel may be fully booked next time we try. The code then goes on to attempt an alternative Taxi company. If this booking fails, we have no option but to cancel the whole transaction as we have no other alternatives.

Conclusion

In this blog post, I showed how a compensation-based transaction could be a good fit for long running transactions. In the next part I'll discuss the status of our API for compensation-based transactions.

Compensating Transactions: When ACID is too much (Part 3: Cross-Domain Distributed Transactions)

In part one in this series I explained why ACID transactions are not always appropriate. I also introduced compensation-based transactions as a possible alternative to ACID transactions. In this post I'll show how compensation-based transactions could be a better fit, than ACID transactions, for distributed applications that cross high latency networks or span multiple business domains.

When your application becomes distributed, and more systems become involved, you inevitably increase the chances of failure. Many of these failures can be tolerated using a distributed ACID transaction. However, an ACID transaction can be seen as impractical for certain distributed applications. The first reason for this, is that distributed transactions, that cross high latency networks (such as the Internet), can take a relatively long time to run. As I showed in part 1, increasing the time to run an ACID transaction can have negative impacts on your application. For example, the holding of database resources for prolonged periods can significantly reduce the throughput of your application. The second reason is due to the tight coupling between the participants of the transaction. This tight coupling occurs because the root coordinator of the transaction ultimately drives all the transactional resources through the 2PC protocol. Therefore, once prepared, a transactional resource has to either make a heuristic decision (bad) or wait for the root coordinator to inform it of the outcome. This tight-coupling may be acceptable if your distributed application resides in a single business domain where you have control of all the parties. However, it is less likely to be acceptable for distributed applications that span multiple business domains.

Compensation-based transactions could prove to be a better solution for these scenarios. As I showed in part 1, compensation-based transactions can be more suitable for longer lived transactions, as they don't need to hold onto database resources until the transaction completes. Compensation-based transactions can also be used to decouple the back-end resources from the transaction coordinator. This can be done by splitting the two phases of the protocol into abstract business operations, such as book/cancel. The 'book' operation makes an update to the database to create the booking. As this update is committed immediately, there is no tight-coupling between the database resources and the transaction coordinator. The cancel operation is invoked by the compensation handler, should the compensation-based transaction need to abort.

I'll also show, through this example, how isolation can be preserved, whilst moving the resource locking from the database-level up to the application-level where it is easier for the application to reason about.

Code Example

In this example we'll look at a simple travel booking example, in which a client makes a hotel and taxi booking with remote services, inside a compensation-based transaction. These remote services live in different business domains and are invoked over the Internet.

public class Client {
 
  @Compensatable
  public void makeBooking() throws BookingException {
 
    // Lookup Hotel and Taxi Web Service ports here...
 
    hotelService.makeBooking("dbl", "paul.robinson@redhat.com");
    taxiService.makeBooking("ncl", "paul.robinson@redhat.com");
  }
}

This code forms part of the client application. The 'makeBooking' method is annotated with '@Compensatable' which ensures that the method is invoked within a compensation-based transaction. The method invokes two Web services. These Web services support WS-BA, so the transaction is transparently distributed over these calls.

@WebService
public class HotelService {
 
 @Inject
 BookingData bookingData;
 
 @Compensatable(MANDATORY)
 @TxCompensate(CancelBooking.class)
 @TxConfirm(ConfirmBooking.class)
 @Transactional(value=REQUIRES_NEW, rollbackOn=BookingException.class)
 @WebMethod
 public void makeBooking(String item, String user) throws BookingException {
 
  //Update the database to mark the booking as pending...
 
  bookingData.setBookingID("the id of the booking goes here");
 }
}

Here's the code for the Hotel's Web Service. As well as the usual JAX-WS annotations that you would expect (some omitted for brevity), there are some extra annotations to manage the transactions. The first is @Compensatable(MANDATORY); this ensures that this method is invoked within the scope of a compensation-based transaction. The second annotation (@TxCompensate) provides the compensation handler, which you should be familiar with from Part 2 in this series. The third annotation (@TxConfirm), may be new to you. This annotation provides a handler that is invoked at the end of the transaction if it was successful. This allows the application to make final changes once it knows the transaction will not be compensated. Hopefully, the need for this feature will become more clear as we discuss this example further. Finally, this example uses the JTA 1.2 @Transactional annotation to begin a new JTA transaction. This transaction is used to make the update to the database and will commit if the 'makeBooking' method completes successfully. The JTA transaction will rollback if a BookingException (see the rollbackOn attribute) or a RuntimeException (or a subclass of) are thrown.

So, why does the JTA transaction commit at the end of this method call even though the compensation-based transaction is still running?

This is done to reduce the amount of time the service holds onto database resources. The application could simply add the booking to the database, but it's possible that at some time in the future it might need to be canceled. Therefore in this example, the application just marks the booking as pending. Therefore, any other transaction that reads the state of the bookings table will see that this particular booking is tentative. Remember, by using a compensation-based transaction, we have relaxed isolation, and this is one way in which the application can be modified to tolerate this.

public class ConfirmBooking implements ConfirmationHandler {
 
 @Inject
 BookingData bookingData;
  
 @Override
 @Transactional(REQUIRES_NEW)
 public void confirm() {
   //Confirm order for '" + bookingData.getBookingID() + "' in Database (in a JTA transaction)
 }
}

As mentioned above, the ConfirmationHandler provides a callback that occurs when the compensation-based transaction completes successfully. In this example, the confirmation handler begins a new JTA transaction and then updates the database to mark the booking as finalized.

public class CancelBooking implements CompensationHandler {
 
 @Inject
 BookingData bookingData;
 
 @Override
 @Transactional(REQUIRES_NEW)
 public void compensate() {
  //Cancel order for bookingData.getBookingID() in Database (in a new JTA transaction)
 }
}

Similarly, we also have a compensation handler that starts a new JTA transaction which cancels the booking.

In this example we are essentially making a trade-off between 2 shorter lived JTA transactions, with application-level locking (in this example) in place of 1 longer lived JTA transaction with database-level locking (had we used a distributed JTA transaction). The isolation level is roughly the same for both cases. We also needed to make a change to the application. Wether this is a sensible trade-off will depend largely on your application and throughput requirements.

It is also worth noting that the middleware can (we don't yet implement this, see here and here) reliably tie the compensation-based and the JTA transaction together. It does this by preparing the JTA transaction when the method completes, but holds off committing it until the compensation handler has been logged to the transaction log. This ensures that the work is only committed if it can later be compensated in the case of failure. The invocation of the CompensationHandler and the ConfirmationHandler is also reliable as they are invoked repeatedly until they complete successfully. Therefore, it is important for thier implementations to be idempotent.

Compensating Transactions: When ACID is too much (Part 2: Non-Transactional Resources)

Introduction

In part one in this series I explained why ACID transactions are not always appropriate. I also introduced compensation-based transactions as a possible alternative to ACID transactions. In this post I'll focus on situations where the application needs to coordinate multiple non-transactional resources and show how a compensation-based transaction could be used to solve this problem.

For the sake of this discussion, I'm defining a transactional resource as one that can participate in a two phase protocol and can thus be prepared and later committed or rolled back. For example, XA-capable databases or message queues would be considered transactional resources. In contrast a non-transactional resource is one that does not offer this facility. For example, the sending of an email or printing of a cheque can not easily participate in this two phase protocol. Third party services can also be hard to coordinate in an ACID transaction. Even though these services might be implemented with ACID transactions, they may not allow participation in any existing transaction.

A compensation-based transaction could be a good fit for these situations. The non-transactional work can be carried out in the scope of the compensation-based transaction. Providing that a compensation handler is registered, the work can later be undone, should the compensation-based transaction need to be aborted. For example, the compensation handler for sending an email, could be to send a second email asking the recipient to disregard the first email. The printing of a cheque could be compensated by canceling the cheque and notifying the recipient of the cancellation. 

It's also possible to coordinate transactional and non-transactional resources in a compensation-based transaction. Here the application just needs to create compensation handlers for the non-transactional resources. You could still use an ACID transaction with the last resource commit optimization (LRCO) if you only have one non-transactional resource, but this approach is not recommended if you have multiple non-transactional resources.

In a nutshell: If you find yourself needing to coordinate multiple non-transactional resources, you should consider using compensations.

Code Example

In this code example, we have a simple service that is used by an EComerce application to sell books. As well as making updates to transactional resources, such as a database, it also needs to send an email notifying the customer that the order was made.

public class BookService {
 
    @Inject
    EmailSender emailSender;
 
    @Compensatable
    public void buyBook(String item, String emailAddress) {
 
        emailSender.notifyCustomerOfPurchase(item, emailAddress);
        //Carry out other activities, such as updating inventory and charging the customer
    }
}

The above class represents the BookService. The 'buyBook' method coordinates updates to the database and notifies the customer via an email. The 'buyBook' method is annotated with '@Compensatable'. Processing of this annotation ensures that a compensation-based transaction is running when the method is invoked. This annotation is processed similarly to the @Transactional annotation (new to JTA 1.2). The key difference being that it works with a compensation-based transaction, rather than a JTA (ACID) transaction. An uncaught RuntimeException (or subclass of) will cause the transaction to be canceled, and any completed work to be compensated. Again, this behavior is based on the Transaction handling behavior of @Transactional in JTA 1.2.

For the sake of brevity, I have excluded the calls to update the other transactional resources. Part 3 of this series will show interoperation with JTA ACID transactions.

public class EmailSender {
 
    @Inject
    OrderData orderData;
  
    @CompensateWith(NotifyCustomerOfCancellation.class)
    public void notifyCustomerOfPurchase(String item, String emailAddress) {
 
        orderData.setEmailAddress(emailAddress);
        orderData.setItem(item);
        //send email here...
    }
}

This class carries out the work required to notify the customer. In this case it simulates the sending of an email. The method 'notifyCustomerOfPurchase' can later be compensated, should the transaction fail. This is configured through the use of the 'CompensateWith' annotation. This annotation specifies which class to use to compensate the work done within the method. For this compensation to be possible, it will need available to it, key information about the work completed. In this case the item ordered and the address of the customer. This data is stored in a CDI managed bean, 'orderData', which as we will see later, is also injected in the compensation handler.

@CompensationScoped
public class OrderData {
 
    private String item;
    private String emailAddress;
    ...
}

This managed bean represents the state required by the compensation handler to undo the work. The key thing to notice here is that the bean is annotated with @CompensationScoped. This scope is very similar to the @TransactionScoped annotation (new in JTA 1.2). This annotation ensures that the lifecycle of the bean is tied to the current running transaction. In this case the lifecycle of the compensation based transaction, but in the case of @TransactionScoped it is tied to the lifecycle of the JTA transaction. The @CompensationScoped bean will also be serialized to the transaction log, so that it is available in the case that the compensation handler needs to be invoked at recovery time.

public class NotifyCustomerOfCancellation implements CompensationHandler {
 
    @Inject
    OrderData orderData;
 
    @Override
    public void compensate() {
        String emailMessage = "Sorry, your order for " + orderData.getItem() + " has been cancelled";
        //send email here...
    }
}

This class implements the compensation handler. For our example it simply takes the details of the order from the injected OrderData bean and then sends an email to the customer informing them that the order failed.

Summary

In this blog post I explained why it's difficult to coordinate non-transational resources in an ACID transaction and showed how a compensation-based transaction can be used to solve this problem.

Part 3, of this series, will look at cross-domain distributed transactions: Here I'll show that ACID transactions are not always a good choice for scenarios where the transaction is distributed, and potentially crossing multiple business domains. I'll show how a compensation-based transaction could be used to provide a better solution.

Compensating Transactions: When ACID is too much (Part 1: Introduction)

ACID transactions are a useful tool for application developers and can provide very strong guarantees, even in the presence of failures. However, ACID transactions are not always appropriate for every situation. In this series of blog posts. I'll present several such scenarios and show how an alternative non-ACID transaction model can be used.

The isolation property of an ACID transaction is typically achieved through optimistic or pessimistic concurrency control. Both approaches can impose negative impacts on certain classes of applications, if the duration of the transaction exceeds a few seconds (see here for a good explanation). This can frequently be the case for transactions involving slow participants (humans, for example) or those distributed over high latency networks (such as the Internet). Also, some actions cannot simply be rolled back; such as, the sending of an email or the invocation of some third-party service.

A common strategy for applications that cannot use ACID, is to throw out transactions altogether. However, with this approach you are missing out on many of the benefits that transactions can provide. There are many alternative transaction models that relax some of the ACID properties, while still retaining many of the strong guarantees essential for building robust enterprise applications. These models are often referred to as "Extended Transaction models" and should be considered before deciding not to use transactions at all.

In the Narayana project, we have support for three Extended Transaction models; Nested Top Level Transactions, Nested Transactions and a compensation-based model based on Sagas. In this series of blog posts I'll be focusing on the compensation-based approach.

What is a ‘Compensation-based transaction’

Transaction systems typically use a two-phase protocol to acheive atomicity between participants. This is the case for both ACID transactions and our compensation-based transactions model. In the first phase, each individual participant, of an ACID transaction, will make durable any state changes that were made during the scope of the transaction. These state changes can either be rolled back or committed later once the outcome of the transaction has been determinned. However, participants in a compensation-based transaction behave slightly differently. Here any state changes, made in the scope of the transaction, are committed during (or prior) to the first phase. In order to make "rollback" possible, a compensation handler is logged during the first phase. This allows the state changes to be 'undone' if the transaction later fails.

What Affect Does this Have on the Isolation property of the Transaction?

The Isolation property of a transaction dictates what, if any, changes are visible outside of the transaction, prior to its completion. For ACID transactions, the isolation property is usually pretty strong with database vendors offering some degree of relaxation via the isolation level configuration. However, in a compensation-based transaction the isolation level is totally relaxed allowing units of work to be completed and visible to other transactions, as the current compensation-based transaction progresses. The benefit of this model is that database resources are not held for prelonged periods of time. However, the down-side is that this model is only applicable for applications that can tolerate this reduced level of isolation.

The following two diagrams show an example, where a client is coordinating invocations to multiple services that each make updates to a database. The diagrams are simplified in order to focus on the different isolation levels offered by a ACID and compensation-based transaction. The example also assumes a database is used by the service, but it could equally apply to other resources.

The diagram above shows a simplified sequence diagram of the interactions that occur in an ACID transaction. After the client begins the (ACID) transaction it invokes the first service. This service makes a change to a database and at this point database resources are held by the transaction. This example uses pessemistic locking. Had optimistic locking been used, the holding or database resources could have been delayed until the prepare phase, but this could result in more failures to prepare. The Client then invokes the other services, who may in turn hold resources on other transactional resources. Depending on the latency of the network and the nature of the work carried out by the services, this could take some time to complete. All the while, the DB resources are still held by the transaction. If all goes well, the client then requests that the transaction manager commit the transaction. The transaction manager invokes the two-phase commit protocol, by first preparing all the participants and then if all goes well, commits all the participants. It's not until the database participant is told to commit, that these database resources are released.

From the diagram, you can see how, in an ACID transaction, DB resources could be held for a relativley long period of time. Also, assuming the service does not wish to make a heuristic decision, this duration is beyond the control of the service. It must wait to be informed of the outcome of the protocol, which is subject to any delays introduced by the other participants.

The diagram above shows a simplified sequence diagram of the intercations that occur in a compensation-based transaction. The client begins a new (compensation-based) transaction and then invokes the first service. The service then sends an update to the database, which is committed imediatlly, in a relativly short, seperate ACID transaction. At this point (not shown in the diagram) the service informs the transaction manager that it has completed it's work, which causes the transaction manager to record the outcome of this participant's work to durable storage along with the details of the compensation handler and any state required to carry out the compensation. It's possible to delay the commit of the ACID tranaction until after the compensation handler has been logged (see here), this removes a failure window in which a non-atomic outcome could occur.

The client now invokes the other services who, in this example, behave similarly to the first service. Finally, the client can request that the Transaction Manager close (commit) or cancel (rollback) the compensation-based transaction. In the case of cancel, the transaction manager calls the compensating action asociated with each participant that previously completed some work. In this example, the compensating action makes an update to the database in a new, relativley short ACID transaction. The service can also be notified if/when the compensation-based transaction closes. We'll cover situations when this is useful later in this series. The notification of (close/compensate) is retried until it is acknowledged by the service. Although this is good for reliability, it does require that the logic of the handlers be idempotent.

From the diagram, you can see that the duration for which DB resources are held, is greatly reduced. This comes at a cost of relaxed isolation (see the 'changes visible' marker). However, in scenarios where compensation is rare, the relaxed isolation could be of little concern as the visible changes are usually valid.

It is also possible to mitigate this loss of isolation by marking the change as tentative in the first phase and then marking the change as confirmed/cancelled in the second phase. For example, the initial change could mark a seat on a plane as reserved; the seat could later be released or marked as booked, depending on the outcome of the transaction. Here we have traded the holding of database-level resources for the holding of application-level resources (in this case the seat). This approach is covered in more detail later in the series.

What's Coming up in the Next Posts?

The following three posts will each focus on particular set of use-cases where compensation-based transactions could prove to be a better fit than ACID transactions. In each part, I'll provide a code example, using the latest iteration of our new API for compensation-based transactions (first introduced here).

• Part 2: Non-transactional Work. This part will cover situations where you need to coordinate multiple non-transactional resources, such as sending an email or invoking a third party service.
• Part 3: Cross-domain Distributed Transactions: This part covers a scenario where the transaction is distributed, and potentially crosses multiple business domains.
• Part 4: Long-lived Transactions. This part covers transactions that span long periods of time and shows how it's possible to continue the transaction even if some work fails.

In part five, I'll cover the status of our support for compensation-based transactions and present a roadmap for our future work.