Narayana LRA: implementation of saga transactions

The Narayana transaction manager implements saga transactional pattern naming it as LRA which is abbreviation to Long Running Action.
This saga implementation is basis for the specification fitting to schema of Eclipse MicroProfile. See at https://github.com/eclipse/microprofile-sandbox/tree/master/proposals/0009-LRA.

The communication is done over HTTP in way of REST principles.
The Java EE technologies which the LRA builds upon are CDI and JAX-RS.

Saga - what we are talking about

Before we start talking about LRA let's introduce the term saga.
Saga consists from independent set of operations which all together forms one atomic action. The operation can be whatever action you think.

Here we took take the well known example of booking a flight and subsequent services. The operations are booking flight, taxi and hotel and these three forms the atomic action which we want all parts would be processed together.

NOTE: From ACID point of view the Saga transaction relaxes the I (isolation) and because of that gets availability and is representation of BASE principle, see http://queue.acm.org/detail.cfm?id=1394128.

Principle of saga requires existence of a compensations actions invoked in case of failure.

The compensation actions undo work done by the "business" operations. The sequence in case of the flight example is first to book flight. If that success this change is overly visible to all the other parties trying to book the flight too (here it's the relaxation of ACID isolation).
Next is the taxi booking, followed by hotel booking. When there is no hotel available in the area for the particular date the whole saga fails and compensation action for the prior actions are invoked. The responsibility of the compensation action is undoing what was done - in this case canceling flight and taxi booking. How this is exactly done depends on the business logic. It could be updating of some database record, call to some API or sending email to the taxi operator.

In comparison to ACID transaction, where developer makes a SQL insertion to database or sends a message to a broker and the potential undoing such action (rollback) is handled by the transaction manager in the background, here in Saga world the responsibility of undoings completely moved to the developer who has to implement the compensations callback.

Responsibility of the transaction manager is to gather information about what operations are part of particular saga (receives participant enlistment) and ensuring the compensation callback is invoked, even in case of failure (either the participant or the manager itself).

The Narayana LRA implementation also adds way to define a complete callback which is invoked when saga ends successfully. This could be used for confirmation actions e.g. customer could be informed with email that order was processed while passing him details needed for the payment. Or database can be enriched with column informing if order was successfully processed and SQL statements could be created with that taken into account.

In summary using saga transactions is a design pattern which needs to be built to the foundation of the application architecture.

LRA - Long Running Actions

The Narayana LRA is implementation of saga for HTTP transport based on the REST principles.

Narayana LRA is represented by coordinator (transaction manager) exposing HTTP endpoint for an incoming remote communication. The coordinator is the core which ensures the LRA saga is processed in atomically. Services enlist to the coordinator, by calling defined REST endpoints. The coordinator calls then back services to confirm saga success or command to undone with compensate.

The coordinator can be placed as separate service or it can be attached to the application too.

For Narayana implementation applies that in case of coordinator packed with the application, the application itself talks to coordinator with in-memory calls.

Let's explain how the LRA communication works on the example. This is diagram showing our usecase.

We can see the LRA coordinator and 4 services talking to each other in the row synchronously (of course your application can be designed in a different way) and communication will looks in following way

1. A client makes a call to the first service (Microservice #1)
2. The Microservice #1 is responsible for starting LRA (creating the saga). It calls LRA coordinator to endpoint starting LRA. The coordinator announces lra identifier in response.
3. The Microservice #1 enlists itself with the created LRA by calling coordinator along with the particular LRA identifier and handing over addresses of REST endpoints for compensation (optionally completion) callbacks. Those are endpoint the coordinator can call back to Microservice #1.
4. The Microservice #1 takes the LRA identifier and adds it as an HTTP header (long-running-action) to the REST call to the next service - Microservice #2. If the Microservice #2 distinguishes the LRA header it can enlist itself (by announcing REST endpoints for compensation/completion callbacks) to coordinator.
5. On way back the first service is responsible for finishing saga by calling close (when finishing with success) on the LRA coordinator with the saga identifier.
6. Some of the other services could fail saga by calling cancel on the LRA coordinator, in which case the close won't succeeds and reports back an error.

Code examples

If you wan to see this example working check out the Red Hat MSA example
enriched with the LRA capabilities.
Detailed installation steps could be found at the page: https://developer.jboss.org/wiki/MSAQuickstartsWithLRAREST-ATOnMinishift

In the next article we will get into CDI annotations used by LRA and their functionality. Meanwhile you can check out how the WildFly Swarm microservice using LRA described in the example looks like
https://github.com/ochaloup/hola/blob/lra/src/main/java/com/redhat/developers/msa/hola/HolaResource.java#L81

or check other implementations run with Spring, Vert.x and Node.js

• https://github.com/ochaloup/ola/blob/lra/src/main/java/com/redhat/developers/msa/ola/OlaController.java#L94
• https://github.com/ochaloup/aloha/blob/lra/src/main/java/com/redhat/developers/msa/aloha/AlohaVerticle.java#L141
• https://github.com/ochaloup/bonjour/blob/lra/lib/api.js#L72

Microservices and transactions - an update

It's almost a year since I wrote my first thoughts on how transactions fit into the world of microservices and it's time for an update. I've had the pleasure of working in the field of fault tolerance and distributed systems for almost 30 years. In that time I've worked with some great friends and colleagues from within the same companies or across different companies on transactions, both traditional atomic (ACID) transactions and extended transactions. Back when I was doing by PhD on transactions and replication, weak consistency replication was in its infancy but there were already a range of extended transaction protocols.

Over the years we've seen these transaction protocols move from research into standards and industrial usage, with efforts such as the OMG's Additional Structuring Mechanisms for the OTS and WS-Transactions from OASIS. Although not as pervasive as ACID transactions, these additions to a developer's repertoire have seen some uptake. Now I'm not someone who believes transactions of any form should be used in all situations, but neither do I believe that they are so bad to be completely useless. Yet throughout the work we did for both Web Services and REST there were some groups that vehemently fought against transactions, often stating that applications should ensure that any transactional changes to state should be isolated within a service and not span multiple services, i.e., only local transactions should be supported.

As I said earlier, I don't believe that transactions, or even distributed transactions, are necessarily right for every application, or even some applications that use them today. Transactions (let's assume ACID for now) provide a nice and simple model for building applications, particularly if the implementation you use supports nested transactions. It's only natural for a developer who finds this structuring mechanism useful to expand it across objects, services and even machines. In a closely coupled environment when transactions last a few seconds this continues to be a useful approach. However, as we've seen and discussed many times before, they become problematical in loosely coupled environments. Hence the development of certain extended transaction models.

Structuring your applications so that all of the state changes which occur do so within a single object or service is often a lot easier said than done. Especially if you are building from components (services or objects) that have been developed over time by different groups or companies. It's easier to do if you build a Big Ball of Mud, which hopefully is not what you want to accomplish by going down the microservices route! Whether your stateful services interact directly with each other via RPC, say, or through a reliable, yet asynchronous messaging bus with queues and topics, such as JMS, it is fairly inevitable that your applications will have state updates which need to occur as some unit of work (note I didn't say "atomic" there). Some of these units of work will need to be atomic (though not necessarily ACID). Some will be fine with relaxed constraints, such as using forward compensation based approaches. Yes, I'm sure we'll hear people suggesting that atomic transactions aren't useful at all in these environments due to performance problems, but if they spent the time understanding the kinds of optimisations that mature transaction implementations have had in place for decades, then perhaps they'd realise that whereas there may be some overhead it's not as black-and-white as they may believe, or want you to believe. And please realise that XA is just one specific standard for transactions - it has its pros and cons, but any downsides you may have with XA shouldn't be assumed to carry over to the plethora of other transaction models and standards out there!

Let's return to the original topic: microservices and transactions. Where do (or should) the two come together? What I really don't want to repeat with microservices is the anti-transaction arguments we had for SOA. Get over it! Some applications will find them useful, whether atomic (ACID) or extended. Therefore, let's just assume that point for the rest of this discussion. As you develop your microservice(s) and hopefully take the approach of making each "do one thing well" as well as "be as simple as possible yet no simpler", you'll want to string them together; you'll want to have an invocation on one service trigger an invocation on another, or even more than one; you'll want to update the state of a number of services together. (I'll talk about weak consistency in a separate article.) You'll need to determine whether or not these updates have to occur atomically - just recognise the trade-offs this may mean to your application and services. As I've mentioned already, atomic transactions (local or distributed/global) aren't your only option though and one of these additional protocols could be better suited to your services and the way in which they have been constructed. And of course you can mix-and-match: just because some groupings of services may be better suited to a compensation-based model does not preclude you from using atomic transactions elsewhere - or even with the same services for for different operations.

In short what I hope anyone developing microservices will get from this is an understanding that transactions, both local and global, are not anathema to SOA/microservices. They may not be the default mechanism for you to choose when building your services, but they most certainly should be part of a good developer's palette. Having to implement equivalent capabilities in your infrastructure or the services themselves (consistency in the presence of arbitrary failures, opaque recovery for services, modular structuring mechanisms, span different communication patterns etc.) is something you shouldn't have to do because it's a monumental effort in its own right. A transaction manager microservice is something that should be available in many enterprise environments!

RESTful Transactions in the Cloud: Now they're for real

A few months back Mark posted on the subject of REST, transactions and the cloud. Well now you can test the validity of his claims for yourselves. RedHat recently announced their OpenShift initiative for deploying applications into the cloud and we have put together a demonstration to show how easy it is to perform end to end transactional requests whilst remaining RESTful/RESTlike throughout and using little more than a JAX-RS container, an HTTP client library and a transaction coordinator.

As our starting point we use an implementation of the draft specification for a RESTful interface to 2-Phase-Commit transactions which we refer to as REST-AT.

The demonstration starts out by showing local clients (written in various languages such as javascript and Ruby) and local (JAX-RS) web services interacting transactionally using HTTP for all communications. The transaction coordinator that implements REST-AT is written using JAX-RS and also runs locally.

The push to cloud based deployments demands that providers support piecemeal migration of applications into the cloud infrastructure. REST-AT together with OpenShift makes the realisation of this goal, at least for transactional services, a relatively painless exercise. The demonstration code and accompanying video shows how to migrate the transaction coordinator component leaving clients and services outside of the infrastructure. We achieve this by using the OpenShift JBoss AS7 cartridge and deploy REST-AT as a war into the application server that this cartridge provides. In a later post we will show how to migrate Java and Ruby services into the OpenShift infrastructure using the AS7 and Ruby cartridges, respectively.

To begin using OpenShift a good starting point would be to watch the various getting started videos (look for the Zero to Cloud in 30 minutes track or go directly to the demo video). To start using REST-AT check out the quickstarts and the sources for the demo.