Software Transactional Memory with WildFly-Swarm

A long time ago (not in a Galaxy Far Far Away!) I wrote about the STM implementation we were adding to Narayana. Over the intervening years this implementation was added to Vert.x and even the RaspberryPi made an appearance! Now whilst the implementation has been available in WildFly for many years, we tend not to mention it because, well, it's not Java EE compliant. However, with the advent of WildFly-Swarm things may be about to change.

Sure, when you're looking at using Swarm it's likely that at least initially you'll be coming at a problem from the perspective of Java EE, but the more you look to decompose your application into constituent (micro) services the more chances there are that you'll also start to look at functionality and frameworks that aren't necessarily just about Java EE. As we've mentioned before, STM is compatible with JTA and JTS transactions as well, as long as you understand what it means to mix-and-match them. Therefore, we've added an example of STM usage within WildFly-Swarm, which hopefully will become part of the mainline Swarm examples eventually. Take a look and give us any feedback, either in the Swarm group/IRC/Twitter or the usual Narayana routes.

JTS Docker Container

So last week our new Docker image became available on https://hub.docker.com. It is a standalone JTS service running the premiere open source transaction manager written by the Narayana team.

In addition to that, it comes with a helper image - JacORB name server, which provides an easy way of sharing the transaction managers IOR.

The transaction manager image can be found here:

https://hub.docker.com/r/jboss/jts-transaction-service.

And the JacORB name server image can be found here:

https://hub.docker.com/r/jboss/jacorb-name-server. 

Usage

This image comes with a quickstart example available on Narayana Github page: https://github.com/jbosstm/quickstart/tree/master/jts-docker. But here I’ll provide a brief extract from the quickstart to explain how the JTS Docker image can be used.

We use Docker linking technology to make interaction between JTS and JacORB containers more smooth. Thus JTS container expects certain environments to be available. Not to worry though, they are automatically created by Docker.

Here is how to start the name server container:

docker run -p 3528:3528 -it --name jacorb-name-server jboss/jacorb-name-server

And this is how to start JTS container on Linux, linking it with the name server:

docker run -p 4711:4711 -it --link jacorb-name-server:jacorb-name-server --name jts-transaction-service jboss/jts-transaction-service

Or if you're running on Boot2Docker start JTS container as follows:

docker run -p 4711:4711 -it -e "PROXY_IP={Boot2Docker IP}" --link jacorb-name-server:jacorb-name-server --name jts-transaction-service jboss/jts-transaction-service

And that is it. Now you can connect to the name server from your application and retrieve the IOR of the transaction manager. Pretty easy.

Of course, since Docker container storage is removed once the container is removed, this is not the best way to use any transaction manager as you will want to make sure your transactions are completed even in case of the system failure. To avoid such scenarios and make the transaction log reliable you have two options: mount a host directory or use JDBC object store.

But you can read more about that in our quickstart readme:

• https://github.com/jbosstm/quickstart/tree/master/jts-docker#mounting-local-repository-as-an-object-store.
• https://github.com/jbosstm/quickstart/tree/master/jts-docker#using-postgres-as-an-object-store.

Additionally, in the quickstart you’ll find an example of how to coordinate JacORB name server and JTS transaction service containers with Kubernetes.

Updating multiple one-phase resources with Narayana

I was recently forwarded a link to an article regarding the use of Springs chained-transaction manager facility wherein the author had utilised this facility to coordinate updates to multiple one-phase resources. This gave me the opportunity to show-case a Narayana facility which has existed for many years that allows you to build something with similar a similar purpose and possibly richer properties.
What we will create is an application that uses multiple one-phase resources (for example, some hypothetical none-XA database and message queue). We will use Narayanas AbstractRecord extension mechanism to order the commit messages to the resource managers in any way that would be appropriate for the application. We will then take a look at some options for failure recovery options.

Notes:

• Applications of this style (i.e. multiple 1PC) are only suited for certain classes of applications. Where possible it is almost always preferable to use 2PC resources to provide spec-compliant transactional outcomes.
• The code I am going to use to demonstrate some of this is derived from a unit test in our repository but I will extract some of the code to below. I won't actually use resource managers in this example to try to illustrate the pattern as clearly as possible.
• If the test ever moves around, you can probably track it via its SHA 8e9f712d5b89762c7b841cf370eb5bdb341fff4d.

Transactional business logic

The general layout of the application follows the same pattern of any other transactional application:

        // Get references to resource managers
        ResourceManager1 rm1 = ...;
        ResourceManager2 rm2 = ...;

        // Create a transaction
        AtomicAction A = new AtomicAction();
        A.begin();

        // Enlist resource manager in transaction
        A.add(new OrderedAbstractRecord(rm1));
        A.add(new OrderedAbstractRecord(rm2));

        // Do business logic
        // rm1.sendMessage()
        // rm2.doSQL()

        // Commit the transaction, the order will be defined in OrderedAbstractRecord rather than
        // the business logic or AtomicAction::add() order
        A.commit();

Guaranteeing the order of commit messages

The ordering of the list for transaction completion events is dictated by the RecordList class. At the most fundamental level, for AbstractRecords of the same type it is determined by the Uid returned in the AbstractRecords order method. As Uids are sequentially numbered at some level, this basically means that if you return a Uid lower to a peer, your record instance will be committed before that one.

So for example, the order of Uid you allocate to the following class will determine the order AbstractRecord::topLevelCommit() is called:

public class OrderedOnePhaseAbstractRecord extends AbstractRecord {
    public OrderedOnePhaseAbstractRecord(Uid uid)
    {
        super(uid);
        order = uid;
    }
    public Uid order()
    {
        return order;
    }
    //...
}

Failure tolerance properties

A final observation to make is that by using the Narayana AbstractRecord facility, it allows you to know that in the presence of a failure, during crash recovery you will receive callback where it may even be possible to re-do some of the work in the later resources.

For example, in the AbstractRecords save_state you could save some of the content of the JMS message which can then be used in a possible recovery scenario to resend a similar message.

Narayana 5.2.1.Final released!

The whole team is proud to announce the latest release of our project, version numbered 5.2.1.Final :)

This was primarily a bug fix release within which however we also polished off a couple of house keeping tasks including moving our dependency on the HornetQ journal to the Artemis journal, plus a new facility to auto-launch an embedded recovery manager when starting the transaction service standalone.  The complete release notes are available from here.

Wildfly has had our upgrade merged in so you should see a nightly build of WildFly that includes the upgrade available soon over here.

XA and microservices

I had a very interesting series of discussions this week with a few friends/colleagues about whether or not transactions are applicable in a microservices architecture. I've already covered these points in an earlier entry so I won't repeat them here. However, several things did come up that are worthy of trying to address, including the assumption that they're too hard to use and too hard to test your application for correctness when using them. Now as I've mentioned in several presentations there are valid concerns like these around using or not using transactions in general, i.e., irrespective of microservices.

These points that came up recently are not just to be dismissed out of hand. Having worked in transactions for almost 30 years I've spent a long time hearing about these and other concerns and trying to address them within our industry with many others from a range of companies. For instance, if you look at what we've achieved within Java EE with CDI annotations for transactions, or within Narayana for similar approaches towards compensations, I think we've come a long way to simplifying how developers can use transactions. The STM work that we've done is also another example of putting transactions into the hands of developers and not requiring them to have to worry even about demarcation or state management.

Testing applications and their use of transactions is obviously important, in just the same way as when adding any component or capability, such as Hibernate, Zookeeper or Map/Reduce. However, once again we've made a lot of progress over the years. Within Narayana we have 1000s of unit and QA tests which not only help to prove the correctness of the underlying implementation but also feed into best practices etc. for developers. Then there's transaction extensions to Arquillian which also help to make testing much easier. Now in terms of proving the correctness of transaction systems we're also pretty safe there (no pun intended). Since transactions have been used in mission critical environments for over five decades there's a large body of associated work to prove the various aspects are correct for implementations and applications. Overall I'm fairly sure that these concerns have been addressed sufficiently well to not be a meaningful reason to avoid transactions, in general or with microservices. As an industry could we do better? Of course. We should always be striving to improve the overall developer experience and how transactions fit in.

However, one other thing which did come up during the conversations I referred to initially is that there's often a concern that transactions equals XA, or put another way, that the problems perceived to be present with transactions are the result of design issues inherent within the original XA protocol and associated implementations. As I keep saying, 2PC and transactions existed before XA and many transaction system implementations aren't tied to only supporting the XA protocol. All of the transaction standards have their quirks, just as most standards do no matter their domain. But XA has a few more than its fair share, often due to the fact it's been around for so long that the ways in which we tend to develop applications have changed radically since it was defined. Yet we're tied to it due to the fact that relational databases still represent the majority way in which we store data. Of course we've developed new transaction standards which don't rely upon XA, such as OTS or WS-AT, but most developers still see and use XA and hence why there may be concerns about transactions which are really more to do with XA.

Back to the topic: I'm not suggesting that XA doesn't have a role to play within microservices and between microservices. At least in the short term it most definitely does for some (small) set of applications. But really when I'm suggesting transactions have a role I'm looking well beyond XA. I'm still considering ACID as well as compensation transactions though because as I've said, some implementations aren't tied to XA and yet provide the optimisations you expect from XA (one-phase commit, read-only optimisation) as well as going beyond (interposition, nested transactions). So whilst I definitely understand concerns about transactions, I still believe they either have been addressed and we need to do a better job of developer education, or we can address them; the benefits they bring are worth it.

Application servers are dead. Apparently.

OK so you may be wondering what the title has to do with transactions. Well it's really because of this article I came across recently. It's slightly old and I'm not sure how I missed it the first time round (maybe I had better things to do at the time!) Anyway, I'm not going to argue about whether or not application servers are dead: you can find enough from me on that topic by searching on JBoss.org or elsewhere.

What I did want to focus on was the transactions references in the article. Basically the author looks at some of the capabilities that an application server provides and tries to show how the same things can be done outside of an application server or with other frameworks/stacks. Well you can probably guess how the comments on transactions went down with me! Let's start by deconstructing the paragraph - yes, it only deserved a paragraph on something I think is so fundamental to enterprise applications! But hey, maybe I'm just strange (don't go there!)

"2PC is used to coordinate multiple transaction resource that should take part in a single transaction. Whether this feature makes sense could be the subject of an article in its own right. " Now of course you could interpret the last sentence in a number of ways, but I'll give the author the benefit of the doubt. Yes, transactions aren't always the right thing to use and in some cases they've definitely been misused.

Then we have: "If 2PC is used to coordinate two databases the architecture might be wrong. Why is data distributed across two databases if it has such a close relation that it should be changed as part of a single transaction?" Erm, exsqueeze me?! Have you heard of separation of concerns? Or maybe different organisations? Or perhaps difference companies? Or what about different levels of security/encryption, so a company stores and updates data in different repositories/data stores with different levels? And of course there's that other example ... replication!

The next bit made be laugh and then cry: "If JMS and access to a database are done in the same transaction synchronization might be an alternative. It offers almost the same guarantees but is a lot less complex and resource consuming." I can almost imagine the conversation: "Yes sir, we lost your money transfer because the message got lost but we were assured the chances of it happening were so small we really didn't need transactions and could use something that was almost exactly the same thing!" Look, a Citroen Belingo is almost the same thing as a Ferrari but I know which one I'd prefer!

"And remember: Also 2 PC can fail – like any IT system." Yes, it can fail. But guess what? A good transaction system implementation will have something called crash recovery (or failure recovery) which will attempt to resolve things for you automatically so you, the developer or sys admin, don't have to. Catastrophic failures may be unrecoverable, but then that would be the case without transactions anyway.

Then it wraps up with: "In modern Systems that use technologies such as REST or NoSQL 2PC cannot be used anyway. These technologies do not support 2PC. So more often than not 2PC is not that useful." OK so what about things like REST Transactions (the clue's in the name!) and the work we've been doing on compensation transactions and NoSQL?

Look, in general the recommendation(s) to stay away from transactions is bogus and ill-informed. If you don't want to use an application server that doesn't mean you should throw away everything that you get from it such as transactions. For instance, you don't need to use Narayana in the scope of an application server - in fact it began life years before there was ever such a thing!

Microservices and transactions - an update

It's almost a year since I wrote my first thoughts on how transactions fit into the world of microservices and it's time for an update. I've had the pleasure of working in the field of fault tolerance and distributed systems for almost 30 years. In that time I've worked with some great friends and colleagues from within the same companies or across different companies on transactions, both traditional atomic (ACID) transactions and extended transactions. Back when I was doing by PhD on transactions and replication, weak consistency replication was in its infancy but there were already a range of extended transaction protocols.

Over the years we've seen these transaction protocols move from research into standards and industrial usage, with efforts such as the OMG's Additional Structuring Mechanisms for the OTS and WS-Transactions from OASIS. Although not as pervasive as ACID transactions, these additions to a developer's repertoire have seen some uptake. Now I'm not someone who believes transactions of any form should be used in all situations, but neither do I believe that they are so bad to be completely useless. Yet throughout the work we did for both Web Services and REST there were some groups that vehemently fought against transactions, often stating that applications should ensure that any transactional changes to state should be isolated within a service and not span multiple services, i.e., only local transactions should be supported.

As I said earlier, I don't believe that transactions, or even distributed transactions, are necessarily right for every application, or even some applications that use them today. Transactions (let's assume ACID for now) provide a nice and simple model for building applications, particularly if the implementation you use supports nested transactions. It's only natural for a developer who finds this structuring mechanism useful to expand it across objects, services and even machines. In a closely coupled environment when transactions last a few seconds this continues to be a useful approach. However, as we've seen and discussed many times before, they become problematical in loosely coupled environments. Hence the development of certain extended transaction models.

Structuring your applications so that all of the state changes which occur do so within a single object or service is often a lot easier said than done. Especially if you are building from components (services or objects) that have been developed over time by different groups or companies. It's easier to do if you build a Big Ball of Mud, which hopefully is not what you want to accomplish by going down the microservices route! Whether your stateful services interact directly with each other via RPC, say, or through a reliable, yet asynchronous messaging bus with queues and topics, such as JMS, it is fairly inevitable that your applications will have state updates which need to occur as some unit of work (note I didn't say "atomic" there). Some of these units of work will need to be atomic (though not necessarily ACID). Some will be fine with relaxed constraints, such as using forward compensation based approaches. Yes, I'm sure we'll hear people suggesting that atomic transactions aren't useful at all in these environments due to performance problems, but if they spent the time understanding the kinds of optimisations that mature transaction implementations have had in place for decades, then perhaps they'd realise that whereas there may be some overhead it's not as black-and-white as they may believe, or want you to believe. And please realise that XA is just one specific standard for transactions - it has its pros and cons, but any downsides you may have with XA shouldn't be assumed to carry over to the plethora of other transaction models and standards out there!

Let's return to the original topic: microservices and transactions. Where do (or should) the two come together? What I really don't want to repeat with microservices is the anti-transaction arguments we had for SOA. Get over it! Some applications will find them useful, whether atomic (ACID) or extended. Therefore, let's just assume that point for the rest of this discussion. As you develop your microservice(s) and hopefully take the approach of making each "do one thing well" as well as "be as simple as possible yet no simpler", you'll want to string them together; you'll want to have an invocation on one service trigger an invocation on another, or even more than one; you'll want to update the state of a number of services together. (I'll talk about weak consistency in a separate article.) You'll need to determine whether or not these updates have to occur atomically - just recognise the trade-offs this may mean to your application and services. As I've mentioned already, atomic transactions (local or distributed/global) aren't your only option though and one of these additional protocols could be better suited to your services and the way in which they have been constructed. And of course you can mix-and-match: just because some groupings of services may be better suited to a compensation-based model does not preclude you from using atomic transactions elsewhere - or even with the same services for for different operations.

In short what I hope anyone developing microservices will get from this is an understanding that transactions, both local and global, are not anathema to SOA/microservices. They may not be the default mechanism for you to choose when building your services, but they most certainly should be part of a good developer's palette. Having to implement equivalent capabilities in your infrastructure or the services themselves (consistency in the presence of arbitrary failures, opaque recovery for services, modular structuring mechanisms, span different communication patterns etc.) is something you shouldn't have to do because it's a monumental effort in its own right. A transaction manager microservice is something that should be available in many enterprise environments!

How we evaluate performance improvements for Narayana

I would like to highlight a piece of work that our team (and special thanks to Mike) have been working on for a while now which is to provide a framework and methodology for us to use when developing performance improvements for Narayana.

The article itself is on our developers wiki over here:
https://developer.jboss.org/wiki/PerformanceGatesForAcceptingPerformanceFixesInNarayana

After you have read the article, he has also started a discussion on our forum where we can answer questions about the approach we have taken and whether our community have any comments/suggestions to what we can do to further refine this:
https://developer.jboss.org/message/922334?et=watches.email.thread#922334

It's worth pointing out that this is all fully integrated into our existing checks for pull requests on the Narayana repo. So not only will we execute about 12 hours of unit and integration tests on your modifications - we will now also run a set of tests to check the performance impact of functional changes or evaluate the effectiveness of improvements targeted at this area!

The article links to various tests that are executed so I won't expand here. What I can say is that although the suite is not exhaustive, if you propose a change that needs a specific style of test to verify its impact we will be really pleased to accept those changes too.

All of this performance testing is open source. You can see the tests we execute, the configurations we use and the results we have obtained on our hardware. If you want to run the tests yourselves it should be a case of "git clone" and a few simple steps as documented in our performance repo.

We have created this framework for anyone that wants to work on our project and we hope you find it useful! As a reminder - if you do wish to contribute to Narayana we recommend you take a look at this article to get started: https://developer.jboss.org/wiki/GetInvolvedWithNarayana

Thanks!