Application servers are dead. Apparently.

OK so you may be wondering what the title has to do with transactions. Well it's really because of this article I came across recently. It's slightly old and I'm not sure how I missed it the first time round (maybe I had better things to do at the time!) Anyway, I'm not going to argue about whether or not application servers are dead: you can find enough from me on that topic by searching on JBoss.org or elsewhere.

What I did want to focus on was the transactions references in the article. Basically the author looks at some of the capabilities that an application server provides and tries to show how the same things can be done outside of an application server or with other frameworks/stacks. Well you can probably guess how the comments on transactions went down with me! Let's start by deconstructing the paragraph - yes, it only deserved a paragraph on something I think is so fundamental to enterprise applications! But hey, maybe I'm just strange (don't go there!)

"2PC is used to coordinate multiple transaction resource that should take part in a single transaction. Whether this feature makes sense could be the subject of an article in its own right. " Now of course you could interpret the last sentence in a number of ways, but I'll give the author the benefit of the doubt. Yes, transactions aren't always the right thing to use and in some cases they've definitely been misused.

Then we have: "If 2PC is used to coordinate two databases the architecture might be wrong. Why is data distributed across two databases if it has such a close relation that it should be changed as part of a single transaction?" Erm, exsqueeze me?! Have you heard of separation of concerns? Or maybe different organisations? Or perhaps difference companies? Or what about different levels of security/encryption, so a company stores and updates data in different repositories/data stores with different levels? And of course there's that other example ... replication!

The next bit made be laugh and then cry: "If JMS and access to a database are done in the same transaction synchronization might be an alternative. It offers almost the same guarantees but is a lot less complex and resource consuming." I can almost imagine the conversation: "Yes sir, we lost your money transfer because the message got lost but we were assured the chances of it happening were so small we really didn't need transactions and could use something that was almost exactly the same thing!" Look, a Citroen Belingo is almost the same thing as a Ferrari but I know which one I'd prefer!

"And remember: Also 2 PC can fail – like any IT system." Yes, it can fail. But guess what? A good transaction system implementation will have something called crash recovery (or failure recovery) which will attempt to resolve things for you automatically so you, the developer or sys admin, don't have to. Catastrophic failures may be unrecoverable, but then that would be the case without transactions anyway.

Then it wraps up with: "In modern Systems that use technologies such as REST or NoSQL 2PC cannot be used anyway. These technologies do not support 2PC. So more often than not 2PC is not that useful." OK so what about things like REST Transactions (the clue's in the name!) and the work we've been doing on compensation transactions and NoSQL?

Look, in general the recommendation(s) to stay away from transactions is bogus and ill-informed. If you don't want to use an application server that doesn't mean you should throw away everything that you get from it such as transactions. For instance, you don't need to use Narayana in the scope of an application server - in fact it began life years before there was ever such a thing!

Bringing Transactional Guarantees to MongoDB: Part 1

In this blog post I'll present some recent work we've been doing to bring stronger transactional guarantees to MongoDB. In part 2 I'll present a code example that shows this in action in WildFly 8.

What requirements are we fulfilling? 

1) Updating multiple MongoDB documents in a single transaction
2) Support for sharded environments, without harming scalability
3) Support for global transactions spanning other datastores and traditional relational databases.
4) A middleware solution that's simple for developers to use.

This post covers the background and explains why a compensating transaction (vs an ACID transaction) could be the best fit to meet the above requirements. Part two in this series is more implementation focused. It presents a code example, showing you how you can use the technology, whilst omitting a lot of the theory (that is covered in this post).

Background

NoSQL datastores were originally built as bespoke, in-house solutions, to meet scalability requirements that it was felt relational databases couldn't meet. The general thinking was that ACID transactions would harm scalability and that it was better to workaround that requirement. However, as NoSQL adoption spread beyond its in-house roots, it became clear that many applications do indeed need a level of reliability that transactions can bring.

Typically a NoSQL datastore offers atomic updates to single items, such as a document or key-value (more generally, an aggregate). Therefore, structuring data into aggregates, can mean that the application never needs to update more than one document at a time, within the same transaction. Mostly, this could be true. However, there are cases in which it's not possible to structure the data in this way. Take the classic example of moving funds from one user's account to another. It doesn't make sense to store all users in the same aggregate as it will create a lot of contention and a very large aggregate! Therefore the only option is to deal with each user's data in separate atomic operations. Without a transaction spanning these operations, the application runs the risk of becoming inconsistent in the event of failure. Another example is when the application needs to make updates to a NoSQL datastore in the same transaction as an RDBMS or JMS interaction. Typically NoSQL datastores don't support this.

MongoDB and other NoSQL datastores scale through a combination of sharding and replica-sets. I won't go into the specifics here on how this works. However, the key point is that the data becomes distributed over several nodes. Updating multiple data items atomically requires a distributed transaction. As well as being complex to implement, under certain workloads, a distributed ACID transaction can limit scalability. I suspect it is for these reasons that very few NoSQL datastores support ACID transactions in a sharded environment.

The blocking nature of an ACID transaction is the key property that limits scalability. For the duration of the transaction, external readers and writers are blocked until the transaction completes. For contended data, this can result in lots of waiting. The longer the transaction takes to run, the worse the problem. As well as delay introduced by applications, distributing data over a cluster or multiple databases can also result in longer running transactions. However, for data with low-contention, it's possible that an ACID transaction doesn't harm your scalability, in which case you should consider using them as they are a lot simpler to deal with.

Compensating transactions offer an alternative to ACID transactions. They remove the blocking property by relaxing Isolation and Consistency. Despite offering fewer guarantees than ACID transactions, they offer significantly more guarantees than forgoing transactions altogether. Furthermore, in many applications these guarantees are enough, and any more are superfluous. ACID vs Compensating transactions are discussed in more detail in my blog series "Compensating Transactions: when ACID is too much". Here I also show a pattern for working around the relaxed properties.

Using Compensating-transactions with MongoDB

Through Narayana and WildFly's compensating transactions feature, we can fullfil the requirements stated at the start of this blog as follows:

1) Multiple document updates. 
A compensation handler is logged with each document update. In the case of failure, or if the application elects to cancel the transaction, any partially completed work is compensated, resulting in an atomic outcome. Narayana can build on the atomic update mechanism provided by MongoDB, by logging a reference to the compensating handler in the updated document, in the same atomic update as the business-logic's update. This ensures that either i) both the business-logic update, and compensating handler are persisted; or ii) neither is persisted. The handler can be removed at the end of the protocol. It is this construct that the rest of the protocol is built on, allowing recovery to be achieved regardless of what stage the protocol is in during failure.


2) Sharded environments. 
This approach builds on the atomic-update primitive offered by MongoDB. As this feature works in a sharded environment, so does the compensating-transaction that is built upon it. Furthermore, scalability is maintained due to i) the units of work, composing the transaction operating relatively-quickly; and ii) external readers and writers not being blocked during the progress of the transaction. This holds true, regardless of the duration of the transaction.

3) Support for global transactions. 
The transaction is coordinated by an external transaction manager which means multiple datastores or databases can be enlisted. As this is a general approach, it should be possible to mix the databases and datastore types. For example, an RDBMS and/or a JMS resource can also be enlisted in the global compensating-transaction as well as multiple NoSQL datastores. Furthermore, not all participants need to enlist as compensating resources. Those, which are more traditionally used with ACID transactions, can enlist as an ACID resource, using traditional XA. Here the ACID (XA) resources would experience the full ACID properties, with the compensating resources experiencing the relaxed-ACID properties of a compensating transaction.

4) A middleware solution that's simple for developers to use. 
Narayana offers an annotation-based API very similar to JTA 1.2, for using compensating transactions in your application. Furthermore, it comes pre-installed in WildFly 8, so you don't need to worry about complex setup. This API is discussed in more detail in the this blog series. Part 2 in this series will show how this API can be used to update two MongoDB documents within a compensating transaction.

Can this be done already with MongoDB?

The MongoDB documentation proposes a pattern for updating multiple documents in a relaxed-ACID transaction. This approach is similar to the Narayana approach in that they are both based on Sagas and result in similar interactions with the datastore. However, where the Narayana approach differs is that it provides a middleware-solution and so doesn't need to be developed within the application. Also, this approach is driven by a transaction manager, making it simpler for the transaction to span multiple resources.

Research Worth Knowing on CAP & ACID

The last couple of blog posts on NoSQL/SOA/large-scale and transactions got me thinking that maybe I hadn't mentioned another interesting research effort that also attempts to show how ACID transactions are possible in environments where some believe they aren't applicable. The work is being done under the banner of HA Transactions (HAT), and there's a more recent paper on the topic from VLDB 2014; they also talk about the use of different transaction models such as Sagas, which were part of the input to WS-TX and our REST transactions work. And of course there's always HPTS over the years!

Transactions and Microservices

I've written elsewhere that I think the term Microservices is really just referring to good SOA principles (why we need another term I really don't quite understand). But for whatever reason, articles and blog entries on Microservices seem to be in vogue at the moment. A recent entry on InfoQ goes into some depth on various aspects of what they are and how to use them. Unfortunately the author talks about transactions in the context of Microservices (aka SOA) and has this to say:

"One solution, of course, is to use distributed transactions. For example, when updating a customer’s credit limit, the CustomerService could use a distributed transaction to update both its credit limit and the corresponding credit limit maintained by the OrderService. Using distributed transactions would ensure that the data is always consistent. The downside of using them is that it reduces system availability since all participants must be available in order for the transaction to commit. Moreover, distributed transactions really have fallen out of favor and are generally not supported by modern software stacks, e.g. REST, NoSQL databases, etc."

Huh? This paragraph is wrong on so many levels that I really don't know where to start! For a start "generally not supported by modern software stacks"? Seriously?! Others have spoken about REST and transactions for years, but we've done our own work for over a decade! You also don't have to look too far on this blog for references to NoSQL and transactions (extended transactions or ACID). And of course there's Google's Spanner! ACID transaction support is a key part of this!

Over the years during the initial SOA, WS-* and REST debates kicking transactions out of the picture was a convenient thing for many people to do. Fortunately sanity and better understanding of where they can and should be used has seen them and their variants returning to these environments. I had hoped that those days were over, but it seems that with Microservices we're turning back the clock yet again. Oh well, time to dust off those old papers, blog posts etc. as it seems there's life left in them thanks to Microservices!

HPTS 2013

I was at HPTS 2013 a week or so ago. It's probably the one conference that I always try to get to no matter what. I've included my presentation here but you should check out the HPTS site to see all of the others. The site isn't complete yet, as some presenters haven't sent in their decks but we'll be updating it as they come in.

When you need ACID and can't get it ...

What happens when you need traditional ACID transactions across your resource managers and they won't behave? By that I mean they're not two-phase aware so can't be driven by a coordinator to achieve all of the necessary properties. Of course if you've just one such resource manager (let's call it one-phase aware for now) then you can probably make use of the LRCO optimisation.

However, what if you've got more than one such resource manager? Well if you've checked out JBossTS in the past then you'll know that we allow you to enlist multiple one-phase resource managers, but there's a very big caveat. This really isn't an option that anyone should choose. Fortunately I blogged about a better option over 6 years ago (!): compensation transactions. There were a few follow up entries, but the original one is the place to start.

Now what got me revisiting this was the article I wrote earlier on banks, ACID and BASE. It wasn't so much the notion that banks don't use ACID as the fact that we still see a lot of popular (NoSQL) databases that don't support transactions. Of course some use transactions internally (local transactions), but what I'm talking about is what is often referred to as global transactions: those transactions that span multiple resource managers. When you want to send a message, update a traditional database and update a NoSQL instance all within the scope of the same transaction, in most cases you're out of luck. And this is something that many enterprise customers want to do or will want to do soon.

That is unless you can use LRCO, which may not be possible if the resource manager(s) don't support the necessary recovery semantics.

Therefore, you're left in a situation that has very few options. One would be to ignore the problem and assume that because failures are rare (they are rare, right?) it's unlikely to ever be a problem. Personally I wouldn't recommend this.

Another option would be to resolve any problems manually. Again, since failures are rare (we're sure, right?) the chances of having to do this are slim and if you do have to resolve then it's a good enough trade-off. Of course you've got to hope that you've got enough information to detect and handle the recovery. Again, this isn't something I'd recommend unless you're a company that can afford to employ people who do nothing each day other than resolve these problems. (Yes, they do exist.)

My recommended option is to use compensation transactions, as I outlined many years ago. They will automate the recovery (and detection) as well as allow you to seamlessly integrate with a range of resource managers which are "well behaved". I'm hoping that we'll get a chance to try these out soon with some enterprise applications that use NoSQL implementations that don't support global transactions. Once we've done this then it'll be a good reason for one of the team to come back and write something here.

Transactions and NoSQL

Just a quick cross-posting in case people are interested.