connecting the dots

Hmm, looks like it's 2012 already, which means we're overdue for another rant. Today's lecture topic is the perils of trying to replace or do without bits of code that you don't understand.

Programming in the large is all about abstraction, loose coupling and separation of concerns. Java EE promotes the ability to use complex functionality without understanding how it is implemented. As long as you follow the instructions this works surprisingly well. With an app server at your disposal you require only the sketchiest understanding of distributed transactions to write code that consumes a message and updates a database in an ACID fashion. Which on the whole is a good thing, since not all that many people actually want to understand all the plumbing that makes it work. There is a snag though: this apparent simplicity can cause some users to think the Java EE container is not doing all that much work and can easily be replaced or done without. Likewise it makes it difficult to tell the difference between a full fledged container that will support the functionality you need and a cut down framework or lightweight container that may not.

Let's debunk a few myths...

Spring is not a JTA.

It's an abstraction layer on top of a transaction manager, not an implementation of one. For transactions involving only a single resource, it simply delegates the hard bits to that resource manager - so called 'native transactions'. Simple, fast and mostly adequate if you need only a single resource. You don't get transaction lifecycle events, but you're probably using an ORM that provides the equivalent of the beforeCompletion hook for its cache anyhow.

For transactions with more than one resource, you need to wire in a real transaction manager to Spring. You can do this with bitronix, atomikos or JBossTS, usually just by specifying the right TransactionManager and UserTransaction implementation bean classes. But your problems don't end there, because...

Spring is not a JCA.

The roles, responsibilities and relationship between the JTA and JCA components of an app server are critical considerations when you're trying to do without one. The JTA manages transaction lifecycle - begin/commit/rollback. Most importantly, it make appropriate calls on any enlisted XAResources as the transaction progresses. But here is the bit that most users don't pay attention to: A JTA does not magically know what resources you want to participate in the transaction. Telling it that is the JCA's job.

In a full on app server, the JCA manages connections to resource managers such as databases and message queues. If you deploy those drivers/connectors in a manner that identifies them as XA enabled, the JCA ensures that they are correctly associated with the transaction. Application code simply e.g. looks up the JNDI name for a connection pool and calls getConnection(). The JCA intercepts the call, get the XAResource for the connection and passes it to the transaction manager.

In some cases you don't need a full JCA. You can often make do with a transaction manager aware XA connection pool, which is essentially a subset of the JCA functionality. But you can't get away with only an XA aware driver or a non-XA connection pool. Trying to do that leads to some interesting behaviour: your app will deploy and run, but you have a transaction and a connection that know nothing about one another. Committing or rolling back the transaction won't commit or rollback the work in the database. oops.

So, you also need to wire in a JCA or suitable connection pooling implementation. Most 'standalone' JTA implementations ship with a simple connection management solution that is suitable for light use. The one in JBossTS is called the TransactionalDriver. For serious deployments you want IronJacamar or some other JCA that has robust and fast connection management.

So now you have wired up your JTA and JCA in Spring, but you are still not done because...

The standard contract between JTA and JCA does not include recovery management setup. Wiring up resources for crash recovery requires a proprietary solution that differs for each transaction manager. The connection manager that ships with the transaction manager may do this more or less automatically, but third party JCAs or XA aware connection pools probably won't. So, go read the transaction manager documentation and write a few test cases.

phew, that was a lot of work, wasn't it? Spring is good at what it does, but its not an out of box replacement for all the transactional plumbing in a Java EE app server. Nor is tomcat - you'll have much the same steps to perform there. In both cases it is possible, but not as simple as unzipping and starting JBossAS. Do you really want to make your life harder than it has to be?

Transactional Android coming soon!

I spent some time this Christmas porting JBossTS to run on Android. It's pretty much done, with the exception of a few workarounds that I need to fix properly over the next few weeks, when I find the time. But once I check this code into the repository, you'll be able to write your own transactional Android applications. Unfortunately I can't guarantee which version this will be in at the moment, but if it is going to take me too long to do the merges then I may create a branch in svn that interested people can just pull from directly, with the usual caveats. If time allows, I may say something about this at JUDCon India too!

Transactions making a comeback? They were never away!

Over the years that I've been involved with transaction processing theory and practice (way too many years for me to admit!) I've seen transactions used, abused and ignored in a wide variety of applications and situations. I've discussed many of these scars in articles here and elsewhere for decades (ouch! that pains me to even use such a timeframe!) But every new software wave, whether it's Web Services, object-orientation, and even the Web itself, has come to the inevitable conclusion that transactions are needed in some way and in one form or another. Maybe not ACID transactions; maybe it's compensation based, or some other variation on the extended transaction theme.

So it is nice to see that being repeated in the past couple of years with Cloud and NoSQL. If you read this blog enough you'll have heard from myself and others on the fact that although some believe that you need to ditch transactions in order to achieve scalability, others aren't quite convinced that it is always necessary to do so - and I count us amongst the latter group. Complete reliance on transactions is sometimes too much. However, completely ignoring them and pushing consistency and recovery up to the application is sometimes too much as well. This is something that appears to be dawning on a number of transaction-less implementations and hopefully it's a trend that will continue.

Anyone who knows me or has followed things I've written or presented over the years will also know that I think transactions are a great structuring technique for concurrent programming. Of course in distributed systems we tend to take this for granted: you've inherently got multiple clients/users manipulating your data, typically at the same time, so transactions with their ACID properties allow developers to concentrate on the functional aspects of the application or service, whilst letting the transaction system do the hard work on ensuring isolation and consistency in the presence of these concurrent users (and possibly failures).

But distribution isn't the only place where you get concurrency, and particularly these days with multi-core processors. Back in the last century (ouch!) some of us in academia and industry, had access to multi-processor machines which weren't as common as you might expect (they were expensive!) But when you had them it quickly became apparent that transactions could help with developing applications that had no distribution in them but were (massively) parallel. From this early work techniques such as software transactional memory were born. Back then it was more of an edge case and people found it hard to understand why you'd need transactions without distribution or even a database involved. Well obviously the advances in hardware have silenced most of those critics and we're seeing more and more vendors, open source projects, languages etc. looking at transactions are a fundamental component and not just an add-on for some scenarios.

So what does all of this mean? Well first of all I think it's great to see transactions continuing to have an impact in these new waves. Fundamental requirements like fault tolerance, concurrency control, security etc. are fundamental for a reason. Secondly I think it's fair to say that as with previous waves, we'll see transaction theory and implementations adapt and change to better address some of the new concerns and requirements that are bound to arise. I'm excited by all of this, as I am whenever there's something new to apply transactions. I'm also excited by the fact that JBossTS (implementation and team) is well placed to help drive some of this as we've done for ... well ... let's just say for quite a long time and leave it at that!

my last commit

Today I created the tag for the JBossTS 4.16.0.Final release, thereby making my last commit to the JBossTS repository as team lead.

It is the end of an era in more ways than one, as 4.16 is planned to be the last feature release for the 4.x line and the last for the JBossTS brand.

Starting in the new year, the project gets a new name, Narayana, a new major version, 5.0, and a new development team lead, Tom Jenkinson. So, exciting times ahead for the project in 2012 and beyond, with fresh blood and fresh ideas.

As for myself, I start 2012 with a shiny new JBoss role looking at Big Data and noSQL. Who knows, there may even be a new blog for you to subscribe to. But first there is the small matter of a seasonal vacation...

Merry Christmas

Jonathan.

HPTS 2011

I've uploaded most of the presentations and posters sessions for HPTS 2011 to the website now. If you interested in transactions, NoSQL, eventual consistency, data provenance and a whole raft of topical subjects, then you really should check them out!

unnecessary

Hot on the heals of thinking about information quality, I came across this little gem regarding Spring and JPA configuration:

"Using multiple session factories in a logical transaction imposes challenging issues concerning transaction management... We quickly abandoned [JTA transaction management] due to the potential cost and complexity of an XA protocol with two-phase commit. Since most of the modules of an application share the same data source, the imposed cost and complexity are definitively unnecessary."

The authors then go on to describe how they built a custom solution that causes the same database connection to be used by the modules, removing the need for transaction coordination across multiple connections.

"Extending Spring’s transaction management with SessionFactory swapping and the Shared Transaction Resource pattern was a very challenging task."

That last bit should probably have read "challenging and unnecessary task".

A good JCA will automatically track connections enlisted with a JTA transaction and will reuse the already enlisted connection to satisfy a further dataSource.getConnection() request. Further, even if it does enlist multiple connections, the JTA will use isSameRM to detect that they relate to the same resource manager and thus still maintain the one phase commit optimisation. All of these challenging tasks are taken care of for you by the application server.

You probably should not bother to invent a better mousetrap until you've determined that current mousetraps don't catch your mice. The imposed cost and complexity are definitively unnecessary.

musings on peer review

I've been reading up in a new field of study recently and as a result have been thinking about information provenance, reputation and collaboration.

Once upon a time, getting up to speed on a new topic in computing science required finding a good library and wading through a stack of conference proceedings and journals. Now it requires an online search and, critically, the ability to evaluate the quality of the massive quantity of material that comes back.

Formal peer review of publications is gone, unable to scale to online use. Meanwhile online systems for review, comment and reputation management are still in their infancy. The best we have right now is an ad-hoc stew of brands, social graphs and distributed conversations.

Those with enough time to invest can build a mental model of a new field that, after the initial investment in learning the landscape, allows them to maintain an ongoing overview of developments with relatively little effort. Those who's work only tangentially involves a deeply technical topic don't have this luxury. They typically perform searches not to learn about the new field in general, but to get specific solutions to a problem outside of their core field of expertise, after which they move on. Such users vastly outnumber the domain experts for niche topics like transactions.

What implications does this new model of communication and information dissemination have for the behaviour of professed experts in technical fields? Clearly our obligation to ensure that information we provide is accurate remains unchanged, and is in any case in our own best interest. Should we consider there to be an implied professional obligation to publish information only in a context that allows feedback to be attached directly to it e.g. blog with comments enabled? Or even allow collaborative editing e.g. wiki? How about taking time out to correct misinformation where we find it - is that now part of our social contract?

Question for the audience: Where do you get your information on transactions, and how do you assess its quality?