Sunday, June 6, 2010

Transactions and volatile state

So far we've looked at some of the basic of writing transactional applications using just JBossTS and TXOJ, as well as the importance of nested transactions. In this entry we'll look at another way of relaxing some of the traditional ACID properties, namely that of durability (persistence).

Hopefully you'll realize that normally when using transactions the Durability aspect kicks in with the transaction log, which maintains a record of the participants and the state of the transaction, as well as transactional participants (e.g., business objects), which have to record any state changes that are made in the context of the transaction. Both of these are needed to ensure atomicity in the presence of failures. But although disk speeds have improved significantly over the years, compared to improvements in processor speeds the disk is still the bottleneck. But it needn't be if all you want is ACI for your objects and applications. Such objects are often referred to as volatile or recoverable in that they have state that cannot survive machine crashes but which is still subject to atomic updates, i.e., previous states can be recovered if the transaction rolls back as long as there are no crashes.

But of course it's no good to simply stop your business objects from writing state, assuming you are even aware that they are doing so within a transactional application; you must have similar recognition by the transaction manager so that it doesn't write the log on behalf of volatile objects. Obviously if there are only volatile objects in the transaction then there's no need for a lot at all and performance can improve significantly. But of course you need to understand the trade-offs that using volatile objects give, which can essentially be summarized as performance vs crash recovery.

Fortunately JBossTS and TXOJ support volatile objects, including recognition by the coordinator. You don't need to do anything special when creating or starting your transactions: the coordinator will only create a log during commit if it detects more than one non-volatile object within its scope. And for your TXOJ objects you have to do very little, as illustrated by the modified AtomicObject code below from one of the previous articles:


public class AtomicObject extends LockManager
{
public AtomicObject ()
{
state = 0;
}

public void incr (int value) throws TestException
{
AtomicAction A = new AtomicAction();

A.begin();

if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
{
state += value;

if (A.commit() != ActionStatus.COMMITTED)
throw new TestException("Action commit error.");
else
return;
}

A.abort();

throw new TestException("Write lock error.");
}

public void set (int value) throws TestException
{
AtomicAction A = new AtomicAction();

A.begin();

if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
{
state = value;

if (A.commit() != ActionStatus.COMMITTED)
throw new TestException("Action commit error.");
else
return;
}

A.abort();

throw new TestException("Write lock error.");
}

public int get () throws TestException
{
AtomicAction A = new AtomicAction();
int value = -1;

A.begin();

if (setlock(new Lock(LockMode.READ), retry) == LockResult.GRANTED)
{
value = state;

if (A.commit() == ActionStatus.COMMITTED)
return value;
else
throw new TestException("Action commit error.");
}

A.abort();

throw new TestException("Read lock error.");
}

public boolean save_state (OutputObjectState os, int ot)
{
boolean result = super.save_state(os, ot);

if (!result)
return false;

try
{
os.packInt(state);
}
catch (IOException e)
{
result = false;
}

return result;
}

public boolean restore_state (InputObjectState os, int ot)
{
boolean result = super.restore_state(os, ot);

if (!result)
return false;

try
{
state = os.unpackInt();
}
catch (IOException e)
{
result = false;
}

return result;
}

public String type ()
{
return "/StateManager/LockManager/AtomicObject";
}

private int state;
}


The only real differences here are that we don't need the constructor call to LockManager with the ANDPERSISTENT flag (the default for all TXOJ objects is that they are recoverable) and it doesn't make sense to have a Uid constructor because there will never be any persistent state for this object, so it can never be recreated later. If you know that the object is only ever going to be recoverable then you may also be able to play tricks with state manipulation during save_state and restore_state calls, which is why there's a second parameter to them, which is either ANDPERSISTENT or RECOVERABLE.

But that's it. With these very minor changes to the code you can create atomic and recoverable objects that have a lot of the benefit of being used within a transaction, including transactional locking, but lacking the overhead of logging. And of course it's possible to mix recoverable and persistent objects in the same transaction seamlessly. But again, there's a price to be paid for turning off persistence. However, for some types of object and application this is a price worth paying, and one of those areas is STM. But it's not alone and you may find the idea of volatile transactional objects appealing for your own needs. In which case give it a go.

Sunday, May 23, 2010

Nested transactions

Here's another periodic entry on our road to discussing the STM work we're doing around JBossTS. We're going to build on the Transactional Objects for Java toolkit that I described before. But before doing so it's worth stressing that everything I'm talking about here is based on the ArjunaCore module, so it doesn't need an ORB or SOAP stack. In fact it doesn't actually need anything from an application server, whether or not that is J(2)EE based. So yes, you could take that module and embed it within your application, framework or whatever and gain full ACID transaction capabilities.

Hopefully if you're reading these entries then you already understand the concept of traditional transactions, which we'll refer to as top-level transactions here. If not then there are enough articles, presentations and books to get you there, and the JBossTS documentation will help too. But suffice it to say that with a top-level transaction, you get the all-or-nothing (Atomic) affect: no matter how many resources are updated within the scope of that transaction, if one of them commits the work then all of them will and likewise, if one of them rolls back the work then all of the others will also roll back their work.

Now this is a great capability to have, especially if you have to do very little as a user to get there. However, for some applications and objects it can be restrictive. For instance, if your transactions last for a long period of time (say minutes), updating different resources throughout, and just at the very end there's a failure, then everything has to be done again. This and other limitations with top-level transactions lead us and others to look at various extended transaction models, and one of the earliest approaches was nested transactions from 1981, also known as subtransactions (not to be confused with subordinate coordinators).

Transactions which are contained within the other transaction are said to be nested (or subtransactions), and the resulting transaction is referred to as the enclosing transaction. The enclosing transaction is sometimes referred to as the parent of a nested (or child) transaction. A hierarchical transaction structure can thus result, with the root of the hierarchy being referred to as the top-level transaction. An important difference exists between nested and top-level transactions: the effect of a nested transaction is provisional upon the commit/roll back of its enclosing transaction(s), i.e., the effects will be recovered if the enclosing transaction aborts, even if the nested transaction has committed.

Subtransactions are a useful mechanism for two reasons:


  • fault-isolation: if subtransaction rolls back (e.g., because an object it was using fails) then this does not require the enclosing transaction to rollback, thus undoing all of the work performed so far.

  • modularity: if there is already a transaction associated with a call when a new transaction is begun, then the transaction will be nested within it. Therefore, a programmer who knows that an object require transactions can use them within the object: if the object’s methods are invoked without a client transaction, then the object’s transactions will simply be top-level; otherwise, they will be nested within the scope of the client’s transactions. Likewise, a client need not know that the object is transactional, and can begin its own transaction.



Imagine what you could do with nested transactions, particularly if you get to control them programmatically. You can start transactions within your application or objects without having to be concerned about whether or not there are transactions already in scope. If you do know and care, then you can utilize them for application needs. For instance, nested transactions give you the ability to try alternate paths in a decision tree without having to worry about the impact on transactional resources; you try each option in a separate nested transaction and roll back the paths you don't chose, relying on the transaction system to undo that work automatically, and then you simply commit the selected path (nested transaction).

So how do you use them? Well in JBossTS there are no special constructs for nesting of transactions: if a transaction is begun while another is running on the thread then it is automatically nested. This allows for a modular structure to applications, whereby objects can be implemented using transactions within their operations without the application programmer having to worry about the applications which use them, i.e., whether or not the applications will use transactions as well. Thus, in some applications those transactions may be top-level, whereas in others they may be nested. Objects written in this way can then be shared between applications and JBossTS will guarantee their consistency. If a nested transaction is rolled back then all of its work will be undone, although strict two-phase locking means that any locks it may have obtained will be retained until the top-level transaction commits or rolls back. As mentioned before, if a nested transaction commits then the work it has performed will only be committed by the system if the top-level transaction commits; if the top-level transaction aborts then all of the work will be undone.

What all of the above means is that if you want to use nested transactions in your applications then all you need to is start them programmatically as we saw earlier. So for example, if we wanted to use the AtomicObject we defined earlier within a top-level transaction, the code could look like:


AtomicAction A = new AtomicAction();
AtomicObject obj = new AtomicObject();

A.begin();

obj.incr(1);

A.commit();


The fact that the AtomicObject incr method uses a transaction internally is hidden from the invoker. In this case it'll be nested. If the code looked like ...


AtomicObject obj = new AtomicObject();

obj.incr(1);


... then the transaction would be top-level. Hopefully the differences in behaviour are pretty obvious by now, but just in case: in the former case we get a chance to undo the increment if we were to roll back the enclosing transaction, whereas in the latter, assuming the incr method didn't fail, then there is no opportunity to roll back; we would have to do an application driven undo which isn't guaranteed to be equivalent from an ACID perspective (e.g., another user could have seen the state between the increment and the decrement). And of course that leads us towards compensating transactions, but they are definitely out of scope for this article.

So I hope I've given you a flavour to nested transactions and some of the power they could bring to applications and objects. Why not give them a go?

Tuesday, May 18, 2010

Thank you HAL Corporation for FUD spreading

I mentioned a few months back how a certain group of people at a certain company that was conjectured to be the basis for HAL 9000 in 2001: A Space Odyssey were spreading FUD about JBossTS. Now not doing your homework is bad enough, but then building a whole marketing exercise around it is even worse! To say it's like building a house of cards is an understatement. It only takes someone to start questioning, probing and pushing for the whole thing to start to come tumbling down and the sheer stupidity of the original exercise comes to the fore.

Now comparing and contrasting projects, products or pretty much anything, should really be done on a scientific basis. (OK there are some things in life where that might be overkill or inappropriate, but not here.) It should be testable and reproduceable, ideally by anyone independent. If you're going to make statements about the pros and cons of this or that, then in some ways your reputation is being put on the line. Of course if you don't value your reputation then this is all moot. However, for JBoss and Red Hat, reputation and integrity are very important, so we're not going to say something bad or potentially maligning unless we have all of the facts, and even then we'll typically do it in a very scientific manner once again. This means that if we're wrong and it's because we made a mistake, e.g., had overlooked something, then we'll admit it.

Another thing we won't do is go around prospective customers spreading FUD, demonstrating FUD and making statements about "fundamental issues" with a competitor's product/project that mean it can never do X, Y or Z, unless there's proof, and in which case it's not really FUD. But of course our principles aren't shared by everyone. But when I heard that that was precisely what was happening it made their previous FUD attempts pale into relative insignificance. Despite my previous blog post, lots of real world deployments showing that recovery works, a history dating back to the 1980's which included being part of HP's foray into the middleware space, it wasn't enough for some who had been bedazzled by the IBM Roadshow (and to a degree you can't really blame them, since this is Big Blue after all.)

So we decided that enough was enough. Our honour had been besmirched (insert smiley). Fortunately recovery is so critical to a transaction system that we have lots of demonstrators and QA tests that cover pretty much every situation you can consider. And most of them are publicly available in our repository. Wow: who knew?! (Insert tongue firmly in cheek). So our teams (TS and QA) got together and in less than a day had produced something that duplicated the scenario, showed that recovery works, and therefore showed what a load of rubbish was being spouted.

Now of course it can be argued that we knew all of this to start with so we just wasted time and effort proving it yet again. I agree in part, and when this impacts on my holidays, as it did, it does make me a little bit mad. But I don't blame anyone other than the original mud slingers. Hopefully they'll learn something from this exercise. Hopefully they'll learn what open source is about and how to use it as well as contribute towards it. Unfortunately I have my doubts. So until the next time I'll leave you with this thought: what's that they say about glass houses?

Sunday, May 2, 2010

Building transactional applications with JBossTS

Towards the end of last year I mentioned that we've been doing some work on STM around JBossTS. I said I'd give some hints at what we've been doing, but in order to do so we need to cover some background on how you can construct transactional applications with the existing code using Transactional Objects for Java (TXOJ). You'll find a lot more information about what I'm going to summarize in the documentation and various papers.

In order for your applications to possess all of the necessary ACID properties when using transactions, the objects manipulated by those transactions need to be both durable and isolated from conflicts. If your application objects aren't backed by a database, then you have the headache of providing these capabilities yourself, unless you have your friendly neighborhood transactional framework like TXOJ. Without going into all of the details (which really are in the documentation) probably the most important class is StateManager which provides all of the basic support mechanisms required by an object for state management purposes. As with concurrency control, which we'll come on to later, durability is obtained through inheritance.

Objects in TXOJ are assumed to be of three possible basic flavours. They may simply be recoverable, in which case StateManager will attempt to generate and maintain appropriate recovery information for the object. Such objects have lifetimes that do not exceed the application program that creates them. Objects may be recoverable and persistent, in which case the lifetime of the object is assumed to be greater than that of the creating or accessing application so that in addition to maintaining recovery information StateManager will attempt to automatically load (unload) any existing persistent state for the object at appropriate times. Finally, objects may possess none of these capabilities in which case no recovery information is ever kept nor is object activation/deactivation ever automatically attempted.

There are quite a few StateManager methods that your application could utilize directly, but for now the only three that are of immediate importance are save_state, restore_state and type. The first two methods are responsible for saving (or restoring) the state of the object, whereas the last is used when positioning the state in the transactional object store. Now although the store may also be the location of the transaction log, the application object states aren't necessarily maintained within the log. (Yes, I'm glossing over subjects such as activating and passivating objects, but we may come back to those in a later posting.)

With these three StateManager methods defined you could manage the state of your object's within the scope of a transaction. But typically you'll also want isolation, so let's move on to LockManager, which inherits from StateManager. So your application objects can become isolated and durable by having their corresponding classes inherit from LockManager. Although you have to be concerned about the three StateManager methods mentioned earlier, there's nothing else here that you need worry about in terms of overriding or defining. But of course you do need to tell the system when and how to isolate your objects. The primary programmer interface to the concurrency controller is via the setlock operation. By default the runtime enforces strict two-phase locking following a multiple reader, single writer policy on a per object basis. Although lock acquisition is under programmer control, lock release is normally under control of the system and requires no further intervention by the programmer. This ensures that the two-phase property can be correctly maintained. (Again, I'll ignore subjects like type specific concurrency control for this posting.)

But sometimes a picture paints a thousand words, so here's an example:


public class AtomicObject extends LockManager
{
public AtomicObject ()
{
super(ObjectType.ANDPERSISTENT);

state = 0;

AtomicAction A = new AtomicAction();

A.begin();

if (setlock(new Lock(LockMode.WRITE), 0) == LockResult.GRANTED)
{
if (A.commit() == ActionStatus.COMMITTED)
System.out.println("Created persistent object " + get_uid());
else
System.out.println("Action.commit error.");
}
else
{
A.abort();

System.out.println("setlock error.");
}
}

public AtomicObject (Uid u)
{
super(u, ObjectType.ANDPERSISTENT);

state = -1;

AtomicAction A = new AtomicAction();

A.begin();

if (setlock(new Lock(LockMode.READ), 0) == LockResult.GRANTED)
{
System.out.println("Recreated object " + u);
A.commit();
}
else
{
System.out.println("Error recreating object " + u);
A.abort();
}
}

public void incr (int value) throws TestException
{
AtomicAction A = new AtomicAction();

A.begin();

if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
{
state += value;

if (A.commit() != ActionStatus.COMMITTED)
throw new TestException("Action commit error.");
else
return;
}

A.abort();

throw new TestException("Write lock error.");
}

public void set (int value) throws TestException
{
AtomicAction A = new AtomicAction();

A.begin();

if (setlock(new Lock(LockMode.WRITE), retry) == LockResult.GRANTED)
{
state = value;

if (A.commit() != ActionStatus.COMMITTED)
throw new TestException("Action commit error.");
else
return;
}

A.abort();

throw new TestException("Write lock error.");
}

public int get () throws TestException
{
AtomicAction A = new AtomicAction();
int value = -1;

A.begin();

if (setlock(new Lock(LockMode.READ), retry) == LockResult.GRANTED)
{
value = state;

if (A.commit() == ActionStatus.COMMITTED)
return value;
else
throw new TestException("Action commit error.");
}

A.abort();

throw new TestException("Read lock error.");
}

public boolean save_state (OutputObjectState os, int ot)
{
boolean result = super.save_state(os, ot);

if (!result)
return false;

try
{
os.packInt(state);
}
catch (IOException e)
{
result = false;
}

return result;
}

public boolean restore_state (InputObjectState os, int ot)
{
boolean result = super.restore_state(os, ot);

if (!result)
return false;

try
{
state = os.unpackInt();
}
catch (IOException e)
{
result = false;
}

return result;
}

public String type ()
{
return "/StateManager/LockManager/AtomicObject";
}

private int state;
}


What we have here is a durable and isolatable integer. It may look like quite a bit of code, but there's a lot of defensive programming here to illustrate explicitly some of the problems that could occur. It's left as an exercise for the reader to optimize the code.

However, what you can see here is the use of AtomicAction for creating transactions (which will be automatically nested if there's already a transaction associated with the thread of control) and then the acquiring of the right types of Lock within the application object code (READ or WRITE to signify whether or not the state of the object may be modified). If you consider what's needed to ensure that the object is transactional, including full recovery, there isn't a lot for the developer to do. OK, it's a bit more than you might expect if you're used to EJB3s, for example, but then there's a bit more going on here, and some of it could well be automated (future posting hint). But when you consider that an entire database system was built on this basis back in the early 1990's (OK, the C++ version of the transaction system and not Java, but very little difference), it helps to illustrate the power of the abstractions as well as the implementation.

OK, that's enough for now. In a later post we may go into more detail on some of the concepts mentioned here (or glossed over so far) before heading off into STM land. If there's anything specific you'd like concentrating on then just suggest via the comments.

Monday, March 15, 2010

Transactions: Over used or just misunderstood?

I gave a presentation at QCon London last week on the above subject. I was part of the Solutions Track and QCon have made the slides available. The room was packed out, which was good though always a little surprising: although transactions are dear to my heart it's not something that lights everyone's fire! It seemed to go down well and there were plenty of questions during and afterwards to convince me that I probably have enough material to do 4 or 5 deep dives into the issues raised by the current presentation. So maybe at the next JUDCon event you'll be able to hear me present on the duality between successful marriages and two-phase commit!

Thursday, January 7, 2010

REST-Atomic Transaction

We've been working a lot on updating the REST transactions work for Bill's REST-* initiative. The latest drafts aren't up there at the moment but are accessible through the mail archives. The current draft 3 of the atomic model is still called REST-ACID due to some WS-ACID history, but the next draft will see a name change to REST-Atomic Transaction to more clearly call out what the model defines. Then of course there's the compensation based model that will see a new draft going up soon.

On a slight side note it's good to see others thinking about the same problem. Though I think this model has some issues it's good that others have recognized that the problem remains undefined within REST. Of course Eric and Phil wrote about some of this in their updated book (worth a read).

Tuesday, January 5, 2010

Someone at IBM needs to do their homework!

I have a lot of respect for IBM, with many good friends and colleagues who work there, or have worked there over the years. I've collaborated with them for a long time on various transaction standards, articles and books. They know a thing or two about transactions, being the home of CICS.

However, sometimes they slip up as do we all. Case in point is this quality article (yes, I'm being sarcastic) from some anonymous author(s) over there in IBM Land. It's a good example of FUD and a really bad example of scientific investigation. I'm not going to help them improve their act on getting those two things to work together, but I will point out a few areas that quickly made me consign the document to the trash.

For a start, making statements like "JBoss AS has a relatively new integration with a third-party transaction manager." and "Public forums seem to indicate that this transaction manager is not proven" could easily be fixed by just reading the community pages. But then I get the impression that the author(s) didn't go near the community pages much except to find "facts" to fit their aims, carefully ignoring anything that didn't fit (not good scientific method!)

Then they concentrate on a specific AS issue which is about configuring AS to automatically recover datasources. Manual editing of files is required at the moment. The author(s) didn't do their homework (or even bother posting on the free forums as far as I can tell) to locate the information they needed. Now maybe they're relatively new to open source and didn't realise you don't have to buy a support contract or buy the software in order to talk with the developers! Maybe even reading the documentation might have helped, although perhaps there was just far too much of it for the author(s).

But does this mean that there are fundamental problems with JBossAS or JBossTS around data integrity? Not that we're aware of and there are enough QA tests and samples to justify that position. The code for any tests that would suggest otherwise is conveniently absent from the author(s) document. If anyone has it or something similar then be a good community member and send it our way. Of course there may be things we've missed (even CICS has issues to this day). Until then I suggest doing your bit for the environment and not waste paper, ink or energy on this article. Your Carbon Footprint will thank you.