Saturday, December 19, 2009

STM for Arjuna

For many years we've discussed the benefits of object-oriented programming for developing transactional applications, with Transactional Objects for Java (TXOJ) as the implementation of those ideas. As we've discussed before, this ties in with other advanced techniques such as nested transactions.

However, in order for developers to use TXOJ you have to use inheritance (remember: object-oriented principles) and modify your classes, whether C++ or Java. These days POJOs (such an overloaded term as well) are the preferred way to develop applications, acquiring enterprise capabilities through the likes of annotations and AOP. So a few months back we started to look at other approaches to TXOJ with some of the researchers at Newcastle University (the historical home of the Arjuna Project). We also looked at Software Transactional Memory and have managed to tie the two requirements together. It's too early to report on everything we've been doing (it's far from complete just yet anyway), but I hope to give some hints in some future postings.

Wednesday, October 21, 2009

Monitoring your JVM using Byteman 1.1.1

Byteman is a bytecode injection tool developed to support testing of Java code using a technique called fault injection. However, it is also very useful for debugging and tracing Java program execution, including execution of the JVM itself. In this post I walk through an example Byteman script which monitors creation, start and exit of threads inside the JBoss Application Server.

The latest version of Byteman (1.1.1) has just been released and it includes the first batch in a series of sample Byteman scripts, all of which provide examples of how to monitor different aspects of JVM operation. The scripts inject tracing code into different JVM classes so that you can watch the JVM as it starts or stops threads, opens sockets, closes files, finalizes instances etc.

Seeing these scripts at work can be very educational -- for example, runnning up the JBoss App Server with script ThreadMonitor.txt installed I could see exactly which threads are created, started and stopped during AS bootstrap.

The script only needs 3 simple rules:

########################################################################
#
# Rule to trace thread creation
#

RULE ThreadMonitor trace create
CLASS java.lang.Thread
METHOD <init>
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceCreate($0)
ENDRULE

########################################################################
#
# Rule to trace thread start
#

RULE ThreadMonitor trace start
CLASS java.lang.Thread
METHOD start
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceStart($0)
ENDRULE

########################################################################
#
# Rule to trace thread exit
#

RULE ThreadMonitor trace exit
CLASS java.lang.Thread
METHOD exit
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT ENTRY
IF TRUE
DO traceExit($0)
ENDRULE

The first rule is injected into the constructor for java.lang.Thread, the second into Thread.start() and the third into Thread.exit(). The conditions are all TRUE because we want to see every call in the trace output. The rule actions call built-in methods traceCreate(Thread), traceStart(Thread) and traceExit(Thread),respectively. The argument to these calls, $0, is the thread object which is the target of the method invocation which triggers the rule, i.e. the one being created, started or exited. The built-in methods access this object to identify the thread's name and it's class name.

The implementation of the three trace methods is provided by a custom HELPER class, ThreadMonitorHelper, which specialises the library class StackTraceHelper provided in the sample lib jar. There is nothing special about this class -- it does not need to use Byteman library code to do its job. It is just a POJO which wraps up the code needed to print meaningful trace with a neat, simple API. All three methods print a line of text identifying the thread event (create, start or exit), thread name and thread class. The first two also use methods inherited from StackTraceHelper to look down the stack for the frame from which new Thread() or Thread.start() was called and print the details of the calling method.

Here's a sample of the output when running JBoss AS with these rules installed:

starting server default
=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT

JAVA: /usr/java/jdk1.6.0_14/bin/java

JAVA_OPTS: -Dprogram.name=run.sh -server -Xms600M -Xmx600M -XX:MaxPermSize=128m -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -Dorg.jboss.byteman.quodlibet -javaagent:/home/adinn/byteman/install/lib/byteman.jar=listener:true,boot:/home/adinn/byteman/install/lib/byteman.jar,boot:/home/adinn/byteman/install/sample/lib/byteman-sample.jar,script:/home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt -Djava.net.preferIPv4Stack=true

CLASSPATH: /home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/bin/run.jar:/usr/java/jdk1.6.0_14/lib/tools.jar

=========================================================================

*** Thread create JDWP Transport Listener: dt_socket java.lang.Thread
from VM runtime
Listening for transport dt_socket at address: 5005
*** Thread create JDWP Event Helper Thread java.lang.Thread
from VM runtime
*** Thread create CompilerThread0 java.lang.Thread
from VM runtime
*** Thread create CompilerThread1 java.lang.Thread
from VM runtime
*** Thread create Low Memory Detector java.lang.Thread
from VM runtime
*** Thread create main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread start main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread exit main java.lang.Thread
*** Thread create DestroyJavaVM java.lang.Thread
from VM runtime
*** Thread create Thread-1 java.util.logging.LogManager.Cleaner
from java.util.logging.LogManager$Cleaner.<init> at LogManager.java:200
*** Thread create Thread-2 org.jboss.Main.ShutdownHook
from org.jboss.Main$ShutdownHook.<init> at Main.java:883
12:38:16,588 INFO [JBossASServerImpl] Server Configuration:

Bootstrap URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/conf/bootstrap.xml
JBOSS_HOME URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/
Common Base URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/common/
Common Library URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/common/lib/
Server Name: default
Server Base URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/
Server Library URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/lib/
Server Config URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/conf/
Server Home URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/
Server Data URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/data/
Server Log URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/log/
Server Temp URL: file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/tmp/

*** Thread create Thread-3 java.lang.Thread
from org.jboss.bootstrap.impl.base.server.AbstractServer.start at AbstractServer.java:350
*** Thread start Thread-3 java.lang.Thread
from org.jboss.bootstrap.impl.base.server.AbstractServer.start at AbstractServer.java:352
12:38:16,590 INFO [AbstractServer] Starting: JBoss Server[6.0.0.SNAPSHOT (build: SVNTag=JBoss_6.0.0-SNAPSHOT date=r93729)]
12:38:17,344 INFO [AbstractMCServerBase] Starting Microcontainer, Main bootstrapURL=file:/home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/conf/bootstrap.xml
*** Thread create Thread-4 java.util.TimerThread
from java.util.TimerThread.<init> at Timer.java:456
*** Thread start Timer-0 java.util.TimerThread
from java.util.Timer.<init> at Timer.java:154
12:38:18,259 INFO [VFSCacheFactory] Initializing VFSCache [org.jboss.virtual.plugins.cache.CombinedVFSCache]
12:38:18,262 INFO [VFSCacheFactory] Using VFSCache [CombinedVFSCache[real-cache: null]]
12:38:18,670 INFO [CopyMechanism] VFS temp dir: /home/adinn/jboss/jbossas/trunk/build/output/jboss-6.0.0-SNAPSHOT/server/default/tmp
12:38:18,675 INFO [ZipEntryContext] VFS force nested jars copy-mode is enabled.
*** Thread create Thread-5 java.util.TimerThread
from java.util.TimerThread.<init> at Timer.java:456
*** Thread start ZipFile Lock Reaper java.util.TimerThread
from java.util.Timer<init> at Timer.java:154

During this initial stage of bootstrap most of the threads are created by the VM itself. The first one actually created and started by the AS is identified by the following lines:
*** Thread create main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread start main java.lang.Thread
from org.jboss.Main.main at Main.java:719
*** Thread exit main java.lang.Thread
The trace shows that the name of the newly created thread is main and it is an instance of class java.lang.Thread. It is created and immediately started by a call to Thread.start() in method org.jboss.Main.main at line 719 of file Main.java. Here's the actual code:

701 Runnable worker = new Runnable() {
702 public void run()
703 {
. . .
716 };
717
718 ThreadGroup threads = new ThreadGroup("jboss");
719 new Thread(threads, worker, "main").start();
The third trace line is a little confusing. The thread which has exited is not actually the JBoss main thread just created. It is in fact the parent thread, created by the VM and also called main. The JBoss main thread goes on to create a whole lot of other threads, connection handlers, pool threads and so on.

Clearly, there are lots of possibilities for developing alternative scripts which provide different, more interesting tracing behaviour. The other scripts in the samples directory provide further simple examples of how you might instrument other JVM classes to provide runtime trace. By coding more selective rule conditions or more complex rule sets it is possible to make the tracing much more fine-grained and specific to particular circumstances. Adding more actions, possibly using other custom helper classes, would allow a variety of different statistics to be gathered and displayed.

I will explain next how you set up your program to inject these rules. The trace above shows Byteman injecting the scripted rules from program start. However, when it comes to tracing execution it would be nice if it were possible to start up your program without any scripts installed and then switch on tracing by loading and unloading rules as needed. Well, of course, it is possible to do just and I'll go on to explain what you need to change to achieve that too.

First off let's run through how JBossAS was configured to start up the byteman agent and execute this script from JVM bootstrap. All that is needed is to supply the java command which starts the JBossAS Main class with a -javaagent argument. Passing this argument to JBossAS is most easily achieved by inserting it into environment variable JBOSS_OPTS ans the running the normal JBossAS startup script.

The actual setting required was displayed in one of the very long lines of trace output:

JAVA_OPTS: -Dprogram.name=run.sh -server -Xms600M -Xmx600M -XX:MaxPermSize=128m -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005  -Dorg.jboss.byteman.quodlibet -javaagent:/home//byteman/install/lib/byteman.jar=listener:true,boot:/home/adinn/byteman/install/lib/byteman.jar,boot:/home/adinn/byteman/install/sample/lib/byteman-sample.jar,script:/home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt -Djava.net.preferIPv4Stack=true
Let's look at the actual -javaagent argument to understand what is needed to point the agent at the script and the relevant libraries. To make it a bit easier to see what is going on I have added line breaks after the equal sign and commas which separate the different options. In reality the argument needs to be supplied as one long string with no spaces or line breaks.
-javaagent:/home/adinn/byteman/install/lib/byteman.jar=
listener:true,
boot:/home/adinn/byteman/install/lib/byteman.jar,
boot:/home/adinn/byteman/install/sample/lib/byteman-sample.jar,
script:/home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt
Before we look at these options in detail I'll also point out that the java command line included the following property setting
-Dorg.jboss.byteman.quodlibet
This property enables injection into java.lang classes. Normally the agent ignores rules which apply to this package on the grounds that by modifying these classes it will likely shoot itself in the foot changing the JVM functionality it relies on to execute rules. Setting this flag tells it to remove this restriction. There is no great harm in trying this but beware! Get your rules wrong and you will probably see a JVM error such as a stack overflow.

Ok, warnings aside let's return to the -javaagent argument. The first value following the -javaagent: prefix is the location of the byteman jar. If BYTEMAN_HOME identifies the directory into which the release has been installed then the library is located in $BYTEMAN_HOME/lib/byteman.jar, in this case /home/adinn/byteman/install/lib/byteman.jar.

The = sign following the jar file name is used to separate it from the agent options which configure various aspects of the agent's behaviour. Where more than one option is supplied then they must be separated using commas. The options provided in this case include a script: option. This points the agent at the script file containing rules which trace thread activity. You can provide multiple script: options if you want to inject several sets of rules. For example, you might also want to use the ClassLoadMonitor script to monitor class loading. The agent options list also contains two boot: options which supply jars to be added to the bootstrap classpath (we'll come to this later) and a listener: option which is used to start the agent listener.

The first option is listener:true. This causes the agent to start a listener thread which can be communicated with using the submit.sh script located in the Byteman bin directory. The submit script allows you to manage the rule base at runtime. You can check which rules have been injected, compiled and executed. You can also load and unload rules while the program is running, including loading updated versions of rules to fix errors or patch the rule behaviour. The listener will not be started if listener:false is provided or if the listener option is omitted.

So, if you want to be able to switch tracing on and off you must provide option listener:true. If not then you cannot add or remove any rules and you will be stuck with whatever rules were loaded at boot time. You can, of course, omit the script: option. It's ok to start with a vanilla JVM. If you do this then you can load the rules later when you want to turn tracing on.

Let's take a closer look at the submit script. Running submit.sh with no arguments lists all the currently installed rules:
[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh
# File /home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt line 66
RULE ThreadMonitor trace start
CLASS java.lang.Thread
METHOD start
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceStart($0)
ENDRULE
Transformed in:
loader: sun.misc.Launcher$AppClassLoader@480457
trigger class: java.lang.Thread
compiled successfully
# File /home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt line 52
RULE ThreadMonitor trace create
CLASS java.lang.Thread
METHOD <init>
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT EXIT
IF TRUE
DO traceCreate($0)
ENDRULE
Transformed in:
loader: sun.misc.Launcher$AppClassLoader@480457
trigger class: java.lang.Thread
compiled successfully
# File /home/adinn/byteman/install/sample/scripts/ThreadMonitor.txt line 80
RULE ThreadMonitor trace exit
CLASS java.lang.Thread
METHOD exit
HELPER org.jboss.byteman.sample.helper.ThreadMonitorHelper
AT ENTRY
IF TRUE
DO traceExit($0)
ENDRULE
Transformed in:
loader: sun.misc.Launcher$AppClassLoader@480457
trigger class: java.lang.Thread
compiled successfully
The output shows each of the 3 rules which were in the original script and shows that they have been injected into class java.lang.Thread loaded by the bootstrap class loader (it actually says sun.misc.Launcher$AppClassLoader@480457 but no other classloader will load this class). The last line of output for each rule indicates that the rule was parsed and type-checked successfully.

Running the submit script with the -u flag causes the rules to be unloaded:
[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh -u \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt
uninstall RULE ThreadMonitor trace create
uninstall RULE ThreadMonitor trace start
uninstall RULE ThreadMonitor trace exit
[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh
no rules installed

The agent responds to the uninstall request by removing any rules defined in the supplied scripts (n.b. if submit.sh -u is run with no script arguments it removes all rules). The agent recompiles the methods of any affected classes to remove the injected trigger code. In this case the rules only apply to class Thread so only one class needs to be recompiled. When submit completes the Thread methods will have reverted to operating as they would do normally and no more trace will be generated. So, tracing has been switched off.

The rules can be reinstated by resubmitting the script:
[adinn@toby trunk]$ $BYTEMAN_HOME/bin/submit.sh \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt
install rule ThreadMonitor trace create
install rule ThreadMonitor trace start
install rule ThreadMonitor trace exit
The agent handles an install request by identifying all classes to which the rules apply and recompiling them, injecting the newly uploaded trigger code. In this case the only class affected is Thread so its methods are recompiled inserting trigger code for the tracing rules once again.

If an uploaded rule has the same name as one that is currently loaded then the old trigger code is removed and new code injected. So, the submit script can be used to modify injected rules on the fly. If your rule shows something interesting and you want to see more details then just redefine and upload your trace rules to print out more details.

In other cases uploaded rules may not match any currently loaded classes. However, they are still added to the agent's rule base. The rule code will be injected when a class is loaded to which the rule applies.

Ok, finally, let's return to the boot: options provided as part of the -javaagent argument. The options are used to ensure that the two jar files are available when the trigger/rule code are type-checked and compiled.

The first jar to consider is the sample library /home/adinn/byteman/install/sample/lib/byteman-sample.jar which contains the implementation of the helper classes StackTraceHelper and ThreadMonitorHelper. When the rules are injected these classes need to be available in the JVM classpath or else the rule code will fail to typecheck and compile.

Normally, it would be sufficient to place the samples library jar in the runtime classpath provided on the java command line. Once this is done it does not usually matter what classloader the trigger class is loaded by since the system classloader is normally a parent for it. So, when type checking the injected rule the helper class will be found by delegation. As an alternative, the agent option sys: can be used to append the jar file to the system classpath. This also ensures the helper classes will be found by delegation.

The only problem with this is that the rules in the thread monitor script need to be injected into JVM classes. These classes are loaded by the bootstrap loader. Unfortunately, the system classloader is not a parent loader for the bootstrap loader -- in fact, the parent relationship is the other way round. Adding the sample library jar to the bootstrap classpath means that it is visible when injecting into JVM classes. Hence, we need to supply the boot: option for this jar.

If you didn't install your helper jar at boot time then you could always use submit.sh to install it at runtime. As long as you have started the listener you can call submit -s my.jar or submit -b my.jar to add a jar to, respectively, the system classpath or bootstrap classpath. Obviously, you need to do this before you load any rules which refer to the helper classes. Unfortunately, there is no way to remove jar files once they have been added in this way.

The other jar supplied using the boot: option is the byteman jar itself. The reasoning here is much the same -- the agent classes need to be visible to bootstrap classes. Normally, when the agent jar is installed its classes get loaded by the system classloader. However, in this case we want to inject a trigger call into a JVM class. If the agent classes such as Rule etc are loaded by the system classloader then they will not be visible from classes such as Thread. So, we also need to supply the boot: option for the byteman jar.

Note that it is not possible to add the byteman jar to the bootstrap classpath using submit. By the time the listener is up and running some agent classes will already have been loaded into the system classloader. Trying to install the byteman jar into the bootstrap classpath at this point will send you straight to classloader hell.

So, let's recap. If you want to start the agent so you can load and unload scripts at runtime and inject into any class, including JVM classes, you need to set
export BJAR=$BYTEMAN_HOME/lib/byteman.jar
and then run your java command with arguments
-javaagent:$BJAR=listener:true,boot:$BJAR
-Dorg.jboss.byteman.quodlibet
When you want to start tracing (firsttime only) you install your helper jar using
$BYTEMAN_HOME/bin/submit.sh \
-b $BYTEMAN_HOME/sample/lib/byteman-sample.jar

You can then load your script using command
$BYTEMAN_HOME/bin/submit.sh \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt

To check that your rules have been injected and compiled use command
$BYTEMAN_HOME/bin/submit.sh
To stop tracing you unload the script using
$BYTEMAN_HOME/bin/submit.sh -u \
$BYTEMAN_HOME/sample/scripts/ThreadMonitor.txt

Tuesday, September 29, 2009

Native transactions - cross post

I had a few things to say about the latest presentation from Spring on transactions. Worth a look if you're thinking about whether or not you need to use a transaction manager (from anyone, not just us) since the presentation isn't always as accurate as it could (should) be.

Friday, September 18, 2009

Byteman 1.1.0 now available

Byteman 1.1.0 has now been tagged, released and made available for download.

This release includes one major new feature:
  • support for dynamic upload of rules into running programs
Note that this includes the ability to retransform already loaded classes, injecting rules into code while the program is executing it. This feature allows Byteman to complement conventional debugging tools, injecting tracing code or fixes into programs on the fly. In order to support this feature Byteman now only runs on JDK 6 upwards.

The release also provides several bug fixes, most notably:
  • full support for injection into bootstrap loader classes (only java.lang.* is exempt -- here be dragons ;-)
  • retrospective injection of rules into bootstrap classes loaded prior to Byteman agent installation
  • fixing of several problems with array types/instances and throw action

Friday, September 4, 2009

New (old) Byteman release now available, New (new) Byteman release in the works

Yesterday I published the latest release of the 1.0 Byteman series, version 1.0.3. That's the new old release in the title. This is a maintenance release which fixes about 15 bugs/features discovered while using Byteman to fix timing issues in various of the JBossTS unit tests. This version has also been used for testing on the MicroContainer project and plans are afoot to extend its use to other JBoss projects.

New features in 1.0.3 include: the ability to reference method parameters by name in rule conditions and actions; option to omit empty BIND clauses; and new helper built-ins to enforce waits on thread exits and to provide better management of rendezvous. Bug fixes include: better parser error detection; and correct reporting of line numbers in parser/type checker errors. Binaries, sources and full release notes are included on the download site.

The new new Byteman release is 1.1.0 which I have been developing in parallel with the maintenance release. It will only run on JDK 6 upwards, allowing it to profit from the retransform capability introduced into the JDK 6 instrumentation package. This new new release will support dynamic loading/reloading of rules into a running program much as is possible using the btrace package. The byteman agent can be configured to open a listener which, currently, can be used either to list all currently applied bytecode transformations or to upload new rules or redefinitions of existing rules. The rule upload capability includes applying rule injection retrospectively to already loaded classes.

This dynamic upload capability should simplify development of rule sets. It should also make it a lot easier to use Byteman as a debugging aid. Hopefully, it will also pave the way for automation of rule injection via IDE plug-ins. Developing tool support is definitely on the agenda for the project so anyone who wants to contribute is more than welcome to respond.

I have just checked in a first draft of the dynamic upload functionality into the byteman repository trunk and I already have a community user lined up to try out the redefine capability. Feedback and suggestions for improvements will be graciously (and, quite possibly, gratefully :-) received.

Tuesday, June 23, 2009

Byteman project goes live on jboss.org

Just in time for my talk at Jazoon I have set up a JBoss project for Byteman -- that's the new, official name for TOAST the fault injection tool we have been using to test Web Services Transactions crash recovery. The project page is pretty minimal at the moment, containing only a welcome and downloads page. However, we will very soon have a user forum and JIRA for the project and will be moving the code from its current temporary location in the JBossTS repository.

I want to to make it as easy as possible for people to try out Byteman and provide feedback on problems encountered, contributions to the code base or new ideas for how to improve or extend its functionality. If you have a Java program you want to test, or even just trace its execution, then take a look at Byteman and see how it can help you.
Link

Tuesday, June 16, 2009

RESTful transactions brings out the crowds

We gave a session on the RESTful transactions work at JavaOne. It was well attended and there was a lot of good discussion afterwards. Then a separate discussion kicked off on the REST-discuss mailing list, followed by some InfoQ article comments. At that point I got a serious case of deja vu (or maybe there's a glitch in the Matrix?)

I've spent many years working in various groups, companies, books and articles on explaining this. I really don't want to have to do it all again! It doesn't matter if you're living in the Land of WS-*, or REST World, or the SOA Nation, we all know that ACID transactions aren't suitable for loosely coupled interactions. We all know that there are problems with forcing atomicity, consistency, isolation and maybe even durability. As they say, we've been there done that and bought the T-shirt! Move along. This is not the argument you're looking for. Move along.

Wednesday, June 10, 2009

Another REST+transactions protocol

During discussions on REST and transactions, Alexadros pointed us at some work they've been doing. Very interesting. Once we get a chance to look through each others approaches I intend to start a discussion on our design forums. If you're interested then join in.

Sunday, March 29, 2009

JBossTS offsite

We had another good offsite meeting earlier this week. As usual Jonathan was spot-on with the selection of venue (the Black Swan Hotel): good food and a great atmosphere for discussions and planning. We had the JBossTS and Blacktie teams together in one place, so were able to cover a lot of things from TOAST through RESTful transactions and OSGi. Jonathan was able to plan out the next 6 months or so for the team and it's a good mix of definite current needs and beliefs on future requirements. I can't wait for the next meeting!

Wednesday, March 25, 2009

Compiled Configurable TOAST

In between completing fully functional crash recovery for the Web Services Transactions (XTS) code and writing a comprehensive test suite for it I've been able to squeeze in a couple of nice extensions to the functionality of TOAST, the code injection tool I have been using to automate XTS crash recovery testing.

Attentive readers may recall that TOAST is a nifty, easy to use tool for injecting side-effects into Java code. I have been using TOAST to test XTS recovery by injecting code which crashes the JVM in the middle of a transaction commit. I also use TOAST to inject further faults during recovery in order to delay or drop messages exchanged between the transaction coordinator and the web service participants it is trying to corral into a successful commit. Finally, I inject trace statements at suitable locations in the code which track the progress of the test and allow the test client to identify successful or failed execution.

None of this requires any modifications to or recompilation of the deliverable application being tested. TOAST just needs to be supplied with a script identifying the trigger points, locations in the application code which need to be tweaked, and the Java expressions which are to be (conditionally) executed at those trigger points. Point your JVM at the TOAST jar and the script when you start your Java application and TOAST makes sure all the required side effects get executed.

For those who have not yet had a look at TOAST I'll recap a little. Scripts comprise a sequence of Event Condition Action (ECA) rules. The Event part is the specification of the trigger point, a class name, a method located in that class and some location in the method code. Locations identify a point in the source code through reference to the code structure. Examples might be: the start of the method (AT ENTRY) when a return occurs (AT EXIT), before the first read of an instance field (AT READ myField), after the 2nd call to a method (AFTER CALL getMyField() 2) and so on. The event specifies WHEN the side-effects coded in the rule are are considered for execution by describing a trigger point. Whenever a thread reaches this trigger point it enters the rule engine to decide whether to fire the associated rule.

The Condition part is a Java expression specifying WHETHER to fire the rule. If this evaluates to true then the rule fires and executes its Action. If the condition is false then control returns to the trigger method and execution continues as normal.

The Action part specifies WHAT side-effects need to occur. It is a sequence of Java expressions to be evaluated one after another. The last expression may be a return or a throw action. These cause an immediate abnormal return of control from the triggering method bypassing normal execution of the rest of the method code. If the method is non-void then the return action must include an expression evaluated to compute the return value. A throw action must include an exception type and arguments for the exception constructor.

In order to simplify testing of multi-threaded code TOAST's rule language also includes built-in methods for use in conditions and actions. These are a variety of operations which simplify the development of multi-threaded tests. For example, one rule might include a call to waitFor() in its action list, suspending any thread which fires the rule. Another rule might include call to signalWake() in its action. If a second thread fires this rule then the call to signalWake() wakes up the suspended thread. This can be used to ensure that the first thread is only allowed to progress once the second thread has reached a suitable point during its execution.

So, what are the two updates which I have sneaked in? Well, the first one is to compile the rule code to bytecode. Up until now the rule execution engine has worked by interpreting the rule parse tree. Method and builtin calls were executed using reflection. Java operations were executed by evaluating their arguments and then combining them based on the operation type. Well, the latest version of TOAST now compiles the rule code into Java bytecode attached to a helper class associated with the rule. This bytecode is handed to the JIT compiler which means it can be compiled to native machine code and optimized like any other Java method. This will not make a lot of difference for rules which are only executed once or twice but it will make a big difference to the usability of TOAST in cases where rules are triggered many times.

The second update is a very nice generalization of the built-in functionality. The rule parser, type-checker and compiler don't actually care much about the presence of builtin methods. As far as they are concerned a call is a call is a call. The type checker, in particular, attempts to resolve calls by looking at the method name and signature i.e. the types of the call arguments. Any calls whose name and signature matches a public instance method of a class called Helper are treated as built-in calls. When the rule is triggered an instance of Helper is created to store the list of event bindings. A built-in call can be executed by calling the matching method with this instance as the method target. The compiler actually generates a subclass of Helper whose execute() method comprises the compiled rule bytecode. So, built-in calls are actually just invocations of methods on this.

Now, this mechanism is completely independent of the details of how class Helper is defined. Any public POJO class could actually be used to define the rule language built-ins. So, a simple declaration in the rule source specifying the name of an alternative class will allow the rules to use a different helper more appropriate to the application in question. A domain-specific helper class makes scripting of side-effects simpler and tidier by encoding complex, application-specific actions as single operations. If you like you can keep the current built-ins and add some more of your own by extending class Helper and adding extra public methods. If you don't like how one of the default built-in operations works you can override the version Helper provides to re-implement it.

These upgrades are described more fully in the TOAST documentation. Please download TOAST, take a look and give it a test run.

Monday, March 16, 2009

Nested transaction support

For well over 20 years we've been saying that JBossTS (aka ArjunaTS aka Arjuna) supports nested transactions. If you check out most of the papers, documents or presentations that have been written on the subject over the years they'll all tell you that nested transactions (subtransactions) are good because:

  • fault-isolation: if subtransaction rolls back (e.g., because an object it was using fails) then this does not require the enclosing transaction to rollback, thus undoing all of the work performed so far.

  • modularity: if there is already a transaction associated with a call when a new transaction is begun, then the transaction will be nested within it. Therefore, a programmer who knows that an object require transactions can use them within the object: if the object’s methods are invoked without a client transaction, then the object’s transactions will simply be top-level; otherwise, they will be nested within the scope of the client’s transactions. Likewise, a client need not know that the object is transactional, and can begin its own transaction.


Now nested transactions have been around for a while (since the early 80's) but they have had limited adoption outside of academia. We did manage to get them adopted within the OTS, even if in a fairly brain-dead/limited approach. But other standards either ignored them (e.g., XA) or made such vague statements about them as to be useless (e.g., JTA). The reason for this is mainly because database vendors have not traditionally supported nested transactions and is primarily why I'm writing this blog entry: I've noticed a few new transaction manager implementations that say they support nested transactions. But what does that actually mean?

If you look at either the JBossTS architecture or even the OTS interfaces, there's not a lot you have to do to the coordinator in order for it to support a parent-child relationship. That's the easy part! (In fact it's easier for the coordinator to support nested transactions because there's no requirement for durability.) But as with a top-level transaction implementation, you need participants (e.g., databases) to be able to do anything useful. The semantics of nested transactions (relaxing ACID properties) mean that a top-level aware participant can't be used for a nested transactions (e.g., your database is going to write the state changes to disk when the XAResource is driven through 2PC and what about lock inheritance?)

Thus whenever we said "JBossTS supports nested transactions" we meant that the coordinator had the parent-child relationship, but we also provided nested transaction-aware resources. Rather than doing very little to the coordinator and pushing most of the real effort on to the application developer or resource implementer, we provide a complete toolkit for developing transactional applications. As a developer you don't have to worry about whether or not a transaction is nested and whether or not you've got nested transaction-aware resources in the mix somehow. All of that is taken care of by JBossTS. So be careful if you come across a transaction manager that says it "supports" nested transactions: it's a meaningless statement unless this covers the coordinator and participants (state and concurrency control). And of course you want to do this in a distributed environment as well as in a flexible manner! (Yes, we do that too!)

Wednesday, March 11, 2009

RESTful transactions reborn

I've mentioned a few times about work we did almost 10 years ago on transactional REST services. This was when JBossTS was part of the HP Netaction suite, we were working on BTP, as well as the precursor to WS-TX/WS-CAF and the world's first Web Services transactions product HP-WST.

Without reminiscing too much, in those days I was participating in the W3C Web Services Architecture working group and the topic of REST came up there quite regularly. So we decided to give it a try as far as transactions were concerned and 'hey presto' RESTful transactions were born. This was way before JAX-RS so we were hand crafting the interactions ourselves, but we demonstrated it could work with the involvement of several companies. And there is remained once HP divested themselves of their middleware division.

But having transactional REST-based services has been a pet project for a long time, so it was bound to rear its head. Especially as some people (rightly) push the Web as an integration platform (shouldn't really be news to people since the WWW has always been about integrating data). Of course there are those who say you don't need transactions in the Web or can (somehow) program around them, but they miss the point. It's a bit like saying you don't need an airbag in your car but can use a pillow (and quick reflexes) yourself to save money. Yes, transactions aren't right for all situations and some bespoke approaches may be more efficient, but if you think about what you get from a transaction system and what you'd have to do in order to obtain the same guarantees, it's more often than not worth using them.

So we originally created a student project to dust off the old REST transactions protocol(s) and implement them today. This started life before JAX-RS (again) but by the time something really happened JAX-RS (and RESTeasy) was around. Michael from the JBossTS team took this work on and did a great job. It's not finished (only the atomic outcome protocol has been implemented so far, for example). Plus the protocols haven't been changed at all since they were created back in 2000/2001. So there are still some things we need to look at and more interesting work ahead.

Is this something you should be looking at for your projects? Should you believe those people who say transactions and REST don't go together? Well as I'm fond of saying "One size does not fit all." Transactions have played a critical part in all distributed systems for over four decades and for very good reasons. Of course we have XTS and Web Services Transactions support if you're using SOAP. But maybe SOAP isn't "your thing" and HTTP is all you can use. Well in that case take a look at this. In just the same way that you wouldn't want to be using a tank in the ocean and a submarine on land, use the right software tool for the right job!

Sunday, February 15, 2009

Lessons learned and magic numbers.

One of the things you do when you start developing software, especially in a new or cutting-edge area, is believe you know what's best for your users and try to hide any complexities from them. That's one of the reasons we pushed RPC as the best way in which to develop distributed applications in the 1980's. Well back when we started developing Arjuna we took hiding complexity to heart, since simplifying the development of transactional applications was core to all of our PhDs. And I think we did a very good job with the initial releases.

However, once we gave the system to people (the source was made freely available by FTP in 1991, but various industry sponsors were using it before then) the feedback we got was very interesting: users will always do things you didn't expect and demand flexibility in what you produce.

In Arjuna this didn't impact the interfaces users saw but many of the internal development parameters that we had used, thinking that they would never need to be configurable (statically or dynamically). For example, when making a remote invocation on an object failures can occur (the network could partition, the machine hosting the object could crash, etc.) In the absence of a perfect failure detector you use failure suspicion and timeouts. Basically if a response does not come back from the object in time T then you assume the object has failed and act accordingly. But if you get the timeout value wrong then you can make the wrong decision with associated consequences. The timeout value really needs to take into account how long work might take to execute, how overloaded a machine may be, network congestion etc. So one value is rarely right for every application. But we didn't take that into account initially and hard-coded a magic number into the system.

There were other examples of magic numbers, including: the number of RPC retransmissions to use before assuming that a request (or response) cannot get through to the endpoint (why should 5 be any better than 2 or worse than 10?); the number of clustered name server instances; the object store location; the orphan detection period; the lock-conflict detection timeout etc. These were all things we believed had the best (sensible) values, but with hindsight it was clear that it was all based on limited deployment knowledge.

What this all lead to quite quickly was a methodology of expect the unexpected and develop accordingly as far as your users are concerned. We made the system extremely flexible and configurable, where many of the old magic numbers could be overridden either as the system ran or during deployment, with sensible defaults (trying to hit Pareto's 80/20 principle). Those magic numbers we didn't make configurable were clearly documented (both so we could remember as much as users could determine why something was behaving the way it was).

From the feedback we received over the past 20 years I think we managed to come close to the right set of compromises. It's true that most users are happy with the default values we set (which were/are revised based on feedback). But those smaller users who really do need the ability to change things now (or since 20 years ago) have the ability to modify them without rebuilding the system. This has been important in the systems adoption and it's a lesson that we continue to apply. So if you're developing software, beware of using too many magic numbers and if you don't make them configurable you need to understand (and believe) why that's the case and, importantly, document them just in case!

Monday, February 9, 2009

JBossTS and Blacktie: the only combination you'll need!

[Cue advertizing jingle.]

Are you a major transaction user, or do you know someone who is? Are you worried about your legacy transaction infrastructure because the vendor may not be reliable? Do you have sleepless nights wondering how to drag your transactional applications into the modern era? Or are just concerned about the ever escalating costs for the transaction system you're currently tied into? Well worry no more: what you need is Blacktie from JBoss, the company that brought you those other useful gadgets like JBossAS, the JBoss T-shirt, the JBoss thong (picture no longer available!), and the JBoss World Virtual Conference.

Unlike offerings from other vendors, Blacktie is entirely open source. It comes in a convenient source or binary bundle for easy deployment to tackle those stubborn legacy applications. Blacktie also builds upon other mature products from your favourite JBoss vendor so it fits nicely into your existing investments from them. And if this is your first entry into open source and JBoss, then it's a great way to start. Blacktie will be the only legacy transaction implementation you'll ever need to solve those annoying problems at a price you'll love!

And coming soon iBlacktie: because everything needs a little i magic!

[Cue advertizing jingle.]

Go on. Check it out. You know you want to!

Tuesday, February 3, 2009

Stay away from pseudo-transactions!

I tried to stay clear of commenting on this article, but while participating in tonight's WS-RA meeting I let my defenses down! In short, the article can be summarized by: "How to use non-transactional resources in a transaction when you can't be bothered to do it right in the first place." Or "How I learned to break transactional semantics and put my data consistency on the line."

I'm fairly sure the authors are trying to help their audience, but they really aren't. They ignore critical problems with their proposal, either deliberately or because they simply haven't given the problem space enough thought. What they are trying to do is emulate the last-resource commit optimization within the application, but whereas a transaction manager will do this and provide support for crash failures, the article ignores that completely. There is no reference to failure recovery at all.

The article also appears to assume that because a datasource is managed by an XAResource it will always honor the business logic agreement when the transaction commits, e.g., if the table update succeeded through the session instance then prepare/commit will work later. I hate to break it to the authors, but that isn't always the case. This works in a transaction manager because we manage the resource ordering and durability very carefully to cope with failures, i.e., there are good reasons we don't require the application programmer to do this!

Oh and the way in which multiple non-transactional resources are managed in the transaction just scares the %$&* out of me! Look, these are resources that do their work when told to and you can't undo that (if you could, then wrap the &%*& things in an XAResource!) So if you crash part way through the normal flow of execution, or part way through the "rollback", what is the state of the application? How do you find out what happened to whom and when? Where's the log?! (I would hate to be a systems administrator in this situation when the sh*t hits the fan!)

The authors say that "Although this is not as robust and comprehensive as a truly transactional interface, it can provide excellent coverage at a fraction of the development cost of a JTA compliant interface." Unfortunately it does not. That's a bit like saying a car with worn brakes is roadworthy in all situations, when it clearly isn't! Ask yourself this: what happens to my data in the cases that aren't covered by this approach and can I really afford to lose it or spend the new hours/days/weeks repairing it manually? If the answer is yes, then you probably don't want to use transactions at all. If the answer is no, then stay away from this approach and go with a transaction system and transactional resources.

In conclusion: use transactions correctly and if you can't make your data items transactional then try using compensating transactions. At least you get logging and recovery!

Building transactional applications

I've been thinking about how we develop transactional applications for a very long time. That thought process got interrupted for a bit, but I came back to it over Christmas and recent events and articles have pushed this further. Plus we're working on transactional aspects for Drools 5 which has covered writing XAResources. Back in HP we had a tutorial on how to do this, which I think we're going to dust off and maybe Jonathan and the team will write some articles around that topic. (I know it's the subject of an update to the book, whenever that happens!)

But it got me thinking that as soon as people start asking about how they can write XAResources to make their data transactional then we really haven't progressed as an industry. XA is a good standard. But it's not perfect. (No standard is!) Plus it was designed with very specific database use cases in mind, which were fine 30 years ago but aren't always a perfect fit for the 21st century. The question should be "how do I make my data transactional?" and leave the implementation specifics to the engine or container.

We spent a great deal of time over the last 20 years making the development of transactional applications easier (HP was using Arjuna 20 years ago and liking it). Some PhD students wrote an entire database system based on it too. Simplicity, flexibility and power were keys to its success (nested transactions rule!) Back then there was little differentiation between the transaction engine and the way in which you developed applications. Subsequently we emphasized that they were different (Arjuna became known as the engine, while Transactional Objects for Java was the framework for developing applications).

However, we've not concentrated on the latter in recent years, which is a shame and something that we're going to remedy, because it's a proven way of developing flexible transactional applications. I think we'll throw in annotations and try to make it slightly less invasive than it was (back in the 1980's there was no such thing as Java let alone annotations). Hopefully we'll then be able to offer this to developers building on JBossTS and they can move away from asking "How do I write an XAResource?" to "What do my users want from my application/framework/service/..?" Leave the transaction complexity to JBossTS.

Side-note: while writing this I was reminded that we have a student project on some of these ideas, though not necessarily tied to TOJ. So if you are a student and looking for something to do, now may be the time to give it a go.

Friday, January 9, 2009