Managing the availability of LRA participants

This post is a continuation of a series of jbosts blogs that discuss the MicroProfile LRA specification.

Services manage their workloads by providing endpoints to an LRA coordinator which in turn uses those endpoints to drive the LRA protocol forward thereby enabling the construction of reliable services. These endpoints may need to be modified over the long run so it ought to be possible to replace them with different ones in response to changes to the environment in which the service executes. Although the specification does not discuss how the endpoints can be replaced, the Narayana LRA REST API for the coordinator includes Microprofile OpenAPI documentation for replacing endpoints.

There are various administrative and management reasons for why the capability can be useful, such as controlling where termination handling is to take place, or to facilitate service replacement, etc. It may also be desirable for work completion, compensation, status reporting and clean up activities to be handled on different endpoints and at different times and this goal is facilitated via annotations including @Compensate, @Complete, @Status, @Forget and @AfterLRA.

When a participant does work in the context of a long running action, a “recovery URL” is created which services may use to associate their work with various management actions such as changing the participant endpoints as the action proceeds, after all a long running action can be of arbitrary duration and the needs of a service may change as the action evolves. The example I created for this post halts the JVM during “complete”, asks the user to send a curl request to the LRA coordinator to provide it with a new participant completion endpoint, restarts the participant on the new endpoint and waits for recovery to resend the completion callback to the new endpoint.

By leveraging the feature admins may proactively react to changing conditions (connectivity, throughput, functionality updates, etc) and be able to tune and or reconfigure the environment accordingly, perhaps bringing up a more reliability aware service that more intelligently operates within the more limited environment.

Build and start a coordinator on port 8080

Use the quarkus-maven-plugin to create a project for the coordinator, adding a dependency on maven artifact org.jboss.narayana.lra:lra-coordinator-jar:0.0.10.Final to the resulting pom. Also specify that the build should produce an uber jar so that the coordinator can run standalone:

    mvn io.quarkus:quarkus-maven-plugin:3.3.1:create -DprojectGroupId=org.acme -DprojectArtifactId=narayana-lra-coordinator -Dextensions="rest-jackson,rest-client"
    cd narayana-lra-coordinator
    rm -rf src/test src/main/java # the sources created by the example aren't required
    echo "quarkus.package.jar.type=uber-jar" > src/main/resources/application.properties
    # don't forget to add a dependency on maven artifact: org.jboss.narayana.lra:lra-coordinator-jar:0.0.10.Final
    ./mvnw clean package

and then start it on port 8080 by running the resulting jar

   java -jar target/narayana-lra-coordinator-1.0.0-SNAPSHOT-runner.jar &

Build and start a participant on port 8081 and run an LRA but halt the JVM before closing it

The service will be quite basic:

@Path("/halt")
public class MigratableResource {
    private static final AtomicBoolean halt = new AtomicBoolean(false);

    @LRA(value = LRA.Type.REQUIRED)
    @PUT
    public void doInTransaction() {
        halt.set(true); // halt when compensate or complete are called
        // when the business method finishes the LRA is closed and the complete endpoint will be called
    }

    @PUT
    @Path("/compensate")
    @Compensate
    public Response compensate() {
        return Response.ok().build();
    }

    @PUT
    @Path("/complete")
    @Complete
    public Response complete(@HeaderParam(LRA_HTTP_RECOVERY_HEADER) String recoveryUrl) {
        if (halt.get()) {
            int port = 8082;
            String completionUrl = String.format("http://localhost:%d/halt/complete", port);

            System.out.printf("Ask the coordinator to send the completion notification on a new endpoint using:%n");
            System.out.printf("curl -X PUT %s -d '<%s>; rel=complete'%n", recoveryUrl, completionUrl);
            Runtime.getRuntime().halt(1);
        }
        System.out.printf("completed%n");
        return Response.ok().build();
    }
}

The interesting part happens during completion where the JVM is halted. Notice that the curl command for migrating the completion endpoint is printed prior to halting.

Now build and run the participant on port 8081 - the maven project is available from the narayana artifacts maven repository.

cd <participant directory>
mvn clean package
java -Dquarkus.http.port=8081 -jar target/quarkus-app/quarkus-run.jar &

and then call the service method using the curl utility, or otherwise:

curl -X PUT -I http://localhost:8081/halt

The service method is annotated with just @LRA(value = LRA.Type.REQUIRED) so when it finishes the completion callback will be invoked by the coordinator. Make a note of the curl request printed by the completion callback just before it halts the JVM. An example is (the Uids will change on each run):

curl -X PUT http://localhost:8080/lra-coordinator/recovery/0_ffffc0a801c7_9d57_677ad0a4_2/0_ffffc0a801c7_9d57_677ad0a4_5 \
  -d '<http://localhost:8082/halt/complete>; rel=complete'

Notice that the payload of the HTTP PUT request includes the specification of the new completion callback, namely <http://localhost:8082/halt/complete>; rel=complete.

The new endpoint will be used on the next recovery pass which is every two minutes by default.

Finally restart the service on the new endpoint (port 8082):

java -Dquarkus.http.port=8082 -jar target/quarkus-app/quarkus-run.jar &

When the coordinator next runs a recovery scan it should use the new endpoint and the service will report that it has completed its' service work by printing the text “completed” when the completion endpoint is by the coordinator.