Wednesday, October 28, 2026

Narayana LRA 2.0.0.Final: propagating JWT identity through a saga

When a single business operation spans several microservices, the caller's identity must travel with it. Without it, the downstream services cannot authorize the work, and the audit trail of who initiated the operation is lost. This blog walks through a Quarkus quickstart that carries a JSON Web Token (JWT) through a MicroProfile LRA saga using the @PropagateToken annotation, and then examines where that propagation ends and explicit configuration takes over.

You can find the runnable code in the quickstart module rts/lra-examples/lra-jwt/quarkus/token-propagation.

The two specifications

MicroProfile LRA coordinates distributed transactions with the saga pattern. Instead of holding database locks across services, each participant does its work immediately and registers a compensation to undo it. A coordinator drives the participants to either @Complete (on success) or @Compensate (on failure). This works across plain HTTP, so it suits microservices far better than classic two-phase commit.

MicroProfile JWT turns a signed bearer token into an authenticated identity. Inject JsonWebToken and you obtain the caller's subject and issuer. The signature makes the token tamper-proof; the runtime verifies it against the issuer's public key (a JWKS).

Combining them raises a practical question: how does the token reach the inventory service, the payment service, and the coordinator?

Environment setup

Clone and build lra:

git clone https://github.com/jbosstm/lra.git
cd lra
mvn clean install -DskipTests
cd ..

Clone and build the quickstart:

git clone https://github.com/marcosgopen/quickstart -b lra-jwt
cd quickstart
mvn clean install -DskipTests
cd rts/lra-examples/lra-jwt/quarkus/token-propagation

The scenario

                       ┌────────────────┐  reserve / release
Client ──Bearer JWT──► │  OrderService    │ ─────────────────► InventoryService
                       │  @LRA(REQUIRED)  │  process / refund
                       │  @PropagateToken │ ─────────────────► PaymentService
                       └─────────────────┘

OrderService opens an LRA and calls two participants. If the payment is declined, the whole order is rolled back and the inventory reservation is compensated.

The forward path: @PropagateToken

The orchestrator method is concise:

@POST
@Path("/create")
@Authenticated
@LRA(LRA.Type.REQUIRED)
@PropagateToken
public Response createOrder(OrderRequest order,
                            @HeaderParam(LRA_HTTP_CONTEXT_HEADER) URI lraId,
                            @HeaderParam(HttpHeaders.AUTHORIZATION) String authHeader,
                            @Context UriInfo uriInfo) {
    // ... calls inventory then payment, forwarding authHeader on each hop
}

@PropagateToken — introduced with the JWT integration in LRA 2.0.0.Final — propagates the caller's identity on outbound calls. It works as follows:

  1. The LRA server filter (ServerLRAFilter) reads the inbound Authorization: Bearer <jwt> header, confirms it is a well-formed JWT, and stores it in a thread-local for the request.
  2. Every LRA-protocol call the service makes to the coordinator — starting the LRA, enlisting a participant, closing or cancelling it — passes through the LRA client request filter, which reads that thread-local (or a CDI-injected JsonWebToken, or a configured client property) and re-attaches Authorization: Bearer <jwt>.
  3. The service-to-service calls forward the same header explicitly, so each participant receives the token, verifies it, and repeats the process when it enlists.

The result is visible in the logs: the inventory and payment endpoints inject the token and record the authenticated caller:

Reserving 2 x ITEM-123 for order ORDER-001 ... (caller=demo-user, authHeader=present)
Processing payment of 99.99 for order ORDER-001 ... (caller=demo-user, authHeader=present)

Identity on the callback path

The callback path works differently from the forward path. The coordinator does not persist the end user's token and replay it on the @Complete/@Compensate callbacks. It attaches a token only when it has been configured to, and the token it then sends is one of:

  • a request-scoped token, when a close/cancel is driven by an inbound request that the coordinator itself authenticates, or
  • a pre-provisioned service token, used by recovery — the background threads that finish interrupted sagas, where no inbound user request exists at all.

That second case is the real reason a stored user token wouldn't work anyway: a long-running action might not complete until minutes, hours, or days later, by which point the user's short-lived access token has long expired. Recovery needs its own durable credential.

In this quickstart the standalone coordinator is not configured with outbound auth providers or a service token, so the callbacks arrive with no Authorization header, as the logs confirm:

Complete callback for LRA ... (caller=null, authHeader=absent): committed ITEM-123

This is the observable default, and it marks the boundary between the two halves of the story: forward propagation requires no configuration, whereas authenticated callbacks require coordinator configuration and a shared identity provider. The next section configures them — see Authenticating the callbacks.

Watching a compensation

Send an order with a negative amount and the payment declines (HTTP 402). OrderService turns that into a failure, the LRA is cancelled, and the reservation that already joined is released:

Reserving 1 x ITEM-123 for order ORDER-002 ... (caller=demo-user, authHeader=present)
Payment declined for order ORDER-002 (negative amount)
Compensate callback for LRA ... (caller=null, authHeader=absent): released ITEM-123

The forward hops are authenticated; the compensation callback is not, following the same rule as above.

Authenticating the callbacks

The section above shows the default: an unconfigured coordinator, with callbacks that carry no token. To complete the picture, use a coordinator configured for authentication. The lra-coordinator-quarkus application provides this — opt-in inbound JWT protection of its own endpoints, an outbound callback filter, and a pluggable service-token provider — and is published as quay.io/jbosstm/lra-coordinator:latest.

Two pieces have to line up:

  1. The participant enforces authentication on its callback endpoints. This module leaves that disabled by default (so the walkthrough above works without a configured coordinator); enable it by starting the participant with -Dparticipant.callback.auth.policy=authenticated. The /inventory/compensate, /payment/complete, and related paths then reject any request without a valid token.
  2. The coordinator presents a token on every callback. Run the image with the callback filter enabled and a service token mounted as a secret volume — the classic Kubernetes pattern. The coordinator re-reads the file every refresh-seconds, so a rotated secret never goes stale:
    # the coordinator's own identity; in production this comes from your IdP, not the demo endpoint
    curl -s "http://localhost:8081/auth/token?subject=lra-coordinator" | sed 's/^Bearer //' > ~/lra-secrets/service-token.jwt
    
    podman run -d --network host \
      -v ~/lra-secrets:/var/run/secrets/lra:ro,Z \
      -e LRA_AUTH_POLICY=authenticated \
      -e MP_JWT_VERIFY_PUBLICKEY_LOCATION=http://localhost:8081/auth/jwks \
      -e MP_JWT_VERIFY_ISSUER=https://demo.narayana.io \
      -e LRA_HTTP_CLIENT_PROVIDERS=io.narayana.lra.coordinator.security.JwtTokenCallbackRequestFilter \
      -e LRA_SECURITY_SERVICE_TOKEN_LOCATION=/var/run/secrets/lra/service-token.jwt \
      quay.io/jbosstm/lra-coordinator:latest

With both sides configured, the callback log shows an authenticated caller, and the token it uses depends on how the callback was driven:

Complete callback for LRA ... (caller=demo-user, authHeader=present): committed ITEM-123
Compensate callback for LRA ... (caller=lra-coordinator, authHeader=present): released ITEM-123

The first ran inline with a close that the coordinator handled on a request thread, so it reused the caller's propagated token. The second was driven by a timeout or recovery on a background thread with no inbound request — the recovery case described in Identity on the callback path — so it fell back to the coordinator's service token (subject=lra-coordinator). This is the purpose of a dedicated service credential: it is available precisely when the user's token is not.

Requiring an authenticated caller

Every forward endpoint in this example — createOrder and the reserve/process participant operations — is annotated @Authenticated, so a request without a valid token is rejected with 401 before any work runs. (@LRA(MANDATORY) only requires an LRA context, not an identity, so the participant methods need @Authenticated as well.) No custom security code is required; authentication is enforced by MicroProfile JWT.

Generating keys

The demo mints its own tokens, so no external identity provider is required to run it. The RSA key pair is generated in memory at start-up — nothing is written to disk or committed to source control — and the public half is published at /auth/jwks, the location that mp.jwt.verify.publickey.location points to. The application therefore verifies the tokens it issues, and the unit tests assert the key properties: a token verifies only against the published key, and the JWKS never exposes private material.

In production, remove JwtKeyGenerator and point mp.jwt.verify.publickey.location at a real provider's JWKS (Keycloak, Entra ID, Auth0, and similar).

Taking it to production

  • Real IdP. Replace the in-process key/token generator with an external identity provider.
  • HTTPS everywhere. The coordinator URL and every inter-service hop should be TLS.
  • Separate credentials for separate jobs. Use short-lived access tokens for the forward path, and give the coordinator a dedicated service token (lra.security.service-token.location) so recovery-driven callbacks can authenticate long after the user's token has expired. Mount it as a secret volume into quay.io/jbosstm/lra-coordinator (as shown above) and rotate it with your secrets tooling — the coordinator re-reads it on an interval.
  • Secure the coordinator's own endpoints too. That is a distinct concern from propagation. The lra-coordinator-quarkus image does it with lra.auth.policy=authenticated plus mp.jwt.verify.* (shown above); the sibling wildfly example shows the WildFly equivalent with endpoint-to-role mappings.

Code: https://github.com/jbosstm/quickstart · LRA: https://github.com/eclipse/microprofile-lra · Narayana LRA: https://github.com/jbosstm/lra · JWT: https://github.com/eclipse/microprofile-jwt-auth · Narayana: https://narayana.io/

Wednesday, July 8, 2026

From Stuck to Fixed: Debugging an LRA Heuristic with an AI Assistant

 

Distributed sagas are hard to debug. When a participant service fails mid-transaction, the

coordinator enters a *heuristic* state — a stuck, partially-compensated mess that requires

manual operator action. Finding out *which* participant failed, *why*, and *what to do about

it* normally means grepping logs across services and reading coordinator state dumps.


This post walks through a live example using the Narayana LRA coordinator and the LRA AI

dashboard — a Quarkus application that lets you chat with the coordinator state in plain English.

We will deliberately break a saga, watch it enter a heuristic state, and then fix it through

a conversation with the AI assistant.


---


## What Is a Heuristic Outcome?


In the LRA (Long Running Actions) model, when you cancel a transaction the coordinator asks

every enrolled participant to *compensate* — roll back its local changes. If a participant

service is down or returns an error, the coordinator retries periodically. When it gives up

after exhausting retries, the LRA enters **FailedToCancel** and the participant is marked

**FailedToCompensate**. At that point automatic recovery is no longer possible and an operator

must intervene.


This is the heuristic outcome: the saga is partially compensated, state is inconsistent, and

a human needs to decide what to do.


---


## Setup


You need three things running before starting:


| Component | Port |

|-----------|------|

| LRA coordinator | 8080 |

| LRA AI assistant | 8082 |

| Mock participant (bad) | 9999 |


**1 — LRA coordinator**


```bash

podman run --network host quay.io/jbosstm/lra-coordinator:latest

```


`--network host` is required so the coordinator can call back to the mock participant running on the host (`localhost:9999`).


**2 — AI assistant**


```bash

git clone https://github.com/marcosgopen/lra-ai-dashboard.git

cd lra-ai-dashboard

mvn quarkus:dev -Dquarkus.http.port=8082

```


Open **http://localhost:8082** — you should see the chat UI.

 

Note: the quarkus application should start a container with the llama3.1 model, if you want to install ollama by yourself you can see the guide here https://github.com/marcosgopen/lra-ai-dashboard#install-and-start-ollama

 

**3 — A deliberately broken participant service**


This one-liner starts an HTTP server on port 9999 that returns `500 FailedToCompensate`.

The body is critical: a bare 500 with no body is treated as a transient error and retried

every 120 seconds indefinitely — the LRA would stay stuck in `Cancelling` forever. The

coordinator only moves the LRA to `FailedToCancel` when the response body is a valid

`ParticipantStatus` enum value (`FailedToCompensate` or `FailedToComplete`).


```bash

python3 -c "

import http.server


class AlwaysFail(http.server.BaseHTTPRequestHandler):

def do_PUT(self):

body = b'FailedToCompensate'

self.send_response(500)

self.send_header('Content-Type', 'text/plain')

self.send_header('Content-Length', str(len(body)))

self.end_headers()

self.wfile.write(body)

print(f' [{self.command}] {self.path} -> 500 FailedToCompensate')

def log_message(self, *a): pass


print('Bad participant listening on :9999')

http.server.HTTPServer(('', 9999), AlwaysFail).serve_forever()

"

```


---


## Creating the LRA and Enrolling the Bad Participant


Open a new terminal. The steps below use `curl` to drive the coordinator REST API directly.


**Start a new LRA:**


```bash

LRA_ID=$(curl -s -X POST \

"http://localhost:8080/lra-coordinator/start?ClientID=payment-saga&TimeLimit=0" \

-H "Accept: text/plain")


echo "LRA ID: $LRA_ID"

# → http://localhost:8080/lra-coordinator/0_ffff7f000001_b07b_68401234_0

```


**Enroll the bad participant:**


The coordinator records participant callback URLs from the `Link` header. It will call

`compensate` on cancel and `complete` on close.


```bash

curl -s -X PUT "$LRA_ID" \

-H "Link: <http://localhost:9999/compensate>;rel=compensate, <http://localhost:9999/complete>;rel=complete, <http://localhost:9999/forget>;rel=forget" \

-H "Long-Running-Action: $LRA_ID" \

-H "Accept: text/plain"

```


A successful enrolment returns a recovery URL — save it, you may need it later:


```

http://localhost:8080/lra-coordinator/recovery/0_ffff7f000001_b07b_68401234_0/1

```


At this point the LRA is `Active` with one enrolled participant. Now switch entirely to the

AI assistant — all remaining steps happen through the chat UI.


---


## Step 1 — AI Lists the Active Transactions


Open the chat UI at **http://localhost:8082** and ask:


> **You:** Show me all active transactions.


The assistant calls `listAllLRAs()` and responds with something like:


> **Assistant:** There is one active LRA:

>

> — **LRA** `http://localhost:8080/lra-coordinator/0_ffff7f000001_b07b_68401234_0`

> — **Status:** Active

> — **ClientID:** payment-saga

>

> This transaction is in progress and has not been closed or cancelled yet.


The AI can see the LRA because it is still in the active store. The coordinator API returns

LRA-level metadata (ID, clientId, status, timestamps) but does not expose participant details

such as participant URLs or per-participant status.


---


## Step 2 — AI Cancels the Saga


Ask the assistant to cancel the transaction:


> **You:** Cancel the payment-saga LRA.


The assistant calls `cancelLRA()` with the LRA ID. The coordinator processes the cancel

synchronously: it calls `PUT http://localhost:9999/compensate`, gets back

`500 FailedToCompensate`, immediately transitions the LRA to **FailedToCancel**, and moves it

to the failed store — all before returning the response. The AI sees the result and explains:


> **Assistant:** I cancelled LRA

> `http://localhost:8080/lra-coordinator/0_ffff7f000001_b07b_68401234_0`.

>

> The cancel completed, but the result is **FailedToCancel** — at least one participant

> returned a permanent failure (`FailedToCompensate`) during compensation. The coordinator

> has given up and moved this LRA to the failed store. There is no automatic retry.

>

> To resolve this you will need to:

> 1. Check the coordinator logs to identify which participant failed and why.

> 2. Fix the failing service.

> 3. Call the participant's compensate endpoint directly — the coordinator will not retry.

> 4. Once done, tell me and I will remove the failed record from the coordinator.


Watch the bad participant terminal — you will see the single compensate call:


```

Bad participant listening on :9999

[PUT] /compensate -> 500 FailedToCompensate

```


---


## Step 3 — Fixing the Participant


In this demo you already know the failing participant is `localhost:9999` because you enrolled

it yourself. In a production system you would identify it from the coordinator logs as the

assistant suggested.


**Restore the participant service**


Stop the broken server (Ctrl-C in its terminal) and start a fixed version that returns `200`:


```bash

python3 -c "

import http.server


class AlwaysOK(http.server.BaseHTTPRequestHandler):

def do_PUT(self):

body = b'Compensated'

self.send_response(200)

self.send_header('Content-Type', 'text/plain')

self.send_header('Content-Length', str(len(body)))

self.end_headers()

self.wfile.write(body)

print(f' [{self.command}] {self.path} -> 200 Compensated')

def log_message(self, *a): pass


print('Fixed participant listening on :9999')

http.server.HTTPServer(('', 9999), AlwaysOK).serve_forever()

"

```


**Manually drive the compensation**


The coordinator will not call the participant again — you must do it directly:


```bash

curl -s -X PUT http://localhost:9999/compensate

# → 200 Compensated

```


This is the actual business compensation: the participant rolls back its local state. In a real

system this is the step where you restore a database row, reverse a payment, release a booking,

and so on.


---


## Step 4 — AI Cleans Up the Failed Record


With the participant compensated, tell the assistant and ask it to close the books:


> **You:** I have manually compensated the participant. Please clean up the failed LRA record.


The assistant calls `listFailedLRAs()` to confirm the record is still there, then calls

`deleteFailedLRA()` to remove it:


> **Assistant:**

> I've deleted the failed LRA record. The participant has been successfully compensated and the transaction can now be considered complete.

>

> **Result:** HTTP 204. The failed LRA record has been removed from the coordinator's

> ObjectStore. The saga is fully resolved — all business-level compensation has been confirmed

> and the coordinator no longer tracks this transaction.


---


## Summary


| Step | What happened |

|------|---------------|

| Create LRA | Coordinator created a new saga in `Active` state |

| Enroll bad participant | Coordinator recorded callback URLs for `localhost:9999` |

| AI: list LRAs | Assistant found the Active LRA via `listAllLRAs()` |

| AI: cancel | Assistant called `cancelLRA()` → coordinator called `/compensate` → got `500 FailedToCompensate` → `FailedToCancel` |

| AI: diagnosis | Assistant explained the failure state and recommended checking coordinator logs for the failing participant |

| Fix participant | Operator restored service and called compensate endpoint directly |

| AI: cleanup | Assistant called `deleteFailedLRA` → coordinator record removed → saga resolved |


The AI assistant drove the entire workflow — monitoring, action, diagnosis, and cleanup — through

a natural-language conversation. The operator made the decisions; the assistant handled the

coordinator API and explained what was happening at each step.


---


## Going Further


- **Real participants:** Replace the Python mock with a Quarkus microservice using the

`@Compensate` and `@Complete` annotations from the MicroProfile LRA API.

- **Nested LRAs:** A parent LRA cancellation propagates to its children — ask the assistant

to trace the compensation cascade down the hierarchy.

- **Proactive alerting:** Add a Quarkus `@Scheduled` job that calls `listFailedLRAs()`

periodically and raises an alert when the count exceeds a threshold.