Distributed sagas are hard to debug. When a participant service fails mid-transaction, the
coordinator enters a *heuristic* state — a stuck, partially-compensated mess that requires
manual operator action. Finding out *which* participant failed, *why*, and *what to do about
it* normally means grepping logs across services and reading coordinator state dumps.
This post walks through a live example using the Narayana LRA coordinator and the LRA AI
dashboard — a Quarkus application that lets you chat with the coordinator state in plain English.
We will deliberately break a saga, watch it enter a heuristic state, and then fix it through
a conversation with the AI assistant.
---
## What Is a Heuristic Outcome?
In the LRA (Long Running Actions) model, when you cancel a transaction the coordinator asks
every enrolled participant to *compensate* — roll back its local changes. If a participant
service is down or returns an error, the coordinator retries periodically. When it gives up
after exhausting retries, the LRA enters **FailedToCancel** and the participant is marked
**FailedToCompensate**. At that point automatic recovery is no longer possible and an operator
must intervene.
This is the heuristic outcome: the saga is partially compensated, state is inconsistent, and
a human needs to decide what to do.
---
## Setup
You need three things running before starting:
| Component | Port |
|-----------|------|
| LRA coordinator | 8080 |
| LRA AI assistant | 8082 |
| Mock participant (bad) | 9999 |
**1 — LRA coordinator**
```bash
podman run --network host quay.io/jbosstm/lra-coordinator:latest
```
`--network host` is required so the coordinator can call back to the mock participant running on the host (`localhost:9999`).
**2 — AI assistant**
```bash
git clone https://github.com/marcosgopen/lra-ai-dashboard.git
cd lra-ai-dashboard
mvn quarkus:dev -Dquarkus.http.port=8082
```
Open **http://localhost:8082** — you should see the chat UI.
Note: the quarkus application should start a container with the llama3.1 model, if you want to install ollama by yourself you can see the guide here https://github.com/marcosgopen/lra-ai-dashboard#install-and-start-ollama
**3 — A deliberately broken participant service**
This one-liner starts an HTTP server on port 9999 that returns `500 FailedToCompensate`.
The body is critical: a bare 500 with no body is treated as a transient error and retried
every 120 seconds indefinitely — the LRA would stay stuck in `Cancelling` forever. The
coordinator only moves the LRA to `FailedToCancel` when the response body is a valid
`ParticipantStatus` enum value (`FailedToCompensate` or `FailedToComplete`).
```bash
python3 -c "
import http.server
class AlwaysFail(http.server.BaseHTTPRequestHandler):
def do_PUT(self):
body = b'FailedToCompensate'
self.send_response(500)
self.send_header('Content-Type', 'text/plain')
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
print(f' [{self.command}] {self.path} -> 500 FailedToCompensate')
def log_message(self, *a): pass
print('Bad participant listening on :9999')
http.server.HTTPServer(('', 9999), AlwaysFail).serve_forever()
"
```
---
## Creating the LRA and Enrolling the Bad Participant
Open a new terminal. The steps below use `curl` to drive the coordinator REST API directly.
**Start a new LRA:**
```bash
LRA_ID=$(curl -s -X POST \
"http://localhost:8080/lra-coordinator/start?ClientID=payment-saga&TimeLimit=0" \
-H "Accept: text/plain")
echo "LRA ID: $LRA_ID"
# → http://localhost:8080/lra-coordinator/0_ffff7f000001_b07b_68401234_0
```
**Enroll the bad participant:**
The coordinator records participant callback URLs from the `Link` header. It will call
`compensate` on cancel and `complete` on close.
```bash
curl -s -X PUT "$LRA_ID" \
-H "Link: <http://localhost:9999/compensate>;rel=compensate, <http://localhost:9999/complete>;rel=complete, <http://localhost:9999/forget>;rel=forget" \
-H "Long-Running-Action: $LRA_ID" \
-H "Accept: text/plain"
```
A successful enrolment returns a recovery URL — save it, you may need it later:
```
http://localhost:8080/lra-coordinator/recovery/0_ffff7f000001_b07b_68401234_0/1
```
At this point the LRA is `Active` with one enrolled participant. Now switch entirely to the
AI assistant — all remaining steps happen through the chat UI.
---
## Step 1 — AI Lists the Active Transactions
Open the chat UI at **http://localhost:8082** and ask:
> **You:** Show me all active transactions.
The assistant calls `listAllLRAs()` and responds with something like:
> **Assistant:** There is one active LRA:
>
> — **LRA** `http://localhost:8080/lra-coordinator/0_ffff7f000001_b07b_68401234_0`
> — **Status:** Active
> — **ClientID:** payment-saga
>
> This transaction is in progress and has not been closed or cancelled yet.
The AI can see the LRA because it is still in the active store. The coordinator API returns
LRA-level metadata (ID, clientId, status, timestamps) but does not expose participant details
such as participant URLs or per-participant status.
---
## Step 2 — AI Cancels the Saga
Ask the assistant to cancel the transaction:
> **You:** Cancel the payment-saga LRA.
The assistant calls `cancelLRA()` with the LRA ID. The coordinator processes the cancel
synchronously: it calls `PUT http://localhost:9999/compensate`, gets back
`500 FailedToCompensate`, immediately transitions the LRA to **FailedToCancel**, and moves it
to the failed store — all before returning the response. The AI sees the result and explains:
> **Assistant:** I cancelled LRA
> `http://localhost:8080/lra-coordinator/0_ffff7f000001_b07b_68401234_0`.
>
> The cancel completed, but the result is **FailedToCancel** — at least one participant
> returned a permanent failure (`FailedToCompensate`) during compensation. The coordinator
> has given up and moved this LRA to the failed store. There is no automatic retry.
>
> To resolve this you will need to:
> 1. Check the coordinator logs to identify which participant failed and why.
> 2. Fix the failing service.
> 3. Call the participant's compensate endpoint directly — the coordinator will not retry.
> 4. Once done, tell me and I will remove the failed record from the coordinator.
Watch the bad participant terminal — you will see the single compensate call:
```
Bad participant listening on :9999
[PUT] /compensate -> 500 FailedToCompensate
```
---
## Step 3 — Fixing the Participant
In this demo you already know the failing participant is `localhost:9999` because you enrolled
it yourself. In a production system you would identify it from the coordinator logs as the
assistant suggested.
**Restore the participant service**
Stop the broken server (Ctrl-C in its terminal) and start a fixed version that returns `200`:
```bash
python3 -c "
import http.server
class AlwaysOK(http.server.BaseHTTPRequestHandler):
def do_PUT(self):
body = b'Compensated'
self.send_response(200)
self.send_header('Content-Type', 'text/plain')
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
print(f' [{self.command}] {self.path} -> 200 Compensated')
def log_message(self, *a): pass
print('Fixed participant listening on :9999')
http.server.HTTPServer(('', 9999), AlwaysOK).serve_forever()
"
```
**Manually drive the compensation**
The coordinator will not call the participant again — you must do it directly:
```bash
curl -s -X PUT http://localhost:9999/compensate
# → 200 Compensated
```
This is the actual business compensation: the participant rolls back its local state. In a real
system this is the step where you restore a database row, reverse a payment, release a booking,
and so on.
---
## Step 4 — AI Cleans Up the Failed Record
With the participant compensated, tell the assistant and ask it to close the books:
> **You:** I have manually compensated the participant. Please clean up the failed LRA record.
The assistant calls `listFailedLRAs()` to confirm the record is still there, then calls
`deleteFailedLRA()` to remove it:
> **Assistant:**
> I've deleted the failed LRA record. The participant has been successfully compensated and the transaction can now be considered complete.
>
> **Result:** HTTP 204. The failed LRA record has been removed from the coordinator's
> ObjectStore. The saga is fully resolved — all business-level compensation has been confirmed
> and the coordinator no longer tracks this transaction.
---
## Summary
| Step | What happened |
|------|---------------|
| Create LRA | Coordinator created a new saga in `Active` state |
| Enroll bad participant | Coordinator recorded callback URLs for `localhost:9999` |
| AI: list LRAs | Assistant found the Active LRA via `listAllLRAs()` |
| AI: cancel | Assistant called `cancelLRA()` → coordinator called `/compensate` → got `500 FailedToCompensate` → `FailedToCancel` |
| AI: diagnosis | Assistant explained the failure state and recommended checking coordinator logs for the failing participant |
| Fix participant | Operator restored service and called compensate endpoint directly |
| AI: cleanup | Assistant called `deleteFailedLRA` → coordinator record removed → saga resolved |
The AI assistant drove the entire workflow — monitoring, action, diagnosis, and cleanup — through
a natural-language conversation. The operator made the decisions; the assistant handled the
coordinator API and explained what was happening at each step.
---
## Going Further
- **Real participants:** Replace the Python mock with a Quarkus microservice using the
`@Compensate` and `@Complete` annotations from the MicroProfile LRA API.
- **Nested LRAs:** A parent LRA cancellation propagates to its children — ask the assistant
to trace the compensation cascade down the hierarchy.
- **Proactive alerting:** Add a Quarkus `@Scheduled` job that calls `listFailedLRAs()`
periodically and raises an alert when the count exceeds a threshold.
