# Observing sagas

> Inspect saga instances, and quarantine one that keeps failing.

[Layer 1: Domain events](/docs/layer-1/)

A saga runs for as long as its process does, so sooner or later you need to ask operational questions about one. Is this instance still running, which step is it waiting in, and which instances have stopped moving? `SagaInstance` answers those and nothing else:

* Java

  ```java
  SagaInstances instances = runningSaga.instances();


  instances.find(orderId).ifPresent(instance ->
          System.out.println(instance.sagaId() + " is " + instance.status() + " in step " + instance.currentStep()));


  // active instances that have not moved for an hour, stalest first
  List<SagaInstance> stalled = instances.findByStatus(SagaStatus.ACTIVE, Instant.now().minus(Duration.ofHours(1)), 50);
  ```

* Kotlin

  ```kotlin
  val instances = runningSaga.instances()


  instances.find(orderId).ifPresent { instance ->
      println("${instance.sagaId()} is ${instance.status()} in step ${instance.currentStep()}")
  }


  // active instances that have not moved for an hour, stalest first
  val stalled = instances.findByStatus(SagaStatus.ACTIVE, Instant.now().minus(Duration.ofHours(1)), 50)
  ```

`SagaInstances` reads the saga’s state store, so it is not the event subscription answering these questions. `SagaSubscription.instances()` is a shortcut that hands you a view over the store the saga already runs against, which is also why it keeps working after you close the handle. Closing stops that instance’s timer poller, but it does not close the store. With no handle at hand, in a separate admin process for instance, build one straight from the store with `SagaInstances.of(stateStore)`.

A `SagaInstance` holds:

* the id and the `SagaStatus`
* the created, updated and completed timestamps
* when the next pending timer is due
* which step a flow saga is waiting in
* what the instance is failing on

`currentStep()` is `null` for a core saga, which names its states in your own state type rather than in a step the executor knows about.

The status is `ACTIVE`, `COMPLETED`, or `QUARANTINED`. A quarantined instance stopped on an event it could not handle, so it is neither running nor finished, and [Quarantined Instances](/docs/layer-1/observing-sagas/#quarantined-instances) below says what to do with one.

A `null` from `failure()` means no failure is on record. The instance can still be failing, because these failures record nothing:

* a failing timeout
* a failing event the saga could not route to an instance
* a failing event on a saga with quarantine off, whether you never turned it on or the runner switched it off at startup because the subscription model cannot guarantee it holds every event it delivers
* a failing event that has no redelivery key
* an `OutOfMemoryError`
* a failing event the instance already counts as handled
* a failure where the store read or the record write throws, for example a store that can only read an instance together with its state, when that state no longer decodes
* a failure whose record write loses its compare-and-set to another writer

When the store read or the record write throws, or the write loses its compare-and-set, the next failure tries the write again.

On an `ACTIVE` instance a non-null answer means an event has failed at least once and the instance has not been quarantined, which can still be true after the budget has run out. On a `QUARANTINED` one it means the instance stopped there.

It leaves out the saga’s own state and the executor’s delivery bookkeeping on purpose. A read model shaped for querying belongs in the [Projection DSL](/docs/layer-1/views/), and reaching into a saga’s private state from outside ties your code to how that process happens to be written.

There is no way to write through this. Nothing here starts, advances, completes, or deletes an instance, because the executor owns those transitions and a compare-and-set save from outside would race the subscription and the timer poller. Retention tooling that really has to remove an instance calls `SagaStateStore.delete(...)`.

`findByStatus` returns the instances in a status whose `updatedAt` falls strictly before the instant you pass, least recently updated first, at most `limit` of them. Pass `Instant.now()` to list everything in a status, or `Instant.now().minus(threshold)` to find the ones that have gone quiet. Stalest-first is what a stuck-instance check wants, because the worst offenders arrive first rather than last. `limit` caps how many instances you get back, there is no paging. Many instances can share the same millisecond `updatedAt`, and resuming a second query from “after that timestamp” would skip the rest of the instances saved in that same millisecond.

`findByStatus(ACTIVE, ...)` does not return a quarantined instance, so once quarantine is on, a check for instances that have stopped moving has to ask for both statuses. Pass `Instant.now()` for the quarantined query rather than a threshold, because a quarantined instance’s `updatedAt` stops at the moment it was quarantined and a threshold would hide the ones quarantined most recently.

Enumeration is an optional store capability. A store implements `SagaStateStoreQueries` to support it, both shipped stores do, and `findByStatus` throws an `UnsupportedOperationException` on a store that does not. `find(sagaId)` works on any store, so a store you wrote yourself to run sagas never has to answer an ordered query it does not need.

Enumerating instances is cheap. Listing flow-saga instances never deserializes their state or received events, so a periodic stuck-instance check costs little even with many instances. Rationale in [ADR 0070](https://github.com/johanhaleby/occurrent/blob/main/doc/architecture/decisions/0070-saga-instance-observation.md).

## Quarantined Instances (Added in 0.34.0)

Each saga gets one subscription, and that subscription delivers the events for all of its instances. An instance that cannot handle an event keeps failing for as long as your subscription model offers that event again.

Quarantine can end that once the instance has been failing for a length of time you set, and only when the conditions in [What else has to be true](/docs/layer-1/observing-sagas/#what-else-has-to-be-true) hold as well. It is off by default, and [Turning quarantine on](/docs/layer-1/observing-sagas/#turning-quarantine-on) says how to turn it on and why it is off.

When an instance is quarantined, this happens:

* The instance never applies the event it failed on, or any later event addressed to it. Those events still reach the saga, which skips them for this instance without recording them as handled.
* Its timers stay armed but never fire, so a timeout it relies on to cancel or compensate never runs.
* The other instances of the saga keep getting their events.
* Nothing is deleted. The events stay in the event store, and the instance’s state stays in the state store.
* Nothing resumes it. Fixing what made it fail doesn’t release it, and deleting it with `SagaStateStore.delete(sagaId)` is the only way out. Whatever business process the instance was running then has to be finished outside the saga.

Before an instance is quarantined, or when quarantine is off, [Delivery Contract](/docs/layer-1/saga-delivery/) says which other events wait behind the failing one.

This lists the quarantined instances of a saga and what each one stopped on:

* Java

  ```java
  List<SagaInstance> stopped = instances.findByStatus(SagaStatus.QUARANTINED, Instant.now(), 50);


  for (SagaInstance instance : stopped) {
      SagaFailure failure = instance.failure();
      System.out.println(instance.sagaId() + " stopped on " + failure.input() + " with " + failure.failureType());
  }
  ```

* Kotlin

  ```kotlin
  val stopped = instances.findByStatus(SagaStatus.QUARANTINED, Instant.now(), 50)


  for (instance in stopped) {
      val failure = instance.failure()
      println("${instance.sagaId()} stopped on ${failure?.input()} with ${failure?.failureType()}")
  }
  ```

### The failure record and the quarantine budget

`failure()` holds the failing event’s redelivery key, its global position where the feed assigns one, the exception’s class name and message, and `firstFailedAt`, the time the instance’s first failure was recorded. A message longer than 1,000 characters is cut short in the record, and the log still has the whole exception.

The runner measures how long the instance has been failing, not how many times it failed. The budget is `SagaRunnerConfig.quarantineAfter`, measured from `firstFailedAt`.

An instance’s first failure writes the failure record and is then rethrown, whether or not the write succeeded, so a subscription model that offers the event again lets the saga try again. An event that fails before its instance exists creates the instance to hold the record.

That first write happens only for a failure that could be quarantined later, and [Observing Saga Instances](/docs/layer-1/observing-sagas/) lists the failures that record nothing. When nothing was recorded, the next failure is treated as a first failure again.

The same event failing again inside the budget writes nothing more. A different event failing rewrites the record to name that event and keeps `firstFailedAt`. The budget is still measured from `firstFailedAt`, so the instance can be quarantined on the new event’s first failure.

Once the instance has been failing for at least the budget, the next failure can move it to `SagaStatus.QUARANTINED` on whichever event it is failing on then. When it does, the runner returns normally instead of rethrowing. The instance never applies that event or any later one of its own, and the other instances of the saga move on to their next events.

Reaching the budget does not quarantine an instance on its own, so an instance past its budget can still be `ACTIVE`. Read its status instead of working it out from the time.

When the event the record names later succeeds, the record is cleared. A different input succeeding does not clear it.

Every way of failing counts once the saga knows which instance the event belongs to. Checking for a redelivery, `evolve`, `react`, your command dispatcher and the state store all do, and an `Error` counts like a `RuntimeException`. The exception is `OutOfMemoryError`. It means the JVM ran out of heap while that instance happened to hold the thread, so the runner rethrows it and the instance keeps its state.

A short MongoDB outage never reaches the runner as a failure. `SpringMongoSagaStateStore` retries every read and write it makes, backing off from 100 ms up to 2 seconds and giving up after ten attempts, so a database that answers again before those run out is not a failure at all. A longer outage does count, but the quarantine is written to that same database, so it quarantines nothing while the outage lasts.

Pass your own [RetryStrategy](/docs/stores/retry/#retry-configuration-blocking) to the store’s five-argument constructor to change that, or `RetryStrategy.none()` to have every failure reach the runner on the first attempt. A `@Saga` that declares no store of its own gets the default, since the Spring Boot starter builds the store the same way.

An event the saga cannot route to an instance stops no instance at all. [Delivery Contract](/docs/layer-1/saga-delivery/) says what happens to it.

### Quarantine on a `SagaStateStore` of your own

Writing your own `SagaStateStore` takes two more members before quarantine works on an instance whose state can no longer be decoded, which is what a renamed event class or a changed converter produces. That instance is very often the one you most want quarantined, since it fails on every event addressed to it.

`findWithoutState(sagaId)` reads an instance without decoding its state, returning an envelope whose `state()` is `null` and whose other members are all filled in, the way `findByStatus` already does. `compareAndSaveWithoutState(sagaId, envelope, expectedVersion)` writes one back under the same compare-and-set rule as `compareAndSave`, leaving the stored state where it is, so the state the instance stopped on is still there for whoever repairs the converter.

Both are `default` methods that call `find` and `compareAndSave`, so a store that does not override them compiles and runs sagas exactly as before. It just never quarantines an instance it cannot decode, because the runner’s own read throws on that instance for the same reason yours does.

Override both or neither. The runner saves the envelope it read, so a store that reads without the state and then writes the whole envelope back replaces the stored state with `null`.

`SpringMongoSagaStateStore` overrides both. `SagaStateStore.inMemory()` does not and does not need to, since it holds each envelope as an object rather than a document, so nothing there can fail to decode. `SagaInstances.find(sagaId)` uses `findWithoutState` too, so looking one instance up by id costs the same as enumerating them and also works for an instance whose state no longer decodes.

### What else has to be true

An instance past its budget is quarantined only when a few other conditions are met too. The next three sections cover the ones you’re most likely to run into, and the javadoc on `SagaStatus.QUARANTINED` lists every one.

### Use a subscription model that keeps every event

Quarantine only works on a subscription model that keeps every event it delivers. A model says it does by implementing `HistoryRetainingSubscriptions` and returning `true` from `retainsEveryEvent()`.

Quarantining an instance returns normally, which acknowledges the event to whatever fed it. Behind a broker source, the source then acknowledges the record to the broker, and nothing can hand that event to the saga a second time.

`NativeMongoSubscriptionModel` and `SpringMongoSubscriptionModel` return `true`, either of them alone or behind `DurableSubscriptionModel`, `CompetingConsumerSubscriptionModel` or `CatchupSubscriptionModel`. The runner looks through those three wrappers to the model inside, so a wrapper around any other model doesn’t qualify.

`CatchupThenPushSubscriptionModel` implements `HistoryRetainingSubscriptions` but returns `false` from `retainsEveryEvent()`, since the live events pushed to it need not be in the event store it replays. A push feed on its own does not implement the interface at all.

When you turn quarantine on for a saga on a model that returns `false` from `retainsEveryEvent()`, or doesn’t implement `HistoryRetainingSubscriptions`, the runner switches it off again at startup and logs a `WARN` saying why. An instance that keeps failing on such a model is never quarantined.

### An event without a redelivery key is never quarantined

The failure record identifies the failing event by its redelivery key, the stream id with its stream version, or the global position when the event has no stream metadata. An event with neither can’t be told apart from its own redelivery, so its budget never runs out and it is never quarantined.

`SagaRunnerConfig.redeliveryDetection` refuses such an event under `REQUIRED`, its default, before the saga sees it. You only meet this case after setting it to `BEST_EFFORT`.

An event store that assigns no global position still gives every event a redelivery key. A store built with `withoutStreamPosition()` gives every event a stream id and a stream version, so a saga on it quarantines like any other and `failure().position()` returns `null`.

### The runner checks that the model still holds the event

Before the runner quarantines an instance, it calls the model’s `retains(event)` for the event the instance stopped on. `retains` returns `true` when acknowledging the event doesn’t delete the only copy of it. A model that returns `true` from `retainsEveryEvent()` by mistake is caught this way, before the event is gone.

When `retains` returns `false`, or throws, the instance isn’t quarantined. The runner logs a `WARN` once per instance, and the instance keeps failing for as long as your model offers the event again.

An event an operator has already deleted from the event store isn’t deleted by the acknowledgement, so `retains` returns `true` for it. The instance can then be quarantined, so it doesn’t keep failing on an event nobody can supply.

### Turning quarantine on

Quarantine is off by default, because nothing brings a quarantined instance back. Deleting the instance with `SagaStateStore.delete(sagaId)` is the only way out of quarantine.

With quarantine off, the saga keeps refusing a failing instance’s event for as long as your subscription model offers it again. On the MongoDB subscription models the saga’s other instances wait behind that event the whole time. Turn quarantine on when you would rather delete one instance than have its failing event hold up the others.

Pass the budget to `withQuarantineAfter`. Five minutes is a reasonable budget.

* Java

  ```java
  SagaRunnerConfig config = SagaRunnerConfig.defaults().withQuarantineAfter(Duration.ofMinutes(5));
  ```

* Kotlin

  ```kotlin
  val config = SagaRunnerConfig.defaults().withQuarantineAfter(Duration.ofMinutes(5))
  ```

`withQuarantineAfter(null)` throws, and so does a zero or negative budget. `disableQuarantine()` turns quarantine off again.

The design, including why releasing an instance back into service is harder than it looks, is in [ADR 0134](https://github.com/johanhaleby/occurrent/blob/main/doc/architecture/decisions/0134-a-saga-instance-that-keeps-failing-is-quarantined-at-its-own-position.md).

> **Using Spring Boot?**
>
> The `@Saga` registrar publishes a `SagaInstancesRegistry` bean, and `occurrent.saga.quarantine-after` turns quarantine on. See [Observing @Saga instances](/docs/spring-boot/saga-annotation/#observing-saga-instances).