# Saga delivery and instances > The delivery contract, running on several instances, and side effects. [Layer 1: Domain events](/docs/layer-1/) Command dispatch is at-least-once. The runner dispatches a reaction’s commands before it saves the resulting state, so a crash between the two, or a compare-and-set retry after a concurrent write, can dispatch the same command twice, but never lose one. This is safe when the receiver is idempotent, which an `ApplicationService`-backed dispatcher is by construction. It replays the authoritative event stream on every call, so the target’s own invariants reject a stale or already-applied command and a duplicate becomes a no-op: * Java ```java CommandDispatcher dispatcher = CommandDispatchers.decider(deciderApplicationService, orderCommandDecider, OrderCommand::orderId); // A duplicate ReservePayment replays the stream and is rejected by the decider's own invariants, so it is a no-op. ``` * Kotlin ```kotlin val dispatcher = CommandDispatchers.decider(deciderApplicationService, orderCommandDecider) { it.orderId } // A duplicate ReservePayment replays the stream and is rejected by the decider's own invariants, so it is a no-op. ``` Timer bookkeeping has no such gap, because `startTimeout` and `cancelTimeout` are saved atomically with the rest of the state in the same write, so timers are exactly-once. When a `SagaConcurrencyException` exhausts its compare-and-set retries on a live event, the exception propagates to the subscription model, and the whole step is retried wherever that model offers the event again. Whether the event is offered again, and which other events wait behind it, depends on what feeds the subscription. On a push feed your listener decides both. On a broker source, the source’s `DeliveryFailurePolicy` setting decides whether the event is offered again. When the saga throws a `RuntimeException` or an `AssertionError`, the Kafka source delays at most the later records in that record’s partition, and it delays nothing once it has parked the record under `PARK`. The RabbitMQ source delays no other message once it has requeued the failed one under `REDELIVER` or parked it under `PARK`. Anything else the saga throws, any other `Error` or a checked exception from Kotlin, stops either source. An instance that keeps failing can be quarantined, so the saga’s other instances keep getting their events. Quarantine is off by default. [Quarantined Instances](/docs/layer-1/observing-sagas/#quarantined-instances) says what happens to a quarantined instance, what has to be true before one is quarantined, how to turn quarantine on for both `SagaRunnerConfig` and `@Saga`, and why it’s off by default. An instance that isn’t quarantined keeps failing for as long as the model keeps offering the event. That ends when: * the retry succeeds * you abandon the instance with `SagaStateStore.delete(sagaId)` * you stop the subscription, which stops the whole saga rather than only the failing instance Only an event the saga routed to an instance can be quarantined. When the converter or `correlate`/`correlateAll` throws, the saga cannot tell which instance the event belongs to, so there is no instance to quarantine. The runner refuses such an event on every delivery, whatever the budget, because it may still belong to an instance and acknowledging it would lose it. Nothing is written to the state store for it, so no `SagaInstance` shows it. The first refusal is logged at `WARN`. After that it is logged at `ERROR` once per interval for as long as the model keeps offering the event. The interval is the quarantine budget when one is configured and five minutes when none is. A model that does not offer the event again gets only the `WARN`. Where the model does offer the event again, fix the converter or the correlation and the saga applies the event in the order it was written, with nothing to feed it by hand. On the timer path the poller catches what the reaction throws, logs it at `WARN` and leaves the timer due, so one failing timer does not stop the poller. A failing timer never quarantines its instance and records no failure. An `OutOfMemoryError` is not caught per instance, so it ends that poll for every instance after it. A failing timer can still keep other instances’ timers from firing. A poll fires at most `timerBatchLimit` instances, 100 by default, and `findWithDueTimers` does not require a store to give a different instance a turn. Once `timerBatchLimit` instances keep failing their timers, every poll can pick those same instances, and then no other instance’s timer fires. `timerBatchLimit` is set through the `SagaRunnerConfig` constructor, and there is no Spring Boot property for it. > **Caution** > > Giving a different instance a turn when the same timers keep failing is tracked in [issue 1003](https://github.com/johanhaleby/occurrent/issues/1003). A failing timer stays armed, so once its reaction stops failing a later poll fires it and the instance needs nothing done to it. Because commands are dispatched before the save and a lost compare-and-set retries the step, a single input can also re-dispatch its whole command list several times, up to the configured `maxCasAttempts`. A receiver can see the same command several times in a row, not just twice. A flow saga does not remember its whole history. A condition, guard, or timeout reaction reads that history through `ReceivedEvents`, which keeps the current step’s own events plus the `historyWindow` most recent earlier ones, 100 by default. Set it with `FlowSaga.Builder.historyWindow(int events)` in Java or `historyWindow(events)` inside the Kotlin `saga { }` block. Raise it for a condition or guard that needs to count back further than 100 events, or lower it to trim what a long-running instance persists. `historyWindow` limits only the history carried over from earlier steps, and it is applied when a step is left. On its own it puts no limit on the current step’s own events, so a condition counting since the step was entered sees every one of them, even with `historyWindow(0)`. The initiating event is kept whatever the window is, since `received.initiating()` is a common lookup. Anything older than the window is dropped and not persisted. `stepWindow(int events)` caps how many of the current step’s own events are kept, and it’s applied on every delivery. An event counts toward the cap when its type is declared, meaning one of the flow’s own `on(...)` branches or an `event(...)` check inside a window condition names it. A repeat of the type that started the instance counts too. An event of any other type is still retained, but it doesn’t count toward the cap. `stepWindow` drops it only when the cap drops a counted event that arrived after it. For example, take `stepWindow(2)` and the events A, x, B, C, D, where x is of a type no step declares. C arriving drops A and keeps x. D arriving drops B, and x goes with it, because the kept events are always an unbroken run ending with the newest. `historyWindow` can drop an event that doesn’t count toward the cap too. Once the flow has left the step the event arrived in, the event is part of the history kept from earlier steps, and a later transition can drop it the same as a counted one. Such an event reaches a flow saga only through a `replacementFilter` wider than the flow’s own types, or a `CloudEventTypeMapper` that maps several domain types onto one CloudEvent type string. A `narrowingFilter` cannot let one in, because it only narrows the filter derived from the flow’s types. There is no cap unless you set one, so an instance that stays in a step while a large number of correlated events arrive keeps all of them, whatever `historyWindow` says. The minimum is 1. * Java ```java FlowSaga.builder() .historyWindow(20) .stepWindow(50) // startsOn, correlateAll and the steps as above ``` * Kotlin ```kotlin saga { historyWindow(20) stepWindow(50) // startsOn, correlateAll and the steps as above } ``` Set both and, at any one moment, an instance keeps at most `historyWindow + 2 * stepWindow + 1` events of the types `stepWindow` counts. Events of any other type come on top of that. `stepWindow` is doubled because a transition keeps the events of the step being left so that step’s reaction can read them, and the step being entered then fills its own cap before anything is dropped. A step condition still completes on the same event it would have without the cap, because its counts are kept in the instance’s state instead of being counted from the events again. What reads less is everything that reads the events directly, a guard’s `onlyIf`, a `timeout` reaction, and a window condition’s reaction. A retry guard counting `PaymentFailed` across a self-looping step therefore needs its threshold to fit inside `stepWindow`, the same requirement `historyWindow` already has. A step that only receives events of types no step declares keeps growing with no limit from `stepWindow`, since none of those events count toward the cap. `SpringMongoSagaStateStore` logs a warning when an instance’s retained events reach 1,000. It doesn’t warn again until the count has dropped below 1,000 and reached it once more. A saga running on a state store of your own gets no such warning. Two things stay true at any cap of 1 or more. `received.initiating()` reaches the start event, which is kept as the first retained event and never counts against the cap, and the event that fired a branch is the last element of `received.asList()`. Before a flow can cap its steps, every `event(...)` condition with a predicate needs a name for that predicate. [Counted Conditions and `stepWindow`](/docs/layer-1/saga-step-conditions/#counted-conditions-and-stepwindow) has the details. Changing what a capped step waits on while instances are still in it, whether that is an event type or a predicate’s name, makes those instances refuse their next delivery with an `IllegalStateException` naming the step. Retrying does not help, because the events those counts would be rebuilt from are gone. Put the previous condition declaration back until those instances have moved on, so they stop refusing deliveries. Delete the instance instead if you don’t need to keep it running. An instance still inside the cap counts its kept events again and continues. A flow saga instance waiting in a step stores that step’s name. Renaming or removing the step while instances are still in it makes each of them refuse its next event or timeout with an `IllegalStateException` naming the step, because the new flow definition has no step by that name. Retrying does not help. To move those instances on, put the step back under its old name, or add a temporary step under the old name that transitions to the new one, until every instance has left it. To unblock an instance before you redeploy, delete it from the `SagaStateStore`. What persists has one compatibility guarantee. The retained domain events serialize as CloudEvents through the application’s `CloudEventConverter`, by their stable `CloudEventTypeMapper` type rather than a Java class name. So a domain event can move to a different package without breaking in-flight saga state, exactly as it can for events in the event store. The executor’s own bookkeeping is not a compatibility surface. A core saga’s state is your own model and serializes like the [snapshot](/docs/layer-1/snapshots/) store. For the full design rationale, including the residual cross-node race a compare-and-set retry can produce and the deferred outbox that would make dispatch exactly-once, see [ADR 0063](https://github.com/johanhaleby/occurrent/blob/main/doc/architecture/decisions/0063-saga-dsl.md). The complete, runnable [order-fulfillment example](https://github.com/johanhaleby/occurrent/tree/occurrent-0.34.0/example/saga/order-fulfillment) wires up both DSLs through `SagaRunner`, with both styles of dispatcher. ## Running Across Multiple Instances Run several instances of your application, and a saga has two things happening per instance. It receives events, and it polls the state store for due timers. The event side is already single-active when the subscription model is a [competing-consumer](/docs/layer-0/competing-consumers/) one,. So for a given saga, only one instance receives events at a time. The timer poller is separate. Left uncoordinated, every instance runs its own poller and queries the store for due timers on its own interval. The timer-query load then grows with the instance count, even though only one instance ever needs to fire a due timer. Firing stays correct either way, since a lost compare-and-set is retried and the dispatch is idempotent, but the extra queries are wasted work. Give the `SagaRunner` a `CompetingConsumerStrategy` with `competingConsumerStrategy(...)`, the same strategy the competing-consumer subscription uses. The poller is gated too. It takes a lease and only polls while it holds it, so exactly one instance queries the store. The lease is keyed apart from the event subscription’s own lease, and released when the `SagaSubscription` is closed, so another instance takes over within about one lease period. A standby instance checks whether it holds the lease in memory, so it costs no query at all: * Java ```java CompetingConsumerStrategy strategy = NativeMongoLeaseCompetingConsumerStrategy.withDefaults(mongoDatabase); SagaSubscription runningSaga = SagaRunner.agnostic(subscriptionModel, cloudEventConverter) .competingConsumerStrategy(strategy) .run("order-fulfillment", orderFulfillment, stateStore, dispatcher); ``` * Kotlin ```kotlin val strategy = NativeMongoLeaseCompetingConsumerStrategy.withDefaults(mongoDatabase) val runningSaga = SagaRunner.agnostic(subscriptionModel, cloudEventConverter) .competingConsumerStrategy(strategy) .run("order-fulfillment", orderFulfillment, stateStore, dispatcher) ``` Run without a strategy and the poller runs on every instance exactly as it did before. Single-node and in-memory setups need none of this. Gating removes the redundant queries. It does not change the residual cross-node race noted in the [delivery contract](/docs/layer-1/saga-delivery/) (an event on one instance interleaving with a timeout fired on another), which stays handled by compare-and-set and an idempotent receiver. See [ADR 0064](https://github.com/johanhaleby/occurrent/blob/main/doc/architecture/decisions/0064-lease-gate-the-saga-timer-poller.md) for the full rationale. ## Side Effects and Compensation A reaction that needs to reach the outside world issues a command, for example `NotifyWarehouse(orderId)`, and points it at a dispatcher that makes the call. That is the only way a saga touches the outside world. There is no “call this API” effect, and that is deliberate, because it keeps `react` a pure function whose output is data. With ordinary command types you can test that output with equality assertions, and with [`Invocation`](/docs/layer-1/running-a-saga/#sagas-without-command-types) you check what the command does instead, since a lambda has no value equality. Either way the reaction performs nothing itself, so a third-party call, whether it runs mid-process or as the last thing a completed saga does, is a command like any other. The terminal reaction, the one whose `Continuation` is `end`, is where a “now that the whole thing is done” effect belongs. Compensation works the same way. A saga does not roll back, it moves forward, so an “undo” is just another command you issue on the branch or timeout that detected the failure. The order-fulfillment saga above already does this. When payment fails or the timeout fires it issues `CancelOrder`, which is the compensation for the `ReservePayment` it issued earlier. You decide which command undoes which, there is no automatic inverse: * Java ```java .on(PaymentFailed.class, Continuation.end(), failure -> List.of(new CancelOrder(failure.orderId(), failure.reason()))) .timeout(Duration.ofMinutes(30), Continuation.end(), received -> List.of(new CancelOrder(received.initiating(OrderPlaced.class).orderId(), "payment timeout"))) ``` * Kotlin ```kotlin on(then = end) { failure -> issue(CancelOrder(failure.orderId, failure.reason)) } timeout(after = Duration.ofMinutes(30), then = end) { received -> issue(CancelOrder(received.initiating().orderId, "payment timeout")) } ``` Both branches issue the same forward-moving `CancelOrder` compensation for the `ReservePayment` issued when the step started, whichever failure mode gets there first. The one thing to watch out for is idempotency, and it follows directly from the [delivery contract](/docs/layer-1/saga-delivery/). Command dispatch is at-least-once, so a compensating or external command can arrive twice. An `ApplicationService`-backed target handles that for free, because it replays the stream and the target’s own invariants reject the duplicate. A raw third-party call does not. When a command triggers a non-idempotent external effect such as an email, a payment capture, or a partner request, give it a stable id derived from the saga and the triggering event, and dedupe at that boundary. The deferred document-local outbox described in [ADR 0063](https://github.com/johanhaleby/occurrent/blob/main/doc/architecture/decisions/0063-saga-dsl.md) would make dispatch exactly-once and remove this caveat, but it is not built yet. > **Using Spring Boot?** > > The starter gates the timer poller for you, and `occurrent.saga.timer-poll-interval` sets its interval. See [Several instances of a @Saga](/docs/spring-boot/saga-annotation/#several-instances-of-a-saga).