Skip to content
OccurrentOccurrent

Saga delivery and instances

Layer 1: Domain events

Command dispatch is at-least-once. The runner dispatches a reaction’s commands before it saves the resulting state, so a crash between the two, or a compare-and-set retry after a concurrent write, can dispatch the same command twice, but never lose one. This is safe when the receiver is idempotent, which an ApplicationService-backed dispatcher is by construction. It replays the authoritative event stream on every call, so the target’s own invariants reject a stale or already-applied command and a duplicate becomes a no-op:

CommandDispatcher<OrderCommand> dispatcher =
        CommandDispatchers.decider(deciderApplicationService, orderCommandDecider, OrderCommand::orderId);
// A duplicate ReservePayment replays the stream and is rejected by the decider's own invariants, so it is a no-op.

Timer bookkeeping has no such gap, because startTimeout and cancelTimeout are saved atomically with the rest of the state in the same write, so timers are exactly-once.

When a SagaConcurrencyException exhausts its compare-and-set retries on a live event, the exception propagates to the subscription model, and the whole step is retried wherever that model offers the event again.

Whether the event is offered again, and which other events wait behind it, depends on what feeds the subscription. On a push feed your listener decides both. On a broker source, the source’s DeliveryFailurePolicy setting decides whether the event is offered again.

When the saga throws a RuntimeException or an AssertionError, the Kafka source delays at most the later records in that record’s partition, and it delays nothing once it has parked the record under PARK. The RabbitMQ source delays no other message once it has requeued the failed one under REDELIVER or parked it under PARK.

Anything else the saga throws, any other Error or a checked exception from Kotlin, stops either source.

An instance that keeps failing can be quarantined, so the saga’s other instances keep getting their events. Quarantine is off by default. Quarantined Instances says what happens to a quarantined instance, what has to be true before one is quarantined, how to turn quarantine on for both SagaRunnerConfig and @Saga, and why it’s off by default.

An instance that isn’t quarantined keeps failing for as long as the model keeps offering the event. That ends when:

  • the retry succeeds
  • you abandon the instance with SagaStateStore.delete(sagaId)
  • you stop the subscription, which stops the whole saga rather than only the failing instance

Only an event the saga routed to an instance can be quarantined. When the converter or correlate/correlateAll throws, the saga cannot tell which instance the event belongs to, so there is no instance to quarantine.

The runner refuses such an event on every delivery, whatever the budget, because it may still belong to an instance and acknowledging it would lose it. Nothing is written to the state store for it, so no SagaInstance shows it.

The first refusal is logged at WARN. After that it is logged at ERROR once per interval for as long as the model keeps offering the event. The interval is the quarantine budget when one is configured and five minutes when none is. A model that does not offer the event again gets only the WARN.

Where the model does offer the event again, fix the converter or the correlation and the saga applies the event in the order it was written, with nothing to feed it by hand.

On the timer path the poller catches what the reaction throws, logs it at WARN and leaves the timer due, so one failing timer does not stop the poller. A failing timer never quarantines its instance and records no failure. An OutOfMemoryError is not caught per instance, so it ends that poll for every instance after it.

A failing timer can still keep other instances’ timers from firing. A poll fires at most timerBatchLimit instances, 100 by default, and findWithDueTimers does not require a store to give a different instance a turn. Once timerBatchLimit instances keep failing their timers, every poll can pick those same instances, and then no other instance’s timer fires.

timerBatchLimit is set through the SagaRunnerConfig constructor, and there is no Spring Boot property for it.

A failing timer stays armed, so once its reaction stops failing a later poll fires it and the instance needs nothing done to it.

Because commands are dispatched before the save and a lost compare-and-set retries the step, a single input can also re-dispatch its whole command list several times, up to the configured maxCasAttempts. A receiver can see the same command several times in a row, not just twice.

A flow saga does not remember its whole history. A condition, guard, or timeout reaction reads that history through ReceivedEvents, which keeps the current step’s own events plus the historyWindow most recent earlier ones, 100 by default.

Set it with FlowSaga.Builder.historyWindow(int events) in Java or historyWindow(events) inside the Kotlin saga { } block. Raise it for a condition or guard that needs to count back further than 100 events, or lower it to trim what a long-running instance persists.

historyWindow limits only the history carried over from earlier steps, and it is applied when a step is left. On its own it puts no limit on the current step’s own events, so a condition counting since the step was entered sees every one of them, even with historyWindow(0).

The initiating event is kept whatever the window is, since received.initiating<T>() is a common lookup. Anything older than the window is dropped and not persisted.

stepWindow(int events) caps how many of the current step’s own events are kept, and it’s applied on every delivery. An event counts toward the cap when its type is declared, meaning one of the flow’s own on(...) branches or an event(...) check inside a window condition names it. A repeat of the type that started the instance counts too.

An event of any other type is still retained, but it doesn’t count toward the cap. stepWindow drops it only when the cap drops a counted event that arrived after it.

For example, take stepWindow(2) and the events A, x, B, C, D, where x is of a type no step declares. C arriving drops A and keeps x. D arriving drops B, and x goes with it, because the kept events are always an unbroken run ending with the newest.

historyWindow can drop an event that doesn’t count toward the cap too. Once the flow has left the step the event arrived in, the event is part of the history kept from earlier steps, and a later transition can drop it the same as a counted one.

Such an event reaches a flow saga only through a replacementFilter wider than the flow’s own types, or a CloudEventTypeMapper that maps several domain types onto one CloudEvent type string. A narrowingFilter cannot let one in, because it only narrows the filter derived from the flow’s types.

There is no cap unless you set one, so an instance that stays in a step while a large number of correlated events arrive keeps all of them, whatever historyWindow says. The minimum is 1.

FlowSaga.<OrderEvent, OrderCommand>builder()
        .historyWindow(20)
        .stepWindow(50)
        // startsOn, correlateAll and the steps as above

Set both and, at any one moment, an instance keeps at most historyWindow + 2 * stepWindow + 1 events of the types stepWindow counts. Events of any other type come on top of that. stepWindow is doubled because a transition keeps the events of the step being left so that step’s reaction can read them, and the step being entered then fills its own cap before anything is dropped.

A step condition still completes on the same event it would have without the cap, because its counts are kept in the instance’s state instead of being counted from the events again. What reads less is everything that reads the events directly, a guard’s onlyIf, a timeout reaction, and a window condition’s reaction. A retry guard counting PaymentFailed across a self-looping step therefore needs its threshold to fit inside stepWindow, the same requirement historyWindow already has.

A step that only receives events of types no step declares keeps growing with no limit from stepWindow, since none of those events count toward the cap.

SpringMongoSagaStateStore logs a warning when an instance’s retained events reach 1,000. It doesn’t warn again until the count has dropped below 1,000 and reached it once more. A saga running on a state store of your own gets no such warning.

Two things stay true at any cap of 1 or more. received.initiating<T>() reaches the start event, which is kept as the first retained event and never counts against the cap, and the event that fired a branch is the last element of received.asList().

Before a flow can cap its steps, every event(...) condition with a predicate needs a name for that predicate. Counted Conditions and stepWindow has the details.

Changing what a capped step waits on while instances are still in it, whether that is an event type or a predicate’s name, makes those instances refuse their next delivery with an IllegalStateException naming the step. Retrying does not help, because the events those counts would be rebuilt from are gone. Put the previous condition declaration back until those instances have moved on, so they stop refusing deliveries. Delete the instance instead if you don’t need to keep it running. An instance still inside the cap counts its kept events again and continues.

A flow saga instance waiting in a step stores that step’s name. Renaming or removing the step while instances are still in it makes each of them refuse its next event or timeout with an IllegalStateException naming the step, because the new flow definition has no step by that name. Retrying does not help.

To move those instances on, put the step back under its old name, or add a temporary step under the old name that transitions to the new one, until every instance has left it. To unblock an instance before you redeploy, delete it from the SagaStateStore.

What persists has one compatibility guarantee. The retained domain events serialize as CloudEvents through the application’s CloudEventConverter, by their stable CloudEventTypeMapper type rather than a Java class name. So a domain event can move to a different package without breaking in-flight saga state, exactly as it can for events in the event store. The executor’s own bookkeeping is not a compatibility surface. A core saga’s state is your own model and serializes like the snapshot store.

For the full design rationale, including the residual cross-node race a compare-and-set retry can produce and the deferred outbox that would make dispatch exactly-once, see ADR 0063. The complete, runnable order-fulfillment example wires up both DSLs through SagaRunner, with both styles of dispatcher.

Running Across Multiple Instances

Section titled “Running Across Multiple Instances”

Run several instances of your application, and a saga has two things happening per instance. It receives events, and it polls the state store for due timers. The event side is already single-active when the subscription model is a competing-consumer one,. So for a given saga, only one instance receives events at a time. The timer poller is separate. Left uncoordinated, every instance runs its own poller and queries the store for due timers on its own interval. The timer-query load then grows with the instance count, even though only one instance ever needs to fire a due timer. Firing stays correct either way, since a lost compare-and-set is retried and the dispatch is idempotent, but the extra queries are wasted work.

Give the SagaRunner a CompetingConsumerStrategy with competingConsumerStrategy(...), the same strategy the competing-consumer subscription uses. The poller is gated too. It takes a lease and only polls while it holds it, so exactly one instance queries the store. The lease is keyed apart from the event subscription’s own lease, and released when the SagaSubscription is closed, so another instance takes over within about one lease period. A standby instance checks whether it holds the lease in memory, so it costs no query at all:

CompetingConsumerStrategy strategy = NativeMongoLeaseCompetingConsumerStrategy.withDefaults(mongoDatabase);


SagaSubscription runningSaga = SagaRunner.<OrderEvent, OrderCommand>agnostic(subscriptionModel, cloudEventConverter)
        .competingConsumerStrategy(strategy)
        .run("order-fulfillment", orderFulfillment, stateStore, dispatcher);

Run without a strategy and the poller runs on every instance exactly as it did before. Single-node and in-memory setups need none of this.

Gating removes the redundant queries. It does not change the residual cross-node race noted in the delivery contract (an event on one instance interleaving with a timeout fired on another), which stays handled by compare-and-set and an idempotent receiver. See ADR 0064 for the full rationale.

Side Effects and Compensation

Section titled “Side Effects and Compensation”

A reaction that needs to reach the outside world issues a command, for example NotifyWarehouse(orderId), and points it at a dispatcher that makes the call. That is the only way a saga touches the outside world. There is no “call this API” effect, and that is deliberate, because it keeps react a pure function whose output is data. With ordinary command types you can test that output with equality assertions, and with Invocation you check what the command does instead, since a lambda has no value equality. Either way the reaction performs nothing itself, so a third-party call, whether it runs mid-process or as the last thing a completed saga does, is a command like any other. The terminal reaction, the one whose Continuation is end, is where a “now that the whole thing is done” effect belongs.

Compensation works the same way. A saga does not roll back, it moves forward, so an “undo” is just another command you issue on the branch or timeout that detected the failure. The order-fulfillment saga above already does this. When payment fails or the timeout fires it issues CancelOrder, which is the compensation for the ReservePayment it issued earlier. You decide which command undoes which, there is no automatic inverse:

.on(PaymentFailed.class, Continuation.end(),
        failure -> List.of(new CancelOrder(failure.orderId(), failure.reason())))
.timeout(Duration.ofMinutes(30), Continuation.end(),
        received -> List.of(new CancelOrder(received.initiating(OrderPlaced.class).orderId(), "payment timeout")))

Both branches issue the same forward-moving CancelOrder compensation for the ReservePayment issued when the step started, whichever failure mode gets there first.

The one thing to watch out for is idempotency, and it follows directly from the delivery contract. Command dispatch is at-least-once, so a compensating or external command can arrive twice. An ApplicationService-backed target handles that for free, because it replays the stream and the target’s own invariants reject the duplicate. A raw third-party call does not. When a command triggers a non-idempotent external effect such as an email, a payment capture, or a partner request, give it a stable id derived from the saga and the triggering event, and dedupe at that boundary. The deferred document-local outbox described in ADR 0063 would make dispatch exactly-once and remove this caveat, but it is not built yet.