Retry

Modules: occurrent-retry

Retry Configuration (Blocking) (Added in 0.7.0)

Occurrent contains a retry module that you can depend on using:

Maven

<dependency>
    <groupId>org.occurrent</groupId>
    <artifactId>occurrent-retry</artifactId>
    <version>0.34.0</version>
</dependency>

Gradle

compile 'org.occurrent:occurrent-retry:0.34.0'

SBT

libraryDependencies += "org.occurrent" % "occurrent-retry" % "0.34.0"

Grape

@Grab(group='org.occurrent', module='occurrent-retry', version='0.34.0')

Leiningen

[org.occurrent/occurrent-retry "0.34.0"]

Buildr

'org.occurrent:occurrent-retry:jar:0.34.0'

Ivy

<dependency org="org.occurrent" name="occurrent-retry" rev="0.34.0" />

Note

Typically you don’t need to depend on this module explicitly since many of Occurrent’s components already uses this library under the hood and is thus depended on transitively.

Occurrent components that support retry (subscription model and checkpoint storage implementations) typically accepts an instance of org.occurrent.retry.RetryStrategy to their constructors. This allows you to configure how they should do retry. You can configure max attempts, a retry predicate, error listener, before/after retry listener, as well as the backoff strategy. Here’s an example:

RetryStrategy retryStrategy = RetryStrategy
                                    .exponentialBackoff(Duration.ofMillis(100), Duration.ofSeconds(2), 2.0)
                                    .retryIf(UncategorizedSQLException.class::isInstance)
                                    .maxAttempts(5)
                                    .onBeforeRetry(throwable -> log.warn("Caught exception {}, will retry.", throwable.getClass().getSimpleName()))
                                    .onError((info, throwable) -> {
                                        if (info.isLastAttempt()) {
                                            log.error("Ended with exception {}.", throwable.getClass().getSimpleName());
                                        }
                                    });

You can then use a RetryStrategy instance to call methods that you want to be retried on exception by using the execute method:

RetryStrategy retryStrategy = ..
// Retry the method if it throws an exception
retryStrategy.execute(Something::somethingThing);

RetryStrategy is immutable, which means that you can safely do things like this:

RetryStrategy retryStrategy = RetryStrategy.retry().fixed(200).maxAttempts(5);
// Uses default 200 ms fixed delay
retryStrategy.execute(() -> Something.something());
// Use 600 ms fixed delay
retryStrategy.backoff(fixed(600)).execute(() -> SomethingElse.somethingElse());
// 200 ms fixed delay again
retryStrategy.execute(() -> Thing.thing());

You can also disable retries by calling RetryStrategy.none().

As of version 0.11.0 you can also use the mapRetryPredicate function easily allows you to map the current retry predicate into a new one. This is useful if you e.g. want to add a predicate to the existing predicate. For example:

// Let's say you have a retry strategy:
RetryStrategy retry = RetryStrategy.exponentialBackoff(Duration.ofMillis(100), Duration.ofSeconds(2), 2.0f).maxAttempts(5).retryIf(WriteConditionNotFulfilledException.class::isInstance);
// Now you also want to retry if an IllegalArgumentException is thrown:
retry.mapRetryPredicate(currentRetryPredicate -> currentRetryPredicate.or(IllegalArgumentException.class::isInstance))

As of version 0.16.3, RetryStrategy now accepts a function that takes an instance of org.occurrent.retry.RetryInfo. This is useful if you need to know the current state of your of the retry while retrying. For example:

RetryStrategy retryStrategy = RetryStrategy
                              .exponentialBackoff(initialDelay, maxDelay, 2.0)
                              .maxAttempts(10)
retryStrategy.execute(info -> {
    if (info.getNumberOfAttempts() > 2 &&  info.getNumberOfAttempts() < 6) {
        System.out.println("Number of attempts is between 3 and 5");
    }
    ...
});

Stopping a Retry When Shutting Down

A retry that’s waiting between attempts keeps going until its attempts run out. That’s a problem when the component doing the retrying is shutting down. You can pass execute a second argument, a predicate that tells it whether to keep retrying:

private volatile boolean running = true;

void store(CloudEvent cloudEvent) {
    retryStrategy.execute(() -> writeToTheDatabase(cloudEvent), __ -> running);
}

public void close() {
    running = false;
}

The retry loop calls the predicate before every retry, and roughly every 50 milliseconds while it waits out a backoff. So if close() is called 1 millisecond into a 2 second backoff, the retry stops the next time the loop calls the predicate, instead of 2 seconds later.

When the loop stops this way, it rethrows the exception from the last attempt, the same as when the attempts run out.

The predicate gets the Throwable from the last attempt, so it can give a different answer for different failures. You can also ignore it and check a flag, as in the example above.

Keep the predicate free of side effects, because it can be called many times between two attempts.

A RetryStrategy you implement yourself, rather than one built from RetryStrategy.retry() or RetryStrategy.none(), ignores the predicate and runs the action the same way execute(..) without a predicate does.

NativeMongoLeaseCompetingConsumerStrategy, SpringMongoLeaseCompetingConsumerStrategy and InMemoryDeadlineConsumerRegistry log a WARN when they’re created with such a strategy, since a shutdown can’t stop their retries then.

Check a flag like running in the predicate you pass to execute, not in retryIf. A retryIf predicate is only called between attempts, so a shutdown that happens during a backoff still waits for the backoff to finish. And retryIf replaces whatever retry predicate the strategy already had instead of adding to it.

Retry and Transactions (Added in 0.31.0)

A retry only works where it also owns the transaction, because only the code that began a transaction can begin a fresh one. A retry running inside somebody else’s transaction spends its attempts on a transaction MongoDB already aborted at the first conflict, and every later attempt fails immediately on its first read. As of version 0.34.0 Occurrent applies that rule across the whole write path.

The two Spring MongoDB event stores check for an active transaction before they retry. When they find one they run the write once and let the conflict reach whoever owns the transaction. That covers the DCB append conflict, the any-version write condition, and the global position counter’s first write. The native driver store has always behaved this way when it finds an ambient ClientSession.

The retry the store gives up is taken over by the layer that does own the transaction. SpringTransactionExecutor and SpringReactiveTransactionExecutor retry a conflict around the transaction they open, and skip the retry when they are themselves joining a caller’s transaction. That is what keeps a synchronous subscription setup retrying, since there Occurrent opens the transaction itself so the event write and the handlers commit together.

When neither the store nor an executor owns the transaction, nothing in Occurrent retries and the conflict reaches you right away. That is the intended outcome, and it means you have to retry at your own transaction boundary, outside the transaction rather than inside it. Catching the conflict and carrying on inside the same transaction does not work: a participating write that throws marks the surrounding transaction rollback-only, so a further attempt runs in a transaction whose inner commit is a no-op, and the outer commit then fails with UnexpectedRollbackException instead of giving you the partial success you expected. The same goes for the RetryStrategy you hand to GenericApplicationService or GenericDcbApplicationService, because it sits inside your transaction too.

With Spring Retry you get the right shape for free, since retry advice sits outside transaction advice and each attempt therefore runs a fresh transaction:

@Retryable(include = [DcbAppendConditionNotFulfilledException::class, DataIntegrityViolationException::class], maxAttempts = 5, backoff = Backoff(delay = 100, multiplier = 2.0, maxDelay = 1000))
operator fun invoke(gameId: GameId, timeOfGuess: Timestamp, playerId: PlayerId, word: Word) {
    applicationService.execute(GameDcbQueries.gameplay(gameId)) { events ->
        guessWord(events, timeOfGuess, playerId, word)
    }
}

DataIntegrityViolationException belongs in that list even though a write conflict is not an integrity violation. MongoDB labels the conflict TransientTransactionError, but Spring translates it to DataIntegrityViolationException, which is not one of Spring’s transient types, so a retry predicate built on TransientDataAccessException alone misses the most common conflict there is. Both DCB versions of the word guessing game retry this way.

A nested append often loses on the counter before it ever reaches the append itself, and that is what a failure here looks like. ADR 21 describes the global position counter as being updated outside the append transaction, which holds only while the store owns that transaction. When the store joins your transaction the counter update joins it as well, so one shared document becomes a conflict point for every concurrent append in that transaction, even for appends to boundaries that have nothing to do with each other.

The full reasoning is in ADR 74.