Missed calls to DynamoDB when using Async client in Spring Webflux (proect Reactor)

[spachari-im]

I am working on a Spring Boot project running Webflux (version 5.3).

In a REST endpoint that has high traffic (approx 500-1000 rpm) , there are 2 async updates to AWS DynamoDB tables, aggregated together with a Mono.when(). There is a Datadog metric reported before calling Mono.when (metric-1) and a metric reported in the DynamoDB update method for success (metric-2) and failure (metric-3).

Here is the pseudo code:

Mono<Void> result = Mono.empty();
        if (//valid check) {
            //Report Datadog metric-1
            result = Mono.when(updateDynamoDBTable(event, dynamoDbAsyncClient1),
                                               updateDynamoDBTable(event, dynamoDbAsyncClient2));
        }
        return result;

//call the given client to update the given record
private updateDynamoDBTable(Event event, DynamoDbAsyncClient dynamoDbAsyncClient){
    return Mono.defer(() -> Mono.just(event))
                .map(this::toDynamoRequest)
                .flatMap(item -> Mono.fromFuture(dynamoDbAsyncClient.updateItem(toDynamoRequest)))
                .doOnNext(uir -> //Report Datadog metric 2)
                .onErrorResume(e -> {
                      //Report Datadog metric 3
                        };
                    return Mono.empty();
                });
}  

I am expecting that all calls to DynamoDB should report either success or failure metric. That is, metric-1 = 2 * (metric2+metric3).

However I notice that a consistent number (about 20%) of calls to Mono.fromFuture(dynamoDbAsyncClient.updateItem(toDynamoRequest) does not report a success or failure metric. I can also see that the data is not being updated in DynamoDB.

I can see the percentage of missed calls reduce to 15% when I reduce the number of requests per host by scaling up the cluster.

Is this possibly due to the default NettyNioAsyncHttpClient unable to process requests above a certain threshold?

How can I change the code to catch requests that are rejected?