Webclient timeout with Httpclient

[5Orange]

Problem statement

While sending high traffic to API, experiencing WebClient timeout but downstream still working and returning response according to timeout configuration time

Expected Behavior

it should return 200 instead of 500 and webclient timeout error

Actual Behavior

ava.util.concurrent.TimeoutException: Webclient timeout
	at********.Test.lambda$call$1(Test.java:37)
	at reactor.core.publisher.Mono.lambda$onErrorResume$32(Mono.java:3887)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.SerializedSubscriber.onError(SerializedSubscriber.java:124)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:295)
	at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:280)
	at reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:419)
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onNext(FluxOnErrorReturn.java:162)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.propagateDelay(MonoDelay.java:271)
	at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:286)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 20000ms in 'flatMap' (and no fallback has been configured)
	... 13 common frames omitted

Steps to Reproduce:

using Jmeter to send 20 requests into GET /test API, and half of them failed

@RestController
@RequiredArgsConstructor
public class Test {

  private final WebClient.Builder webClientBuilder;

  public Mono<?> call() {
    return webClientBuilder.build().method(HttpMethod.GET)
        .uri("http://localhost:8080/ms-test-webclient/delay")
        .headers(httpHeaders -> httpHeaders.set("Content-Type", "application/json"))
        .retrieve()
        .toEntity(String.class)
        .flatMap(Mono::just)
        .timeout(Duration.ofSeconds(20))
        .onErrorResume(TimeoutException.class, ex -> {
          String message = "Webclient timeout";
          return Mono.error(new TimeoutException(message).initCause(ex));
        });
  }

  @GetMapping("test")
  public Mono<?> test() {
    return call();
  }

  @GetMapping("delay")
  @SneakyThrows
  public String delayresponse() {
    Thread.sleep(15000);
    return "success";
  }
}

Reactor version(s) used: 1.0.23

+--- org.springframework.boot:spring-webflux:5.3.31^M

| | --- io.projectreactor.netty:reactor-netty-http:1.0.39^M
JVM version (java -version): 11

[violetagg]

Stackoverflow question https://stackoverflow.com/questions/78470231/webclient-timeout-exception-while-sending-high-traffic

[violetagg]

@mswindowsxp I'm not able to reproduce the described problem. See the reproducible example which is a compilation from the description here and the recommendation in the stackoverflow thread.

https://github.com/violetagg/GH-3240

[5Orange]

@mswindowsxp I'm not able to reproduce the described problem. See the reproducible example which is a compilation from the description here and the recommendation in the stackoverflow thread.

https://github.com/violetagg/GH-3240

hi @violetagg , I created a repo to reproduce, please refer this one: https://github.com/5Orange/webclient-error
video reproduce: https://www.youtube.com/watch?v=91e4DT7_rso&ab_channel=mswindows

[violetagg]

@5Orange Stackoverflow thread already provided an answer for the reproducible example that you prepared - you cannot block the event loop ...

https://github.com/5Orange/webclient-error/blob/a375992e789eabe6ecf9965deaa7fa8bbdeb8dd1/src/main/java/com/example/demo/controller/TestController.java#L42

Reactor Netty is using just a few threads and if you block them then you will observe the behaviour that you described above.

If you need to block or you have CPU intensive operations then you need to offload.

[5Orange]

@violetagg Vio
i updated to use Flux instead of Thread.sleep, and got the same error, it just indicate the slow response from downstream
@GetMapping("delay") @SneakyThrows public String delayresponse() { return Flux.interval(Duration.ofSeconds(15)) .next() .map(any -> "success"); }

[violetagg]

@5Orange The project is now broken. The method should return Mono<String> and should look like this

    @GetMapping("delay")
    @SneakyThrows
    public Mono<String> delayresponse() {
        return Flux.interval(Duration.ofSeconds(15))
        .next()
        .map(any -> "success");
    }

Both my client that sends requests in parallel and curl are working. I don't have JMeter, may be the problem is there.

My client

public class ClientApplication {
	public static void main(String[] args) {
		HttpClient client = HttpClient.create().port(8080);

		System.out.println(Flux.range(0, 20)
				.flatMap(i -> client.get()
						.uri("/test")
						.responseContent()
						.aggregate()
						.asString())
				.collectList()
				.block());
	}
}

curl

curl -Z --config urls.txt

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[hypr2771]

Can't reproduce either using either ab or drill.
Could you please provide a fully reproducible exemple, using for instance https://github.com/jmeter-maven-plugin/jmeter-maven-plugin to allow us to simply run into your issue with the same exact configuration. Otherwise it's quite impossible for us to pinpoint that very specific issue.

[github-actions]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[github-actions]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open.