Spring Rest Template is not thread safe for headers when there are lot of concurrent threads

Affects: Spring Boot 2.2.6

---

When we use a shared RestTemplate across multiple threads, there are issues when lot of concurrent threads are setting different headers while sending the request.

We could see that RestTemplate is sending wrong headers (set by another request thread) in outgoing request for a request thread.

Our code is simply autowiring and using a single common instance using @Bean and rest template builder.

Config Class

@Configuration
public class MyRestTemplateConfig {
  @Autowired
  private RestTemplateBuilder restTemplateBuilder;

  @Bean
  public RestTemplate myRestTemplate() {
    return restTemplateBuilder.build();
  }
}

Client where it's injected

  @Autowired
  @Qualifier("myRestTemplate")
  public RestTemplate myRestTemplate;

Code where it's used

HttpEntity<Object> requestEntity = new HttpEntity<>(payload, headers);
myRestTemplate.exchange(url, method, requestEntity, responseType);

Someone else also ran into this issue: https://stackoverflow.com/questions/59621966/spring-resttemplate-seems-to-not-be-thread-safe-wrt-headers

This problem can be reproducible by sending multiple concurrent requests (with different headers) with JMeter. Add unique IDs to those requests to trace. You can check the incoming headers in target server. RestTemplate is mismatching headers sometimes.

Comment From: bclozel

Could you provide us with a sample application that reproduces this problem? Note that it's critical to provide a minimal application: the issue could be from RestTemplate, JMeter's threading model, they way you're generating those unique IDs, the underlying HTTP client library, etc. Removing as much as possible from the picture is really important to find the problem.

Thanks!

Comment From: rstoyanchev

Also are the headers passed into HttpEntity the same instance for all requests involved? We generally expect that the headers you use for a request are separate from those for other requests and we don't make a copy.

Comment From: rahulrastogi8

@rstoyanchev Yes we were creating new object of HttpHeaders for each request: HttpHeaders headers = new HttpHeaders();

Though one of the value was picked from a cached map. But it should not impact IMO.

Comment From: rahulrastogi8

@bclozel It happened in our enterprise code which can't be shared. I'll try to build a sample application. But it may take time.

The original internal issue was not reported via JMeter. It came with custom test code. But I could repro this easily with JMeter.

The unique IDs are generated as requestIds and set via MDC context. So those IDs were proper.

Comment From: rstoyanchev

If you're creating separate HttpHeaders instances then the issue likely is not in the Spring Framework as those are simply passed through as arguments and used to populate the request object for the underlying HTTP client. Perhaps also review the application code that populates the headers for any potential concurrency issues.

If not we really will need a sample.

Comment From: mdeinum

The unique IDs are generated as requestIds and set via MDC context. So those IDs were proper.

The MDC is a stored in a ThreadLocal which should be cleared after a request. If you don't values will leak into other threads. Make sure you clear those values after the request ends. This is especially the case when using a pool of threads, like Tomcat does for request handling!

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.