JDK 11.0.6 Spring Boot 2.2.6
Started getting the exception below. Worked fine in 2.2.5.
2020-03-30 12:30:51.549 WARN 58892 --- [O dispatcher 13] org.elasticsearch.client.RestClient : request [PUT http://localhost:9200/
kibana_sample_data_flights/_mapping/_doc?master_timeout=30s&include_type_name=true&timeout=30s] returned 1 warnings: [299 Elasticsearch-
7.6.1-aa751e09be0a5072e8570670309b1f12348f023b "[types removal] Using include_type_name in put mapping requests is deprecated. The param
eter will be removed in the next major version."]
Exception in thread "I/O dispatcher 13" java.lang.AssertionError
at org.elasticsearch.client.Response.assertWarningValue(Response.java:193)
at org.elasticsearch.client.Response.extractWarningValueFromWarningHeader(Response.java:183)
at org.elasticsearch.client.Response.getWarnings(Response.java:205)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:546)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-03-30 12:30:52.498 ERROR 58892 --- [pool-3-thread-1] o.a.h.i.n.c.InternalHttpAsyncClient : I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar:4.1.4]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.lang.AssertionError: null
at org.elasticsearch.client.Response.assertWarningValue(Response.java:193) ~[elasticsearch-rest-client-6.8.7.jar:6.8.7]
at org.elasticsearch.client.Response.extractWarningValueFromWarningHeader(Response.java:183) ~[elasticsearch-rest-client-6.8.7.jar:6.8.7]
at org.elasticsearch.client.Response.getWarnings(Response.java:205) ~[elasticsearch-rest-client-6.8.7.jar:6.8.7]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:546) ~[elasticsearch-rest-client-6.8.7.jar:6.8.7]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537) ~[elasticsearch-rest-client-6.8.7.jar:6.8.7]
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[httpcore4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar:4.4.13]
... 1 common frames omitted
Comment From: scottfrederick
@SledgeHammer01 It appears from this stacktrace that the Elasticsearch client is getting a response from the server with a Warning header containing the value 299 Elasticsearch-, and the value of the header that doesn't match the client's expected format. What's the version of the server that you're running against?
It would be very helpful if you could share a small sample project that re-creates the issue, either as a zip attached to this issue or pushed on a separate github repo.
Comment From: SledgeHammer01
@SledgeHammer01 It appears from this stacktrace that the Elasticsearch client is getting a response from the server with a
Warningheader containing the value299 Elasticsearch-, and the value of the header that doesn't match the client's expected format. What's the version of the server that you're running against?It would be very helpful if you could share a small sample project that re-creates the issue, either as a zip attached to this issue or pushed on a separate github repo.
I am running against 7.6.1. 2.2.5 works against 7.6.1 with no changes. 2.2.6 causes the exception. Sharing a project will be difficult as its tied to the data. I just have the dependency:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
</dependency>
I do disable the reactive drivers: spring.data.elasticsearch.repositories.enabled=false
Other then that I just specify the host, username and password.
Here is my config class. I'm just using the newer rest highlevel client:
@Configuration
@EnableElasticsearchRepositories(basePackages="org.xxx.xxx.flights.repositories")
public class FlightsDataSourceConfig {
@Autowired
private Environment environment;
private final Logger logger = LoggerFactory.getLogger(FlightsDataSourceConfig.class);
@Bean
public RestHighLevelClient client() {
RestClientBuilder restClientBuilder = RestClient.builder(getHost())
.setHttpClientConfigCallback(new HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
return httpClientBuilder.setDefaultCredentialsProvider(getCredentialsProvider());
}
});
RestHighLevelClient client = new RestHighLevelClient(restClientBuilder);
try {
client.ping(RequestOptions.DEFAULT);
}
catch (ElasticsearchStatusException | IOException e) {
this.logger.error(String.format("Unable to resolve ElasticSearch host: %s", getHost().toHostString()));
return null;
}
return client;
}
@Bean
ElasticsearchOperations elasticsearchTemplate() {
return new ElasticsearchRestTemplate(client());
}
private HttpHost getHost() {
HostAndPort hostAndPort = HostAndPort.fromString(Preconditions.checkNotNull(this.environment.getProperty("flights.server.host")));
return new HttpHost(hostAndPort.getHost(), hostAndPort.getPort());
}
private CredentialsProvider getCredentialsProvider() {
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, getUsernamePasswordCredentials());
return credentialsProvider;
}
private UsernamePasswordCredentials getUsernamePasswordCredentials() {
return new UsernamePasswordCredentials(Preconditions.checkNotNull(this.environment.getProperty("flights.server.username")),
Preconditions.checkNotNull(this.environment.getProperty("flights.server.password")));
}
}
Comment From: SledgeHammer01
@SledgeHammer01 It appears from this stacktrace that the Elasticsearch client is getting a response from the server with a
Warningheader containing the value299 Elasticsearch-, and the value of the header that doesn't match the client's expected format. What's the version of the server that you're running against?It would be very helpful if you could share a small sample project that re-creates the issue, either as a zip attached to this issue or pushed on a separate github repo.
In addition, if there's any logging I can turn on that can help you, I can do that too :).
Comment From: scottfrederick
@SledgeHammer01 If you can inspect the responses you're getting back from Elasticsearch and show the value of any Warning headers, that might be useful.
Comment From: wilkinsona
We upgraded to version 6.8.7 of the Elasticsearch client in 2.2.6. I think that's the most likely cause of the change in behaviour. @SledgeHammer01 Can you please try downgrading to 6.8.6 and see if that helps? You can use the elasticsearch.version property to do so.
Comment From: wilkinsona
Here is the change in Elasticsearch that introduced the change in behaviour. It's new in 6.8.7. @SledgeHammer01 thank you for bringing this to our attention but, as the change is in Elasticsearch, there's nothing we can do about it. Generally speaking we do not downgrade a dependency as it would be a step backwards for those that are not affected by the problem. Please open an Elasticsearch issue and comment here with a link to it so that we can follow along.
Comment From: SledgeHammer01
Here is the change in Elasticsearch that introduced the change in behaviour. It's new in 6.8.7. @SledgeHammer01 thank you for bringing this to our attention but, as the change is in Elasticsearch, there's nothing we can do about it. Generally speaking we do not downgrade a dependency as it would be a step backwards for those that are not affected by the problem. Please open an Elasticsearch issue and comment here with a link to it so that we can follow along.
I opened a bug here:
https://github.com/elastic/elasticsearch/issues/54501
Thanks.