Versions
Spring Parent: 2.7.4 Spring Cloud Version: 2021.0.4 Java Version: 11
Issue
My Spring service has been using Eureka to connect to the config server for a long time, but I want to upgrade to Spring 2.7.4. I understand that as of Spring 2.4, the bootstrap context has been deprecated (source) and I need to make some adjustments to the old bootstrap properties and move them over to application.properties.
The documentation for Spring Cloud specifies that in order for me to continue to use discovery-first config lookup, I need to define a spring.config.import property with an optional configserver entry (source). Since I'm also using Vault, I define the property as follows:
spring.config.import = optional:configserver:placeholder,vault://<my-generic-backend>/dev
Next, I need to define the following properties (source). These properties were already defined in my old bootstrap.properties, so all I need to do is copy and paste.
spring.cloud.config.discovery.enabled = true
spring.cloud.config.discovery.serviceId = config-server
eureka.client.serviceUrl.defaultZone = <my-eureka-url>
Unless I'm missing something, these are all the steps I need to take in order to upgrade to 2.7.4. However, when I run the Spring service, it complains that it can't find the config server (via Eureka, or via URL), then it registers successfully with Eureka, and then continues trying and failing to find the config server.
Here is some of the output of the program:
> Running with Spring Boot v2.7.4, Spring v5.3.23
> Could not locate configserver via discovery: No instances found of configserver (config-server)
> Could not locate PropertySource ([ConfigServerConfigDataResource@2aa6311a uris = array<String>['placeholder'], optional = true, profiles = list['local']]): Invalid URL: placeholder
...
> DiscoveryClient_<my-project-name>/local - registration status: 204
Full logs in the spoiler below.
Logs from Failure
Starting <my-spring-service> using Java 11.0.14.1 on ...
Running with Spring Boot v2.7.4, Spring v5.3.23
The following 1 profile is active: "local"
Could not locate configserver via discovery
java.lang.IllegalStateException: No instances found of configserver (config-server)
at org.springframework.cloud.config.client.ConfigServerInstanceProvider.getConfigServerInstances(ConfigServerInstanceProvider.java:59) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.refresh(ConfigServerInstanceMonitor.java:90) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerConfigDataLocationResolver.lambda$resolveProfileSpecific$8(ConfigServerConfigDataLocationResolver.java:228) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.boot.DefaultBootstrapContext.getInstance(DefaultBootstrapContext.java:119) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.DefaultBootstrapContext.getOrElseThrow(DefaultBootstrapContext.java:111) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.DefaultBootstrapContext.get(DefaultBootstrapContext.java:88) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.load(ConfigServerConfigDataLoader.java:85) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.load(ConfigServerConfigDataLoader.java:61) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.boot.context.config.ConfigDataLoaders.load(ConfigDataLoaders.java:107) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataImporter.load(ConfigDataImporter.java:128) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataImporter.resolveAndLoad(ConfigDataImporter.java:86) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataEnvironmentContributors.withProcessedImports(ConfigDataEnvironmentContributors.java:116) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataEnvironment.processWithProfiles(ConfigDataEnvironment.java:311) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataEnvironment.processAndApply(ConfigDataEnvironment.java:232) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:102) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:94) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:102) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:87) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66) ~[spring-boot-2.7.4.jar:2.7.4]
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) ~[na:na]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:344) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:302) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.4.jar:2.7.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.4.jar:2.7.4]
at <my-package>.<my-spring-service>.main(<my-spring-service>.java:37) ~[classes/:na]
Could not locate PropertySource ([ConfigServerConfigDataResource@2aa6311a uris = array<String>['placeholder'], optional = true, profiles = list['local']]): Invalid URL: placeholder
Bootstrapping Spring Data JPA repositories in DEFAULT mode.
Finished Spring Data repository scanning in 76 ms. Found 6 JPA repository interfaces.
Bootstrapping Spring Data JPA repositories in DEFAULT mode.
Finished Spring Data repository scanning in 5 ms. Found 1 JPA repository interfaces.
Tomcat initialized with port(s): 8083 (http)
Starting service [Tomcat]
Starting Servlet engine: [Apache Tomcat/9.0.65]
Initializing Spring embedded WebApplicationContext
Root WebApplicationContext: initialization completed in 1857 ms
Initialized JPA EntityManagerFactory for persistence unit 'default'
spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
Eureka HTTP Client uses RestTemplate.
Spring Cloud LoadBalancer is currently working with the default cache. While this cache implementation is useful for development and tests, it's recommended to use Caffeine cache in production.You can switch to using Caffeine cache, by adding it and org.springframework.cache.caffeine.CaffeineCacheManager to the classpath.
Setting initial instance status as: STARTING
Initializing Eureka in region us-east-1
Resolving eureka endpoints via configuration
Disable delta property : false
Single vip registry refresh property : null
Force full registry fetch : false
Application is null : false
Registered Applications size is zero : true
Application version is -1: true
Getting all instance registry info from the eureka server
The response status is 200
Starting heartbeat executor: renew interval is: 30
InstanceInfoReplicator onDemand update allowed rate per min is 4
Discovery Client initialized at timestamp 1666838114593 with initial instances count: 136
Registering application <my-spring-service> with eureka with status UP
Saw local status change event StatusChangeEvent [timestamp=1666838114602, current=UP, previous=STARTING]
DiscoveryClient_<my-spring-service>/local: registering service...
Tomcat started on port(s): 8083 (http) with context path ''
Updating port to 8083
Started <my-spring-service> in 7.966 seconds (JVM running for 8.619)
The <my-spring-service> Application has started...
DiscoveryClient_<my-spring-service>/local - registration status: 204
Could not locate configserver via discovery
java.lang.IllegalStateException: No instances found of configserver (config-server)
at org.springframework.cloud.config.client.ConfigServerInstanceProvider.getConfigServerInstances(ConfigServerInstanceProvider.java:59) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.refresh(ConfigServerInstanceMonitor.java:90) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.heartbeat(ConfigServerInstanceMonitor.java:81) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.onApplicationEvent(ConfigServerInstanceMonitor.java:69) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.cloud.netflix.eureka.CloudEurekaClient.onCacheRefreshed(CloudEurekaClient.java:119) ~[spring-cloud-netflix-eureka-client-3.1.4.jar:3.1.4]
at com.netflix.discovery.DiscoveryClient.fetchRegistry(DiscoveryClient.java:1031) ~[eureka-client-1.10.17.jar:1.10.17]
at com.netflix.discovery.DiscoveryClient.refreshRegistry(DiscoveryClient.java:1531) ~[eureka-client-1.10.17.jar:1.10.17]
at com.netflix.discovery.DiscoveryClient$CacheRefreshThread.run(DiscoveryClient.java:1498) ~[eureka-client-1.10.17.jar:1.10.17]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Could not locate configserver via discovery
java.lang.IllegalStateException: No instances found of configserver (config-server)
at org.springframework.cloud.config.client.ConfigServerInstanceProvider.getConfigServerInstances(ConfigServerInstanceProvider.java:59) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.refresh(ConfigServerInstanceMonitor.java:90) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.heartbeat(ConfigServerInstanceMonitor.java:81) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.cloud.config.client.ConfigServerInstanceMonitor.onApplicationEvent(ConfigServerInstanceMonitor.java:69) ~[spring-cloud-config-client-3.1.4.jar:3.1.4]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) ~[spring-context-5.3.23.jar:5.3.23]
at org.springframework.cloud.netflix.eureka.CloudEurekaClient.onCacheRefreshed(CloudEurekaClient.java:119) ~[spring-cloud-netflix-eureka-client-3.1.4.jar:3.1.4]
at com.netflix.discovery.DiscoveryClient.fetchRegistry(DiscoveryClient.java:1031) ~[eureka-client-1.10.17.jar:1.10.17]
at com.netflix.discovery.DiscoveryClient.refreshRegistry(DiscoveryClient.java:1531) ~[eureka-client-1.10.17.jar:1.10.17]
at com.netflix.discovery.DiscoveryClient$CacheRefreshThread.run(DiscoveryClient.java:1498) ~[eureka-client-1.10.17.jar:1.10.17]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
I understand why it's failing to find a config server at URL: placeholder since that's not a valid URL, but I don't understand how the service can successfully register with Eureka yet not be able to find the config server. I know the service is registered because the output of the program says it registered correctly (and I can see it in the registry), and I know that the config server has the correct entity ID (config-server) because it was copied and pasted from the old bootstrap (and I can see config-server in the registry).
Workaround with Hardcoded URL
When I hardcode the config server URL like this (and set spring.cloud.config.discovery.enabled to false), the config is loaded properly from the server:
spring.config.import=configserver:https://<my-hardcoded-config-url>.com,vault://<my-generic-backend>/dev
I'll put the logs in a spoiler below.
Logs from Hardcoded URL Workaround
Starting <my-spring-service> using Java 11.0.14.1 on ...
Running with Spring Boot v2.7.4, Spring v5.3.23
The following 1 profile is active: "local"
Fetching config from server at : https://<my-hardcoded-config-url>.com
Located environment: name=<my-spring-service>, profiles=[local], label=null, version=20fd39f513d6c5c933cc19c2c7496756a6dfddc2, state=null
Bootstrapping Spring Data JPA repositories in DEFAULT mode.
Finished Spring Data repository scanning in 75 ms. Found 6 JPA repository interfaces.
Bootstrapping Spring Data JPA repositories in DEFAULT mode.
Finished Spring Data repository scanning in 4 ms. Found 1 JPA repository interfaces.
Tomcat initialized with port(s): 8083 (http)
Starting service [Tomcat]
Starting Servlet engine: [Apache Tomcat/9.0.65]
Initializing Spring embedded WebApplicationContext
Root WebApplicationContext: initialization completed in 1797 ms
Initialized JPA EntityManagerFactory for persistence unit 'default'
Eureka HTTP Client uses RestTemplate.
Spring Cloud LoadBalancer is currently working with the default cache. While this cache implementation is useful for development and tests, it's recommended to use Caffeine cache in production.You can switch to using Caffeine cache, by adding it and org.springframework.cache.caffeine.CaffeineCacheManager to the classpath.
Setting initial instance status as: STARTING
Initializing Eureka in region us-east-1
Resolving eureka endpoints via configuration
Disable delta property : false
Single vip registry refresh property : null
Force full registry fetch : false
Application is null : false
Registered Applications size is zero : true
Application version is -1: true
Getting all instance registry info from the eureka server
The response status is 200
Starting heartbeat executor: renew interval is: 30
InstanceInfoReplicator onDemand update allowed rate per min is 4
Discovery Client initialized at timestamp 1666837765059 with initial instances count: 136
Registering application <my-spring-service> with eureka with status UP
Saw local status change event StatusChangeEvent [timestamp=1666837765068, current=UP, previous=STARTING]
DiscoveryClient_<my-spring-service>/local: registering service...
Tomcat started on port(s): 8083 (http) with context path ''
Updating port to 8083
Started <my-spring-service> in 9.532 seconds (JVM running for 10.179)
The <my-spring-service> Application has started...
DiscoveryClient_<my-spring-service>/local - registration status: 204
Workaround with Bootstrap
It's possible to return to using the bootstrap context and still use Spring 2.7.4 with discovery-first config lookup by adding the "spring-cloud-starter-bootstrap" dependency. So I added the dependency to my POM and moved these properties back to bootstrap.properties from application.properties.
spring.cloud.config.discovery.enabled=true
spring.cloud.config.discovery.service-id=config-server
I moved the Vault and Eureka properties back into bootstrap.properties as well. The new application.properties now contains no values relating to Eureka, Vault, and Cloud Config.
When I run the service, it does indeed find the address for the config server through Eureka, as expected (although it fails to connect because it's the internal address and I'm running locally). I'll put the logs in a spoiler below.
Logs from Bootstrap Workaround
Multiple Config Server Urls found listed.
Fetching config from server at : http://<my-internal-config-server-ip-addr-1>/
Connect Timeout Exception on Url - http://<my-internal-config-server-ip-addr-1>/. Will be trying the next url if available
Fetching config from server at : <my-internal-config-server-ip-addr-2>/
Connect Timeout Exception on Url - <my-internal-config-server-ip-addr-2>/. Will be trying the next url if available
Could not locate PropertySource: I/O error on GET request for "<my-internal-config-server-ip-addr-2>/<my-application-name>/dev": connect timed out; nested exception is java.net.SocketTimeoutException: connect timed out
The following 1 profile is active: "dev"
Bootstrapping Spring Data JPA repositories in DEFAULT mode.
Finished Spring Data repository scanning in 71 ms. Found 6 JPA repository interfaces.
Bootstrapping Spring Data JPA repositories in DEFAULT mode.
Finished Spring Data repository scanning in 4 ms. Found 1 JPA repository interfaces.
Tomcat initialized with port(s): 8080 (http)
Starting service [Tomcat]
Starting Servlet engine: [Apache Tomcat/9.0.65]
Initializing Spring embedded WebApplicationContext
Root WebApplicationContext: initialization completed in 1720 ms
Initialized JPA EntityManagerFactory for persistence unit 'default'
Eureka HTTP Client uses RestTemplate.
Spring Cloud LoadBalancer is currently working with the default cache. While this cache implementation is useful for development and tests, it's recommended to use Caffeine cache in production.You can switch to using Caffeine cache, by adding it and org.springframework.cache.caffeine.CaffeineCacheManager to the classpath.
Setting initial instance status as: STARTING
Initializing Eureka in region us-east-1
Resolving eureka endpoints via configuration
Disable delta property : false
Single vip registry refresh property : null
Force full registry fetch : false
Application is null : false
Registered Applications size is zero : true
Application version is -1: true
Getting all instance registry info from the eureka server
The response status is 200
Starting heartbeat executor: renew interval is: 30
InstanceInfoReplicator onDemand update allowed rate per min is 4
Discovery Client initialized at timestamp 1666837087594 with initial instances count: 132
Registering application <my-application-name> with eureka with status UP
Saw local status change event StatusChangeEvent [timestamp=1666837087604, current=UP, previous=STARTING]
DiscoveryClient_<my-application-name>/local_bootstrap: registering service...
Tomcat started on port(s): 8080 (http) with context path ''
Updating port to 8080
Started <my-spring-service> in 30.077 seconds (JVM running for 30.717)
The <my-application-name> Application has started...
DiscoveryClient_<my-application-name>/local_bootstrap - registration status: 204
Conclusion
While these are valid workarounds, it's frustrating to not be able to have a dynamic URL for the config server (as is the entire point of using Eureka). Right now, it looks like my choices are either to use a hard-coded URL and risk having to change every property file, or use a deprecated behavior that Spring documentation specifically disfavors (source).
I would appreciate any guidance you have on the issue, and I thank you in advance.
Comment From: ryanjbaxter
optional:configserver:placeholder
You do not need placeholder in the URL it should just be optional:configserver:
Comment From: raphaelpreston
Thanks @ryanjbaxter! I do understand that I can leave it blank, but in this particular example I left in placeholder so it would be clear why the connection to placeholder failed instead of having to explain that http://localhost:8888 is the default config connection address and that I didn't have a config server running at this location. @spencergibb actually left a similar comment on my stackoverflow post as well.
Comment From: ryanjbaxter
Got it, so when you remove placeholder it still does not work for you?
Could you open an issue in spring-cloud-vault regarding the documentation?
Comment From: raphaelpreston
Yeah, unfortunately then it just fails to connect to http://localhost:8888. Just opened up this issue here in spring-cloud-vault for the documentation :smile:
Comment From: ryanjbaxter
Using this application.yaml it works fine for me
spring:
config:
import: "optional:configserver:"
cloud:
config:
discovery:
enabled: true
service-id: configserver
Comment From: spring-cloud-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: spring-cloud-issues
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.