Hi Team,
We are experiencing an unusual issue with our config server. After upgrading our Spring Boot 2.7.12 and Spring Cloud versions 2021.0.7, our micro-services in Kubernetes are failing to start because the config server (3.1.7) is unable to load the application properties. Initially, we thought it was a one-time occurrence, but it has become a recurring problem. We have had to retry the application startup multiple times for the config server to successfully retrieve the properties. Despite checking the config server logs, we have not found any error messages. The config server was up and running.
Due to technical difficulties with version 17, we are unable to switch to Spring Boot 3.x.x at the moment.
Below is the error from the applications trying to connect to config server to load the properties:
2023-07-21 07:45:01.339 INFO 10796 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://localhost:8888
2023-07-21 07:48:06.418 INFO 10796 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Connect Timeout Exception on Url - http://localhost:8888. Will be trying the next url if available
2023-07-21 07:48:06.418 WARN 10796 --- [ main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/test-assignment-process/fin-sit": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
2023-07-21 07:48:06.420 INFO 10796 --- [ main] com.mutual.fin.CommonStepDefs : The following 1 profile is active: "fin-sit"
2023-07-21 07:48:06.583 INFO 10796 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=f173beaa-62bd-3636-b713-bd549510b7f8
2023-07-21 07:48:06.625 WARN 10796 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'sessionContextStaticWrapper': Unsatisfied dependency expressed through field 'testProps'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'testProperties': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'fin.url' in value "${fin.url}"
2023-07-21 07:48:06.628 INFO 10796 --- [ main] ConditionEvaluationReportLoggingListener :
Error from the Jenkins:
+ curl http://localhost:8888/actuator/health
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
Cancelling nested steps due to timeout
Sending interrupt signal to process
/home/centos/workspace/s_fin-service-ent-article_master@tmp/durable-f74b09a1/script.sh: line 1: 32723 Terminated
curl http://localhost:8888/actuator/health
script returned exit code 143
Could you provide any suggestions?
Comment From: ryanjbaxter
Just to be clear it is the confit server having the problems not the config client correct? You provided the client logs in the issue.
Comment From: skkadium
Yes it is the config server. So basically, whenever the service pods are getting refreshed, most of the time are failing to start up because the properties were unable to load up. I may be wrong, but just wanted to understand, do we need to make amendments at the server side or the client side. Any thoughts?
Comment From: ryanjbaxter
So its the clients failing to load configuration from the config server, not the config server failing to load the configuration from the back end environment repository, correct?
Comment From: skkadium
So its the clients failing to load configuration from the config server, not the config server failing to load the configuration from the back end environment repository, correct?
Yes, thats correct.
Comment From: ryanjbaxter
So its not an error with the config server its an error with the config clients.
Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/test-assignment-process/fin-sit": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
There is not much to go on with the error above, but ultimately it seems like a network issue. You said this only started happening after an upgrade, if you revert to the previous version does the problem go away?
Comment From: skkadium
So its not an error with the config server its an error with the config clients.
Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/test-assignment-process/fin-sit": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed outThere is not much to go on with the error above, but ultimately it seems like a network issue. You said this only started happening after an upgrade, if you revert to the previous version does the problem go away?
We dint try to revert to the old version, as we were thinking, if something can be done to make sure the client will be compatible with the new version.
Comment From: ryanjbaxter
Compatible? Are the config client and config server using different versions?
Comment From: skkadium
Compatible? Are the config client and config server using different versions?
Unfortunately, we have combination of services that are using Spring versions < 2.5.x and that uses a config client of 3.0.X and few other services are inline with the config server spring boot version i.e 2.7.12
Comment From: ryanjbaxter
What version of Spring Cloud is the config server using? What version(s) of Spring Cloud are you using in your clients? In the example above with the failures, what version of Spring Cloud was that app using?
Comment From: skkadium
What version of Spring Cloud is the config server using? What version(s) of Spring Cloud are you using in your clients? In the example above with the failures, what version of Spring Cloud was that app using?
The Spring cloud for the config server is 2021.0.7 and few applications are using 2021.0.7 and few of them are using 2020.0.3. The failure is observed in the apps that are using cloud version as 2020.0.3
Comment From: ryanjbaxter
While its not something we would recommend, I dont see how this issue could be cause by using a 2021.0.x config server and a 2020.0.3 config client. Like I said above the errors appear to be network related. That said we no longer support 2020.0.x its end of life at this point.
Have you tried just making the request from within the same container yourself? For example, curl http://localhost:8888/test-assignment-process/fin-sit?
Comment From: skkadium
While its not something we would recommend, I dont see how this issue could be cause by using a 2021.0.x config server and a 2020.0.3 config client. Like I said above the errors appear to be network related. That said we no longer support 2020.0.x its end of life at this point.
Have you tried just making the request from within the same container yourself? For example,
curl http://localhost:8888/test-assignment-process/fin-sit?
We are thinking to move ahead and refactor our services to use 2021.0.x. When we encountered this error, which we are encountered even today, I did try to make the request from the service container, it failed, but from within the config server, it gave the expected result.
Comment From: ryanjbaxter
If it’s failing also using curl it’s not an issue with the config client, again this seems like a network issue outside of our control.
Comment From: BorisSpring
im issuing same problem my services canot fetch properties when i run compose.yml file
Comment From: BorisSpring
i was searching for 10 days stackoverflow,slack,udemy,google,youtube and nothing helps me .... how to solve this issue
version: '3.8' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.11.1 ports: - "9288:9288" jms: container_name: jms image: vromero/activemq-artemis ports: - "8161:8161" - "61616:61616" zipkin: container_name: zipkin image: openzipkin/zipkin ports: - "9411:9411" eureka: container_name: eureka image: sfgbeerworks/mscc-eureka-server ports: - "8761:8761" config: container_name: config image: sfgbeerworks/mscc-cloud-config ports: - "8888:8888" depends_on: - eureka environment: EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://username:password@eureka:8761/eureka/ EUREKA_INSTANCE_PREFER_IP_ADDRESS: 'true' inventory: container_name: inventory image: sfgbeerworks/mscc-beer-inventory-service:latest ports: - "9501:9501" depends_on: - jms - eureka - config environment: EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://username:password@eureka:8761/eureka/ EUREKA_INSTANCE_PREFER_IP_ADDRESS: 'true' SPRING_ARTEMIS_HOST: jms SPRING_DATASOURCE_URL: jdbc:mysql://host.docker.internal:3306/beer_inventory_2?useUnicode=true&characterEncoding=UTF-8 SPRING_PROFILES_ACTIVE: local-discovery,local SPRING_CONFIG_IMPORT: optional:configserver:http://config:8888 SPRING_CLOUD_CONFIG_URI: http://config:8888 SPRING_CLOUD_CONFIG_PROFILE: local SPRING_APPLICATION_NAME: inventory
restart: on-failure
beer: container_name: beer image: sfgbeerworks/mscc-beer-service:latest ports: - "9500:9500" depends_on: - jms - eureka - config - inventory environment: EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://username:password@eureka:8761/eureka/ EUREKA_INSTANCE_PREFER_IP_ADDRESS: 'true' SPRING_ARTEMIS_HOST: jms SPRING_DATASOURCE_URL: jdbc:mysql://host.docker.internal:3306/beer_service_2?useUnicode=true&characterEncoding=UTF-8 SPRING_PROFILES_ACTIVE: local-discovery,local SFG_BEER_INVENTORY_SERVICE: http://inventory:9501 SFG_BEER_ORDER_SERVICE: http://order:9502 SPRING_CONFIG_IMPORT: optional:configserver:http://config:8888 SPRING_CLOUD_CONFIG_URI: http://config:8888 SPRING_CLOUD_CONFIG_PROFILE: local SPRING_APPLICATION_NAME: beer-service
restart: on-failure
order: container_name: order image: sfgbeerworks/mscc-beer-order-service ports: - "9502:9502" depends_on: - jms - eureka - config - beer environment: EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://username:password@eureka:8761/eureka/ EUREKA_INSTANCE_PREFER_IP_ADDRESS: 'true' SPRING_PROFILES_ACTIVE: local-discovery,local SPRING_ARTEMIS_HOST: jms SPRING_DATASOURCE_URL: jdbc:mysql://host.docker.internal:3306/beer_order_service_2?useUnicode=true&characterEncoding=UTF-8 SFG_BREWERY_BEER-SERVICE-HOST: http://beer:9500 SPRING_CONFIG_IMPORT: optional:configserver:http://config:8888 SPRING_CLOUD_CONFIG_URI: http://config:8888 SPRING_CLOUD_CONFIG_PROFILE: local SPRING_APPLICATION_NAME: order
restart: on-failure
gateway: image: sfgbeerworks/mscc-gateway ports: - "9090:9090" depends_on: - eureka - config - inventory - beer - order environment: EUREKA_CLIENT_SERVICEURL_DEFAULTZONE: http://username:password@eureka:8761/eureka/
Comment From: BorisSpring
when i check url in my browser it works perfecly fine and when i run everytihng localy outside of container it works fine
Comment From: ryanjbaxter
Please open another issue with a sample to reproduce the problem
Comment From: BorisSpring
Sorry when i create issue like this https://github.com/spring-cloud/spring-cloud-config/issues/2355 is it visible to everyone because no one answer me :) im just cuirous . thanks