Spring Cloud Config How to get stable set of properties from the Spring Cloud Config Server when trying to fetch properties from multiple sources

I have a requirement to fetch stable set of properties from Spring cloud config server. Example : I have configured multiple like spring.profiles.active=jdbc,git. So, as per the priority the cloud config server fetches properties from JDBC i.e, from properties table and later it will fetches from git repository. The org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository#findOne(java.lang.String, java.lang.String, java.lang.String, boolean) method is consolidating in to a Environment object containing List propertySources. In this case the list is multiple PropertySource objects. Problem statement: I have a property called app.scheduler.timeout=9000 in properties table(JDBC backend) and the same property is also present in the git -profile.properties file as app.scheduler.timeout=9001. If you see the final Environment object, it will have the properties in both the PropertySources. I know that the merging will be done at the Cloud Config Client Side by preparing the bootstrap property source.

Is there any way we can prepare or get only stable set of propertySource from the Spring Cloud Config Server itself? i.e, single propertySource object in the List.

Note: In the above example I have mentioned only one property but I want this merging should happen for all the properties in response sent to the cloud config client.

Comment From: ryanjbaxter

I am not sure what you mean by "stable set of propertySource". Depending on the order of the backends specified that will determine the order of the property sources and which ones will take priority.

Comment From: Santhoshinftech

Thank you for replying @ryanjbaxter. Let me elaborate the scenario. Based on the property spring.profiles.active=jdbc,git the cloud config server fetches the List propertySources (based on the order configured for the respective profile) and returns org.springframework.cloud.config.environment.Environment object in response to the Cloud Config Client.

Consider, I have a property configured for the application i.e., "app.scheduler.timeout=9000" and inserted same in the properties table as I am fetching from JDBC Backend. I have added the same property in the -.yml present in the git repository with a different value i.e., "app.scheduler.timeout=9999". If I request for latest properties from the Cloud Config Client then the Cloud Config Server will return org.springframework.cloud.config.environment.Environment object containing List propertySources. In this scenario, I will have atleast 2 PropertySource in the list(One from the JDBC and another one from git .yml file). As per the design the property value "app.scheduler.timeout=9000" from JDBC will take precedence due to the higest priority/precedence order given to it. In the Cloud Config client side it will consider this property and the property from the git file will be ignored for the property "app.scheduler.timeout".

Here my question is: Can we get single PropertySource inside the environment response object instead of List? Do we have any option to merge the List (in my case there are more than 2 PropertySource objects inside the List) into a single PropertySource based on the priority in Cloud Config Server side? Or any hook points to customize the code to get the desired result? As I feel it's a redundant property for the git source which is going to be ignored by the Cloud Config Client anyways.

Is there any other way to achieve this in Cloud Config Server so that I will get consolidated/single PropertySource inside the Environment object from the Cloud Config Server which will have properties merged from both JDBC backend and Git(redundant properties will be discarded based on priority)?

Please let me know if you need any other details. Thanks in advance!

Comment From: ryanjbaxter

Can we get single PropertySource inside the environment response object instead of List?

Why? It sounds like you just want to discard properties that wont be used by the Spring Boot app.

Comment From: Santhoshinftech

No, Not exactly. Here I am just trying to suppress/ignore the redundant properties from the List of PropertySources. Below is the example code what I wrote(just for reference). I have added a custom Environment controller and called the Spring cloud config server EnvironmentController from this:

`@RequestMapping(path = "/{name}/{profiles}/{label:.*}", produces = MediaType.APPLICATION_JSON_VALUE) public Environment labelled(@PathVariable String name, @PathVariable String profiles, @PathVariable String label) { Environment environment = environmentController.labelled(name, profiles, label); return mergeProperties(null); }

private Environment mergeProperties(Environment environment) {
    if(null != environment.getPropertySources() && !environment.getPropertySources().isEmpty()){
        List<PropertySource> propertySources = environment.getPropertySources();
        Collections.reverse(propertySources);
        Map<String, String> sourceMap = new LinkedHashMap<>();
        propertySources.forEach(propSource -> {
            if(null != propSource.getSource() && !propSource.getSource().isEmpty()){
                propSource.getSource().forEach((propName, propVal) -> sourceMap.put(String.valueOf(propName), String.valueOf(propVal)));
            }
        });
        String name = isEmptyString(environment.getName()) ? "" : environment.getName();
        String profile = isEmptyString(String.join(",", environment.getProfiles())) ? "" : String.join(",", environment.getProfiles());
        PropertySource propertySource = new PropertySource(name +"-"+profile, sourceMap);
        environment.getPropertySources().clear();
        environment.getPropertySources().add(propertySource);
    }
    return environment;
}`

The above code will reverses the PropertySource list and creates new propertySource from that by overriding the existing keys and values in the map.

I know the above implementation looks dirty. I am looking for some way of hook point or customization option to handle the above scenario.

Comment From: Santhoshinftech

Additional Information: Let me give you an example: Exactly, If the Cloud Config Client anyways going to discard the properties which are already present in the PropertySource of higher precedence then that property is no longer required for the cloud config client.

Consider the below example: The cloud Config Client is making below call to the Cloud Config Server. http://localhost:8888/sample-service/dev

The Cloud Config Server has spring.profiles.active=jdbc,git I have created and added below properties to the PROPERTIES table:

threadpool_min_size=10 threadpool_size=50 app.scheduler.timeout=9000

In the same way I have git cloned in the Cloud Config Server and have sample-service-dev.yml file in the cloned repo.

threadpool_max_size=100 threadpool_queue_size=100 threadpool_core_size=25 app.scheduler.timeout=9999

If I execute the above URL, Firstly the properties will be fetched from JDBC as the order for JDBC is 1 and later it will fetch from the git location as the order is given is 2. The cloud config server is going to return an Environment object which contains list of PropertySources one from JDBC and another one from class path source from git location.

Here what I am expecting is the PropertySource from JDBC should only have all the 3 properties from the table. And the second PropertySource from Git repo should have only 3 properties i.e,

threadpool_max_size=100 threadpool_queue_size=100 threadpool_core_size=25

as these properties are not fetched from the JDBC table. the last property i.e, app.scheduler.timeout=9999 should be ignored/removed in the Cloud Config server itself before returning to the Cloud Config Client application. as these property is only referred/used in the client application which was provided by the JDBC propertySource.

Is there any way to do this? what I feel is, the PropertySources should be merged based on the order/priority (into single propertySource or list of PropertySource). I want only final set of properties here from the Cloud Config server.

Currently, I am trying to customize this but unable to get proper hook point to do this. Any help will be appreciated to achieve the above.

Comment From: ryanjbaxter

I would start here https://github.com/spring-cloud/spring-cloud-config/blob/main/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/environment/CompositeEnvironmentRepository.java#L62

Comment From: Santhoshinftech

Thank you for replying ryanjbaxter.

Can you please give a sample code? I tried to do customization for the above mentioned class but It didn't help. Can you please give me more inform with same sample code? Thanks in advance.

Comment From: ryanjbaxter

If you provide your own CompositeEnvironmentRepository bean our default configuration should not kick in https://github.com/spring-cloud/spring-cloud-config/blob/b62216d814f8bfccb8c0c27024f82175a8993282/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/config/CompositeConfiguration.java#L38

Comment From: Santhoshinftech

Thank you for replying Ryan. But still the flow goes through the CompositeEnvironmentRespository class. and my CustomCompositeEnvironmentRespository bean repository has been added to the repository list of CompositeEnvironmentRespository class for iteration. Below is the bean I have added: @Bean @ConditionalOnMissingBean(SearchPathLocator.class) public CustomCompositeEnvironmentRepository customCompositeEnvironmentRepository() { return new CustomCompositeEnvironmentRepository(this.environmentRepos, properties.isFailOnCompositeError()); } I have also added CustomCompositeEnvironmentRepository class(which is a code replica of CompositeEnvironmentRepository class) as per your above comment It should go through my custom code for fetching the properties and run in loop similar like .CompositeEnvironmentRepository's findOne method. how to avoid execution of CompositeEnvironmentRepository's findOne method and execute my CustomCompositeEnvironmentRepository's findOne method for looping through the configured active profiles? Do I am missing anything here? Please let me know. Thank you!

Comment From: ryanjbaxter

Can you try returning CompositeEnvironmentRepository in your Bean declaration instead of CustomCompositeEnvironmentRepository?

If that doesn't work can you provide a sample so I can take a look?

Comment From: Santhoshinftech

@ryanjbaxter I tried returning CompositeEnvironmentRepository in your Bean declaration instead of CustomCompositeEnvironmentRepository. But it didn't help. I am able to create a CustomCompositeEnvironmentRepository bean and CompositeEnvironmentRepository bean is not getting called from CompositeConfiguration class. But from the environment controller, the call is still going to org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository#findOne(java.lang.String, java.lang.String, java.lang.String, boolean) ans newly added CustomCompositeEnvironmentRepository object is being added to the environmentRepositories and iterated further.

you can refer to the code in the below link: https://github.com/Santhoshinftech/cloud-config-custom-svc Please correct me if I am doing anything wrong. thanks!

Comment From: ryanjbaxter

CustomCompositeEnvironmentRepository should extend ComponsiteEnvironmentRepository.

Then in your configuration class add @AutoConfigureBefore(EnvironmentRepositoryConfiguration.class).

And you bean should be defined as follows

    @Bean
    @Primary
    @ConditionalOnMissingBean(SearchPathLocator.class)
    public CompositeEnvironmentRepository customCompositeEnvironmentRepository() {
        return new CustomCompositeEnvironmentRepository(this.environmentRepos, properties.isFailOnCompositeError());
    }

That appears to work.

Comment From: Santhoshinftech

Hi @ryanjbaxter ,

Thank you! the above solution worked for me. Now I am able to configure CustomCompositeEnvironmentRepository.

The code changes for the same is updated in the below Github repo for reference: https://github.com/Santhoshinftech/cloud-config-custom-svc

Thanks again.

Comment From: ryanjbaxter

Great glad that helped!

Comment From: Santhoshinftech

Hi @ryanjbaxter ,

I continued with integrating other profiles like Native, Git and Vault. Here, I am seeing an issue with the above code changes. Previously, I only integrated JDBC and Native profiles and it worked fine. Now, If I add Git profile with JDBC and Native profile in the Cloud Config Server I am getting the below error in the application start: `***** APPLICATION FAILED TO START

---

Description:

Parameter 0 of method resourceRepository in org.springframework.cloud.config.server.config.ResourceRepositoryConfiguration required a single bean, but 2 were found: - nativeEnvironmentRepository: defined by method 'nativeEnvironmentRepository' in class path resource [org/springframework/cloud/config/server/config/NativeRepositoryConfiguration.class] - defaultEnvironmentRepository: defined by method 'defaultEnvironmentRepository' in class path resource [org/springframework/cloud/config/server/config/GitRepositoryConfiguration.class]

Action:

Consider marking one of the beans as @Primary, updating the consumer to accept multiple beans, or using @Qualifier to identify the bean that should be consumed ` Without the previous custom code changes it is working without any issues. I tried to find a solution, but nothing helped. Can you please help me in fixing this issue?

I have added the application.properties change and pushed to the below repository: https://github.com/Santhoshinftech/cloud-config-custom-svc

Thanks in advance.

Comment From: Santhoshinftech

@ryanjbaxter , Tried the solving the issue by referring https://github.com/spring-cloud/spring-cloud-config/issues/984 but this also didn't help. Can you please help me in this?

Comment From: ryanjbaxter

I am busy getting ready for Spring One this week I will try and get back to you next week

Comment From: ryanjbaxter

Can you try defining your composite configuration using spring.cloud.config.server.composite? https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#composite-environment-repositories

Comment From: Santhoshinftech

Hi @ryanjbaxter , Not sure how, but after defining composite configuration using spring.cloud.config.server.composite it worked.

Thanks a lot.