I'm deploying a spring boot application in PCF, using the CredHub service for storing credentials. However, when I'm using version 2.2.5.RELEASE of spring boot, the placeholders from CredHub can't be resolved when words are separated with a . (or dash) in the placeholder. For example

vcap.services.credhub-instance-name.credentials.ssl.trust-password // not working vcap.services.credhub-instance-name.credentials.someplaceholder // working

I tried using version 2.2.4.RELEASE, that one works fine for both placeholders.

Comment From: wilkinsona

Thanks for the report. This could be a duplicate of #20432 or #20438. To eliminate the former, could you please try with a 2.2.6 snapshot available from https://repo.spring.io/.snapshot

Comment From: Vladg90

I tried with 2.2.6.BUILD-SNAPSHOT, same issue.

Comment From: wilkinsona

Thanks for trying a snapshot. To allow us to investigate further, can you please provide a small sample that reproduces the problem? You can share it with us by zipping it up and attaching it to this issue or by pushing it to a separate repository.

Comment From: Vladg90

Just attached a sample app.

properties-credhub-issue.zip

Comment From: wilkinsona

Thanks for the sample. Unfortunately, it leaves more moving parts than I would like to be able to effectively diagnose the problem. Let's try a different approach. Could you please share the value of the VCAP_SERVICES environment variable when the app is deployed in your CloudFoundry environment?

Comment From: Vladg90

Not sure if this is helpful, but here it is: System-Provided:

{
 "VCAP_SERVICES": {
  "credhub": [
   {
    "binding_name": null,
    "credentials": {
     "credhub-ref": "/credhub-service-broker/credhub/bbcf88dd-1a62-44d6-842b-1ef4c611d2de/credentials"
    },
    "instance_name": "credhub-instance",
    "label": "credhub",
    "name": "credhub-instance",
    "plan": "default",
    "provider": null,
    "syslog_drain_url": null,
    "tags": [
     "credhub"
    ],
    "volume_mounts": []
   }
  ]
 }
}

Comment From: wilkinsona

Thanks. It helps a little, but still leaves quite a bit of unknown, particularly as you're using a custom buildpack. The key thing that's missing is what is populating the Environment with credentials that are stored in CredHub such that ${vcap.services.credhub-instance.credentials.demo.property} can be resolved.

I can reproduce the symptom that you've reported by running the same app with a VCAP_SERVICES environment variable set to { "credhub": [ { "credentials": { "demo.property": "demoValue" }, "name": "credhub-instance" } ] }. It starts successfully with Spring Boot 2.2.4, fails with 2.2.5, but works with 2.2.6.BUILD-SNAPSHOT. This is what I expected and is why I suggested that this may be a duplicate of #20432.

Can you please double-check that you continue to experience the problem with the latest 2.2.6 snapshot? If you do and you'd like us to spend some more time investigating this problem, we're going to need a more complete sample that describes everything that's necessary to reproduce it.

Comment From: scottfrederick

The key thing that's missing is what is populating the Environment with credentials that are stored in CredHub such that ${vcap.services.credhub-instance.credentials.demo.property} can be resolved.

Cloud Foundry's container runtime does some magic in the application container to replace any credhub-ref field in the credentials structure of a service binding with the contents of the JSON text stored in CredHub. It does this using the CredHub interpolation endpoint.

While a Cloud Foundry user running a command like cf env app-name will see VCAP_SERVICES as in the example above, the application's environment might contain something more like: 

{
 "VCAP_SERVICES": {
  "credhub": [
   {
    "binding_name": null,
    "credentials": {
      "demo": {
        "property": "demoValue"
      }
    },
    ...
  ]
 }
}

The best way to see what the application actually gets in its Environment is to enable the Spring Boot Actuators /actuator/env endpoint and view the PropertySource added by the VcapEnvironmentPostProcessor.

Comment From: Vladg90

I double-checked, it worked indeed with 2.2.6 snapshot. Sorry I didn't get back to you yesterday, I was pretty busy with something else.

Comment From: wilkinsona

Great. Thanks for letting us know.