It's pretty common to want a separate application.properties to contains passwords and keys that shouldn't be checked into version control. Currently this can be done with a profile but it would be nice to support something out of the box.
Comment From: cemo
+1
Comment From: amr
I believe currently this can be one by placing a application.properties at the root and making sure it doesn't get added to VCS (using .gitignore, etc). Slightly related is #2240 which I just closed.
Comment From: philwebb
Another good work-around is to add spring.profiles.active=secrets to the main application.properties then add a application-secrets.properties that is in .gitignore.
Comment From: dsyer
Isn't that the same as "supporting something out of the box"? What else can we do? We could scan for .gitignore and ensure that the app won't start locally if application-secrets.properties is not ignored or something?
Comment From: philwebb
I think just adding -secrets as an additional suffix would be enough. Something to save you needing to add spring.profiles.active.
Comment From: philwebb
We should probably use a different convention secrets.properties for example.
Comment From: amr
Do you think this should be loaded by default with tests too? I don't think so myself.
Comment From: dsyer
Yuck. What's the point if it isn't loaded the same in all executions? I see what you mean though - CI builds will fail if the tests rely on a local value. I wonder if this is such a good idea after all.
Comment From: joshiste
In our team we had a similar discussion if we should take the same approach to our sensitive configuration parts and decided not to and rather stick to spring-clouds encryption feature. Imho, since loading additional config-files is already possible, i wouldn't do anything here by default.
Comment From: philwebb
I think we'll just live with the current situation for now and just suggest use the spring.profiles.active technique.
Comment From: alimate
@philwebb @dsyer As the likes of Docker Swarm, Kubernetes, and other container scheduling tools are becoming more common, How about providing an out of the box support for Swarm Secrets or Kubernetes Secrets in Spring Boot configuration and property source hierarchy?
How about adding a couple of PropertySource providers to scan for these container secrets and add their provided properties (Mostly secrets) to the Environment?
Comment From: wilkinsona
@alimate How do you see that working? Looking at the Kubernetes documentation, it appears that secrets can be mounted as a volume with a name chosen by the user, projected into files with names chosen by the user, or mapped to environment variables. In the first two cases it's not clear to me how Spring Boot would know where to look. In the third case, the existing support for consuming environment variables looks like it's already sufficient.
Comment From: spencergibb
We are working on that in spring cloud kubernetes that will be part of the Greenwich release train
Comment From: alimate
How do you see that working?
Both Docker Swarm and Kubernetes can provide secrets by mounting them to a specific mounting point. Suppose the exposed secrets are mounted on /run/secrets, then the PropertySource provider should scan the /run/secrets directory and for each file add a property named after the filename with the file content being the property value.
For example, if:
> ls /run/secrets
database_password
and:
> cat /run/secrets/database_password
123
Then this new PropertySource provider should add a property named database_password with 123 as its value.
In the first two cases it's not clear to me how Spring Boot would know where to look
We can consider sensible default values for the Mounting Point, say the /run/secrets directory. Also, we could allow developers to customize that directory.
In one of our projects, We've added the support for Swarm/Kubernetes secrets by registering an EnvironmentPostProcessor:
/**
* Adding first class support for Swarm and Kubernetes Secrets in Spring Boot configuration
* and property source hierarchy. This way any Spring Boot applications can read properties
* from Docker Swarm/Kubernetes secrets.
*
* <h3>Implementation Details</h3>
* Currently we only support Swarm style secrets file. By default when Swarm manager exposes a
* secret to a service, it would be mounted under {@code /run/secrets/{secret_name}} file. With
* this arrangement, each file in the {@code /run/secrets} directory would be the property name
* and its content would be the property value.
*
* <a href="https://docs.docker.com/engine/swarm/secrets/">Swarm Secrets</a>
*
* @see EnvironmentPostProcessor
* @see SystemEnvironmentPropertySource
*
* @author Ali Dehghani
*/
public class ContainerSecretsAwareEnvironmentPostProcessor implements EnvironmentPostProcessor {
/**
* The logger.
*/
private static final Logger log = LoggerFactory.getLogger(ContainerSecretsAwareEnvironmentPostProcessor.class);
/**
* This property allows to customize the default {@code /run/secrets/} directory for
* container property lookups.
*/
private static final String SECRETS_DIR_PROPERTY = "container.secrets-dir";
/**
* The default directory inside the container to lookup for secrets that pushed by
* the container scheduler.
*/
private static final String DEFAULT_SECRET_DIR = "/run/secrets/";
/**
* Scans the secrets directory and for each file found there:
* <pre>
* for each_file in secrets_dir:
* addProperty(filename, fileContent)
* </pre>
*
* @param environment The Spring environment to customize
* @param application The Spring Boot bootstrapper
*/
@Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
String secretsDir = environment.getProperty(SECRETS_DIR_PROPERTY);
if (secretsDir == null || secretsDir.trim().isEmpty()) secretsDir = DEFAULT_SECRET_DIR;
log.info("About to read container secrets from {}", secretsDir);
Map<String, Object> readProperties = readFromSharedSecrets(secretsDir);
if (!readProperties.isEmpty()) {
log.info("Adding {} secrets from container scheduler", readProperties.size());
SystemEnvironmentPropertySource propertySource =
new SystemEnvironmentPropertySource("container-secrets", readProperties);
// A property with lowest possible priority
environment.getPropertySources().addLast(propertySource);
}
}
/**
* Scans the {@code secretsDir} directory and for each file, add a property named after the
* file paired with its content as the property value.
*
* @param secretsDir Represents the secrets directory
* @return Map of property names and property values.
*/
private Map<String, Object> readFromSharedSecrets(String secretsDir) {
Path directory;
try {
directory = Paths.get(secretsDir);
} catch (Exception e) {
log.warn("The provided container secrets directory was invalid: {}", secretsDir);
return Collections.emptyMap();
}
if (!Files.exists(directory)) {
log.warn("The provided container secrets directory not found: {}", secretsDir);
return Collections.emptyMap();
}
try {
return Files.list(directory)
.filter(this::isFile)
.collect(toMap(this::filename, this::content));
} catch (Exception e) {
log.warn("Failed to read secrets from secrets directory", e);
return Collections.emptyMap();
}
}
/**
* @param path The path to inspect
* @return Is the given {@code path} represents a file?
*/
private boolean isFile(Path path) {
return path.toFile().isFile();
}
/**
* @param path The path to extract its filename
* @return The filename
*/
private String filename(Path path) {
return path.getFileName().toString();
}
/**
* @param path Represents a path to a file
* @return The corresponding file content
*/
private String content(Path path) {
try {
return Files.lines(path).filter(line -> !line.trim().isEmpty()).collect(joining());
} catch (IOException e) {
log.warn("Failed to read the secret file", e);
return "";
}
}
}
Comment From: dsyer
Please consider contributing that to Spring Cloud (e.g. in the project that @spencergibb mentioned).
Comment From: spencergibb
AFAIK it should be in the works if not working already
Comment From: philwebb
@spencergibb This might be a good candidate to pull up at some point if it's a minimal piece of code. Keep us posted.
Comment From: molexx
I see this is now supported in spring-cloud-kubernetes. Could it be available for Docker Swarm deployments too please?
Comment From: nickspacek
I'm trying to find out if this was added to Spring Cloud for Docker Swarm; I don't see any Docker or Docker Swarm-related projects in the Spring Cloud group of projects. Is Docker Swarm intentionally excluded from this group?
Comment From: dsyer
I'm not sure what you mean by the "Intentional", but I wouldn't read anything into it. The code above is still interesting IMO, but it didn't make it's way into any Spring Boot (or Spring Cloud) projects yet. You can get the same result with a very simple and generic init container in Kubernetes. Probably Swarm has similar features?
The layout of the "secrets" above is very close to what is starting to come out of the Cloud Native Buildpacks specs (https://github.com/buildpack/spec/blob/master/extensions/bindings.md), so I wouldn't be surprised to see something that worked with that, and could be adapted to the example above. Whether that ends up being an init container or native in Spring Boot is an open question. It's not really anything to do with secrets (as in credentials or encrypted data) per se though and there are other plans in the works that might reflect the feature better than this issue.