We're trying to upgrade Spring Cloud Server app from Spring Cloud 2020.0.4 to 2021.0.0, corresponding upgrade for Spring Boot: 2.5.7 -> 2.6.1 We're using git SSH configuration using properties, repo is hosted on bitbucket. When running with 2020.0.4 everything works fine, no exceptions, config server starts as expected. We use this config server app for a while.
Our configuration in application.yaml:
spring:
cloud:
config:
server:
git:
uri: git@bitbucket.org:my-repo-name.git
skipSslValidation: true
timeout: 10
search-paths: '{profile}'
cloneOnStart: true
ignoreLocalSshSettings: true
privateKey: |
-----BEGIN RSA PRIVATE KEY-----
....
-----END RSA PRIVATE KEY-----
After changing version to 2021.0.0 and starting the app, different behaviour is observed (not seen in 2020.0.4) re ssh when running locally. Somehow this line is printed now, requiring to enter passphrase:
Enter passphrase for /Users/my-user-name/.ssh/my-user-Bitbucket:
After entering valid password , or just pressing Enter following exception is thrown and app is aborting:
2021-12-17 17:59:18.308 WARN 18400 --- [ main] .c.s.e.MultipleJGitEnvironmentRepository : Error occured cloning to base directory.
org.eclipse.jgit.api.errors.TransportException: git@bitbucket.org:my-repo-name.git: failed to send channel request at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:224) at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:303) at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:178) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.cloneToBasedir(JGitEnvironmentRepository.java:658) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.initClonedRepository(JGitEnvironmentRepository.java:363) at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.afterPropertiesSet(JGitEnvironmentRepository.java:284) at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.afterPropertiesSet(MultipleJGitEnvironmentRepository.java:66) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:730) at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:412) at org.springframework.boot.SpringApplication.run(SpringApplication.java:302) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1290) at com.healthyinteractions.configserver.ConfigServerApplication.main(ConfigServerApplication.java:12) Caused by: org.eclipse.jgit.errors.TransportException: git@bitbucket.org:my-repo-name.git: failed to send channel request at org.eclipse.jgit.transport.JschSession$JschProcess.(JschSession.java:167) at org.eclipse.jgit.transport.JschSession.exec(JschSession.java:77) at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.(TransportGitSsh.java:289) at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:153) at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:142) at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:94) at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1309) at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:213) ... 24 common frames omitted Caused by: com.jcraft.jsch.JSchException: failed to send channel request at com.jcraft.jsch.Request.write(Request.java:65) at com.jcraft.jsch.RequestEnv.request(RequestEnv.java:52) at com.jcraft.jsch.ChannelSession.sendRequests(ChannelSession.java:222) at com.jcraft.jsch.ChannelExec.start(ChannelExec.java:41) at com.jcraft.jsch.Channel.connect(Channel.java:152) at org.eclipse.jgit.transport.JschSession$JschProcess.(JschSession.java:159) ... 31 common frames omitted
Same exception is thrown when trying to run the app in the cloud.
I didn't find any breaking changes mentioned in the documentation re Spring Cloud Config Server due to 2021.0.0/2.6.x release. Am I missing something here? Please advise.
Comment From: j0rzsh
Happening the same for me with a passwordless privateKey and these versions:
Spring boot: 2.6.2 Spring-cloud: 2021.0.0
The exact same configuration works on this setup:
Spring boot: 2.5.4 Spring-cloud: 2020.0.5
In my case the configuration is:
Application.yml:
spring:
application:
name: configserver
cloud:
config:
server:
git:
uri: git@bitbucket.org:workspace/my-private-repository.git
ignoreLocalSshSettings: true
And environment variable set SPRING_CLOUD_CONFIG_SERVER_GIT_PRIVATEKEY with the content of the private key with access to that private repository
Compiled using maven:3.8.4-openjdk-17 from docker hub and run using openjdk:17-jdk
Comment From: blazeej
I think the problem occurs when the remote git repository doesn't support git protocol version V2. Starting from version 5.11 JGit uses protocol V2 by default (release notes: https://wiki.eclipse.org/JGit/New_and_Noteworthy/5.11). According to the documentation, JGit should fall back to older protocol, but I did some debugging and it looks like it doesn't. Spring-cloud-config uses JGit version 5.12.0.202106070339-r (change: https://github.com/spring-cloud/spring-cloud-config/commit/8d8451b694f22cebb64781b412fa78c45da0db97).
I was able to configure the gitlab instance to use protocol V2, which fixed the issue. Before that I tried to override the JGit version to the older one in maven dependencyManagement which also worked fine - the config server was running without any issues.
Comment From: j0rzsh
@blazeej Wow that is a great answer really! I was able to make it work with that. Many many thanks.
However, I've tried to use the previous version that uses as default the V2 protocol (5.10.0.202012080955-r) and wasn't working either. The same result with 5.8.1.202007141445-r version. It worked for me with 5.7.0.202003110725-r version.
I am not a Java expert nor a developer so I wasn't able to find the reason of that, sorry! But this is happening with Bitbucket Cloud and I presume (according to the logs) that is the same git vendor that @healthy-interac is using so maybe is time to open a ticket to Bitbucket :)
Many thanks again.
PS: I have this in the pom.xml:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>5.7.0.202003110725-r</version>
</dependency>
</dependencies>
</dependencyManagement>
Comment From: blazeej
@j0rzsh I'm glad I could help!
I found out an alternative workaround, which doesn't require switching the JGit dependency in runtime - you can configure git client to always use protocol version 1. This works, event if you have ignoreLocalSshSettings: true in config-server configuration.
Just put the following section into the global git config file (default location is ~/.gitconfig):
[protocol]
version = 1
There are cases when one can't or don't want to modify the global git config, so I think there should be an option to specify protocol version in org.springframework.cloud.config.server.environment.JGitEnvironmentProperties to keep all the config in a single spring configuration file.
Comment From: healthy-interac
@j0rzsh if you want to keep maven-based workaround suggested by @blazeej you need to specify 2 more dependencies of JGit with version 5.10:
<properties>
<jgit.version>5.10.0.202012080955-r</jgit.version>
</properties>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>${jgit.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit.http.apache</artifactId>
<version>${jgit.version}</version>
</dependency>
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit.ssh.jsch</artifactId>
<version>${jgit.version}</version>
</dependency>
I already opened a support ticket with Atlassian Bitbucket, will update if they have easy configuration solution similar to what GitLab offers.
Comment From: healthy-interac
@j0rzsh I'm glad I could help!
I found out an alternative workaround, which doesn't require switching the JGit dependency in runtime - you can configure git client to always use protocol version 1. This works, event if you have
ignoreLocalSshSettings: truein config-server configuration.Just put the following section into the global git config file (default location is
~/.gitconfig):
[protocol] version = 1
@blazeej thank you for offering more options! ;) This one will be hard to get working for cloud deployments though?
Comment From: yesmanmx
There are cases when one can't or don't want to modify the global git config, so I think there should be an option to specify protocol version in org.springframework.cloud.config.server.environment.JGitEnvironmentProperties to keep all the config in a single spring configuration file.
Bad Hack: git config file, created by jgit can be tweaked using next code (can be placed into main method before spring application run)
Repository.getGlobalListenerList().addConfigChangedListener((it) -> {
if (
StringUtils.isBlank(
it.getRepository().getConfig().getString(
ConfigConstants.CONFIG_PROTOCOL_SECTION,
null,
ConfigConstants.CONFIG_KEY_VERSION
)
)
) {
it.getRepository().getConfig().setString(
ConfigConstants.CONFIG_PROTOCOL_SECTION,
null,
ConfigConstants.CONFIG_KEY_VERSION,
TransferConfig.ProtocolVersion.V0.version()
);
}
});
Comment From: j0rzsh
@j0rzsh if you want to keep maven-based workaround suggested by @blazeej you need to specify 2 more dependencies of JGit with version 5.10:
```
5.10.0.202012080955-r
org.eclipse.jgit org.eclipse.jgit ${jgit.version} org.eclipse.jgit org.eclipse.jgit.http.apache ${jgit.version} ``` org.eclipse.jgit org.eclipse.jgit.ssh.jsch ${jgit.version} I already opened a support ticket with Atlassian Bitbucket, will update if they have easy configuration solution similar to what GitLab offers.
That is working perfectly thanks!
As @healthy-interac said the workaround for the .ssh/config file is not the best imo for cloud/docker based solutions I suppose. Also I will give a try at @YesmanMaxim's solution.
Comment From: healthy-interac
As @healthy-interac said the workaround for the .ssh/config file is not the best imo for cloud/docker based solutions I suppose.
I wonder if setting protocol v1 as default in Git local config (instead of global one initially suggested by @blazeej ) will do the job. If it does for both local & cloud environments, then it will be cleaner than downgrading the JGit versions, for a time being.
git config --local protocol.version 1
Comment From: healthy-interac
Bad Hack: git config file, created by jgit can be tweaked using next code (can be placed into main method before spring application run)
Repository.getGlobalListenerList().addConfigChangedListener((it) -> { if ( StringUtils.isBlank( it.getRepository().getConfig().getString( ConfigConstants.CONFIG_PROTOCOL_SECTION, null, ConfigConstants.CONFIG_KEY_VERSION ) ) ) { it.getRepository().getConfig().setString( ConfigConstants.CONFIG_PROTOCOL_SECTION, null, ConfigConstants.CONFIG_KEY_VERSION, TransferConfig.ProtocolVersion.V0.version() ); } });
Literals of ConfigConstants do not exist anymore, TransferConfig.ProtocolVersion is not public. No go.
Comment From: healthy-interac
Bitbucket Cloud opened a ticket to add full Git V2 support, you're welcome to vote: https://jira.atlassian.com/browse/BCLOUD-21649
Comment From: tomaswolf
This is a bug in JSch. The bitbucket.org server does not allow setting environment variables, and with protocol V2, one needs to set the environment variable GIT_PROTOCOL to "version=2". JSch sends the request to set this environment variable in the SSH channel session with a flag indicating that it wants a reply, so the server sends back an error reply, and JSch terminates the SSH connection.
This behavior in JSch is wrong. OpenSSH and also Apache MINA sshd send this SSH request with a flag indicating that they don't want a reply, so the server just silently doesn't set that environment variable, and protocol V0/V1 is used.
Compare Eclipse bug 576922.
JGit does no longer support JSch. Spring Cloud should migrate to the newer supported transport using Apache MINA sshd (bundle org.eclipse.jgit.ssh.apache).
(Or migrate to using https://github.com/mwiede/jsch instead, but the JGit team cannot give any support for the combination of that JSch fork and JGit. See also mwiede/jsch#93.)
Comment From: ryanjbaxter
Thanks @tomaswolf
Related to https://github.com/spring-cloud/spring-cloud-config/issues/1901
Comment From: QuinnBast
How is this completed? There is still no way of setting the git protocol version in spring (via environment variables maybe?) and the workarounds suggested are just that -- workarounds, and are hacky at-best. Running in containers/cloud these workarounds are extremely difficult to apply.
Comment From: healthy-interac
@QuinnBast As Bitbucket still "gathering interest" on this one, we've ended up with switching from ssh to https...
Comment From: ryanjbaxter
That would be a separate enhancement request
Comment From: mladenSyn
@j0rzsh if you want to keep maven-based workaround suggested by @blazeej you need to specify 2 more dependencies of JGit with version 5.10:
```
5.10.0.202012080955-r
org.eclipse.jgit org.eclipse.jgit ${jgit.version} org.eclipse.jgit org.eclipse.jgit.http.apache ${jgit.version} ``` org.eclipse.jgit org.eclipse.jgit.ssh.jsch ${jgit.version} I already opened a support ticket with Atlassian Bitbucket, will update if they have easy configuration solution similar to what GitLab offers.
Thank you very much!