We are running with the Edgware SR2 build.
We have set the spring.cloud.config.server.git.timeout value (which is seconds for the unit) to 10 for one release and 120 for another. It does not seem to have any effect on the actual timeout. We see git response timeout errors on occasion. In every case, it reports timeout after 5 seconds:
Caused by: java.io.InterruptedIOException: Read timed out after 5,000 ms at org.eclipse.jgit.util.io.TimeoutInputStream.readTimedOut(TimeoutInputStream.java:141) at
org.eclipse.jgit.util.io.TimeoutInputStream.read(TimeoutInputStream.java:114) at
org.eclipse.jgit.util.IO.readFully(IO.java:246) at
org.eclipse.jgit.transport.PacketLineIn.readLength(PacketLineIn.java:186) at
org.eclipse.jgit.transport.PacketLineIn.readString(PacketLineIn.java:138) at
org.eclipse.jgit.transport.BasePackConnection.readAdvertisedRefsImpl(BasePackConnection.java:195) at
org.eclipse.jgit.transport.BasePackConnection.readAdvertisedRefs(BasePackConnection.java:176)
How can we change the timeout? Is this broken in the Edgware SR2 build? Are there separate settings for connection and response timeouts perhaps?
Comment From: spencergibb
Can you try with the latest Edgware.SR5?
Comment From: spring-projects-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: marnee01
I tried with Edgwar SR5. The result was the same.
Comment From: spencergibb
During debugging, I can see the jgit CloneCommand.setTimeout() called with the correct value.
Can you post the whole stack trace and your configuration?
Comment From: marnee01
Here is the relevant config:
spring:
application:
name: configserver
cloud:
config:
server:
git:
# Seconds
timeout: 1
# Seconds
refreshRate: 60
# Default
uri : ssh://git@bitbucket.company.com/project_configs/{application}.git
repos:
AWE:
pattern: AWE*
searchPaths: HubWrapper
uri : ssh://git@bitbucket.company.com/project_configs/differentrepo.git
security:
basic:
enabled: true
path: ["/**"]
Here is the full stack trace:
org.springframework.cloud.config.server.environment.NoSuchRepositoryException: Cannot clone or checkout repository: ssh://git@bitbucket.company.com/project_configs/differentrepo.git
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:261)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:213)
at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:42)
at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository$PatternMatchingJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:288)
at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:164)
at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:45)
at org.springframework.cloud.config.server.environment.EnvironmentEncryptorEnvironmentRepository.findOne(EnvironmentEncryptorEnvironmentRepository.java:53)
at org.springframework.cloud.config.server.environment.EnvironmentController.labelled(EnvironmentController.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:166)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jgit.api.errors.TransportException: Read timed out after 5,000 ms
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:139)
at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:187)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:127)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.cloneToBasedir(JGitEnvironmentRepository.java:547)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.copyRepository(JGitEnvironmentRepository.java:522)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.createGitClient(JGitEnvironmentRepository.java:505)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:234)
... 109 common frames omitted
Caused by: org.eclipse.jgit.errors.TransportException: Read timed out after 5,000 ms
at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:370)
at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:301)
at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:291)
at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:247)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:160)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:122)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1138)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:130)
... 115 common frames omitted
Caused by: java.io.InterruptedIOException: Read timed out after 5,000 ms
at org.eclipse.jgit.util.io.TimeoutInputStream.readTimedOut(TimeoutInputStream.java:141)
at org.eclipse.jgit.util.io.TimeoutInputStream.read(TimeoutInputStream.java:114)
at org.eclipse.jgit.util.IO.readFully(IO.java:246)
at org.eclipse.jgit.transport.PacketLineIn.readLength(PacketLineIn.java:186)
at org.eclipse.jgit.transport.SideBandInputStream.needDataPacket(SideBandInputStream.java:158)
at org.eclipse.jgit.transport.SideBandInputStream.read(SideBandInputStream.java:140)
at org.eclipse.jgit.transport.PackParser.fill(PackParser.java:1131)
at org.eclipse.jgit.transport.PackParser.readPackHeader(PackParser.java:838)
at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:476)
at org.eclipse.jgit.internal.storage.file.ObjectDirectoryPackParser.parse(ObjectDirectoryPackParser.java:194)
at org.eclipse.jgit.transport.PackParser.parse(PackParser.java:448)
at org.eclipse.jgit.transport.BasePackFetchConnection.receivePack(BasePackFetchConnection.java:762)
at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:363)
... 122 common frames omitted
Comment From: marnee01
I want to set the timeout to 10s, but had it set to 1s in order to test that the setting was working.
Comment From: marnee01
I ran it through the debugger (still using Edgware SR5) with a breakpoint in CloneCommand.setTimeout(...). The value passed to that method is 1, as expected.
Comment From: spencergibb
That's what I did. I wonder if that is the connect timeout and there is a different read timeout?
Comment From: marnee01
Often for services, there is a connection timeout setting and a response timeout setting. I was wondering which operation this value was really supposed to affect.
We do have a repo that might be unusually large, so maybe this really never came up before. If I right-click on the "config-repo" local cache folder, the "Size on disk" is 6 MB. (We have migrated ~115 heavy-weight non-native-Spring desktop apps and two dozen non-native Spring services to using ConfigServer. One file alone has many thousands of lines, though not sure how it all adds up to 6 MB.)
We are worried about switching fully to Config Server in our production environment now that we have almost all of our apps and services set up to use it. We are seeing that the average Config Server response time keeps rising the more properties we add. That would be expected, but whether the rise would be negligible or not was something we were not sure of. We see now it is not negligible.
So maybe most users have set the timeout because of slow internet connection or a large # of calls to Git that causes connection itself to be slow. They increase the timeout, and it works for them, so no one has complained. But we might be hitting a read timeout due to our large size.
Tonight I was testing over VPN and hit the timeout maybe 10x as often as usual. Another thing to suggest it's more probably that the error I'm seeing is a read versus connect timeout since of course downloading a lot of data over my cantankerous VPN is always slow.
I don't know what I could do locally to determine if it's a read versus connection timeout if the provided stack trace doesn't make it obvious.
Comment From: marnee01
I just read in another issue that the spring.cloud.config.server.git.basedir property is not a global property and applies only to the default git uri. It wouldn't apply to spring.cloud.config.server.git.repo.
So, is that also true for timeout and refreshRate? Do I have to set those for each repo? E.g. spring.cloud.config.server.git.repo.MessagingService.timeout and spring.cloud.config.server.git.repo.MessagingService.refreshRate? If I don't hear back by the time I get a chance, I will do some testing of that and will update this with my findings.
Comment From: marnee01
refreshRate does seem to be global.
Comment From: ryanjbaxter
So is there anything that needs to be done on our side then? Sounds like you just need to specify the timeouts for the git uri....
Comment From: marnee01
No, I believe this is an issue with the existing timeout parameter or is a lack of functionality. The timeout parameter does not seem to affect the Git read timeout. If it is supposed to affect the Git read timeout, it is a defect. If it does not affect the read timeout:
1) Rename it to connectionTimeout or something more descriptive and deprecate the original parameter. Update the documentation if there is any. 2) Please add a global Git readTimeout value.
Comment From: spencergibb
You say it is a bug, but I can not see in jgit how to set anything else.
Comment From: marnee01
To clarify: If it is supposed to affect Git read timeout, then it is a bug. If it does not appear to be a bug, then this is the request. Should I submit this request separately?
- Rename timeout to connectionTimeout or something more descriptive and deprecate the original parameter. Update the documentation if there is any.
- Please add a global Git readTimeout value.
Comment From: ryanjbaxter
What @spencergibb is saying is that there is no concept of a read timeout in all cases. When using a http url that applies but when you don't there is no read timeout.
Another thing to note is that there were some changes made around this code in the Finchley release train. If you try Finchley.SR3 and use an http git url if possible I am suspecting things might work better.
Comment From: marnee01
There would not be a connection timeout in all cases, either, would there? Yet there is still a timeout parameter. Having both read and connection timeout values be configurable is very common in software components, so it's unfortunate that this seems to be missing for read timeout.
We will be trying Finchley SR3 eventually, but not in the next few weeks. It's not a completely trivial change. I don't believe our architects want to switch to http git url. I guess this can be closed, then.
Comment From: spencergibb
My issue is that there is nowhere in jgit to set a read timeout.
Comment From: marnee01
It would be an enhancement to do update the JGitEnvironmentRepository to support a read time, correct? That is what I'm requesting.
(There is something I think I'm not understanding about this communication channel. Is this not a place to submit requests? JGitEnvironmentRepository class is in this project. Is there some reason it can not be modified for an enhancement such as a read timeout? I think I have seen other conversations here under spring-cloud-config where people have asked for enhancements and it was at least considered. Am I wrong about that? Do I need to submit the request some other way? Or are you saying that you actually can't modify code in this project for some reason?)
Comment From: spencergibb
We can't unless there is a place to set it in jgit. If there is, we can certainly plug into it.
Comment From: marnee01
By jgit, I thought you meant the JGitEnvironmentRepository class. Are you referring to something else?
Comment From: spencergibb
https://www.eclipse.org/jgit/
Comment From: marnee01
Sorry...thought you were using JGit as shorthand for that class. What you said above makes sense now.
I found a different conversation that pointed to code in jgit/TransportHttp that shows that JGit sets the same timeout value for both connect and read, but this is for Http. That ties in with what Ryan said, "When using a http url that applies but when you don't there is no read timeout."
When we can, we'll try Finchley SR3. I don't know if we'll ever be able to switch to https, though.
This can be closed.
Comment From: marnee01
For anyone interested in this topic:
I was able to affect the git read timeout value when using https for the git repo connection protocol, but only by specifying timeout on each git repo (in addition to the value for the default repo).
spring:
cloud:
config:
server:
git:
# Connection timeout in seconds. (Also the read timeout when using https.)
timeout: 10
uri : https://someuri
repos:
Service:
pattern: '*Service'
searchPaths: Service/{applicaton}
timeout: 10
uri : https://someotheruri
Additional Notes: - I am using Edgware SR5 and spring boot 1.5.14.RELEASE. - The timeout setting is still being ignored for ssh, even when setting it on each repo, as expected based on the other information in this conversation.
Comment From: danielkwinsor
Under spring-cloud-config Hoxton.SR11 (server 2.2.8) and jgit org.eclipse.jgit:org.eclipse.jgit:jar:5.1.3.201810200350-r I have tested that timeouts work for SSH.