springboot References to external types in Boot's javadoc are no longer links

We appear to have lost the links to third-party types in Boot's javadoc. Links are present in 2.1.5 but missing in 2.1.6 (and later as far as I can tell). Those two examples are for Spring Framework. Links to types from other projects also appear to be affected.

Comment From: dreis2211

I tried reproducing this locally, but wasn't able to do so.

What I find interesting though is the following output taken from one of the latest builds:

[INFO] --- maven-javadoc-plugin:3.1.1:jar (attach-javadocs) @ spring-boot-docs ---
[INFO] The goal 'org.apache.maven.plugins:maven-javadoc-plugin:3.1.1:javadoc' has not been previously called for the module: 'org.springframework.boot:spring-boot-starters:pom:2.3.0.BUILD-SNAPSHOT'. Trying to invoke it...
[WARNING] Creating fake javadoc directory to prevent repeated invocations: /tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-starters/target/apidocs
[INFO] The goal 'org.apache.maven.plugins:maven-javadoc-plugin:3.1.1:javadoc' has not been previously called for the module: 'org.springframework.boot:spring-boot-starter-web:jar:2.3.0.BUILD-SNAPSHOT'. Trying to invoke it...
[WARNING] Creating fake javadoc directory to prevent repeated invocations: /tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-starters/spring-boot-starter-web/target/apidocs
Jan 08, 2020 2:31:25 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://docs.spring.io:80: Connection reset
Jan 08, 2020 2:31:25 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://docs.spring.io:80
Jan 08, 2020 2:31:25 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://docs.spring.io:80: Connection reset
Jan 08, 2020 2:31:25 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://docs.spring.io:80
Jan 08, 2020 2:31:25 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://docs.spring.io:80: Connection reset
Jan 08, 2020 2:31:25 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://docs.spring.io:80
Jan 08, 2020 2:31:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://docs.spring.io:80: Connection reset
Jan 08, 2020 2:31:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://docs.spring.io:80
Jan 08, 2020 2:31:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://docs.spring.io:80: Connection reset
Jan 08, 2020 2:31:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://docs.spring.io:80
Jan 08, 2020 2:31:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://docs.spring.io:80: Connection reset
Jan 08, 2020 2:31:26 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://docs.spring.io:80
[ERROR] Error fetching link: /tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-starters/target/apidocs. Ignored it.
[ERROR] Error fetching link: /tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-starters/spring-boot-starter-web/target/apidocs. Ignored it.

Shouldn't it be https://docs.spring.io?

Cheers, Christoph

Comment From: dreis2211

Interestingly, running the following command

curl -I https://docs.spring.io/spring-framework/docs/5.2.3.BUILD-SNAPSHOT/javadoc-api

shows the following Location header:

HTTP/2 301
...
location: http://docs.spring.io/spring-framework/docs/5.2.3.BUILD-SNAPSHOT/javadoc-api/
...

Could it be that the Concourse setup is blocking http requests? Maybe that helps identifying the root cause. If not, sorry for the noise.

Comment From: dreis2211

In general I'd say the behaviour is correct though when I read through the javadoc documentation:

When javadoc is run without the -link option and encounters a name that belongs to an externally referenced class, it prints the name with no link

When I take a look at the docs for FlywayEndpointAutoConfiguration in 2.1.5 for example, it also shows org.flywaydb.core.Flyway.class fully qualified already.

So I'd say the link to the Spring-Framework docs is broken and thus creates more fully qualified unlinked classes than usual. We could add a couple of more <links> to the javadoc config for some more libraries, though.

Let me know what you think.

Comment From: wilkinsona

Thanks, @dreis2211. That's really helpful.

Could it be that the Concourse setup is blocking http requests?

Yep, all our CI environments block port 80 outbound. This was done as part of our response to the ecosystem-wide MITM vulnerability that was disclosed in June last year.

The http link with the trailing slash will redirect again to an equivalent https location but I suspect that the javadoc tool only follows a single redirect. Going via http is undesirable anyway as it creates the possibility of a MITM attack. It sounds like we should be able to work around it by including trailing slashes in the links. I'll also raise it internally to see if we can redirect straight to https.

Comment From: dreis2211

Is that matching the timeframe of the 2.1.6.RELEASE by any chance? 😉

As MJAVADOC-532 apparently got fixed that made the removal of the trailing slash necessary, I guess that's the most pragmatic solution. I'd be happy to provide a PR if you want me to.

Comment From: wilkinsona

So I'd say the link to the Spring-Framework docs is broken and thus creates more fully qualified unlinked classes than usual.

That's what I think is happening too.

We could add a couple of more to the javadoc config for some more libraries, though.

I'd like to use this issue to get things back to how they used to be. We should then consider a separate issue to link to more third-party javadoc. The list of projects that we link to seems rather arbitrary at the moment.

Comment From: mbhave

Closing in favor of PR #19581.

@wilkinsona not sure if you wanted to keep this issue open to verify that we redirect to https directly once that's been fixed internally. Feel free to reopen in that case.