springboot Unpacked War won't start with tomcat on JVM classpath

An unpacked war won't start when there is an embedded tomcat on the JVM classpath.

An MVCE can be found here: https://github.com/larsgrefer/tomcat-exception-demo

Just execute ./gradlew triggerBug

The actual exception can be found here: https://github.com/larsgrefer/tomcat-exception-demo/runs/488549568?#step:5:50

java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Error starting the loader
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[na:na]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) ~[tomcat-embed-core.jar:9.0.31]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core.jar:9.0.31]
        at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) ~[na:na]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.startup.Tomcat.start(Tomcat.java:467) ~[tomcat-embed-core.jar:9.0.31]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.initialize(TomcatWebServer.java:107) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.<init>(TomcatWebServer.java:88) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getTomcatWebServer(TomcatServletWebServerFactory.java:438) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory.getWebServer(TomcatServletWebServerFactory.java:191) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.createWebServer(ServletWebServerApplicationContext.java:180) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onRefresh(ServletWebServerApplicationContext.java:153) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:544) ~[spring-context-5.2.3.RELEASE.jar:5.2.3.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.4.RELEASE.jar:2.2.4.RELEASE]
        at de.larsgrefer.tomcatexceptiondemo.TomcatExceptionDemoApplication.main(TomcatExceptionDemoApplication.java:10) ~[classes/:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) ~[exploded/:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) ~[exploded/:na]
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) ~[exploded/:na]
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:58) ~[exploded/:na]
Caused by: org.apache.catalina.LifecycleException: Error starting the loader
        at org.apache.catalina.loader.WebappLoader.startInternal(WebappLoader.java:414) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4997) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) ~[tomcat-embed-core.jar:9.0.31]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) ~[tomcat-embed-core.jar:9.0.31]
        at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) ~[na:na]
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) ~[tomcat-embed-core.jar:9.0.31]
        ... 37 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[na:na]
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[na:na]
        at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
        at java.base/java.lang.Class.forName(Class.java:315) ~[na:na]
        at org.apache.catalina.loader.WebappLoader.createClassLoader(WebappLoader.java:504) ~[tomcat-embed-core.jar:9.0.31]
        at org.apache.catalina.loader.WebappLoader.startInternal(WebappLoader.java:390) ~[tomcat-embed-core.jar:9.0.31]
        ... 46 common frames omitted

Context

We package our application in the war format, because it contains JSP pages, which won't work with other packaging options.

In the start script for the application we include an additional folder in the classpath which allows our customer to supply additional jars to the application which aren't part of our distribution. This is mainly used for jdbc drivers and commercial libraries. Today one of our customers tried to update the tomcat version by placing newer tomcat-jars there which resulted in the above error.

Comment From: wilkinsona

Thanks for the sample.

The presence of the Tomcat classes on the classpath of the system class loader will result in them being loaded in favour of those on the classpath of Boot’s LaunchedURLClassloader. These Tomcat classes will then be unable to load anything only available to the LaunchedURLClassLoader. This results in the ClassNotFoundException shown above.

I don’t think there’s anything we can do about this as it is standard JVM behaviour with a parent-first ClassLoader. Switching to parent-last may solve this specific problem but it would potentially introduce a whole host of other problems.

You may want to consider adding the jars in your extra directory to the classpath of the LaunchedURLClassLoader rather than the system class loader. I think you could probably achieve this with a custom subclass of WarLauncher

Comment From: larsgrefer

That makes sense. Thanks for the answer.

I'll see if I can get everything in the same classloader.

Comment From: larsgrefer

Just in case someone else with the same problem comes here. This is the solution I found:

Instead of running

java -classpath "/opt/extra_libs/*:." org.springframework.boot.loader.WarLauncher

We now use this:

java -classpath "/opt/extra_libs/*:./WEB-INF/classes:./WEB-INF/lib-provided/*:./WEB-INF/lib/*" com.example.app.Main

Doing so, both the Tomcat (from /opt/extra_libs) and Spring Boot (from ./WEB-INF/lib) are in the same classloader, so everthing works as expected.

This solution was inspired by https://docs.spring.io/spring-boot/docs/2.2.5.RELEASE/reference/html/deployment.html#containers-deployment