TestContainers related memory leaks

[rwinch]

When running the build I see warnings like this:

:spring-session-sample-javaconfig-redis:jar UP-TO-DATE
:spring-session-sample-javaconfig-redis:integrationTest
:spring-session-sample-javaconfig-redis:appAfterIntegrationTest
19:43:55.657 [localhost-startStop-1] INFO  o.s.w.c.s.AnnotationConfigWebApplicationContext - Closing Root WebApplicationContext: startup date [Thu Sep 07 19:43:50 CDT 2017]; root of context hierarchy
19:43:55.658 [localhost-startStop-1] INFO  o.s.c.s.DefaultLifecycleProcessor - Stopping beans in phase 2147483647

19:43:57 WARN  The web application [ROOT] appears to have started a thread named [dockerjava-netty-1-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:705)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:325)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:745)
19:43:57 WARN  The web application [ROOT] appears to have started a thread named [threadDeathWatcher-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.testcontainers.shaded.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:745)
19:43:57 WARN  The web application [ROOT] appears to have started a thread named [dockerjava-netty-1-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:705)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:325)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:745)

we should ensure all the threads are shutdown properly

[rwinch]

@vpavic Can you take a look at this?

[vpavic]

@rwinch I think the problem is related to the fact that Netty's ThreadPool in docker-java is starting non-daemon threads as explained by @bsideup in testcontainers/testcontainers-java#369. Not sure we can do much about that from our side.

[bsideup]

Hey guys! Yes, our beloved docker-java dependency brings the issue :)
I don't think it's a real memory issue because 1) it's testing 2) we start a single pool for all tests, but I definitely want to fix it one day. Does it cause issues to you, or so far it's just an annoying warning?

[rwinch]

@bsideup I think the leak is contributing to memory being consumed in our build since we run lots of different samples war applications for integration tests which cannot be cleaned up.

[vpavic]

I'm afraid we're blocked here until testcontainers/testcontainers-java#369 is addressed.

[bsideup]

@vpavic does it mean that the workaround I provided didn't help?

[vpavic]

Hi @bsideup, your workaround did help us to get things working, this issue is only about memory leak warnings we're encountering.

[bsideup]

@vpavic ok, I will take a look again and try to fix it in docker-java because AFAIR we can't control it in TestContainers

[vpavic]

We're still experiencing these with TestContainers 1.7.1:

Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [ducttape-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [threadDeathWatcher-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.testcontainers.shaded.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-4] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-5] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-6] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-7] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-ryuk] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Object.wait(Native Method)
 org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:127)
 org.testcontainers.utility.ResourceReaper$$Lambda$99/254873768.run(Unknown Source)
 java.lang.Thread.run(Thread.java:748)
Apr 20, 2018 2:26:54 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [ROOT] appears to have started a thread named [testcontainers-netty-1-8] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait0(Native Method)
 org.testcontainers.shaded.io.netty.channel.epoll.Native.epollWait(Native.java:111)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:231)
 org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:255)
 org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
 java.lang.Thread.run(Thread.java:748)

[vpavic]

testcontainers/testcontainers-java#651 is tracking this on TestContainers side.

[kiview]

As explained in more detail in testcontainers/testcontainers-java#651 (comment), I'm pretty sure this could be solved (or more correctly, the warnings omitted), by setting the skipMemoryLeakChecksOnJvmShutdown Tomcat context attribute.

If this would be a desired solution, I'd propose to make this attribute configurable via TomcatWebServerFactoryCustomizer (is this the correct class?) and enabling it for the tests.