undeploying existing web app causes other web apps go blank in multiple war files environment #3389
Description
RHEL: 7
Tomcat: 9.0.13
jdk: 11.0.2
ctags: built on 01/15/19
opengrok: 1.5.9
We are having multiple war files under tomcat/webapps directory and each war file contains one branch (no project enabled).
I notice that if I remove existing war file, tomcat shows blank screen when I go to branches's root by click the remainng
branch from tomcat web application manager. (e.g. url is hostname:port/test2/) However, if I go to xref by typing in url
field of the browser, correct screen is displayed. (e.g. url is hostname:port/test2/xref/) I need to restart tomcat to
workaround this issue.
This issue was started to appear around 1.4.7, 8, or 9.
I also notice that tomcat restarting time gets extraordinally increased (from 8 min to 40+ min; with 70 war files) starting from above these releases, too.
So, restarting tomcat to fix this issue is not a good option for us.
Recreation Steps:
- start tomcat
- add test1.war (test1 branch) which has index generated by 1.5.9
- add test2.war (test2 branch) which has index generated by 1.5.9
- remove test1.war from tomcat's webapps directory
- click test2 from tomcat web application manager
- blank screen displayed
[
catalina.2020-12-02.zip
](url)
I am attaching catalina.log. (sorry for the big file)
I can see "java.nio.file.NoSuchFileException: /opt/pisces/ogtest/webapps/test1/WEB-INF/lib/tablesorter-2.31.3.jar" at the last.
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
vladak commentedon Dec 2, 2020
The log file contains lot of noise due to #3384. This issue may actually be slowing the deploy.
I wonder if removal of a .war file is supported way how to undeploy a web application in Tomcat. Why don't you undeploy the webapp first using the application manager ?
vladak commentedon Dec 2, 2020
I assume that the
CONFIGURATIONparameter in theWEB-INF/web.xmlfile in eachtest1.warandtest2.warpoints to different file and the 2 files specify different source/data root. How exactly are these .war files created ?vladak commentedon Dec 2, 2020
Browsing the log I don't see any indication of times in minutes (modulo the overall time):
and
vladak commentedon Dec 2, 2020
Could you try with 1.5.10 ?
ghost commentedon Dec 3, 2020
catalina.2020-12-02.part2.zip
I tried with 1.5.10 but got the same symptom/results unfortunately.
Let me add some for your questions.
Thank you very much for your support!
vladak commentedon Dec 3, 2020
Looking at
catalina.2020-12-02.part2.log, the container has a mixture of web application versions:I was hoping that the fix for #3384 in 1.5.10 would shed some time during the deploy (generating log entries for all these exceptions takes some time). To benefit from that all the applications within the container would have to be upgraded.
Anyhow, I don't see any significant times, everything is in matter of seconds:
and
What are these times in the production instance ?
How do you measure the Tomcat restart time exactly ? It would be good idea to take a look at the Tomcat process in terms of I/O and CPU load during these 40 minutes with the usual tools.
vladak commentedon Dec 3, 2020
Back to the original problem:
To reproduce this:
At this point I checked that both web apps display the usual OpenGrok index page on http://localhost:8080/foo/ and http://localhost:8080/bar/. Then I triggered undeploy of
foowithrm ~/apache-tomcat-8.5.47/webapps/foo.warand displayed http://localhost:8080/bar/ which produces the HTTP 500 internal error (in the browser you get the blank page however if you look at the response e.g. using Developer tab/Network in Firefox you will see the same HTTP 500 error):Looking into the logs I can see this exception:
Obviously
/home/vkotal/apache-tomcat-8.5.47/webapps/foo/WEB-INF/lib/showdown-1.9.1.jaris gone since the/home/vkotal/apache-tomcat-8.5.47/webapps/foodirectory is gone. The/home/vkotal/apache-tomcat-8.5.47/webapps/bar/WEB-INF/lib/showdown-1.9.1.jarfile is present.It looks as if there was some routing/sharing/caching problem in Tomcat.
vladak commentedon Dec 3, 2020
When I did the 'Reload' on the
barweb application from the Tomcat Web Application Manager it has started working again. I think it basically did a redeploy.[-]Removing existing war file causes Tomcat issue on multiple war files environment[/-][+]undeploying existing web app causes other web apps go blank in multiple war files environment[/+]vladak commentedon Dec 3, 2020
If
baris undeployed instead offoothefoostill works.vladak commentedon Dec 3, 2020
Looking at
straceoutput tomcat.log that captures the undeploy offoo:and then the request to
/bar:When the request is being dealt with, the following .jar files are being opened:
So,
showdown-1.9.1.jaris successfully opened for thebarand then again unsuccessfully forfoo.vladak commentedon Dec 3, 2020
These 2 lines in the stack trace are interesting:
so perhaps this is some broken caching in Tomcat. It is still a question why it needs to stat the same file again.
vladak commentedon Dec 3, 2020
After reviewing bunch of Tomcat caching related questions on Stackoverflow I created https://stackoverflow.com/questions/65126838/how-to-prevent-tomcat-from-accessing-jar-files-from-undeployed-web-application
vladak commentedon Dec 3, 2020
Tried to disable the caching by putting:
into
conf/context.xml, did not help. It just leads to slightly different stack trace:That matches
Tomcat-8.5/java/org/apache/catalina/webresources/StandardRoot.java#getResource()source code: