weblogic uses the external mapped port inside the container when 7201:7001 docker port mapping is used

[zappee]

Hello,
I am starting two WebLogic 12.2.1.4 domains in docker from my docker-compose.yml file. So I am starting 4 containers:

• admin-server-A + managed-server-A1
• admin-server-B + managed-server-B1

The following Docker port mapping is used:

• admin-server-A: 7101:7001
• managed-server-A1: 8101:8001
• admin-server-B: 7201:7001
• managed-server-B1: 8201:8001

Everything works fine, except this:

<Feb 12, 2021 5:11:21,002 PM UTC> <Notice> <JMX> <BEA-149535> <JMX Resiliency Activity Server=All Servers : Resolving connection list DomainRuntimeServiceMBean> 
javax.ws.rs.ProcessingException: java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: 'localhost', port: '7201'
failed reasons:
 [0] address:'localhost/127.0.0.1',port:'7201' : java.net.ConnectException: Connection refused

The 2nd WL admin server tries to use port 7201 in the container but actually, it is listening on the default 7001 port inside the container. Port 7201 is used from the host machine, and of course, WL is not listening on 7201 inside the container.

Why WL sees and wanna use the host port, instead of use the default 7001?

[rjeberhard]

These port mappings are for clients external to the container, correct? But, the managed server and the admin server running in the same container also need to communicate with each other. This would be done based on the ports in the config.xml.

[zappee]

Correct. 7201 is an external port only used from the host machine where docker runs.
All 4 servers run in different hosts (different containers). I am checking the config.xml.

[zappee]

Thee is no 7201 in the confix.XML It seems that this is some kikind of automatic port mapping that weblogic does automatically somehow. WL sees and wanna use somehow the mapped external docker port.

But why?

The DomainRuntimeServiceMBean is mentioned in the error message. Is it possible that the 7201 port comes from this mbean?

Can I check \ see somehow this mbean?

[zappee]

Additional info on the topic:
That exception constantly appears when I click on the "Deployment" menu on the WL console.

1. If I use the console of the "admin-server-A" then WL wanna open a connection to localhost:7101 instead of port 7001.

2. If I use the console of the "admin-server-B" then WL wanna open a connection to localhost:7201 instead of port 7001.

So WL trying to collect deployment info using the wrong port: port from docker port mapping

But why?

[zappee]

It seems that this issue is NOT related to the multi-domain docker configuration.
It appears when I only have an admin and managed server as well.

I have a _docker-compose.yml, starting the following 2 containers:

• admin-server-A
• managed-server-A1

I use the following Docker port mappings:

• admin-server-A: 7101:7001
• managed-server-A1: 8101:8001

The same issue then before: WL is trying to use 7101 instead of using the default 7001 port.

That exception constantly appears when I click on the "Deployment" menu on the WL console.

Full exception:

javax.ws.rs.ProcessingException: java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: 'localhost', port: '7101'
 failed reasons:
  [0] address:'localhost/127.0.0.1',port:'7101' : java.net.ConnectException: Connection refused

	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287)
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:255)
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684)
	at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444)
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:411)
	at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:311)
	at com.bea.console.utils.rest.RestClient.checkRestConnectivity(RestClient.java:53)
	at com.bea.console.utils.rest.RestClient.isRestfulManagementAvailable(RestClient.java:95)
	at com.bea.console.actions.app.DeploymentsControlTableAction.getCollection(DeploymentsControlTableAction.java:196)
	at com.bea.console.actions.BaseTableAction.display(BaseTableAction.java:48)
	at com.bea.console.actions.BaseDisplayAction.execute(BaseDisplayAction.java:95)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431)
	at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.access$201(PageFlowRequestProcessor.java:97)
	at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor$ActionRunner.execute(PageFlowRequestProcessor.java:2044)
	at org.apache.beehive.netui.pageflow.interceptor.action.internal.ActionInterceptors.wrapAction(ActionInterceptors.java:91)
	at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processActionPerform(PageFlowRequestProcessor.java:2116)
	at com.bea.console.internal.ConsolePageFlowRequestProcessor.processActionPerform(ConsolePageFlowRequestProcessor.java:275)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:237)
	at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processInternal(PageFlowRequestProcessor.java:556)
	at org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:853)
	at org.apache.beehive.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:631)
	at org.apache.beehive.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:158)
	at com.bea.console.internal.ConsoleActionServlet.process(ConsoleActionServlet.java:266)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:416)
	at com.bea.console.internal.ConsoleActionServlet.doGet(ConsoleActionServlet.java:135)
	at org.apache.beehive.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1199)
	at org.apache.beehive.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1129)
	at com.bea.portlet.adapter.scopedcontent.framework.internal.PageFlowUtilsBeehiveDelegate.strutsLookupInternal(PageFlowUtilsBeehiveDelegate.java:43)
	at com.bea.portlet.adapter.scopedcontent.framework.PageFlowUtils.strutsLookup(PageFlowUtils.java:108)
	at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:707)
	at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.renderInternal(ScopedContentCommonSupport.java:265)
	at com.bea.portlet.adapter.scopedcontent.StrutsStubImpl.render(StrutsStubImpl.java:105)
	at com.bea.netuix.servlets.controls.content.NetuiContent.preRender(NetuiContent.java:292)
	at com.bea.netuix.nf.ControlLifecycle$6.visit(ControlLifecycle.java:428)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:727)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walkRecursivePreRender(ControlTreeWalker.java:739)
	at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:146)
	at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:399)
	at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:361)
	at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:208)
	at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:162)
	at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:465)
	at com.bea.netuix.servlets.manager.UIServlet.doPost(UIServlet.java:291)
	at com.bea.netuix.servlets.manager.UIServlet.doGet(UIServlet.java:231)
	at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:216)
	at com.bea.netuix.servlets.manager.SingleFileServlet.service(SingleFileServlet.java:275)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.bea.console.utils.MBeanUtilsInitSingleFileServlet.service(MBeanUtilsInitSingleFileServlet.java:64)
	at weblogic.servlet.AsyncInitServlet.service(AsyncInitServlet.java:125)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295)
	at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
	at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
	at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353)
	at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at com.bea.console.internal.ParamFilter.doFilter(ParamFilter.java:38)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:32)
	at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3797)
	at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3763)
	at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
	at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
	at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
	at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
	at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2451)
	at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2299)
	at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2277)
	at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1720)
	at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1680)
	at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
	at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
	at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
	at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
	at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
	at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused by: java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: 'localhost', port: '7101'
 failed reasons:
  [0] address:'localhost/127.0.0.1',port:'7101' : java.net.ConnectException: Connection refused

	at weblogic.net.http.HttpClient.openServer(HttpClient.java:410)
	at weblogic.net.http.HttpClient.openServer(HttpClient.java:513)
	at weblogic.net.http.HttpClient.New(HttpClient.java:315)
	at weblogic.net.http.HttpClient.New(HttpClient.java:294)
	at weblogic.net.http.HttpURLConnection.connect(HttpURLConnection.java:364)
	at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:754)
	at weblogic.net.http.SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:42)
	at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1616)
	at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:394)
	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285)
	... 108 more

[zappee]

I would like to add one more piece of info to this topic.
I hope that it helps to understand the issue better and makes it easier to help me ;)

So the issue appears too when I start ONLY a WL 12.2.1.4 ADMIN server. No managed server(s), nothing. Only a simple admin server. When the server starts, log in to the web admin console and click on the "Deployment" menu. The exception mentioned above appears.

My workaround is the following:

1. Check the IP address of the admin-server container with docker inspect ...
2. Open the WL console using the container private IP address, e.g.: http://172.19.0.2:7001/console
3. In this case, the exception does not appear
4. But if I open the WL console from http://localhost:7101/console which is the mapped port to the host machine by docker, then the exception appears

Maybe this is a WL user interface issue.

[zappee]

Any news on this, please?

[xiancao]

Can you list the reproduce steps here?

[robertpatrick]

@zappee localhost does not work in a docker-compose environment for communication between containers. localhost will only work for applications running outside of docker-compose communicating with a container. All container-to-container communication must use the docker-compose service name as the DNS name.

[zappee]

Okay, but how to instruct the WebLogic admin server in Docker to use the correct port and host instead of using the Docker host port?

WebLogic insist on this:

weblogic-admin-server        | javax.ws.rs.ProcessingException: java.net.ConnectException: Tried all: '1' addresses, but could not connect over HTTP to server: 'localhost', port: '7201'
weblogic-admin-server        |  failed reasons:
weblogic-admin-server        |   [0] address:'localhost/127.0.0.1',port:'7201' : java.net.ConnectException: Connection refused

This exception appears if I click on the "Deployments" menu on the WebLogic web console.

Port 7201 is only defined in the docker-compose file:

ersion: '3'
services:
    weblogic-admin-server:
        image: weblogic-admin-11.4:1.3
        container_name: weblogic-admin-server
        hostname: weblogic-admin-server
        ports:
            - "7201:7001" # weblogic listening port
            - "4201:4001" # debug port

How the WL knows this external port inside the container?
Why WL want to use port 7201 instead of using port 7001 in the Docker container?