Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection refused connecting from another container #31

Closed
rjvq85 opened this issue Aug 16, 2024 · 5 comments
Closed

Connection refused connecting from another container #31

rjvq85 opened this issue Aug 16, 2024 · 5 comments
Assignees

Comments

@rjvq85
Copy link

rjvq85 commented Aug 16, 2024

Description

I'm building a Testcontainers' solution to test my java Quarkus service.

Up until now, I have successfully created 4 running docker containers:

  • Generic Container running my service's image (let's call it Service A)
  • Generic container running Azurite's
  • Mock server container running a mock of an external API service
  • Generic container running Event Hub emulator's

All of these containers are running in the same network.
The issue, now, is that my service is not able to connect to the event hub emulator's container.

I have tried all different kind of options for the connection string:

  • Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;
  • Endpoint=sb://host.docker.internal;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;
  • "Endpoint=sb://" + eventHubIpAddress + ";SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"
  • Endpoint=sb://eventhubs-emulator;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;

I even removed the emulator and azurite from being managed by Testcontainers and just ran docker compose -f docker/docker-compose.yaml up -d, re-adapted the configuration for the Testcontainers' containers. I even added them to the same network of the emulator, but nothing works. I always get connection refused.

Logs

[com.azu.cor.amq.imp.han.ConnectionHandler] (reactor-executor-3) {"az.sdk.message":"onTransportError","connectionId":"MF_3002fd_1723802485997","errorCondition":"proton:io","errorDescription":"Connection refused","hostName":"localhost"}
[com.azu.cor.amq.imp.ReactorConnection] (reactor-executor-3) {"az.sdk.message":"Error occurred while processing connection state.","exception":"Connection refused, errorContext[NAMESPACE: localhost. ERROR CONTEXT: N/A]","connectionId":"MF_3002fd_1723802485997"}

Desktop

  • OS: MacOS

Image Platform-Architecture Used

  • Linux-arm64

Docker Version

  • Docker version 24.0.7, build afdd53b

Arguments && Environment variables to start Emulator

  • ACCEPT_EULA: Y
  • CONFIG_PATH: ./config.json

Emulator Launch Method

  • Using Testcontainers' Generic Container

Event Hub Emulator

this.eventHubContainer = new GenericContainer<>("mcr.microsoft.com/azure-messaging/eventhubs-emulator:1.2.4-preview-arm64")
                .withCreateContainerCmdModifier(cmd -> cmd.withName("eventhubs-emulator"))
                .withNetworkAliases("eventhubs-emulator")
                .withExposedPorts(5672)
                .withNetwork(this.network)
                .withEnv("BLOB_SERVER", "azurite")
                .withEnv("METADATA_SERVER", "azurite")
                .withEnv("ACCEPT_EULA", "Y")
                .withCopyToContainer(MountableFile.forClasspathResource("config.json"), "/Eventhubs_Emulator/ConfigFiles/Config.json")  // Copy config.json
                .waitingFor(Wait.forListeningPort().withStartupTimeout(Duration.ofMinutes(2)));
this.eventHubContainer.start();

Service A

this.serviceA = new GenericContainer<>(FILTER_STREAM_IMAGE_NAME)
                .withCreateContainerCmdModifier(cmd -> cmd.withName("serviceA"))
                .withNetwork(this.network)
                .withExposedPorts(8080)
                .withExposedPorts(5005)
                .withEnv("quarkus.rest.client.filter.management-service.url".replace(".", "_").toUpperCase(), "http://external-api:1080")
                .withEnv("EVENT_HUB_CONNECTION_STRING", "Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;")
                .withEnv("BLOB_STORAGE_CONNECTION_STRING", String.format(BLOB_STORAGE_CONNECTION_STRING, "azurite", azuriteContainer.getMappedPort(10000)))
                .withStartupTimeout(Duration.ofMinutes(2))
                .withLogConsumer(new Slf4jLogConsumer(log))
                .withNetwork(this.network);
this.serviceA.start();

Expected Behaviour

Service A's container should successfully connect to Event Hub Emulator's container

Actual Behaviour

Service A's container isn't able to connect to Event Hub Emulator's container (connection refused)

@krishankumar95
Copy link
Contributor

@rjvq85 ; can you provide more details around :

  1. Logs from EH Emulator container. (Console & File Logs)
  2. SDK version used in Application A to interact with Event-Hub.

@rjvq85
Copy link
Author

rjvq85 commented Aug 16, 2024

Hi @krishankumar95,

Thanks!

Sure, here it is:

1. Logs

info: a.a.aap[0]
      Emulator Service is Launching On Platform:CBL-Mariner/Linux,Arm64
info: a.a.aap[0]
      Instantiating service components
info: a.a.aap[0]
      Creating namespace, entities and consumer groups
info: a.a.aaw[0]
      Emulator is launching with config : {"LoggingConfig":{"Type":"File"},"NamespaceConfig":[{"Type":"EventHub","Name":"emulatorns1","Entities":[{"Name":"eh1","PartitionCount":2,"ConsumerGroups":[{"Name":"eh1-consumer"},{"Name":"eh1-consumer-intest"},{"Name":"$default"}]},{"Name":"eh2","PartitionCount":2,"ConsumerGroups":[{"Name":"eh2-consumer"},{"Name":"eh2-consumer-intest"},{"Name":"$default"}]},{"Name":"process-result","PartitionCount":2,"ConsumerGroups":[{"Name":"consumer-service"},{"Name":"consumer-dashboard"},{"Name":"$default"}]}]}]}
info: a.a.aap[0]
      Emulator Service is Successfully Up! ; Use connection string : "Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"

/home/app/EmulatorLogs:

2024-08-16 13:55:44.227 +00:00 [WRN] <Trc Id="60006" Ch="Debug" Lvl="Warning" Kw="1000000000011110" UTC="2024-08-16T13:55:44.227Z" Msg="An exception was handled at TenantCacheComponent.OnEndRefresh. Exception message ExceptionId: e68e7d28-1229-48ab-bd2b-69a087770691-System.NullReferenceException: Object reference not set to an instance of an object.&#xA;   at a.a.aax.z(IAsyncResult)." />
2024-08-16 13:55:44.243 +00:00 [WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-08-16T13:55:44.243Z" Msg="Exception occurred while creating performance counters. Exception message &amp;apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.&#xA;   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)&#xA;   at h.hM.ad()&#xA;   at h.hM.ab()&#xA;   at h.hh.A(PerformanceCounterLevel)&amp;apos;." />
2024-08-16 13:55:45.254 +00:00 [WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-08-16T13:55:45.254Z" Msg="Exception occurred while creating performance counters. Exception message &amp;apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.&#xA;   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)&#xA;   at h.hK.ab()&#xA;   at h.hh.A(PerformanceCounterLevel)&amp;apos;." />
2024-08-16 13:55:45.269 +00:00 [WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-08-16T13:55:45.269Z" Msg="Exception occurred while creating performance counters. Exception message &amp;apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.&#xA;   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)&#xA;   at h.hK.ab()&#xA;   at h.hh.A(PerformanceCounterLevel)&amp;apos;." />
2024-08-16 13:55:45.272 +00:00 [WRN] <Trc Id="60006" Ch="Debug" Lvl="Warning" Kw="1000000000011110" UTC="2024-08-16T13:55:45.271Z" Msg="An exception was handled at TenantCacheComponent.OnEndRefresh. Exception message ExceptionId: 9f204705-e8d3-419f-b95e-fd9452f700f1-System.NullReferenceException: Object reference not set to an instance of an object.&#xA;   at a.a.aax.z(IAsyncResult)." />
2024-08-16 13:55:45.343 +00:00 [WRN] <Trc Id="60000" Ch="Operational" Lvl="Warning" Kw="4000000000011110" UTC="2024-08-16T13:55:45.343Z" Msg="Exception occurred while creating performance counters. Exception message &amp;apos;System.PlatformNotSupportedException: Performance Counters are not supported on this platform.&#xA;   at System.Diagnostics.PerformanceData.CounterSet..ctor(Guid providerGuid, Guid counterSetGuid, CounterSetInstanceType instanceType)&#xA;   at h.hp.ac()&#xA;   at h.hp.ab()&#xA;   at h.hh.A(PerformanceCounterLevel)&amp;apos;." /

These were the only logs I found.

2. SDK

  • com.azure:azure-core-amqp:2.9.0
  • com.azure:azure-messaging-eventhubs:5.18.0
  • com.azure:azure-messaging-eventhubs-checkpointstore-blob:1.19.0
  • Quarkus 3.8.3
  • Java 17

@krishankumar95
Copy link
Contributor

krishankumar95 commented Aug 16, 2024

@rjvq85 ; For interacting with EH Emulator please use the Java SDK version >=5.18.6 (com.azure:azure-messaging-eventhubs:5.18.6). Any version prior will not work.
Reference : SDK Changelog

@krishankumar95 krishankumar95 self-assigned this Aug 16, 2024
@rjvq85
Copy link
Author

rjvq85 commented Aug 19, 2024

@krishankumar95 thank you. I'm no longer getting connection refused after updating to v.5.18.6.

However, when trying to connect to the emulator (and subsequently trying to send a message) I get this error:

[main] INFO  com.azure.messaging.eventhubs.EventHubClientBuilder - {"az.sdk.message":"Emitting a single connection.","connectionId":"MF_7f4bcd_1724053418140"}
[main] DEBUG com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"getConnectionState","connectionId":"MF_7f4bcd_1724053418140","state":"UNINITIALIZED"}
[main] INFO  com.azure.messaging.eventhubs.implementation.EventHubConnectionProcessor - {"az.sdk.message":"Setting next AMQP channel.","entityPath":"eventhub1"}
[main] INFO  com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Creating and starting connection.","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator","port":5672}
[main] INFO  com.azure.core.amqp.implementation.ReactorExecutor - {"az.sdk.message":"Starting reactor.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onConnectionInit","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator","namespace":"eventhubs-emulator"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ReactorHandler - {"az.sdk.message":"reactor.onReactorInit","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onConnectionLocalOpen","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onConnectionBound","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator"}
[reactor-executor-1] WARN  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onTransportError","connectionId":"MF_7f4bcd_1724053418140","errorCondition":"proton:io","hostName":"eventhubs-emulator"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Disposing of active sessions due to error.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Disposing of ReactorConnection.","connectionId":"MF_7f4bcd_1724053418140","isTransient":false,"isInitiatedByClient":false,"shutdownMessage":"errorContext[NAMESPACE: eventhubs-emulator. ERROR CONTEXT: N/A]"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Closed CBS node.","connectionId":"MF_7f4bcd_1724053418140","signalType":"onComplete"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Scheduling closeConnection work.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Closed reactor dispatcher.","connectionId":"MF_7f4bcd_1724053418140","signalType":"onComplete"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Closed management nodes.","connectionId":"MF_7f4bcd_1724053418140","signalType":"onComplete"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onTransportClosed","connectionId":"MF_7f4bcd_1724053418140","errorCondition":"proton:io","hostName":"eventhubs-emulator"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.GlobalIOHandler - {"az.sdk.message":"onTransportClosed","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onConnectionUnbound","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator","state":"ACTIVE","remoteState":"UNINITIALIZED"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Closing executor.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onConnectionLocalClose","connectionId":"MF_7f4bcd_1724053418140","hostName":"eventhubs-emulator"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.ReactorExecutor - {"az.sdk.message":"Processing all pending tasks and closing old reactor.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorExecutor - {"az.sdk.message":"Had more tasks to process on reactor but it is shutting down.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.ReactorDispatcher - {"az.sdk.message":"Reactor selectable is being disposed.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] INFO  com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"onConnectionShutdown. Shutting down.","connectionId":"MF_7f4bcd_1724053418140","isTransient":false,"isInitiatedByClient":false,"shutdownMessage":"connectionId[MF_7f4bcd_1724053418140] Reactor selectable is disposed.","namespace":"eventhubs-emulator"}
[reactor-executor-1] DEBUG com.azure.core.amqp.implementation.ReactorExecutor - {"az.sdk.message":"Completing close and disposing scheduler. Finished processing pending tasks.","connectionId":"MF_7f4bcd_1724053418140"}
[reactor-executor-1] WARN  com.azure.core.amqp.implementation.ReactorConnection - {"az.sdk.message":"Error occurred while processing connection state.","exception":"errorContext[NAMESPACE: eventhubs-emulator. ERROR CONTEXT: N/A]","connectionId":"MF_7f4bcd_1724053418140"}
com.azure.core.amqp.exception.AmqpException: errorContext[NAMESPACE: eventhubs-emulator. ERROR CONTEXT: N/A]

The connection string is Endpoint=localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;.
This first warning:

[reactor-executor-1] WARN  com.azure.core.amqp.implementation.handler.ConnectionHandler - {"az.sdk.message":"onTransportError","connectionId":"MF_7f4bcd_1724053418140","errorCondition":"proton:io","hostName":"eventhubs-emulator"}

Update: this error occurs when I try to send a message from a client that I create in the host machine. Because of this I'd expect that I could connect to event hub with the standard connection string Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true; since I expose all the necessary ports in the docker-compose.

I don't see any explicit error that could point me into any direction.

@rjvq85
Copy link
Author

rjvq85 commented Aug 19, 2024

Update:

I was looking at the python example and I tried to run the simple_sender and simple_receiver locally and it worked.

I have then tried to replicate the methods using Java's SDK with

private static void sendMsg() {
      final String connectionString = "Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;";
      final String eventHubName = "eh1";


      final EventHubProducerClient producerClient = new EventHubClientBuilder()
              .connectionString(connectionString, eventHubName)
              .buildProducerClient();
      final CreateBatchOptions createBatchOptions = new CreateBatchOptions();
      createBatchOptions.setPartitionId("1");
      EventDataBatch eventDataBatch = producerClient.createBatch(createBatchOptions);

      final Random random = new Random();
      final String eventMessageLeft = "{\"direction_data\":{\"direction\": \"left\"}}";
      final String eventMessageRight = "{\"direction_data\":{\"direction\": \"right\"}}";

      for (int i = 0; i < 10; i++) {
          final String message = (random.nextInt(100) % 2 == 0) ? eventMessageRight : eventMessageLeft;
          final EventData eventData = new EventData(message);
          if (eventDataBatch.tryAdd(eventData)) {
              producerClient.send(eventDataBatch);
              System.out.println("Sent message: " + message);

              eventDataBatch = producerClient.createBatch(createBatchOptions);
              if (!eventDataBatch.tryAdd(eventData)) {
                  throw new IllegalArgumentException("Event is too large for an empty batch. Max size: "
                                                     + eventDataBatch.getMaxSizeInBytes());
              }
              if (eventDataBatch.getCount() > 0) {
                  producerClient.send(eventDataBatch);
              }
          }
      }

      producerClient.close();
      System.out.println("All messages sent.");
  }

and

private static void receiveMsg() {
        final String connectionString = "Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;";
        final String eventHubName = "eh1";
        final String consumerGroup = "cg1";

        final EventHubConsumerClient cc = new EventHubClientBuilder()
                .connectionString(connectionString, eventHubName)
                .consumerGroup(consumerGroup).buildConsumerClient();

        cc.receiveFromPartition("1", 10, EventPosition.earliest())
                .forEach(event -> {
                    final EventData eventData = event.getData();
                    System.out.println("Received event: " + eventData.getBodyAsString());
                });

        cc.close();
    }

and it worked. The connection string is the same. I don't really understand what's changed, because I have gone back to the original config.json, but after successfully testing it I re-tested with my own config.json (multiple entities) and still works.

I will close the issue.

Thanks.

@rjvq85 rjvq85 closed this as completed Aug 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants