EC2 instance metadata and ECS Fargate

[TumbleMagnet]

Describe the bug
When running in AWS ECS Fargate, my spring boot app using spring cloud AWS is not able to get region from instance metadata. From AWS documentation, it seems that EC2 and ECS might have different metadata endpoints. Can you please clarify whether Spring Cloud AWS should be able to use ECS instance metadata or not?

Application is using Spring Cloud AWS for publishing to SNS and listening from SQS queues.

Sample
Here is the error I get at startup:

InstanceMetadataServiceResourceFetcher : Fail to retrieve token
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:402) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:371) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:367) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.util.EC2MetadataUtils.getInstanceInfo(EC2MetadataUtils.java:247) ~[aws-java-sdk-core-1.11.764.jar:na]
at org.springframework.cloud.aws.core.region.Ec2MetadataRegionProvider.getCurrentRegion(Ec2MetadataRegionProvider.java:48) ~[spring-cloud-aws-core-2.2.2.RELEASE.jar:2.2.2.RELEASE]
...
Caused by: java.net.ConnectException: Invalid argument (connect failed)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na]
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na]
at java.base/java.net.Socket.connect(Socket.java:609) ~[na:na]
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[na:na]
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[na:na]
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[na:na]
at java.base/sun.net.www.http.HttpClient.(HttpClient.java:242) ~[na:na]
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[na:na]
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1248) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1227) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) ~[na:na]
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015) ~[na:na]
at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[aws-java-sdk-core-1.11.764.jar:na]
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[aws-java-sdk-core-1.11.764.jar:na]

[maciejwalkowiak]

I am not sure if this would suit you, but I think that if you switch to use default AWS credentials chain and default region chain it should work well on ECS. In 2.2.3 there will be an ability to configure cloud.aws.region.useDefaultAwsRegionChain.

In current production version (2.2.2) you can configure custom region provider bean (with name regionProvider) with implementation like:

public class DefaultAwsRegionProviderChainDelegate implements RegionProvider {

	private final DefaultAwsRegionProviderChain delegate = new DefaultAwsRegionProviderChain();

	@Override
	public Region getRegion() {
		return RegionUtils.getRegion(delegate.getRegion());
	}

}

[bowczarek]

Hi/Cześć:) @maciejwalkowiak

Unfortunately, the AwsSecretsManagerBootstrapConfiguration from latest release 2.2.3 still uses DEFAULT_REGION (US_WEST_2) instead of resolving that region from the DefaultAwsRegionProviderChain:

	@Bean
	@ConditionalOnMissingBean
	AWSSecretsManager smClient(AwsSecretsManagerProperties awsSecretsManagerProperties) {
		AWSSecretsManagerClientBuilder builder = AWSSecretsManagerClientBuilder.standard()
				.withClientConfiguration(
						SpringCloudClientConfiguration.getClientConfiguration());
		return StringUtils.isNullOrEmpty(awsSecretsManagerProperties.getRegion())
				? builder.withRegion(Regions.DEFAULT_REGION).build()
				: builder.withRegion(awsSecretsManagerProperties.getRegion()).build();
	}

Whereas on master branch I see the change done 2 days ago :) :

I guess we need to wait for the very next release then right?

[cbm-afoulfoin]

Same exception for me at startup when upgrading from 2.2.2 to 2.2.3 for an SQS on my local environment:

com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
	at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)
	at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)
	at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
	at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:402)
	at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:371)
	at org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38)
	at org.springframework.cloud.aws.context.annotation.OnAwsCloudEnvironmentCondition.matches(OnAwsCloudEnvironmentCondition.java:38)

in my application.yml:

cloud:
  aws:
    region:
      static: eu-west-1
    stack:
      auto: false

Since it's on my local pc it's not on AWS cloud, the issue seems to come from here: AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment
When reverting to 2.2.2 everything works back.

[maciejwalkowiak]

@bowczarek thanks for reporting. This issue has been removed in 17c3f91 and released in 2.2.4

[maciejwalkowiak]

@cbm-afoulfoin regarding the exception logging issue, it is logged by AWS SDK and we can't do much about it. Spring Cloud AWS Reference has one paragraph about this issue: https://docs.spring.io/spring-cloud-aws/docs/2.2.4.RELEASE/reference/html/#amazon-sdk-configuration