Destination S3 Data Lake: avoid System.setProperty in assume role mode #50971
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
octavia-squidington-iii
added
area/connectors
octavia-squidington-iii
added
connectors/destination/s3-data-lake
and removed
CDK
...ke/src/main/kotlin/io/airbyte/integrations/destination/iceberg/v2/GlueCredentialsProvider.kt
Outdated
Show resolved
Hide resolved
edgao
changed the title
|
fyi I pushed 3cd2c47, which seems to work? It kills the it works locally on the tests I spot-checked 🤷 (cc @subodh1810 since you ran into the problem originally - is there some other problem I'm missing?) |
| AwsProperties.CLIENT_ASSUME_ROLE_EXTERNAL_ID to externalId, | ||
| AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER to | ||
| GlueCredentialsProvider::class.java.name, | ||
| "${AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER}.$AWS_CREDENTIALS_MODE" to |
There was a problem hiding this comment.
The fact that this is the expected way of handling those scenarios is blowing my mind.
octavia-squidington-iii
added
the
area/documentation
.../src/main/kotlin/io/airbyte/integrations/destination/s3_data_lake/GlueCredentialsProvider.kt
Show resolved
Hide resolved
| AssumeRoleRequest.builder() | ||
| .externalId(properties[ASSUME_ROLE_EXTERNAL_ID]) | ||
| .roleArn(properties[ASSUME_ROLE_ARN]) | ||
| .roleSessionName("airbyte-sts-session") |
There was a problem hiding this comment.
It looks like this end up in CloudTrail and is intended to be used as an audit tool. I wonder if we should have some sort of client/workspace identifier in there?
There was a problem hiding this comment.
I just copied this from https://github.com/airbytehq/airbyte/blob/master/airbyte-cdk/bulk/toolkits/load-s3/src/main/kotlin/io/airbyte/cdk/load/file/s3/S3Client.kt#L202 :P (probably should actually point at a shared constant)
we don't actually have the workspace/connection ID at runtime unfortunately (... we really should get platform to pass that through to us though)
DefaultS3FileIOAwsClientFactorycallss3FileIOProperties.applyCredentialConfigurations, which callsawsClientProperties.credentialsProvider, which prioritizes the accessKey property over AwsClientProperties.CLIENT_CREDENTIALS_PROVIDER.so stop setting that property entirely in assume role mode. Also make our creds provider a bit better.
(also, kill the region system property, which means we're now doing everything via the iceberg props 🎉 )