Skip to content
This repository was archived by the owner on Jul 16, 2024. It is now read-only.

Commit a000619

Browse files
lmouhibflochazFlorian ChazalAutomationvgkowski
authored
fix: add condition for trust policy to execution (#365)
* feat: upgrade to CDK v2 (#343) * move to cdkv2 * e2e utils * fix for cdk-nag for cdkv2 * fix e2e * fix API.md * Force delete retry * chore: self mutation * fix: change the layer version for batch replayer to match python 3.9 runtime * chore: self mutation Co-authored-by: Florian Chazal <[email protected]> Co-authored-by: Lotfi Mouhib <[email protected]> Co-authored-by: Automation <[email protected]> Co-authored-by: Vincent Gromakowski <[email protected]> * fix: emr eks instance profile name (#354) * fix: notebookplatform fix for type mismatch (#353) * fix: resolve issue failing the deployment of managedendpoint * fix: adding e2e test for notebook-platform.ts * fix: add managednode groups for notebook platform without podtempalte labels * fix: add default version for emr on eks to be used if no version is provided in the managedendpoint definition * fix: amend the unit tests to pass the tests. Added the new nodegroup -notebookwithoutpodtemplate- Co-authored-by: Lotfi Mouhib <[email protected]> * fix: remove non unique rolename for emr on eks instance profile * fix: upgrade the EMR default version * fix: notebookplatform fix for type mismatch (#353) * fix: resolve issue failing the deployment of managedendpoint * fix: adding e2e test for notebook-platform.ts * fix: add managednode groups for notebook platform without podtempalte labels * fix: add default version for emr on eks to be used if no version is provided in the managedendpoint definition * fix: amend the unit tests to pass the tests. Added the new nodegroup -notebookwithoutpodtemplate- Co-authored-by: Lotfi Mouhib <[email protected]> * fixing tests * chore: self mutation Co-authored-by: lmouhib <[email protected]> Co-authored-by: Lotfi Mouhib <[email protected]> Co-authored-by: Automation <[email protected]> * feat: batch replayer v2 (#362) * change BatchReplayer to support requester pay and regional dataset location * add requester pay to batchreplayer * chore: self mutation Co-authored-by: Automation <[email protected]> * fix: iam scoping -not finished yet code does not synth- * fix: iam scoping for execution role * fix: add base 36 conversion library to THIRD-PARTY-LICENSES * fix: add yarn lock and fix unit tests * doc: add documentation * chore: self mutation Co-authored-by: Florian Chazal <[email protected]> Co-authored-by: Florian Chazal <[email protected]> Co-authored-by: Automation <[email protected]> Co-authored-by: Vincent Gromakowski <[email protected]> Co-authored-by: Vincent Gromakowski <[email protected]>
1 parent 8d244e2 commit a000619

File tree

11 files changed

+611
-484
lines changed

11 files changed

+611
-484
lines changed

‎core/.projen/deps.json

+4
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎core/.projen/tasks.json

+1-1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎core/.projenrc.js

+1
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,7 @@ const project = new awscdk.AwsCdkConstructLibrary({
8383
'uuid',
8484
'aws-sdk',
8585
'@exodus/schemasafe',
86+
'simple-base',
8687
],
8788

8889
python: {

‎core/API.md

+13-3
Original file line numberDiff line numberDiff line change
@@ -2627,7 +2627,7 @@ the EmrEksNodegroup [properties]{@link EmrEksNodegroupOptions}.
26272627
##### `createExecutionRole` <a name="createExecutionRole" id="aws-analytics-reference-architecture.EmrEksCluster.createExecutionRole"></a>
26282628

26292629
```typescript
2630-
public createExecutionRole(scope: Construct, id: string, policy: IManagedPolicy, name?: string): Role
2630+
public createExecutionRole(scope: Construct, id: string, policy: IManagedPolicy, namespace: string, name: string): Role
26312631
```
26322632

26332633
Create and configure a new Amazon IAM Role usable as an execution role.
@@ -2658,11 +2658,21 @@ the execution policy to attach to the role.
26582658

26592659
---
26602660

2661-
###### `name`<sup>Optional</sup> <a name="name" id="aws-analytics-reference-architecture.EmrEksCluster.createExecutionRole.parameter.name"></a>
2661+
###### `namespace`<sup>Required</sup> <a name="namespace" id="aws-analytics-reference-architecture.EmrEksCluster.createExecutionRole.parameter.namespace"></a>
26622662

26632663
- *Type:* string
26642664

2665-
for the Managed Endpoint.
2665+
The namespace from which the role is going to be used.
2666+
2667+
MUST be the same as the namespace of the Virtual Cluster from which the job is submitted
2668+
2669+
---
2670+
2671+
###### `name`<sup>Required</sup> <a name="name" id="aws-analytics-reference-architecture.EmrEksCluster.createExecutionRole.parameter.name"></a>
2672+
2673+
- *Type:* string
2674+
2675+
Name to use for the role, required and is used to scope the iam role.
26662676

26672677
---
26682678

‎core/THIRD-PARTY-LICENSES

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
** @exodus/schemasafe -- https://github.com/ExodusMovement/schemasafe
2+
3+
Copyright (c) 2014 Mathias Buus
4+
Copyright (c) 2020 Exodus Movement
5+
6+
The MIT License
7+
Permission is hereby granted, free of charge, to any person obtaining a copy of
8+
this software and associated documentation files (the 'Software'), to deal in
9+
the Software without restriction, including without limitation the rights to
10+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
11+
of the Software, and to permit persons to whom the Software is furnished to do
12+
so, subject to the following conditions:
13+
14+
The above copyright notice and this permission notice shall be included in all
15+
copies or substantial portions of the Software.
16+
17+
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
18+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
19+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
20+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
21+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
22+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
23+
SOFTWARE.
24+
25+
------
26+
27+
** Geekoosh/flyway-lambda -- https://github.com/Geekoosh/flyway-lambda
28+
29+
MIT License
30+
31+
Copyright (c) 2021 Assaf Kamil
32+
33+
Permission is hereby granted, free of charge, to any person obtaining a copy
34+
of this software and associated documentation files (the "Software"), to deal
35+
in the Software without restriction, including without limitation the rights
36+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
37+
copies of the Software, and to permit persons to whom the Software is
38+
furnished to do so, subject to the following conditions:
39+
40+
The above copyright notice and this permission notice shall be included in all
41+
copies or substantial portions of the Software.
42+
43+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
44+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
45+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
46+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
47+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
48+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
49+
SOFTWARE.
50+
51+
------
52+
53+
** g-plane/simple-base -- https://github.com/g-plane/simple-base
54+
55+
MIT License
56+
57+
Copyright (c) 2017-present Pig Fang
58+
59+
Permission is hereby granted, free of charge, to any person obtaining a copy
60+
of this software and associated documentation files (the "Software"), to deal
61+
in the Software without restriction, including without limitation the rights
62+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
63+
copies of the Software, and to permit persons to whom the Software is
64+
furnished to do so, subject to the following conditions:
65+
66+
The above copyright notice and this permission notice shall be included in all
67+
copies or substantial portions of the Software.
68+
69+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
70+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
71+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
72+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
73+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
74+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
75+
SOFTWARE.

‎core/package.json

+2
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎core/src/data-generator/index.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
11
export { BatchReplayerProps, BatchReplayer } from './batch-replayer';
2-
export { PreparedDataset, PreparedDatasetProps } from './prepared-dataset';
2+
export { PreparedDataset, PreparedDatasetProps } from './prepared-dataset';

‎core/src/emr-eks-platform/emr-eks-cluster.ts

+17-5
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import {
2727
import { LogGroup, RetentionDays } from 'aws-cdk-lib/aws-logs';
2828
import { Bucket, BucketEncryption, Location } from 'aws-cdk-lib/aws-s3';
2929
import { BucketDeployment, Source } from 'aws-cdk-lib/aws-s3-deployment';
30-
import { Aws, CfnOutput, CustomResource, Duration, Fn, Stack, Tags, RemovalPolicy } from 'aws-cdk-lib';
30+
import { Aws, CfnOutput, CustomResource, Duration, Fn, Stack, Tags, RemovalPolicy, CfnJson } from 'aws-cdk-lib';
3131
import { AraBucket } from '../ara-bucket';
3232
import { SingletonKey } from '../singleton-kms-key';
3333
import { SingletonCfnLaunchTemplate } from '../singleton-launch-template';
@@ -36,6 +36,7 @@ import { EmrEksNodegroup, EmrEksNodegroupOptions } from './emr-eks-nodegroup';
3636
import { EmrEksNodegroupAsgTagProvider } from './emr-eks-nodegroup-asg-tag';
3737
import { EmrManagedEndpointOptions, EmrManagedEndpointProvider } from './emr-managed-endpoint';
3838
import { EmrVirtualClusterOptions } from './emr-virtual-cluster';
39+
import * as SimpleBase from 'simple-base';
3940
import * as configOverrideSchema from './resources/k8s/emr-eks-config/config-override-schema.json';
4041
import * as CriticalDefaultConfig from './resources/k8s/emr-eks-config/critical.json';
4142
import * as NotebookDefaultConfig from './resources/k8s/emr-eks-config/notebook.json';
@@ -175,7 +176,6 @@ export class EmrEksCluster extends TrackedConstruct {
175176
private readonly awsNodeRole: Role;
176177
private readonly ec2InstanceNodeGroupRole: Role;
177178
private readonly defaultNodeGroups: boolean;
178-
179179
/**
180180
* Constructs a new instance of the EmrEksCluster construct.
181181
* @param {Construct} scope the Scope of the CDK Construct
@@ -900,15 +900,27 @@ ${userData.join('\r\n')}
900900
* @param {Construct} scope of the IAM role
901901
* @param {string} id of the CDK resource to be created, it should be unique across the stack
902902
* @param {IManagedPolicy} policy the execution policy to attach to the role
903-
* @param {string} name for the Managed Endpoint
903+
* @param {string} namespace The namespace from which the role is going to be used. MUST be the same as the namespace of the Virtual Cluster from which the job is submitted
904+
* @param {string} name Name to use for the role, required and is used to scope the iam role
904905
*/
905-
public createExecutionRole(scope: Construct, id: string, policy: IManagedPolicy, name?: string): Role {
906+
public createExecutionRole(scope: Construct, id: string, policy: IManagedPolicy, namespace: string, name: string): Role {
906907

907908
const stack = Stack.of(this);
908909

910+
let irsaConditionkey: CfnJson = new CfnJson(this, 'irsaConditionkey', {
911+
value: {
912+
[`${this.eksCluster.openIdConnectProvider.openIdConnectProviderIssuer}:sub`]: 'system:serviceaccount:' + namespace + ':emr-containers-sa-*-*-' + Aws.ACCOUNT_ID.toString() +'-'+ SimpleBase.base36.encode(name),
913+
},
914+
});
915+
909916
// Create an execution role assumable by EKS OIDC provider
910917
return new Role(scope, `${id}ExecutionRole`, {
911-
assumedBy: this.eksOidcProvider,
918+
assumedBy: new FederatedPrincipal(
919+
this.eksCluster.openIdConnectProvider.openIdConnectProviderArn,
920+
{
921+
StringLike: irsaConditionkey,
922+
},
923+
'sts:AssumeRoleWithWebIdentity'),
912924
roleName: name ? name : undefined,
913925
managedPolicies: [policy],
914926
inlinePolicies: {

‎core/src/notebook-platform/notebook-platform.ts

+4
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,7 @@ export class NotebookPlatform extends TrackedConstruct {
186186
private readonly federatedIdPARN : string | undefined;
187187
private readonly authMode :string;
188188
private studioServiceRole: IRole;
189+
private vcNamespace: string;
189190

190191
/**
191192
* @public
@@ -208,6 +209,7 @@ export class NotebookPlatform extends TrackedConstruct {
208209
this.studioSubnetList = [];
209210
this.managedEndpointExecutionPolicyArnMapping = new Map<string, string>();
210211
this.authMode = props.studioAuthMode;
212+
this.vcNamespace = props.eksNamespace ? props.eksNamespace : 'default';
211213

212214
if (props.idpArn !== undefined) {
213215
this.federatedIdPARN = props.idpArn;
@@ -392,6 +394,8 @@ export class NotebookPlatform extends TrackedConstruct {
392394
this,
393395
`${user.identityName}${index}`,
394396
notebookManagedEndpoint.executionPolicy,
397+
this.vcNamespace,
398+
`${endpointName}-execRole`,
395399
),
396400
emrOnEksVersion: emrOnEksVersion ? emrOnEksVersion : NotebookPlatform.DEFAULT_EMR_VERSION,
397401
configurationOverrides: configOverride ? configOverride : undefined,

‎core/test/unit/emr-eks-platform/emr-eks-cluster.test.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ const policy = new ManagedPolicy(emrEksClusterStack, 'testPolicy', {
3030
],
3131
}),
3232
});
33-
cluster.createExecutionRole(emrEksClusterStack, 'test', policy);
33+
cluster.createExecutionRole(emrEksClusterStack, 'test', policy, 'default', 'myExecRole');
3434
const template = Template.fromStack(emrEksClusterStack);
3535

3636
test('EKS cluster created with correct version and name', () => {

0 commit comments

Comments
 (0)