feat: add aiml-access-diagnostics skill for Bedrock and SageMaker - #53
Open
ams-thakkar wants to merge 14 commits into
Open
feat: add aiml-access-diagnostics skill for Bedrock and SageMaker#53ams-thakkar wants to merge 14 commits into
ams-thakkar wants to merge 14 commits into
Conversation
added 14 commits
August 12, 2026 14:03
Read-only skill that diagnoses IAM and access failures for Amazon Bedrock and Amazon SageMaker calls. Walks the six-hop authorization chain (caller action, iam:PassRole, role trust policy, role permissions, resource policy, organization SCP) and names the hop that denied the call. Key design points: CloudTrail is treated as evidence and simulation as explanation, never inverted. Three verdict states with no plain 'allowed', so an unreadable policy is never reported as an absent one. Implicit and explicit denials are distinguished because the remediations differ. The two iam:PassRole failure modes are separated. Bedrock non-IAM causes (model access, Marketplace subscription, propagation, region mismatch) are ruled out explicitly. Adds iam:SimulatePrincipalPolicy to the CloudFormation skill policies; it is the one action the skill needs that AIDevOpsAgentAccessPolicy does not grant.
…mmunity agent SOPs Applies five conventions from the UNO-Community agent-sops standard that transfer to this skill: 1. Mandatory AI-generated banner on every report. Their guardrail requires an AI disclaimer on output-generating SOPs; this skill's output proposes IAM policies, making an unreviewed application the highest-consequence failure mode. 2. Output discipline: no narration of API calls, plans, or reasoning, and no post-delivery summary that could be read in place of the report. 3. All arithmetic computed, never estimated, per their 'all math through code' rule; applies to the grant-to-denial interval. 4. Sections are never silently dropped; empty sections render a heading plus a muted explanation, since an absent section is indistinguishable from a check that never ran. 5. User-facing error handling table with explicit graceful degradation on every condition. Pre-render checks go from 12 to 14, renumbered 1-14 contiguously, with claim, subtotals, and CHANGELOG kept in sync.
…ns found in live testing End-to-end validation against real Bedrock and SageMaker denials in a test account surfaced four cases the 1.0.0 logic would have diagnosed incorrectly. - CloudTrail selection filtered on AccessDenied only, but two of the four SageMaker failure modes return ValidationException. Hop 3 (trust policy) returns 'Could not assume role' and hop 4 create-time S3 returns 'No S3 objects found under S3 URL'. Neither would have been found. - SageMaker validates the S3 input path using the execution role, so a role lacking s3:ListBucket makes an existing object appear absent. Confirmed with a control job differing only in S3 permissions. Previously diagnosed as missing data. - cloudtrail:LookupEvents is region-scoped even with a multi-region trail, so the region-mismatch cause the skill claims to diagnose was invisible when querying only the default region. - A legacy model returns ResourceNotFoundException whose message begins 'Access denied'; added as a non-IAM cause so it is not diagnosed as a permissions gap. Adds the two ValidationException signals to the activation description so the skill triggers on them. Bumps to 1.1.0.
…ext in simulation Simulating the live test fixtures exposed two more defects, both of which produce confidently wrong findings rather than degraded ones. iam:PassRole simulated without an iam:PassedToService context entry returns implicitDeny for a correctly configured caller, because AWS's own recommended scoping pattern conditions on that key. Verified: denied without it, allowed with it, and still correctly denied when the caller's condition names a different service. This is the worst possible failure for this skill - it sends the customer to add a permission they already have while the real cause at hop 3 goes unreported. Hop 2 now refuses to emit a denial when MissingContextValues is non-empty and returns CANNOT_DETERMINE. Omitting ResourceArns errs in both directions: an Allow on * plus a resource-specific Deny simulates as allowed (false negative), while a region-scoped Allow simulates as implicitDeny (false positive, blaming hop 1). Documented with the measured results. Also generalizes the rule: a denial carrying MissingContextValues is never evidence of a permission gap.
…load limit The DevOps Agent upload validator rejects a skill whose frontmatter description exceeds 1024 characters. The description was 1233 after adding the ValidationException trigger phrases, so the upload failed. Condensed to 998 characters with every trigger and exclusion phrase preserved, including ThrottlingException as a negative trigger. Verified no other skill in the repo exceeds the limit; this was the only one.
…ent can load them SKILL.md linked its six reference files by absolute github.com URL, so at runtime the agent tried to fetch remote pages instead of reading the bundled files. The reference bundle never loaded. The absolute URLs came from over-applying a real constraint. Relative .md links do break 'mkdocs build --strict', but only for README.md, which the catalog hook copies verbatim into docs/skills/<id>.md. SKILL.md and references/ never enter the docs build at all - only SKILL.md's frontmatter is read, for catalog metadata. The two consumers want opposite link styles, and this skill had it backwards for the runtime. Every other skill in the repo with a references/ directory uses relative links or plain text mentions; this was the only one using absolute URLs. Zip layout was already identical to the working skills, so structure was not a factor. Verified: strict docs build still passes with no reference to this skill, all six relative links resolve on disk, and README.md remains free of relative links.
…t the archive root The documented zip command archived the parent directory, prefixing every file with 'aiml-access-diagnostics/'. AWS DevOps Agent expects SKILL.md at the root of the archive. The failure mode is quiet. Upload succeeds and the skill activates, because the platform locates SKILL.md by scanning the archive. But reference files are retrieved by their manifest path (references/access-chain-model.md), which no longer matches the stored path, so every reference fails with 'Failed to get skill resource' and the skill silently runs on SKILL.md alone. Adds the expected archive layout, an 'unzip -l' verification step, and -D to omit directory entries, which carry no file extension and can trip the extension validator. Verified the documented command reproduces the intended bundle byte-for-byte: 7 files, SKILL.md at root, no README/CHANGELOG/evals leaked. The same incorrect command appears in the root README, project conventions, and the nine other skills that ship a references/ directory. Reported separately to keep this PR scoped.
…the IAM grant Running the skill against live denials in a real Agent Space showed that both operations the previous design treated as central are not callable in the DevOps Agent runtime, and that the skill misdiagnosed that as its own misconfiguration. cloudtrail:LookupEvents and iam:SimulatePrincipalPolicy are read-only, are granted in IAM, and sit inside the DevOps Agent permission guardrail, yet are refused before reaching AWS. The observed pattern is that operations whose verb is not Get, List or Describe are treated as potentially mutating. Every other read works. Policy documents are now the primary evidence and the diagnosis stands without CloudTrail or simulation. This is not a downgrade. Simulation cannot evaluate trust policies at all (hop 3), and at hop 2 it returns a false implicitDeny for callers using AWS's own recommended iam:PassedToService scoping, so policy reads are the only correct evidence for both. Simulation now contributes one thing exclusively, AllowedByOrganizations at hop 6, and where the two disagree the policy read wins. Removes the CloudFormation grant entirely. iam:SimulatePrincipalPolicy was the sole action needed beyond AIDevOpsAgentAccessPolicy, and granting it changes nothing. cloudformation/devops-agent-skill-policies.yaml is restored byte-identical to main, so this skill now requires no IAM changes. Fixes found in the same testing: - A blocked operation is no longer reported as a missing permission. The skill told a user to deploy a template that was already deployed and could not have helped. Adds a RuntimeUnavailable status distinct from AgentAccessDenied, splits the notice in two, and adds a pre-render check that fails the report if it proposes a grant for a runtime-blocked operation. - Adds the WOULD_ALSO_DENY verdict. A run marked hop 4 "Allows (unverified)" in the chain table while the body said the job would fail again on S3; there was no vocabulary for a second defect below the root cause. - Closes the verdict vocabulary. SKILL.md specified three verdicts while report-format.md defined five markers, and a run emitted NOT_EVALUATED as a heading verdict outside the documented set. - Removes a dependency on s3:ListBucket, which the agent does not have. The finding never needed it: a missing S3 list permission on the execution role produces "No S3 objects found" whether or not the objects exist. - Stops inferring one operation's availability from another's failure. After s3:ListBucket was refused, s3:GetBucketPolicy was assumed unavailable and hop 5 was reported undeterminable, while the bucket had a readable aws:SecureTransport deny. Pre-render validation grows from 14 checks to 16.
…rate the two not-run markers Two findings from the explicit-deny test case. A run cited the account's SCPs as 'established in earlier diagnoses this session' instead of reading them for that diagnosis. Inherited evidence is not auditable, propagates any error in the earlier read, and may describe a configuration that has since changed. Findings must now cite reads performed for the current diagnosis, and pre-render check 17 rejects phrases such as 'established earlier'. The same run marked hop 5 NOT_EVALUATED for a plain foundation-model call while explaining in the body that an AWS-owned model carries no customer resource policy. That makes it NOT_APPLICABLE - there was never anything to read. NOT_EVALUATED is reserved for a hop whose evidence exists but was not collected because an earlier hop denied. svc-bedrock.md now states this directly rather than leaving it inferable from the applicability matrix. Pre-render validation grows from 16 checks to 17.
…ntent after the report Two validated runs appended a paragraph grading the diagnosis - 'the skill worked end to end' and 'the skill nailed the deceptive case'. Output discipline already barred a post-delivery summary but not self-assessment, and the rule was lost twice out of two runs. Praising the output lends unearned confidence to findings whose limitations the report has just carefully enumerated, and the reader evaluates the report rather than the report evaluating itself. Nothing may now follow the report except a single line offering a next action.
… architecture Two functional evals still asserted the pre-1.2.0 design and would have failed for anyone running Agent Skill Eval against the shipped skill. aiml-access-verdict-states named only three verdicts and justified the absence of a plain allow with 'simulation can diverge from live behavior'. There are now six tokens in a closed vocabulary, and the justification is that reading a policy cannot account for session policies, conditional SCPs, or service-side gates outside IAM. aiml-access-read-only-boundary credited write prevention to the agent role's IAM permissions alone. The actual guarantee is the DevOps Agent permission guardrail, a session policy capping effective permissions at roughly ReadOnlyAccess. Adds four evals covering behaviour that had no coverage: the WOULD_ALSO_DENY post-denial verdict, evidence precedence (policy reads primary, policy read wins over simulation except AllowedByOrganizations), the rule that a runtime-blocked operation is not a permission gap and must not prompt a CloudFormation recommendation, and the three access failures that do not surface as AccessDenied. Functional evals now 12 (was 8). Trigger queries unchanged at 12 with 7 negatives; they are architecture-independent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
aiml-access-diagnostics, a read-only skill that diagnoses why an Amazon Bedrock orAmazon SageMaker call was denied. It walks a six-hop authorization chain — caller identity,
iam:PassRole, role trust policy, role permissions, resource policies, organization SCP —names the hop that denied the call, and proposes a scoped policy for human review.
The problem it targets is that AI/ML access failures are routinely misattributed. The
caller's permissions are frequently correct while the service role's are not, and several
failure modes surface under error codes and messages that point away from the real cause.
Three concrete examples the skill handles, all verified against live failures:
ValidationException: Could not assume role, notAccessDenied. Filtering CloudTrail for access errors never finds it.s3:ListBucketfails withValidationException: No S3 objects found under S3 URL, which reads as missing data.SageMaker validates the input path as the execution role, so an object that exists is
reported absent.
ResourceNotFoundExceptionwhose message begins"Access denied", which is not a permissions problem at all.
No IAM changes are required. Everything the skill uses is already granted by
AIDevOpsAgentAccessPolicy.cloudformation/devops-agent-skill-policies.yamlis untouchedby this PR.
Design note — why policy reads rather than the policy simulator. An earlier revision
depended on
iam:SimulatePrincipalPolicyand shipped a CloudFormation grant for it. Livetesting showed that operation is refused by the DevOps Agent runtime, as is
cloudtrail:LookupEvents, even though both are read-only, are granted in IAM, and sitinside the agent's permission guardrail. The observed pattern is that operations whose verb
is not
Get,List, orDescribeare treated as potentially mutating.Inverting to policy-read-first turned out to be more correct regardless of that constraint:
simulation cannot evaluate trust policies at all (hop 3), and at hop 2 it returns a false
implicitDenyfor callers using AWS's own recommendediam:PassedToServicescoping.Policy documents are the only correct evidence for both hops. Simulation now contributes
exactly one thing —
AllowedByOrganizationsat hop 6 — and where the two disagree thepolicy read wins. Both operations are used opportunistically when the environment permits
them, and reported as an environment characteristic when it does not, never as a missing
permission.
Design choices worth reviewer attention:
ALLOWED_BUT_UNVERIFIABLE, because readable policies cannot account for sessionpolicies, conditional SCPs, or service-side gates outside IAM. False reassurance is the
primary failure mode for this class of tool.
iam:PassRolefailures are separated. The caller missingiam:PassRoleandthe role's trust policy omitting the service principal present near-identically and have
different fixes in different places.
subscription for third-party models, region mismatch, cross-region inference-profile
dual-resource requirements — not assumed absent.
permissions commonly required but not observed. Wildcard resources are never emitted, and
no policy is proposed for an explicit-deny root cause, since adding an allow cannot
override one.
Type of change
Testing
Validated end to end against purpose-built failures in a real Agent Space, not by review
alone. Twelve deliberately broken IAM roles were created in a test account, each producing
a genuine denial, and the skill was run against them through the DevOps Agent chat surface.
Scenarios run and the expected discrimination confirmed:
sagemaker.amazonaws.comiam:PassRoleAllowon*plus a resource-specificDenyThe inference-profile destination regions in the last case were independently verified
against
bedrock:GetInferenceProfile—us-east-1,us-east-2,us-west-2— so theproposed policy is materially correct rather than plausible.
Testing found and fixed nine defects that review had not, including four that produced
confidently wrong output rather than degraded output. The most consequential:
iam:PassRolewithout aniam:PassedToServicecontext entry returnsimplicitDenyfor a correctly configured caller. This would have told users to add apermission they already held while the real cause at hop 3 went unreported.
ResourceArnsfrom a simulation errs in both directions — anAllowon*with a resource-specific
Denysimulates asallowed, while a region-scopedAllowsimulates as
implicitDeny.recommendation to deploy a CloudFormation template that was already deployed and could
not have helped.
while the body stated the job would fail there. Resolved by adding a
WOULD_ALSO_DENYverdict for a second defect below the root cause.
Each is recorded in
CHANGELOG.mdwith the behaviour that exposed it.Also verified locally:
mkdocs build --strictpasses against the merged tree including theMCP docs hook added in #49/#51; the skill's docs page and catalog entry generate correctly;
the upload bundle is structurally valid with
SKILL.mdat the archive root; and thefrontmatter description is within the 1024-character upload limit.
Evals are included (
.skilleval.yaml,evals/evals.json,evals/eval_queries.json) butwere deprioritised in favour of manual end-to-end validation, which is what surfaced every
defect above.
License confirmation