fix(#6019): sagemaker-core 2.15.0: role validation raises false-positive RoleValidationError under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

[sagemaker-bot]

Resolves aws#6019.

Draft PR authored by the SageMaker PySDK issue agent. The reproduction, fix, and validation below were produced and checked automatically; a human review is required before this is marked ready.

---

Agent Run Report — issue aws#6019

sagemaker-core 2.15.0: role validation raises false-positive RoleValidationError under condition-based SCPs (IAM simulator can't evaluate conditional SCPs)

Outcome: ready_for_pr

1. Triage

• SDK version: v3
• Sub-package: sagemaker-core
• Kind: bug
• Feature area: IAM role validation

2. Reproduction

• Tier: tier0_hermetic
• Repro test: sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py

Repro test

"""Reproduction for issue #6019.

Under an AWS Organization that uses condition-based SCPs, the IAM policy
simulator cannot faithfully evaluate the SCP layer, so
``iam:SimulatePrincipalPolicy`` returns ``EvalDecision: implicitDeny`` with an
empty ``MatchedStatements`` list and ``OrganizationsDecisionDetail.
AllowedByOrganizations == "false"`` — even for actions the role is actually
allowed to perform at run time.

The client-side pre-check in ``resolve_and_validate_role`` treats this org-layer
verdict as a definitive denial and raises ``RoleValidationError`` — a false
positive. It should instead treat an org-layer-only denial (no matched identity
statement) as *unverifiable* (warn + proceed), the same as the "caller can't
simulate" path.
"""
from unittest.mock import MagicMock

import pytest

from sagemaker.core.helper.iam_role_resolver import resolve_and_validate_role


def _make_session(caller_arn, account="123456789012"):
    mock_session = MagicMock()
    mock_iam = MagicMock()
    mock_sts = MagicMock()

    def client_factory(service, **kwargs):
        return mock_iam if service == "iam" else mock_sts

    mock_session.boto_session.client.side_effect = client_factory
    mock_sts.get_caller_identity.return_value = {"Arn": caller_arn, "Account": account}
    return mock_session, mock_iam, mock_sts


def _trusted_doc():
    return {
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {"Service": "sagemaker.amazonaws.com"},
                "Action": "sts:AssumeRole",
            }
        ]
    }


def test_condition_based_scp_denial_is_not_a_false_positive():
    """An org-layer-only ``implicitDeny`` (SCP condition unresolvable) must not raise.

    The role's own identity policies grant the smoke-test actions (no matched
    identity Deny statement), but the simulator reports ``implicitDeny`` because
    ``AllowedByOrganizations`` is ``false`` — an artifact of the simulator being
    unable to evaluate condition-based SCPs. This is unverifiable, not a real
    denial, so the resolver should proceed (returning the ARN) rather than raise.
    """
    arn = "arn:aws:iam::123456789012:role/TrainingExecRole"
    mock_session, mock_iam, _ = _make_session(
        "arn:aws:sts::123456789012:assumed-role/Other/sess"
    )
    mock_iam.get_role.return_value = {
        "Role": {"AssumeRolePolicyDocument": _trusted_doc()}
    }

    def paginate(**kwargs):
        # Every simulated smoke-test action comes back as an org-layer denial:
        # implicitDeny, NO matched identity statements, AllowedByOrganizations=false.
        return [
            {
                "EvaluationResults": [
                    {
                        "EvalActionName": action,
                        "EvalDecision": "implicitDeny",
                        "MatchedStatements": [],
                        "OrganizationsDecisionDetail": {
                            "AllowedByOrganizations": False
                        },
                    }
                    for action in kwargs["ActionNames"]
                ]
            }
        ]

    paginator = MagicMock()
    paginator.paginate.side_effect = paginate
    mock_iam.get_paginator.return_value = paginator

    # The real CreateTrainingJob would succeed; the pre-check must not block it.
    result = resolve_and_validate_role(
        provided_role=arn, role_type="training", sagemaker_session=mock_session
    )
    assert result == arn

Failing output (before fix)

F                                                                        [100%]
=================================== FAILURES ===================================
___________ test_condition_based_scp_denial_is_not_a_false_positive ____________

    def test_condition_based_scp_denial_is_not_a_false_positive():
        """An org-layer-only ``implicitDeny`` (SCP condition unresolvable) must not raise.
    
        The role's own identity policies grant the smoke-test actions (no matched
        identity Deny statement), but the simulator reports ``implicitDeny`` because
        ``AllowedByOrganizations`` is ``false`` — an artifact of the simulator being
        unable to evaluate condition-based SCPs. This is unverifiable, not a real
        denial, so the resolver should proceed (returning the ARN) rather than raise.
        """
        arn = "arn:aws:iam::123456789012:role/TrainingExecRole"
        mock_session, mock_iam, _ = _make_session(
            "arn:aws:sts::123456789012:assumed-role/Other/sess"
        )
        mock_iam.get_role.return_value = {
            "Role": {"AssumeRolePolicyDocument": _trusted_doc()}
        }
    
        def paginate(**kwargs):
            # Every simulated smoke-test action comes back as an org-layer denial:
            # implicitDeny, NO matched identity statements, AllowedByOrganizations=false.
            return [
                {
                    "EvaluationResults": [
                        {
                            "EvalActionName": action,
                            "EvalDecision": "implicitDeny",
                            "MatchedStatements": [],
                            "OrganizationsDecisionDetail": {
                                "AllowedByOrganizations": False
                            },
                        }
                        for action in kwargs["ActionNames"]
                    ]
                }
            ]
    
        paginator = MagicMock()
        paginator.paginate.side_effect = paginate
        mock_iam.get_paginator.return_value = paginator
    
        # The real CreateTrainingJob would succeed; the pre-check must not block it.
>       result = resolve_and_validate_role(
            provided_role=arn, role_type="training", sagemaker_session=mock_session
        )

sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py:89: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

provided_role = 'arn:aws:iam::123456789012:role/TrainingExecRole'
role_type = 'training', sagemaker_session = <MagicMock id='140101816247616'>

    def resolve_and_validate_role(
        provided_role: Optional[str],
        role_type: str,
        sagemaker_session=None,
    ) -> str:
        """Resolve the role to use and validate it (read-only; does not mutate IAM).
    
        Resolution:
            1. ``provided_role`` given → resolve it to an ARN (must exist).
            2. Otherwise → resolve the caller's own identity role.
    
        The resolved role is then VALIDATED (read-only, via iam:SimulatePrincipalPolicy
        + trust inspection):
            * permissions allowed AND trusted → return the ARN.
            * a required permission is definitively denied → raise RoleValidationError.
            * the trust policy definitively excludes the service → raise RoleValidationError.
            * permissions cannot be verified (caller lacks iam:SimulatePrincipalPolicy,
              the common Studio/notebook case) → return the ARN with a WARNING.
    
        Args:
            provided_role: User-supplied role name or ARN. If set, used directly.
            role_type: One of ROLE_TYPES.
            sagemaker_session: SageMaker session (used to get the boto session).
    
        Returns:
            IAM role ARN.
    
        Raises:
            RoleValidationError: The resolved role lacks required permissions/trust,
                or no role could be resolved.
            ValueError: ``provided_role`` doesn't exist, or ``role_type`` is invalid.
        """
        if role_type not in ROLE_TYPES:
            raise ValueError(f"Invalid role_type '{role_type}'. Must be one of: {ROLE_TYPES}")
    
        boto_session = _get_boto_session(sagemaker_session)
        iam_client = boto_session.client("iam")
    
        if provided_role:
            role_arn = _resolve_explicit_role(provided_role, sagemaker_session)
        else:
            sts_client = boto_session.client("sts")
            caller_identity = sts_client.get_caller_identity()
            caller_arn = caller_identity["Arn"]
            account_id = caller_identity["Account"]
            partition = _partition_from_arn(caller_arn)
            role_arn = _resolve_caller_role_arn(iam_client, caller_arn, account_id, partition)
            if not role_arn:
                raise RoleValidationError(_build_validation_error_message(None, role_type))
    
        # Permission check (definitive denial blocks; unverifiable warns).
        verdict, denied = _evaluate_permissions(iam_client, role_arn, role_type)
        if verdict is False:
>           raise RoleValidationError(
                _build_validation_error_message(role_arn, role_type, missing_actions=denied)
            )
E           sagemaker.core.helper.iam_role_resolver.RoleValidationError: IAM role 'arn:aws:iam::123456789012:role/TrainingExecRole' cannot be used for 'training' workloads.
E           Missing permissions: cloudwatch:PutMetricData, ec2:CreateNetworkInterface, ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface, ec2:DeleteNetworkInterfacePermission, ec2:DescribeDhcpOptions, ec2:DescribeNetworkInterfaces, ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs, ecr:GetAuthorizationToken
E           
E           To fix this, either:
E             1. Grant the permissions above (and ensure 'sagemaker.amazonaws.com' is a trusted principal) on a role, then pass it via the interface's 'role'/'role_arn' parameter. If y
... [truncated]

3. Fix

Fixed a false-positive RoleValidationError under condition-based SCPs (issue aws#6019). The client-side permission pre-check in sagemaker-core's iam_role_resolver.py relies on iam:SimulatePrincipalPolicy, which per AWS docs cannot evaluate SCPs that carry any conditions. In organizations using condition-based SCPs the simulator returns EvalDecision=implicitDeny with empty MatchedStatements and OrganizationsDecisionDetail.AllowedByOrganizations=false even for actions the role is actually allowed to perform at run time, and the resolver treated this as a definitive denial and raised. The root-cause fix is in _simulate_denied_actions: a new helper _is_org_layer_only_denial detects verdicts denied solely at the Organizations/SCP layer (AllowedByOrganizations false, no matched identity statement) and excludes them from the denied set, so they are treated as unverifiable (warn + proceed) — the same as the existing "caller can't simulate" path — letting the real API call be the source of truth. Explicit identity Deny statements still surface as matched statements and continue to block. Changed file: sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py.

Changed files:

• sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py
• sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py

Diff

diff --git a/sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py b/sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py
index eee9b0e..b2751cf 100644
--- a/sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py
+++ b/sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py
@@ -309,6 +309,33 @@ def _resolve_caller_role_arn(
 # ---------------------------------------------------------------------------
 # Read-only permission / trust validation
 # ---------------------------------------------------------------------------
+def _is_org_layer_only_denial(result: dict) -> bool:
+    """Return True if a simulate verdict is denied *only* at the Organizations/SCP layer.
+
+    The IAM policy simulator does not evaluate SCPs that carry any conditions, so
+    in an account whose organization uses condition-based SCPs it returns
+    ``AllowedByOrganizations == false`` (with ``EvalDecision: implicitDeny`` and no
+    matched statements) even for actions the role is actually allowed to perform at
+    run time. Such a verdict is *unverifiable*, not authoritative — there is no
+    matched identity Deny and no non-org reason for the denial. We treat it like the
+    "caller can't simulate" path (warn + proceed) rather than as a real permission
+    gap, letting the real API call be the source of truth. See issue #6019 and
+    https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_testing-policies.html
+    """
+    org_detail = result.get("OrganizationsDecisionDetail")
+    if not org_detail:
+        return False
+    allowed_by_orgs = org_detail.get("AllowedByOrganizations")
+    # Normalize the boolean (the API/serializers may surface it as a string).
+    if isinstance(allowed_by_orgs, str):
+        allowed_by_orgs = allowed_by_orgs.lower() == "true"
+    if allowed_by_orgs is not False:
+        return False
+    # Only unverifiable when nothing else matched — an explicit identity Deny would
+    # show up as a matched statement and must still block.
+    return not result.get("MatchedStatements")
+
+
 def _simulate_denied_actions(iam_client, role_arn: str, actions: List[str]) -> List[str]:
     """Return the subset of ``actions`` that ``role_arn`` is NOT allowed to perform.
 
@@ -317,6 +344,10 @@ def _simulate_denied_actions(iam_client, role_arn: str, actions: List[str]) -> L
     An empty list means every action is allowed. The pagination matters: a
     truncated first page must not produce a false "all allowed" verdict.
 
+    Denials that come *only* from the Organizations/SCP layer (the simulator can't
+    evaluate condition-based SCPs) are excluded — they are unverifiable, not real
+    permission gaps. See :func:`_is_org_layer_only_denial`.
+
     Raises ClientError on failures the caller must interpret (e.g. AccessDenied
     when the principal can't self-simulate, NoSuchEntity when the role is gone).
     """
@@ -328,7 +359,7 @@ def _simulate_denied_actions(iam_client, role_arn: str, actions: List[str]) -> L
     return [
         result["EvalActionName"]
         for result in evaluation_results
-        if result["EvalDecision"] != "allowed"
+        if result["EvalDecision"] != "allowed" and not _is_org_layer_only_denial(result)
     ]
 
 

4. Validation

• Repro test passes after fix: yes
• Pre-existing failures ignored (baseline): 2
• New failures introduced: none

Post-fix test output

.                                                                        [100%]
1 passed in 2.00s

5. Review

• Reviewer approved: yes
• Rounds: 1 · confidence: 0.92

5b. Adversarial verification

• Confirmed (symptom resolved, no regression): yes
• Notes: Reproduced the exact user scenario: resolve_and_validate_role with a simulate verdict of implicitDeny + empty MatchedStatements + AllowedByOrganizations="false" (string, as in the issue's CLI JSON). On unfixed code this raises the identical RoleValidationError ("Missing permissions: cloudwatch:PutMetricData, ..."); with the fix it returns the role ARN — the end-to-end outcome the reporter cares about (role usable, no false-positive block). Also tested the caller-identity path and provided-ARN path. Regression probes all pass: (1) genuine identity denial in an org account where org allows (AllowedByOrganizations=true) still blocks; (2) non-org account real denial (no OrganizationsDecisionDetail) still blocks; (3) explicit identity Deny with matched statement still blocks even when AllowedByOrganizations=false; (4) mixed page keeps real denials and drops org-only ones. Handles both bool False and string "false". Full existing helper unit suite (209 tests, incl. HyperPod caller perms, ECR scoping false-positive regressions, trust checks) passes. Confirmed my probes fail on unfixed code and pass on fixed code. Note: left an emptied scratch test file test_probe_scp_6019.py (no delete tool available); it collects nothing and is harmless.

6. Routing

• Owning team: ?
• CTI: ?
• Discovery candidates: sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py

Timeline

• triage: v3/sagemaker-core/bug area='IAM role validation'
• workspace acquired: /tmp/agent-xfkhrsqz/issue-6019
• discovery: candidate files ['sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py']
• reproduced (tier0_hermetic) via sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py
• note: 2 pre-existing failure(s) in baseline (ignored)
• fix authored (round 1, turns<=40); changed: sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py, sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py
• validated (round 1): repro passes + no new failures
• review (round 1): APPROVED
• verification: confirmed=True regressions=[] — Reproduced the exact user scenario: resolve_and_validate_role with a simulate verdict of implicitDeny + empty MatchedStatements + AllowedByOrganizations="false"
• routed to SageMaker Codex (AWS/SageMaker/SageMaker Python SDK)

Agent actions (26 tool calls)

Tool-by-tool transcript

1. read_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py'} → """IAM role resolution, validation, and (opt-in) creation for the SageMaker Python SDK. The default path is read-only: :func:resolve_and_validate_role re
2. search {'pattern': 'simulate_principal_policy|_simulate_denied_actions|_evaluate_permissions', 'path': 'sagemaker-core/tests'} → sagemaker-core/tests/unit/helper/test_iam_role_resolver.py:20: _simulate_denied_actions, sagemaker-core/tests/unit/helper/test_iam_role_resolver.py:881:
3. read_file {'path': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver.py'} → """Unit tests for the read-only IAM role resolver (validate, never create).""" import json import logging from unittest.mock import MagicMock, patch import pyt
4. write_file {'path': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py', 'bytes': 3480} → wrote 3480 bytes to sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py
5. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py'} → exit=1 F [100%] =================================== FAILURES ============================
6. read_file {'path': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py'} → """Reproduction for issue sagemaker-core 2.15.0: role validation raises false-positive RoleValidationError under condition-based SCPs (IAM simulator can't evaluate conditional SCPs) aws/sagemaker-python-sdk#6019. Under an AWS Organization that uses condition-based SCPs, the IAM policy simulator cannot faithfully evaluate the SCP layer, s
7. read_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py'} → """IAM role resolution, validation, and (opt-in) creation for the SageMaker Python SDK. The default path is read-only: :func:resolve_and_validate_role re
8. edit_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py', 'old_bytes': 1044, 'new_bytes': 2801} → edited sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py (replaced 1044→2801 bytes)
9. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py'} → exit=0 . [100%] 1 passed in 1.99s
10. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/'} → exit=0 ........................................................................ [ 36%] ........................................................................
11. read_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py'} → """IAM role resolution, validation, and (opt-in) creation for the SageMaker Python SDK. The default path is read-only: :func:resolve_and_validate_role re
12. search {'pattern': '_simulate_denied_actions|_is_org_layer_only_denial|EvaluationResults|OrganizationsDecisionDetail', 'path': 'sagemaker-core/tests'} → sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py:6: empty MatchedStatements list and ``OrganizationsDecisionDetail. sagemaker-c
13. search {'pattern': 'simulate_principal_policy|EvaluationResults', 'path': 'sagemaker-core/tests'} → sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py:70: "EvaluationResults": [ sagemaker-core/tests/unit/helper/test_i
14. read_file {'path': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py'} → """Reproduction for issue sagemaker-core 2.15.0: role validation raises false-positive RoleValidationError under condition-based SCPs (IAM simulator can't evaluate conditional SCPs) aws/sagemaker-python-sdk#6019. Under an AWS Organization that uses condition-based SCPs, the IAM policy simulator cannot faithfully evaluate the SCP layer, s
15. read_file {'path': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver.py'} → """Unit tests for the read-only IAM role resolver (validate, never create).""" import json import logging from unittest.mock import MagicMock, patch import pyt
16. write_file {'path': 'sagemaker-core/tests/unit/helper/test_probe_scp_6019.py', 'bytes': 8698} → wrote 8698 bytes to sagemaker-core/tests/unit/helper/test_probe_scp_6019.py
17. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/test_probe_scp_6019.py'} → exit=0 .......... [100%] 10 passed in 1.97s
18. edit_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py', 'old_bytes': 88, 'new_bytes': 63} → edited sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py (replaced 88→63 bytes)
19. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/test_probe_scp_6019.py'} → exit=1 F........F [100%] =================================== FAILURES ============================
20. edit_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py', 'old_bytes': 63, 'new_bytes': 88} → edited sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py (replaced 63→88 bytes)
21. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/test_iam_role_resolver.py sagemaker-core/tests/unit/helper/test_iam_role_resolver_scp_false_positive.py sagemaker-core/tests/unit/helper/test_probe_scp_6019.py'} → exit=0 ........................................................................ [ 85%] ............
22. search {'pattern': 'EvalDecision.*allowed.*not _is_org_layer_only_denial', 'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py'} → (no matches)
23. read_file {'path': 'sagemaker-core/src/sagemaker/core/helper/iam_role_resolver.py'} → """IAM role resolution, validation, and (opt-in) creation for the SageMaker Python SDK. The default path is read-only: :func:resolve_and_validate_role re
24. run_pytest {'selection': 'sagemaker-core/tests/unit/helper/'} → exit=0 ........................................................................ [ 34%] ........................................................................
25. write_file {'path': 'sagemaker-core/tests/unit/helper/test_probe_scp_6019.py', 'bytes': 0} → wrote 0 bytes to sagemaker-core/tests/unit/helper/test_probe_scp_6019.py
26. read_file {'path': 'sagemaker-core/tests/unit/helper/test_probe_scp_6019.py'} →