feat: Add CloudFormation Language Extensions support (Fn::ForEach)#8637
feat: Add CloudFormation Language Extensions support (Fn::ForEach)#8637
Conversation
github-actions
bot
added
area/package
bnusunny
force-pushed
the
feat-language-extension
branch
from
0be94d0 to
5d6cbf3
Compare
|
Integration test: https://github.com/aws/aws-sam-cli/actions/runs/21808626015 |
bnusunny
force-pushed
the
feat-language-extension
branch
7 times, most recently
from
e68efa0 to
4ed8396
Compare
bnusunny
force-pushed
the
feat-language-extension
branch
15 times, most recently
from
5324dad to
707baad
Compare
- sam validate: valid ForEach, invalid syntax, cloud-dependent collections, dynamic CodeUri, nested depth validation (5 valid, 6 invalid) - sam local invoke: expanded function names from ForEach - sam local start-api: ForEach-generated API endpoints
Track CFNLanguageExtensions as a UsedFeature event when templates with AWS::LanguageExtensions transform are expanded. Emitted once per expansion in expand_language_extensions().
Remove redundant and AWS-dependent integration tests, keeping 9 essential tests across build, package, validate, local invoke, and start-api. Delete 34 orphaned testdata directories.
YAML parsing produces Python booleans for bare true/false values, but parameter overrides from --parameter-overrides are always strings. Fn::Equals was using Python == which returns False for 'true' == True. CloudFormation Fn::Equals performs string comparison, so convert both operands to their string representations before comparing. Booleans are lowercased to produce 'true'/'false' matching CFN serialization.
… only Language extension functions are only supported in these three sections per AWS::LanguageExtensions transform documentation. Previously the intrinsic resolver also processed Parameters, Mappings, Metadata, etc.
The name iter_regular_resources better conveys that ForEach blocks are skipped. Removes the backward-compatible alias.
Extract duplicated _to_boolean logic from condition_resolver.py and fn_if.py into IntrinsicFunctionResolver.to_boolean() static method. Replace os.path.isfile() + os.path.getmtime() two-step check with a single try/except around getmtime() to eliminate the race condition.
Remove 9 integration tests whose test data directories were removed in
an earlier commit: validate/language-extensions/, buildcmd/language-
extensions-dynamic-imageuri/, language-extensions-foreach/, and
language-extensions-nested-foreach-{valid,invalid}/.
- Move inline imports to top level in sam_stack_provider.py and test_template.py - Add missing assertion in test_handles_empty_mappings - Uncomment Fn::ForEach::Topics block to test non-Lambda resource types - Update mock patch paths to match top-level import locations
bnusunny
force-pushed
the
feat-language-extension
branch
from
e3ff752 to
75df927
Compare
roger-zhangg
left a comment
roger-zhangg
left a comment
There was a problem hiding this comment.
Seems we are missing some failure tests, e.g. when there's intrinsic that can't be resolved locally
| [default.deploy.parameters] | ||
| stack_name = "language-extensions-nested-foreach-dynamic-codeuri" | ||
| resolve_s3 = true | ||
| s3_prefix = "language-extensions-nested-foreach-dynamic-codeuri" |
There was a problem hiding this comment.
s3 max length is 64, this might be too long
| from tests.testing_utils import get_sam_command, run_command | ||
|
|
||
|
|
||
| class TestInvokeLanguageExtensions(InvokeIntegBase): |
There was a problem hiding this comment.
we should mark few integ test as tier 1 so it would be run windows/finch
| """ | ||
| template_path = self.test_data_path.joinpath("language-extensions-dynamic-codeuri", "template.yaml") | ||
|
|
||
| with tempfile.NamedTemporaryFile(delete=False, suffix=".yaml") as output_template_file: |
There was a problem hiding this comment.
can we avoid NamedTemporaryFile but use a test fixture directly. This has been causing issue on windows tests. And as we are using delete=False there's no reason to use this function right?
| self.assertEqual(codeuri["Fn::FindInMap"][0], mapping_name) | ||
| self.assertEqual(codeuri["Fn::FindInMap"][1], {"Ref": "FunctionName"}) | ||
|
|
||
| def test_build_nested_foreach_dynamic_codeuri_generates_mappings(self): |
There was a problem hiding this comment.
might worth to test on windows
| For list-of-lists, items can also contain intrinsics | ||
| 3. template_body: A dictionary containing the template to expand | ||
|
|
||
| For list-of-lists format: |
There was a problem hiding this comment.
does CloudFormation actually accept a list of identifiers? I don't see it mentioned here https://docs.aws.amazon.com/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference-foreach.html
|
|
||
| def _substitute_identifier(self, template: Any, identifier: str, value: str) -> Any: | ||
| """ | ||
| Substitute ${identifier} and {"Ref": "identifier"} with value throughout the template. |
There was a problem hiding this comment.
This only handles ${identifier} substitution. CloudFormation also supports &{identifier} syntax which strips non-alphanumeric characters from the value for use in logical IDs (see [docs
example](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-foreach-example-resource.html#intrinsic-function-reference-foreach-example-non-
alphanumeric)). Should we either implement it or detect and raise a clear error?
|
|
||
| This method iterates through the section, validates any | ||
| Fn::ForEach:: prefixed keys found, and expands them. | ||
| Used for top-level sections (Resources, Conditions, Outputs), |
There was a problem hiding this comment.
Nit: The docstring says "Used for top-level sections (Resources, Conditions, Outputs)" but _process_section is not called directly for the Resources section — that goes through
_process_resources_section. This method is used for Conditions, Outputs, resource Properties, and nested ForEach bodies.
|
|
||
| Identifiers can be: | ||
| - A string: "VariableName" | ||
| - A list of strings: ["LogicalId", "TopicId"] |
There was a problem hiding this comment.
Again, can identifiers be a list of strings for existing cloudformation language extension?
| Identifiers can be: | ||
| - A string: "VariableName" | ||
| - A list of strings: ["LogicalId", "TopicId"] | ||
| - A list containing intrinsics: [{"Ref": "ParamName"}, "TopicId"] |
There was a problem hiding this comment.
Can intrinsic function be an identifier? For example,
yaml
Parameters:
IdentifierParam:
Type: String
Default: Name
Fn::ForEach::Functions:
- !Ref IdentifierParam # resolves to "Name"
- [Alpha, Beta]
- ${Name}Function:
It seems pointless, i think we can't use ${!Ref IdentifierParam} rather we have to use ${Name}, it that is true, then using intrinsic function as identifier seems pointless.
If we can use intrinsic function as identifier, then in the logic below we miss a case where identifier is a single intrinsic function, i.e. a dict but not dict in a list.
| MAX_FOREACH_NESTING_DEPTH = 5 | ||
| _LAYOUT_ERROR_FMT = "{} layout is incorrect" | ||
|
|
||
| def __init__(self, intrinsic_resolver: Optional[Any] = None) -> None: |
There was a problem hiding this comment.
Nit: intrinsic_resolver is typed as Optional[Any] but it's always an IntrinsicResolver instance as mentioned in the docstring. Could we use Optional["IntrinsicResolver"] to get proper type checking here?
|
|
||
| Args: | ||
| intrinsic_resolver: Optional IntrinsicResolver instance for resolving | ||
| collections that contain intrinsic functions. |
There was a problem hiding this comment.
Nit: The docstring says the intrinsic_resolver is for "resolving collections that contain intrinsic functions," but it's also used to resolve intrinsic functions in identifiers (in _resolve_identifiers). Maybe we could broaden the description to reflect all its uses.
| for ident in identifiers: | ||
| self._check_identifier_conflicts(ident, key, context, parent_identifiers) | ||
|
|
||
| # Check for loop name conflicts with parameter names |
There was a problem hiding this comment.
Why loop name conflict with parameter name would be a problem?
| if not isinstance(value, dict) or len(value) != 1: | ||
| return False | ||
| key = next(iter(value.keys())) | ||
| return key.startswith("Fn::") or key in ("Ref", "Condition") |
There was a problem hiding this comment.
Why key is Condition counted as intrinsic function?
|
|
||
| if language_extension_result is not None: | ||
| # Use pre-computed Phase 1 results | ||
| self._original_template = language_extension_result.original_template |
There was a problem hiding this comment.
_original_template and _dynamic_artifact_properties are set on the wrapper but get_original_template() and get_dynamic_artifact_properties() are never called anywhere in the PR. Are these fields and their getters still needed, or can they be removed?
| Raises: | ||
| InvalidTemplateException: If the policy is invalid. | ||
| """ | ||
| if "Ref" in policy: |
There was a problem hiding this comment.
_resolve_intrinsic_policy handles Ref to parameters, but by the time this processor runs, the IntrinsicResolverProcessor should have already resolved !Ref xxxx
to a plain string. Is there a case where a Ref in DeletionPolicy/UpdateReplacePolicy would survive step 3 and reach this code? If not, this might be dead code.
| If the template contains invalid language extension syntax | ||
| """ | ||
| # --- cache lookup --- | ||
| cache_key: Optional[Tuple[str, float, str]] = None |
There was a problem hiding this comment.
how would the cache here be helpful? I thought for each command, we just expand the template for one time and then the sam cli process is killed and cache is gone. Will the cache be used?
| if not check_using_language_extension(template): | ||
| result = LanguageExtensionResult( | ||
| expanded_template=copy.deepcopy(template), | ||
| original_template=template, |
There was a problem hiding this comment.
This is not deepcopy while all others are deepcopy, is this intentional?
| return artifacts | ||
|
|
||
|
|
||
| def get_template_function_resource_ids(template_file, artifact): |
There was a problem hiding this comment.
Seems this method has no caller in the codebase.
samcli/commands/_utils/template.py
Outdated
| # Only these property names in SAM-generated Mappings represent local file paths | ||
| # that need relative path adjustment. Other properties (like LayerOutputKey for | ||
| # auto dependency layer references) are CloudFormation references, not file paths. | ||
| _ARTIFACT_PATH_PROPERTIES = { |
There was a problem hiding this comment.
_ARTIFACT_PATH_PROPERTIES and PACKAGEABLE_RESOURCE_ARTIFACT_PROPERTIES define overlapping but inconsistent sets of artifact property names. For example, "Code" (from
AWS::Lambda::Function) and "Content" (from AWS::Lambda::LayerVersion) are in PACKAGEABLE_RESOURCE_ARTIFACT_PROPERTIES but "Code" is missing from _ARTIFACT_PATH_PROPERTIES. Is this an issue?
There was a problem hiding this comment.
We seems have three lists for similar purpose?
_ARTIFACT_PATH_PROPERTIES,
PACKAGEABLE_RESOURCE_ARTIFACT_PROPERTIES,
RESOURCES_WITH_LOCAL_PATHS
Should we use one as the source of truth?
| if has_build_artifact: | ||
| ApplicationBuilder._update_built_resource( | ||
| built_artifacts[full_path], properties, resource_type, store_path | ||
| built_artifacts[full_path], properties, resource_type or "", store_path |
| # Packageable resource types and their artifact properties that can be dynamic in Fn::ForEach blocks. | ||
| # These properties reference local files/directories that SAM CLI needs to package. | ||
| # Dynamic values (using loop variables) are supported via Mappings transformation. | ||
| PACKAGEABLE_RESOURCE_ARTIFACT_PROPERTIES: Dict[str, List[str]] = { |
There was a problem hiding this comment.
Should this list sync with existing RESOURCES_WITH_LOCAL_PATHS?
| parameter_values = dict(IntrinsicsSymbolTable.DEFAULT_PSEUDO_PARAM_VALUES) | ||
|
|
||
| try: | ||
| result = expand_language_extensions(child_template_dict, parameter_values, template_path=abs_template_path) |
There was a problem hiding this comment.
Only pseudo-parameters are passed to expand_language_extensions here. If the child template's Fn::ForEach collection references a parameter supplied by the parent stack (
via AWS::CloudFormation::Stack Parameters property), expansion will fail to resolve it and fall back to the unexpanded template. This works today because CloudFormation handles it server-
side, but it means dynamic artifact properties in such child templates won't get the Mappings-based S3 URI rewriting — sam package would produce a template with unresolved local paths for
those resources.
5 participants
Description
This PR adds support for CloudFormation Language Extensions in SAM CLI, addressing GitHub issue #5647.
Features
Fn::Ifin resource policiesKey Design Decisions
Fn::ForEachblocks with dynamic artifact properties (e.g.,CodeUri: ./src/${Name}) are supported via a Mappings transformationFn::ForEachcollections must be resolvable locally; cloud-dependent values (Fn::GetAtt,Fn::ImportValue) are not supported with clear error messagesSupported Commands
sam build- Builds all expanded functions, preserves original templatesam package- PreservesFn::ForEachstructure with S3 URIssam deploy- Uploads original template for CloudFormation to processsam validate- Validates language extension syntaxsam local invoke- Invokes expanded functions by namesam local start-api- Serves ForEach-generated API endpointssam local start-lambda- Serves all expanded functionsExample
Resolves #5647
Testing
Checklist