Skip to content

fix(parameters): handle invalid UTF-8 transform errors - #8609

Merged
leandrodamascena merged 3 commits into
aws-powertools:developfrom
Golden007-prog:fix/parameter-transform-invalid-utf8
Oct 29, 2026
Merged

leandrodamascena merged 3 commits into
aws-powertools:developfrom
Golden007-prog:fix/parameter-transform-invalid-utf8

Conversation

@Golden007-prog

Copy link
Copy Markdown
Contributor

Issue number: closes #8600

Summary

Changes

Resolve the transform from the parameter name before decoding bytes. Identity transforms preserve the original byte value; UTF-8 decoding for JSON/base64 transforms runs inside the existing error handler. This wraps decoding failures in TransformParameterError, or returns None when transformation errors are suppressed.

User experience

An invalid UTF-8 SecretBinary now follows the documented transformation-error policy. With transform="auto" and an unmatched suffix, both valid and invalid UTF-8 binary secrets remain bytes instead of being decoded or raising an unexpected exception.

Validation

  • Nine new regression cases failed on the original implementation and pass with the fix, including SecretsProvider calls with botocore Stubber; no AWS access required.
  • Parameter functional suite: 147 passed.
  • Ruff formatting/lint and mypy for the changed source pass.
  • Broader Windows unit/functional run: 3,594 passed, 72 skipped, 11 failed. Ten failures reproduced with the original parameter implementation (Windows paths/newlines and bytecode-loader behavior); the Datadog logging failure passed when rerun in isolation. These are disclosed rather than reported as a green full suite.
  • git diff --check passes.

The full make pr pipeline, live AWS tests, and Valkey native extra were not run. The Valkey extra could not build in this Windows environment. Implementation and verification assisted by OpenAI Codex.


By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@Golden007-prog
Golden007-prog requested a review from a team as a code owner October 18, 2026 12:43
@Golden007-prog
Golden007-prog requested a review from svozza October 18, 2026 12:43
@boring-cyborg

boring-cyborg Bot commented Oct 18, 2026

Copy link
Copy Markdown

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

@boring-cyborg boring-cyborg Bot added the tests label Oct 18, 2026
@powertools-for-aws-oss-automation powertools-for-aws-oss-automation Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 18, 2026
@leandrodamascena leandrodamascena added this to the 3.39.0 milestone Oct 19, 2026
Merge upstream develop while retaining the invalid UTF-8 transform error policy and raw-byte identity behavior. Preserve the adjacent upstream AppConfig environment/application regression.

Signed-off-by: Oikantik Basu <140976895+Golden007-prog@users.noreply.github.com>

@leandrodamascena leandrodamascena left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Golden007-prog, this is exactly the fix I had in mind when I opened the issue. I ran it against real Secrets Manager with invalid and valid binary secrets across every transform, and the error policy now holds everywhere: TransformParameterError when asked to raise, None in a batch when not, and raw bytes when auto finds no suffix to act on.

One thing I want to be explicit about: with transform="auto" and no matching suffix, a valid UTF-8 binary secret used to come back as str and now comes back as bytes, same as transform=None. I think that's the right behavior, the old str was a side effect of decoding too early, but I'll call it out in the release notes.

I'm approving now. This goes out in 3.39.0, so it will sit here for a while before I merge it. No rebase needed in the meantime, and you'll be credited in the release notes.

@sonarqubecloud

Copy link
Copy Markdown

@leandrodamascena
leandrodamascena merged commit 92a28eb into aws-powertools:develop Oct 29, 2026
12 checks passed
@powertools-for-aws-oss-automation

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@boring-cyborg

boring-cyborg Bot commented Oct 29, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!

@codecov

codecov Bot commented Oct 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.95%. Comparing base (7188793) to head (c1ff178).
⚠️ Report is 1 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #8609      +/-   ##
===========================================
- Coverage    96.95%   96.95%   -0.01%     
===========================================
  Files          321      321              
  Lines        16260    16258       -2     
  Branches      1456     1455       -1     
===========================================
- Hits         15765    15763       -2     
  Misses         353      353              
  Partials       142      142              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/M Denotes a PR that changes 30-99 lines, ignoring generated files. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: non-UTF-8 SecretBinary with transform='binary' or 'auto' raises UnicodeDecodeError instead of TransformParameterError

2 participants