"Short circuiting" in base64's b64decode, decode, decodebytes #79013

Activity

1. pwmichaelharris commented on Sep 28, 2018

   

   pwmichaelharrismannequin

   on Sep 28, 2018

   MannequinAuthor

   More actions

   When given an invalid base64 string that starts with a valid base64 substring, the functions will return the decoded bytes only up to the substring rather then ignoring the non-alphabet character.

   Examples:
   >>> base64.b64decode("AAAAAAAA")
   b'\x00\x00\x00\x00\x00\x00'
   >>> base64.b64decode("AA=AAAAAA")
   b'\x00\x00\x00\x00\x00\x00'
   >>> base64.b64decode("AAA=AAAAA")
   b'\x00\x00'
2. pwmichaelharris

   added

   stdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directory

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   on Sep 28, 2018
3. fbidu commented on Sep 28, 2018

   

   fbidumannequin

   on Sep 28, 2018

   Mannequin

   More actions

   I am not sure if simply ignoring the non-valid character is the best way to go. Feels like silencing errors.

   b64decode does accept the 'validate' flag - defaulted to False - that will halt the execution and throw an error.

   What might be a good idea is to implement an 'errors' argument that accepts 'ignore' as a value, like we do for bytes.decode (https://docs.python.org/3/library/stdtypes.html#bytes.decode)
4. fbidu commented on Sep 28, 2018

   

   fbidumannequin

   on Sep 28, 2018

   Mannequin

   More actions

   Actually, I'm not even sure if it makes sense to decode the 'first valid substring'... IMHO, we should warn the user
5. fbidu commented on Sep 29, 2018

   

   fbidumannequin

   on Sep 29, 2018

   Mannequin

   More actions

   For reference in future discussions, Python's base64 module implements RFC 3548 (https://tools.ietf.org/html/rfc3548) whose section 2.3 (https://tools.ietf.org/html/rfc3548#section-2.3) discusses about "Interpretation of non-alphabet characters in encoded data".

   The section's content is:

   Base encodings use a specific, reduced, alphabet to encode binary
   data. Non alphabet characters could exist within base encoded data,
   caused by data corruption or by design. Non alphabet characters may
   be exploited as a "covert channel", where non-protocol data can be
   sent for nefarious purposes. Non alphabet characters might also be
   sent in order to exploit implementation errors leading to, e.g.,
   buffer overflow attacks.

   Implementations MUST reject the encoding if it contains characters
   outside the base alphabet when interpreting base encoded data, unless
   the specification referring to this document explicitly states
   otherwise. Such specifications may, as MIME does, instead state that
   characters outside the base encoding alphabet should simply be
   ignored when interpreting data ("be liberal in what you accept").
   Note that this means that any CRLF constitute "non alphabet
   characters" and are ignored. Furthermore, such specifications may
   consider the pad character, "=", as not part of the base alphabet
   until the end of the string. If more than the allowed number of pad
   characters are found at the end of the string, e.g., a base 64 string
   terminated with "===", the excess pad characters could be ignored.

   In my opinion, the RFC is rather permissive about strange characters in the encoded data. The RFC refers to the MIME specification that ignores the data and hints the possibility of rejecting the pad symbol '=' unless it is found in the end of the string.

   I think that our best option if we would like to address this issue is to add an 'errors' argument whose default value will keep the current behavior for backwards compatibility but will accept more options in order to both ignore the strange characters and carry on with the processing - like bytes.decode's errors=ignore flag - and to raise an error in such situations, like bytes.decode's errors=strict.
6. ezio-melotti

   transferred this issue fromon Apr 10, 2022
7. simon-friedberger commented on Aug 8, 2023

   

   simon-friedberger

   on Aug 8, 2023

   Last edited by simon-friedberger

   More actions

   @fbidu, I agree with your reasoning. The default behavior should error on any non-base-alphabet characters. validate=True should be the default. I would even suggest removing the option.

   There are inconsistencies in the API, b64_decode has validate(default False) but urlsafe_b64decode and b32_decode do not. b32_decode implicitly has validate=True but has a switch for allowing lower-case characters.

   A single corrupt character will silently corrupt the rest of the data. Disagreement about which alphabet to use (urlsafe?) will lead to data that decodes fine but isn't the right data.

   Since it's fairly trivial (something like re.sub(r'[a-zA-Z0-9]', ' ', s)) to remove unwanted characters people who want that could just do that.

   Related: People want to ignore incorrect padding: #73613
   This can easily be done using +"==" as long as those characters get ignored so a solution which does remove validate should probably add an ignore_missing_padding.
8. serhiy-storchaka commented on Apr 6, 2026

   

   serhiy-storchaka

   on Apr 6, 2026

   Member

   More actions

   The originally reported issue was fixed in #145264. Base64 decoder has now the ignorechars parameter which allows finer control on what characters to ignore (see #143214). If '=' is included in ignorechars or in non-strict mode (default), then '=' in the middle of string and excess padding will be ignored, otherwise they will be errors. #73613 added new parameter padded which controls whether the padding is required.

   As for urlsafe_b64decode(), it has less options, but there are plans to make it stricter by default. Use b64decode() with altchars=b'-_' if you need more flexibility.
9. serhiy-storchaka

   closed this as a duplicateof Excess Base64 data ignored after padding by default #145264 on Apr 6, 2026