Repository navigation
base64.b85encode uses significant amount of RAM #101178
Description
Activity
I can confirm, that this still happens on
main:» time ./python.exe -c "import base64; print(len(base64.b85encode(b'a' * 250 * 1024 * 1024)) / 1024 / 1024)" 312.5 ./python.exe -c 160.32s user 74.28s system 87% cpu 4:29.26 totalCompare it to almost-instant
64version (it usesbinascii's C implementation):» time ./python.exe -c "import base64; print(len(base64.b64encode(b'a' * 250 * 1024 * 1024)) / 1024 / 1024)" 333.33333587646484 ./python.exe -c 0.91s user 0.26s system 98% cpu 1.187 totalI looked into the implementation of
b85encode(in_85encode). The memory problems are caused by the fact that internally copies of the data are made. The copies consist of small chuncks, where each chuck has the memory overhead of a python object.- First with
words = struct.Struct('!%dI' % (len(b) // 4)).unpack(b). There is aiter_unpackmethod in theStructclass, but it does not work as expected. - And then with a list comprehension
chunks = [ ... for word in words]. In the casepadding=Falsewe can replace this with a generator which would save memory. Forpadding=Truewe need to modify the last element of the list, which does not work for the generator.
The poor performance (compared to b64encode) is mainly due to the part
(chars2[word // 614125] + chars2[word // 85 % 7225] + chars[word % 85]).To make any significant gains an implementation in C seems the most worthwhile.
- First with
- added a commit that references this issue
on Mar 3, 2024 - added a commit that references this issue
on Apr 21, 2025 Pinging this issue in hopes of finding a reviewer for my PR gh-102753. It's been over two years 😅
1 remaining item
- added a commit that references this issue
on Dec 25, 2025 - added a commit that references this issue
on Dec 28, 2025 - added a commit that references this issue
on Jan 14, 2026 - added a commit that references this issue
on Jan 18, 2026 - added a commit that references this issue
on Feb 24, 2026
Bug report
On the same string:
b85encode takes up my entire RAM and crashes
On IPython:
Here is the GNU time stats:
I have gotten same results in Python 3.6, 3.7, 3.8, 3.9
Your environment
Linked PRs
binascii.b2a_ascii85#148166