Repository navigation
Optimize _PyUnicodeWriter implementation #119396
Description
Activity
- added a commit that references this issue
on May 22, 2024 I do not know how much this is needed, but the code looks correct.
Could you please collect some data about types used with
%Sand%R? And how often%Sand%Rare used with and without the width and the precision? Just add writing the type name in the special log file and run the test suite, then count most common names.Could you please collect some data about types used with %S and %R?
Sure, here you have (I didn't check width or precision):
Details
%R, str: 113038 (80%) %S, str: 24825 (18%) %R, bool: 770 (1%) %R, tuple: 696 (0%) %R, int: 507 (0%) %R, list: 393 (0%) %S, int: 136 (0%) %S, _GenericAlias: 120 (0%) %R, bytes: 116 (0%) %R, dict: 113 (0%) %R, NoneType: 85 (0%) %R, frame: 62 (0%) %R, type: 52 (0%) %R, types: 43 (0%) %R, function: 42 (0%) %R, _io: 33 (0%) %S, types: 30 (0%) %R, datetime: 24 (0%) %R, object: 19 (0%) %R, float: 18 (0%) %R, _StoreAction: 16 (0%) %R, Element: 15 (0%) %R, _CallableGenericAlias: 12 (0%) %R, typing: 12 (0%) %S, NoneType: 11 (0%) %R, _asyncio: 11 (0%) %R, QueryTestCase: 10 (0%) %S, type: 10 (0%) %A, NoneType: 7 (0%) %R, FixedOffset: 7 (0%) %R, EnumType: 6 (0%) %A, str: 6 (0%) %S, _AnnotatedAlias: 6 (0%) %R, NewType: 6 (0%) %R, CustomPyPicklerClass: 6 (0%) %R, ArrayBinASCIITest: 5 (0%) %R, Mock: 5 (0%) %R, _contextvars: 5 (0%) %S, object: 4 (0%) %S, Exception: 4 (0%) %R, Foo: 4 (0%) %S, _ProtocolMeta: 4 (0%) %R, _thread: 4 (0%) %R, complex: 4 (0%) %R, Task: 4 (0%) %R, _ALWAYS_EQ: 3 (0%) %R, collections: 3 (0%) %R, builtin_function_or_method: 3 (0%) %R, Test: 3 (0%) %S, _CallableGenericAlias: 3 (0%) %R, functools: 3 (0%) %R, CPartialSubclass: 3 (0%) %R, Future: 3 (0%) %R, Derived: 3 (0%) %R, StopIteration: 2 (0%) %S, BadStr: 2 (0%) %R, code: 2 (0%) %S, dict: 2 (0%) %S, NotImplementedType: 2 (0%) %S, ellipsis: 2 (0%) %S, Cheese: 2 (0%) %S, SubBytes: 2 (0%) %S, list: 2 (0%) %S, _TypedDictMeta: 2 (0%) %S, MutatesYourDict: 2 (0%) %R, Meta: 2 (0%) %R, SSLSocket: 2 (0%) %R, CustomStr: 2 (0%) %S, ValueError: 2 (0%) %R, C: 2 (0%) %R, socket: 2 (0%) %R, TarInfo: 2 (0%) %R, sub: 1 (0%) %R, MyFileIO: 1 (0%) %R, Completer: 1 (0%) %R, aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: 1 (0%) %R, StackObject: 1 (0%) %R, bytearray: 1 (0%) %R, CustomBytes: 1 (0%) %R, CustomByteArray: 1 (0%) %R, memoryview: 1 (0%) %R, array: 1 (0%) %R, posix: 1 (0%) %R, xml: 1 (0%) %R, sqlite: 1 (0%) %R, Base: 1 (0%) %R, expr: 1 (0%)For str, int and float:
%R, str: 113038 (80%) %S, str: 24825 (18%) %R, int: 507 (0%) %S, int: 136 (0%) %R, float: 18 (0%) %A, str: 6 (0%)Patch:
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 480b671390..d2df7bf62d 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -2750,6 +2750,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, PyObject *obj = va_arg(*vargs, PyObject *); PyObject *str; assert(obj); +fprintf(stderr, "@@@ PyUnicode_FromFormat(%%S): %s\n", Py_TYPE(obj)->tp_name); str = PyObject_Str(obj); if (!str) return NULL; @@ -2766,6 +2767,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, PyObject *obj = va_arg(*vargs, PyObject *); PyObject *repr; assert(obj); +fprintf(stderr, "@@@ PyUnicode_FromFormat(%%R): %s\n", Py_TYPE(obj)->tp_name); repr = PyObject_Repr(obj); if (!repr) return NULL; @@ -2782,6 +2784,7 @@ unicode_fromformat_arg(_PyUnicodeWriter *writer, PyObject *obj = va_arg(*vargs, PyObject *); PyObject *ascii; assert(obj); +fprintf(stderr, "@@@ PyUnicode_FromFormat(%%A): %s\n", Py_TYPE(obj)->tp_name); ascii = PyObject_ASCII(obj); if (!ascii) return NULL;
Thank you. According to these results we can ignore
intandfloatand only optimizestr. Of course, the data obtained in such way is very rough estimation of real world usage, but the difference is too large.Unfortunately,
%Rforstris not optimized in this PR, and the usage ofstrwith%Sis not overwhelming, so this optimization can have very minor effect. I am not sure now that the gain is worth the cost.- added a commit that references this issue
on Jun 3, 2024
To prepare gh-119182 implementation, I propose to optimize first the _PyUnicodeWriter implementation. For example, optimize the UTF-8 decoder in PyUnicode_FromFormat() by avoiding the creation of a temporary buffer: write directly characters in the writer.
Linked PRs