Repository navigation
namedtuple row factory for sqlite3 #57508
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-10-31.00:32:25.001>
labels = ['type-feature']
title = 'namedtuple row factory for sqlite3'
updated_at = <Date 2021-08-05.07:15:53.393>
user = 'https://github.com/ncoghlan'bugs.python.org fields:
activity = <Date 2021-08-05.07:15:53.393>
actor = 'intellimath'
assignee = 'ghaering'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2011-10-31.00:32:25.001>
creator = 'ncoghlan'
dependencies = []
files = ['26909', '26945', '26946', '37673']
hgrepos = []
issue_num = 13299
keywords = ['patch']
message_count = 19.0
messages = ['146670', '146738', '147474', '168617', '168748', '168761', '168762', '168763', '218657', '218668', '223704', '223725', '224702', '224707', '233839', '234424', '394912', '395065', '398970']
nosy_count = 11.0
nosy_names = ['rhettinger', 'ghaering', 'ncoghlan', 'eric.araujo', 'petri.lehtinen', 'serhiy.storchaka', 'dlenski', 'Russell.Sim', 'YoSTEALTH', 'erlendaasland', 'intellimath']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13299'
versions = ['Python 3.5']
Activity
ncoghlan commentedon Oct 31, 2011
Currently, sqlite3 allows rows to be easily returned as ordinary tuples (default) or sqlite3.Row objects (which allow dict-style access).
collections.namedtuple provides a much nicer interface than sqlite3.Row for accessing ordered data which uses valid Python identifiers for field names, and can also tolerate field names which are *not* valid identifiers.
It would be convenient if sqlite3 provided a row factory along the lines of the one posted here:
http://peter-hoffmann.com/2010/python-sqlite-namedtuple-factory.html
(except with smarter caching on the named tuples)
rhettinger commentedon Oct 31, 2011
+1
merwok commentedon Nov 12, 2011
Definitely!
RussellSim commentedon Aug 20, 2012
Hi,
Here is an implementation using lru_cache to prevent regeneration of the named tuple each time.
Cheers,
Russell
rhettinger commentedon Aug 21, 2012
Caching based on the cursor going to be problematic because a single cursor can be used multiple times with different descriptions:
It might make more sense to cache the namedtuple() factory itself:
Also, the example in the docs is too lengthy and indirect. Cut-out the step for creating an populating the database -- just assume db created in the example at the top of the page:
For example::
>>> conn.row_factory = sqlite3.NamedTupleRow >>> c = conn.cursor() >>> for record in c.execute('select * from stocks'): print recordNo need to go into a further lesson on how to use named tuples.
Also, the patch uses star-unpacking: _namedtuple_row(cursor)(*row)
Instead, it should use _make: _namedtuple_row(cursor)._make(row)
(u'2006-04-05', u'BUY', u'MSFT', 1000, 72.0)
RussellSim commentedon Aug 21, 2012
Raymond, Thanks for the comprehensive feedback! It's fantastic! I have updated the patch with most of you feedback... but there was one part that I couldn't follow entirely. I am now using the _make method but I have had to use star unpacking to allow the method to be cached, lru_cache won't allow a key to be a list because they aren't hash-able.
ncoghlan commentedon Aug 21, 2012
You should be able to just use "tuple(col[0] for col in cursor.description)" instead of the current list comprehension in order to make the argument hashable.
RussellSim commentedon Aug 21, 2012
Nick, Thanks for the tip. I have removed the star unpacking.
glangford commentedon May 16, 2014
In abstract, I like the namedtuple interface for sqlite3 as well. One caution is that the approach suggested at
http://peter-hoffmann.com/2010/python-sqlite-namedtuple-factory.html
can have a dramatic impact on performance. For one DB-intensive application, I experienced 20+ seconds run time with the row factory (under 3.4), versus sub second without (identified with cProfile). Many thousands of calls to namedtuple_factory were not good. :)
glangford commentedon May 16, 2014
...if I understand the proposed caching scheme, then repeated executions of the query
SELECT a,b,c FROM table
would result in cache hits, since the column names remain the same. I'm guessing this would resolve the performance problem in the app I saw, but it would be good to verify that performance is broadly similar with/without named tuples.
BreamoreBoy commentedon Jul 22, 2014
I'd like to see this in 3.5 as I often use sqlite so what needs doing here?
serhiy-storchaka commentedon Jul 23, 2014
There is significant overhead. Microbenchmark results:
It would be easier to add __getattr__ to sqlite3.Row.
dlenski commentedon Aug 4, 2014
Serhiy,
52 usec/loop doesn't seem like much overhead. This is not 52 usec per row fetched, but just 52 usec per cursor.execute(). An example where >1 row is fetched for each cursor would show this more clearly.
The advantage of namedtuple is that it's a very well-known interface to most Python programmers. Other db-api modules have taken a similar approach; psycopg2 has a dict-like cursor similar to Row, but has added NameTupleCursor in recent versions. (http://initd.org/psycopg/docs/extras.html#namedtuple-cursor)
serhiy-storchaka commentedon Aug 4, 2014
Yes, above microbenchmarks measure the time of execute() + the time of fetching one row. Here is more precise microbenchmarks.
Here sqlite3.Row is about 1.5 times slower than tuple, but sqlite3.NamedTupleRow is about 7 times slower than sqlite3.Row.
With C implementation of lru_cache() (bpo-14373) the result is much better:
100 loops, best of 3: 3.16 msec per loop
And it will be even more faster (up to 1.7x) when add to the Cursor class a method which returns a tuple of field names.
9 remaining items
erlend-aasland commentedon Jul 17, 2022
FTR, here's a benchmark using @serhiy-storchaka's approach w/LRU cache:
Lib/sqlite3/dbapi2.py patch
bench-row.py
Following up Serhiy's suggestion to add a
cursor.fieldnames()method shows a small improvement:merwok commentedon Jul 17, 2022
These years, I would prefer a dataclass conversion rather than a named tuple.
erlend-aasland commentedon Jul 18, 2022
I'm curious why?
IMO, named tuples are a better fit. A dataclass factory
would be considerably slower than using named tuples1, and it does not offer subscripting nor slicing out of the box.benchmark
Footnotes
I re-ran the benchmark with an optimised build, and now it performed exactly like the namedtuple factory. ↩
merwok commentedon Jul 18, 2022
I don’t like using named tuples after all! I prefer having attribute access only, it’s more explicit than tuple positions.
Nowadays I see named tuples as a great thing to add attributes to some data structure that used to be a tuple only, but as a tool to define new lightweight data classes I prefer the dataclasses module.
erlend-aasland commentedon Jul 19, 2022
More importantly; is it worth it to pursue this issue further? Is the performance acceptable?
merwok commentedon Jul 19, 2022
I think that named tuples still make sense for a row factory. But I am the wrong person to evaluate performance numbers.
erlend-aasland commentedon Jul 20, 2022
An alternative approach is simply to add attribute access to sqlite3.Row.
erlend-aasland commentedon Nov 15, 2022
FYI, I've added a namedtuple row factory as an example in the proposed "row factory how-to" in gh-99507. Perhaps it is suffictient to keep this as an example/recipe in the docs instead of adding it as a new feature.
erlend-aasland commentedon Nov 25, 2022
The sqlite3 docs now include a "row factory how-to" that includes a recipe for a simple
namedtuplerow factory (though without the class creation cache optimisation). I'm not sure it is worth it to add thenamedtuplerow factory as a feature, thought; some users want a named tuple, some want a dataclass, some want something else.In the light of the docs update, I now suggest to close this feature request.