Repository navigation
Native blob attribute silently stores str(array) on MySQL — no error on insert or fetch #1527
Description
Summary
Declaring an attribute with a native MySQL blob type (longblob, mediumblob,
blob, tinyblob) and inserting a numpy array stores str(array) — numpy's
text repr, with the middle elided for large arrays — and no error is raised on
insert or on fetch. The data is unrecoverable afterwards.
Reported downstream against element-facemap, which declares 16 such
attributes: datajoint/element-facemap#45. Credit to
@akshay-jaggi for the reproduction. This issue is the upstream half.
Reproduction
import numpy as np, datajoint as dj
schema = dj.Schema("probe")
@schema
class Native(dj.Manual):
definition = """
id : int32
---
trace : longblob
"""
a = np.arange(5000, dtype="float32") / 7.0 # 20,000 bytes
Native().insert1({"id": 1, "trace": a})>>> (Native & "id=1").fetch1("trace")
b'[0.0000000e+00 1.4285715e-01 2.8571430e-01 ... 7.1385712e+02 7.1400000e+02\n 7.1414288e+02]'
90 bytes stored. 4,994 of the 5,000 values are gone. information_schema
reports the column as longblob, so nothing about the schema looks wrong.
Why it happens
declare.py:72—NATIVE_BLOBmatches, and:86puts it inNATIVE_TYPES,
notSPECIAL_TYPES. So no:<type>:marker is written into the column
comment, and the attribute reloads permanently asis_blob=True, codec=None.
The only signal is the generic warning atdeclare.py:916-921, which talks
about portability and says nothing about how the value will be stored.table.py:1325__make_placeholder— withattr.codecNone, the encode
branch at:1362is skipped, socodec.validate(value)never runs.
attr.uuid,attr.json, andattr.numericare all False, and theelif
chain ends in a bare comment with no code (:1416). The ndarray is returned
unmodified as a bound parameter.table.py:899hands it to the driver. PyMySQL has no encoder for
np.ndarray, soconverters.escape_itemfalls back toescape_str, which
is"'%s'" % escape_string(str(value)).codecs.py:645—if attr.is_blob: return data # Raw bytes.
So the stringification is PyMySQL's fallback, not ours. On PostgreSQL the same
declaration raises instead: adapters/postgres.py:146-162 registers psycopg2
adapters for numpy scalars only, so an ndarray gives
ProgrammingError: can't adapt type. The silent-corruption path is
MySQL-specific, which is why nothing in our own code reads as wrong.
The 2.x semantics themselves are fine — adapters/mysql.py:27,42 map
bytes ↔ longblob, so a native blob column is a raw binary column by design,
and <blob> is the codec. The bug is that we accept a value that column cannot
hold.
Proposed fixes
- Reject the value. In
__make_placeholder, whenattr.is_bloband
attr.codec is None, raiseDataJointErrorunless the value is
bytes/bytearray/memoryview. Message should name<blob>as the fix.
This is the part that matters — it converts silent data loss into a loud
error, and it makes MySQL behave like PostgreSQL already does. - Sharpen the declaration warning for native blob types specifically: say
the column stores raw bytes and that<blob>is wanted for arrays, rather
than the generic "consider a core DataJoint type for better portability". - Add the missing test.
tests/integration/test_blob.py:171 test_insert_longblobdoes not cover this — its table
(tests/schema.py:450-455) declaresdata: <blob>. The codec-less native
blob path has no coverage at all.
Not affected
Schemas populated under 0.14.x and migrated with dj.migrate are fine —
analyze_blob_columns / migrate_blob_columns (migrate.py:81-86, 322, 509)
rewrite the column comments to carry :<blob>:, and 0.14 wrote real packed
blobs, so the bytes on disk are valid. Only rows written under 2.x into a bare
native blob column are lost.
Activity
fix: validate attribute types at declaration and insert (#1527, #1528, …