[antonwolfy]

pickle.dumps(a) and copy.deepcopy(a) failed on any dpnp.ndarray with TypeError: no default __reduce__ due to non-trivial __cinit__. The default pickling tries to serialize the underlying usm_ndarray together with its SYCL queue, and a SYCL queue can't be pickled.

Closes #2602.

import copy, pickle
import dpnp

a = dpnp.arange(6.0).reshape(2, 3)
b = pickle.loads(pickle.dumps(a))  # TypeError before, works now
c = copy.deepcopy(a)               # TypeError before, works now

How pickling works (__reduce__):

• The data is copied to the host with asnumpy(order="A") and pickled together with the array type and the USM type. The SYCL queue isn't pickled.
• Unpickling allocates the array on the default device with the original USM type, so it can end up on a different device than the one it was pickled from. For example, an array pickled from a CPU device comes back on the default GPU. The device isn't recorded because a filter string or device index can name a missing or a different device on another machine, or with a different ONEAPI_DEVICE_SELECTOR.
• The dtype is kept. Unpickling raises ValueError on a device that doesn't support it, e.g. float64 on a device without fp64, instead of silently downcasting to float32 as dpnp.asarray would.
• The memory layout follows NumPy: an F-contiguous array stays F-contiguous, and anything else becomes C-contiguous.
• Subclasses of dpnp.ndarray are preserved.

copy.deepcopy (__deepcopy__) behaves like copy.copy: it returns a.copy(order="K") on the array's own queue, with no host round trip. It has to be defined explicitly; otherwise copy.deepcopy would go through __reduce__ and move the array to the default device.

Docs: ndarray.__reduce__ is now listed in doc/reference/ndarray.rst.

• Have you provided a meaningful PR description?
• Have you added a test, reproducer or referred to an issue with a reproducer?
• Have you tested your changes locally for CPU and GPU devices?
• Have you made sure that new changes do not introduce compiler warnings?
• Have you checked performance impact of proposed changes?
• Have you added documentation for your changes, if necessary?
• Have you added your changes to the changelog?

[github-actions]

View rendered docs @ https://intelpython.github.io/dpnp/pull/3142/index.html

[github-actions]

Array API standard conformance tests for dpnp=0.22.0dev0=np2py314h8d9cdd5_7 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

[coveralls]

coverage: 82.64% (+0.004%) from 82.636% — support-pickle-ndarray into master

[ndgrigorian]

@antonwolfy
I am working to make SyclQueue picklable under the right conditions. In the future, when that PR is open and then merged to dpctl, we can use this as a fallback for unpicklable arrays (cases where non-default context is used, etc.)

[antonwolfy]

@ndgrigorian
That sounds great! What do you think we need to wait with adding pickle support to dpnp?
Or it sounds reasonable to add a limited one with the default device/context now and to extend that by later PR, once SyclQueue is pickable?

Comparing to CuPy, they seem also supporting only the default device.

[ndgrigorian]

Or it sounds reasonable to add a limited one with the default device/context now and to extend that by later PR, once SyclQueue is pickable?

I think this sounds reasonable, if the dpctl PR ends up open soon (which it may be) we can rework this one slightly

[ndgrigorian]

IntelPython/dpctl#2406 is now open