Resolve into_shape() limitations

[jturner314]

The .into_shape() method works in some cases, but compared to NumPy's reshape() has the following limitations:

1. It only works for arrays with contiguous memory layout.
2. The result depends on the layout of the data in memory. (Calling .into_shape() on an array with Fortran memory layout will give a different result from calling .into_shape() on the logically equivalent array with C memory layout.)
3. All of the new axis lengths need to be known. (NumPy has the nice ability to use -1 to indicate that an axis length should be inferred.)

I'd like to add a method or methods to resolve at least limitations 1 and 2.

In order to handle arrays without contiguous memory layout, the data needs to be cloned in some cases. If the input array owns its data, this is simple – just return a new owned array. However, if the input array is an ArrayView, then you need to return something like Cow. If the input array is an ArrayViewMut, you need some way to return a mutable view with the correct shape but make sure that any changes to that view are copied back to the original underlying data.

A prototype of one approach is at jturner314/ndarray@0157df8. This introduces a few new types, including ArrayViewTemp which handles the temporary array if the input is an ArrayView, and ArrayViewMutTemp, which handles the case where the input is an ArrayViewMut (using Drop to make sure the data is written from the temporary array back to the original underlying data).

It's worth noting that this type of problem (needing a temporary array in some cases) also occurs for operations other than reshaping. For example, I'm working on a similar approach for ndarray-linalg to convert arrays/views to Fortran layout to pass to LAPACK. If something like ArrayViewTemp and ArrayViewMutTemp get added to ndarray, I can reuse those types in ndarray-linalg for the changing-layout problem.

What do you think?

Edit: After thinking about this a little more, I realized:

1. Instead of wrapping arrays in ArrayViewTemp and ArrayViewMutTemp enums, it would be cleaner to create new reprs for the S type parameter in ArrayBase that implement Data/DataMut.
2. It might be possible for drop_hook in ArrayViewMutTempRepr to own the view of the original data so that the view field and Do type parameter could be removed.

[bluss]

The drawback 2 seems critically bad, but at the same time, it's in the interest of not biasing too much in favour of the default c order.

[jturner314]

The drawback 2 seems critically bad, but at the same time, it's in the interest of not biasing too much in favour of the default c order.

The issue is that in a large program, it can be difficult to determine the memory layout of an array by just reading the source code, so the result of .into_shape() is fairly unpredictable. For example, slicing often creates custom-layout views, and I typically create column-major arrays for use with ndarray-linalg (for efficiency when calling LAPACK routines). The memory layout of the result of some methods isn't even specified in the documentation (e.g. .dot()). Every time before I call .into_shape(), I have to check if the memory layout is what I think it is and explicitly copy the data into a new row-/column-major array if it isn't.

I really like the way NumPy's reshape() handles this. It takes a parameter order for the desired logical interpretation of the reshaping operation. If order='C', then the data is reshaped following row-major order; if order='F', then the data is reshaped following column-major order; and if order='A', then the ordering is chosen based on the memory layout of the array (similar to the existing .into_shape()). The 'C' and 'F' options are completely independent of the underlying memory layout of the array; they dictate the logical interpretation of the reshape() operation, similar to how indices in ndarray are logically row-major even if the underlying memory layout is column-major or some custom layout.

The challenge with directly applying NumPy's reshape() to ndarray is that in some cases, the result can just be a view of the original array, while in other cases, the data actually needs to be copied. For owned arrays, it's simple – just take ownership of the array and return a new owned array (copying if necessary.) This doesn't work for views, though. Personally, I think the best approach for views is to add new types for the S parameter in ArrayBase. For read-only views, this would be something like:

enum TmpViewRepr<'a, A> {
    View(ViewRepr<&'a A>),
    Temporary(OwnedRepr<A>),
}

impl<'a, A> Data for TmpViewRepr<'a, A> {
    // ...
}

which can represent a read-only view of the data or a temporary copy. (This could also be called CowRepr if you want clone-on-write functionality.) The other type would be something like

enum TmpViewMutRepr<'a, A> {
    View(ViewRepr<&'a mut A>),
    Temporary {
        tmp: OwnedRepr<A>,
        drop_hook: Box<FnMut(&mut Self)>
    },
}

impl<'a, A> Data for TmpViewMutRepr<'a, A> {
    // ...
}

impl<'a, A> DataMut for TmpViewMutRepr<'a, A> {
    // ...
}

which can represent a mutable view of the data or a temporary copy. When this type gets dropped, if it's the Temporary variant, it would call drop_hook, which could be used to write the modified data in the temporary copy back to the original array. (The drop_hook could be a closure with a reference to the original array.)

These types are useful not only for improving .into_shape() in ndarray but also for client code. In my own programs, I sometimes need to either view existing data or create a temporary array, which could be handled by TmpViewRepr. In ndarray-linalg, these types would be useful when changing the memory layout of arrays to pass to LAPACK routines.

What do you think?

Edit: Yet another place where TmpViewMutRepr would be useful is if a function needs &mut [A] as an argument, but you only have an ArrayViewMut1<A> which might not be contiguous. You could do .into_contiguous().as_slice_mut().unwrap(), where .into_contiguous() would return an ArrayBase<S::TmpRepr, D> where ViewRepr<&'a mut A>::TmpRepr = TmpViewMutRepr<'a, A>.

[bluss]

Just a brief note, I'm sorry; I've been swamped this week. RcArray is much more like numpy's array :-P But in the end we went for the tempting Rust model of Array vs views.

So a solution RcArray could also be an option; note that your "Temp" should probably be called Cow.

[jturner314]

Now that #632 is merged, we can add an .as_shape() method.

I think we should require a logical order parameter (similar to NumPy's .reshape()). For example, the user might write

// row-major order
arr.as_shape((3, 4), Order::RowMajor)?
// column-major order
arr.as_shape((3, 4), Order::ColMajor)?

and the API would be

/// This is analogous to the `order` parameter in
/// [`numpy.reshape()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html#numpy.reshape).
pub enum Order {
    /// C-like order
    RowMajor,
    /// Fortran-like order
    ColMajor,
    #[doc(hidden)]
    __NonExhaustive,
}

impl<A, S, D> ArrayBase<S, D>
where
    S: RawData<Elem = A>,
    D: Dimension,
{
    pub fn into_shape<E>(self, shape: E, order: Order) -> Result<ArrayBase<S, E::Dim>, ShapeError>
    where
        S: DataOwned,
        E: IntoDimension,
    {
    }

    pub fn as_shape<E>(&self, shape: E, order: Order) -> Result<CowArray<'_, A, E::Dim>, ShapeError>
    where
        S: Data,
        E: IntoDimension,
    {
    }
}

We could add other variants to Order, e.g. Automatic (like NumPy's 'A') or Custom(D) (for arbitrary axis order).

Edit: Earlier, I included a second approach, but I later realized that it didn't actually work, so I've removed it from this comment.

Thoughts?

[bluss]

Yes, implementing @jturner314's plan here is important. We shouldn't just add a new method but need to rethink how to handle all of reshape, into_shape and the new method.

The as_ name doesn't sound like it fits as well as the into_ one.. Maybe to_ if it can't be into_?