Skip to content

BitShuffle#shuffle for arrays of primitive datatypes does not consider the byte-order (leading to wrong results) #297

Open

Description

@foosmate

The underlying BitShuffle library works internally with an algorithm that was designed for the little-endian format. For this reason, the input data must always be passed in little-endian format.

However, the snappy-java wrapper directly passes an int[] array with the native byte order. Thus the Java implementation does not give the same results on little-endian and big-endian platforms for the shuffle / unshuffle methods.

The shuffle/unshuffle methods still work bijective so this issue is not easily noticeable.

How to fix:

The data array based on primitive types (e.g. int[], short[], long[], etc.) must be converted to little-endian format in the Java implementation if executed on big-endian platforms (before passing the buffer to the external library).

Activity

  1. xerial commented on May 23, 2023

    @xerial
    Owner

    Related #443

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions