[IPC][ZSTD] Compressed buffer prefix can be written as 0 while ZSTD frame has non-zero content size

Describe the bug, including details regarding any error messages, version, and platform.

Describe the bug

We observed Arrow IPC streams written by Arrow Java where a ZSTD-compressed buffer has an invalid 8-byte uncompressed-length prefix.

The compressed payload itself appears to be a valid ZSTD frame, but the Arrow IPC prefix is 0, while the ZSTD frame reports a non-zero decompressed content size.

This makes downstream readers fail. In our case ClickHouse 26.1 fails with:

Error while reading batch of Arrow data:
IOError: ZSTD decompression failed: Destination buffer is too small

The same file also fails when loaded with Arrow Java ArrowStreamReader.

This has been observed with both:

Arrow Java 18.3.0 + zstd-jni 1.5.7-6
Arrow Java 19.0.0 + zstd-jni 1.5.7-7

Environment

Observed with both dependency sets:

Original environment
- Apache Arrow Java: 18.3.0
- org.apache.arrow:arrow-vector:18.3.0
- org.apache.arrow:arrow-memory-netty:18.3.0
- org.apache.arrow:arrow-compression:18.3.0
- com.github.luben:zstd-jni:1.5.7-6
Also reproduced with
- Apache Arrow Java: 19.0.0
- org.apache.arrow:arrow-vector:19.0.0
- org.apache.arrow:arrow-memory-netty:19.0.0
- org.apache.arrow:arrow-compression:19.0.0
- com.github.luben:zstd-jni:1.5.7-7

Other context:

Java: 17
Writer API:
- ArrowStreamWriter
- CommonsCompressionFactory.INSTANCE
- CompressionUtil.CodecType.ZSTD

Writer creation:

new ArrowStreamWriter(
    root,
    null,
    channel,
    IpcOption.DEFAULT,
    CommonsCompressionFactory.INSTANCE,
    CompressionUtil.CodecType.ZSTD
);

What happened

One generated Arrow IPC stream contains 44 record batches and a valid end marker. The total row count from record batch metadata matches the sidecar count file.

However, batch 21 fails to load.

Inspection results:

file size: 26090728 bytes
schema fields: 504
record batches: 44
batch metadata row sum: 92936
sidecar count: 92936

The failure occurs on batch 21.

batch=21
rows=3469
field[446]=update_date type=Timestamp(MILLISECOND, null)
node length=3469
node nullCount=3469

For this Timestamp(MILLISECOND) vector, the data buffer should decompress to:

3469 rows * 8 bytes = 27752 bytes

But the compressed buffer contains:

buffer[1264]
compressedBytes=27
declaredUncompressed=0
zstdContentSize=27752

So the Arrow IPC compressed buffer prefix says the uncompressed length is 0, but the ZSTD frame itself reports 27752.

Neighboring batches for the same buffer look correct:

batch=20 buffer[1264]
declaredUncompressed=22552
zstdContentSize=22552

batch=22 buffer[1264]
declaredUncompressed=20672
zstdContentSize=20672

Expected behavior

The 8-byte Arrow IPC compressed buffer prefix should match the uncompressed length of the compressed payload.

For the failing buffer, it should have been:

declaredUncompressed=27752

Actual behavior

The prefix is written as:

declaredUncompressed=0

while the ZSTD frame content size is:

zstdContentSize=27752

This causes readers to allocate a zero-sized or too-small destination buffer, then fail during ZSTD decompression.

Arrow Java reader failure

Loading the file with Arrow Java fails at batch 21:

failedAfterBatches=20
rowsBeforeFailure=39866
bytesReadBeforeFailure=12083504

java.lang.IndexOutOfBoundsException:
index: 0, length: 512 (expected: range(0, 504))
    at org.apache.arrow.memory.ArrowBuf.checkIndex(ArrowBuf.java:690)
    at org.apache.arrow.memory.ArrowBuf.setBytes(ArrowBuf.java:942)
    at org.apache.arrow.vector.BaseFixedWidthVector.reAlloc(BaseFixedWidthVector.java:443)
    at org.apache.arrow.vector.BaseFixedWidthVector.setValueCount(BaseFixedWidthVector.java:764)
    at org.apache.arrow.vector.VectorSchemaRoot.setRowCount(VectorSchemaRoot.java:247)
    at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:90)
    at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:213)
    at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:161)

ClickHouse reader failure

ClickHouse fails on the same Arrow stream:

Code: 33. DB::Exception:
Error while reading batch of Arrow data:
IOError: ZSTD decompression failed: Destination buffer is too small:
While executing WaitForAsyncInsert. (CANNOT_READ_ALL_DATA)

Diagnostic code used

The following kind of check was used to inspect the failing compressed buffer:

long declared = ByteBuffer.wrap(bytes, 0, 8)
    .order(ByteOrder.LITTLE_ENDIAN)
    .getLong();

byte[] frame = Arrays.copyOfRange(bytes, 8, len);
long zstdContentSize = Zstd.decompressedSize(frame);

System.out.println("declaredUncompressed=" + declared);
System.out.println("zstdContentSize=" + zstdContentSize);

For the failing buffer:

declaredUncompressed=0
zstdContentSize=27752

Notes

We have not yet produced a small deterministic reproducer. The issue was found in production-generated Arrow IPC streams with a wide schema and repeated batch writes using the same VectorSchemaRoot.

The writer pattern is:

for each batch:
    root.clear();
    for (FieldVector vector : root.getFieldVectors()) {
        vector.allocateNew();
    }
    root.setRowCount(batchSize);

    // populate vectors

    for (FieldVector vector : vectors) {
        vector.setValueCount(batchSize);
    }

    writer.writeBatch();

The failing column is a nullable Timestamp(MILLISECOND) vector where the entire failing batch has nulls for that column.

A minimal test with only one nullable timestamp column and all-null values did not reproduce the issue, so this may require a wider schema, vector reuse, or a particular buffer lifecycle pattern.

Additional reproduction note

We also tested with Apache Arrow Java 19.0.0 and zstd-jni 1.5.7-7, and the same class of corruption still occurs: the ZSTD frame has a non-zero decompressed content size, but the Arrow IPC compressed buffer prefix is written as 0.

This suggests the issue is not limited to Arrow Java 18.3.0 or zstd-jni 1.5.7-6.

Workaround

We worked around this locally by replacing the default ZSTD codec with a custom codec that captures input.writerIndex() before compression, writes that value into the first 8 bytes of the compressed buffer, and verifies the prefix immediately after writing.

This avoids producing a buffer where the ZSTD frame content size and Arrow IPC declared uncompressed length diverge.

Questions

Is this a known issue in Arrow Java IPC ZSTD compression around AbstractCompressionCodec.compress() / ZstdCompressionCodec.doCompress()?
Could uncompressedBuffer.writerIndex() become stale or zero between doCompress() and writeUncompressedLength() in the parent codec?
Are there known issues with reusing VectorSchemaRoot across many compressed IPC batches in Arrow Java 18.3.0 or 19.0.0?

Address review: drop concurrency framing from compress() comment

Address review: drop ZSTD round-trip tests that don't reflect apacheG…

GH-1116: [Java] Fix compressed buffer prefix write and ZSTD dstCapaci…

We merged a fix that may relate to this, but we were unable to replicate the issue. Generally, reusing a VectorSchemaRoot is encouraged, however, concurrency is not supported.

D34: 修复压缩读链泄漏——上游 apache/arrow-java#1116 前缀损坏 → VectorLoader 异常路径泄漏 →…

fg-p7 (D34): https presign 直取挂 ResilientCompressionFactory——apache#1116…

We hit what looks like the other face of this defect in Apache DataFusion Comet (apache/datafusion-comet#7305): an executor JVM with Native Memory Tracking enabled aborted with

fatal error: NMT corruption: Block at 0x...: footer canary broken at 0x... (buffer overflow?)

while ZstdCompressionCodec was compressing Arrow IPC buffers. The corrupted block was 512 bytes and held a zstd frame that inflates to 437 bytes, with no length prefix: the buffer that AbstractCompressionCodec.compress closes when the output turns out larger than the input.

The dstCapacity overstatement that #1119 corrected is a real overrun, not only a semantic nit. zstd guarantees that it never writes past the capacity it is given, not that it never writes past compressBound: while it entropy-codes the literals of a block that turns out not to compress, it fills the destination up to dstCapacity before falling back to a stored block, so with 8 + compressBound(n) as the capacity it can write up to 8 bytes past the allocation. The allocator's power-of-two rounding hides it unless compressBound(n) + 8 lands within 8 bytes below a power of two (compressBound(437) + 8 = 509, allocated as 512), and without NMT the overrun lands silently in padding or in the next block, which may be why neither of us could get a minimal reproducer from the original symptom.

A deterministic reproducer for the 18.3.0 code, run with -XX:NativeMemoryTracking=detail (classpath: arrow-compression, arrow-vector, arrow-memory-core, arrow-memory-unsafe, arrow-format, flatbuffers, jackson, slf4j, zstd-jni):

import java.util.Random;
import org.apache.arrow.compression.ZstdCompressionCodec;
import org.apache.arrow.memory.*;

public class ZstdCanary {
  public static void main(String[] args) {
    try (BufferAllocator allocator = new RootAllocator()) {
      ZstdCompressionCodec codec = new ZstdCompressionCodec(1);
      for (int len = 430; len <= 441; len++) {
        for (int seed = 0; seed < 2000; seed++) {
          ArrowBuf in = allocator.buffer(len);
          in.setBytes(0, protoLike(len, seed));
          in.writerIndex(len);
          codec.compress(allocator, in).close(); // aborts the JVM on the free of an overrun block
        }
      }
      System.out.println("no abort");
    }
  }

  // Random bytes with a scatter of repeated values: structured enough for zstd to attempt
  // entropy coding, random enough for the attempt to fail and run to the end of dstCapacity.
  static byte[] protoLike(int len, int seed) {
    Random rnd = new Random(seed * 7919L + len);
    byte[] out = new byte[len];
    rnd.nextBytes(out);
    int tags = 8 + rnd.nextInt(24);
    for (int i = 0; i < tags; i++) out[rnd.nextInt(len)] = (byte) (0x08 + 8 * rnd.nextInt(4));
    return out;
  }
}

On Arrow 18.3.0 this aborts within the first few hundred inputs at len=437 (and at other lengths in the exposed window), with the Java stack AbstractCompressionCodec.compress -> BufferLedger.release -> Unsafe.freeMemory0. With doCompress changed as in #1119 (maxSize as dstCapacity), the same 24000 inputs run clean. The behaviour is the same on macOS/aarch64 and Linux/x86_64.

Since the fix is a one-argument change and 18.3.0 and 19.0.0 both ship the overrun, would a patch release carrying #1119 be possible? Until then we are carrying a subclass with the corrected doCompress in Comet.

@jbonofre

@jbonofre is working on a 20.0.0 release. We don't have the capacity for patch releases, I would say (frankly: the rise of automated PRs and issues has blown out what was already a very limited base of maintainers)

@lidavidm

@lidavidm @m8928 I propose to freeze the changes to move forward on the release.

Navigation Menu

Repository navigation

[IPC][ZSTD] Compressed buffer prefix can be written as 0 while ZSTD frame has non-zero content size #1116

Describe the bug, including details regarding any error messages, version, and platform.

Describe the bug

Environment

What happened

Expected behavior

Actual behavior

Arrow Java reader failure

ClickHouse reader failure

Diagnostic code used

Notes

Additional reproduction note

Workaround

Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[IPC][ZSTD] Compressed buffer prefix can be written as 0 while ZSTD frame has non-zero content size #1116

Description

Describe the bug, including details regarding any error messages, version, and platform.

Describe the bug

Environment

What happened

Expected behavior

Actual behavior

Arrow Java reader failure

ClickHouse reader failure

Diagnostic code used

Notes

Additional reproduction note

Workaround

Questions

Activity

lidavidm commented on Jun 30, 2026

andygrove commented on Oct 28, 2026

lidavidm commented on Oct 29, 2026

jbonofre commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions