HADOOP-19855. Replace native zstd C bindings with zstd-jni library by pan3793 · Pull Request #8399 · apache/hadoop

pan3793 · 2026-04-02T16:21:49Z

Description of PR

Replaces Hadoop's custom native C bindings for ZStandard compression (ZStandardCompressor.c, ZStandardDecompressor.c) with the com.github.luben:zstd-jni JVM library. This eliminates the requirement to build or install libzstd natively, making ZStandard compression available on all platforms without a native build.

This is similar to

zstd-jni is the de facto choice for JVM applications to the zStandard compression algorithm, it was adopted by many projects - Parquet, Spark, Avro, Kafka, HBase, ElasticSearch, etc.

It's relatively painful to set up / upgrade native libraries on each Hadoop client and node, downstream projects like Parquet/Spark/HBase used to choose to implement their own zstd codec to overcome such a challenge, resulting in a lot of repetitive work:

zstd-jni has been used and bundled by Hadoop, hadoop-3.5.0/share/hadoop/tools/lib/zstd-jni-1.5.6-4.jar

How was this patch tested?

Pass all existing UTs:

TestZStandardCompressorDecompressor
TestCodec
TestCompressionStreamReuse

Manually tested the implementation is compatible with the existing Parquet/Spark zstd codec:

Compatible with Parquet ZstandardCodec PARQUET-1866

Modify Parquet to allow it to use Hadoop ZstandardCodec

https://github.com/apache/parquet-java/blob/apache-parquet-1.17.0/parquet-common/src/main/java/org/apache/parquet/hadoop/metadata/CompressionCodecName.java#L32

public enum CompressionCodecName {
  ...
  ZSTD(getZstdCodec(), CompressionCodec.ZSTD, ".zstd"),
  ...;

  static String getZstdCodec() {
    if ("1".equals(System.getenv("PARQUET_USE_HADOOP_ZSTD_CODEC"))) {
      System.err.println("PARQUET_USE_HADOOP_ZSTD_CODEC is set to 1, using Hadoop's ZStandardCodec");
      return "org.apache.hadoop.io.compress.ZStandardCodec";
    } else {
      return "org.apache.parquet.hadoop.codec.ZstandardCodec";
    }
  }
  ...
}

Then replace the Spark shipped parquet-common jar with the modified one.

Create a Parquet Zstd table with Parquet ZstandardCodec

$ bin/spark-sql \
  --packages org.apache.kyuubi:kyuubi-spark-connector-tpcds_2.13:1.11.1 \
  --conf spark.sql.parquet.compression.codec=zstd \
  --conf spark.sql.catalog.tpcds=org.apache.kyuubi.spark.connector.tpcds.TPCDSCatalog
spark-sql (default)> create table spark_catalog.default.store_returns_sf100_parquet_zstd as select * from tpcds.sf100.store_returns;

Create a Parquet Zstd table with Hadoop ZstandardCodec

$ PARQUET_USE_HADOOP_ZSTD_CODEC=1 bin/spark-sql \
  --packages org.apache.kyuubi:kyuubi-spark-connector-tpcds_2.13:1.11.1 \
  --conf spark.sql.parquet.compression.codec=zstd \
  --conf spark.sql.catalog.tpcds=org.apache.kyuubi.spark.connector.tpcds.TPCDSCatalog
spark-sql (default)> create table spark_catalog.default.store_returns_sf100_parquet_zstd_hadoop as select * from tpcds.sf100.store_returns;
PARQUET_USE_HADOOP_ZSTD_CODEC is set to 1, using Hadoop's ZStandardCodec

Test two tables using different ZstandardCodec that can be read by each other

$ bin/spark-sql -e 'select sum(hash(*)) from spark_catalog.default.store_returns_sf100_parquet_zstd_hadoop'
-4268887280300

$ PARQUET_USE_HADOOP_ZSTD_CODEC=1 bin/spark-sql -e 'select sum(hash(*)) from spark_catalog.default.store_returns_sf100_parquet_zstd'
PARQUET_USE_HADOOP_ZSTD_CODEC is set to 1, using Hadoop's ZStandardCodec
-4268887280300

Compatiable with SPARK-52482

Apply SPARK-47829 first

[SPARK-47829][SQL] Text Datasource supports Zstd compression codec spark#46026

Create a JSON zstd table using Hadoop ZstandardCodec

$ bin/spark-sql \
  --packages org.apache.kyuubi:kyuubi-spark-connector-tpcds_2.13:1.11.1 \
  --conf spark.sql.catalog.tpcds=org.apache.kyuubi.spark.connector.tpcds.TPCDSCatalog
spark-sql (default)> create table spark_catalog.default.store_returns_sf100_json_zstd_hadoop USING json OPTIONS ('compression'='zstd') as select * from tpcds.sf100.store_returns;

Try reading using Spark ZStdCompressionCodec, unable to read the table directly, but able to read using path syntax, it's by design.

(Use the unmodified Spark 4.1.1)

spark-sql (default)> select * from spark_catalog.default.store_returns_sf100_json_zstd_hadoop limit 10;
[CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] The codec zstd is not available. Available codecs are bzip2, deflate, uncompressed, snappy, none, lz4, gzip. SQLSTATE: 56038
spark-sql (default)> select * from json.`/Users/chengpan/spark-warehouse/store_returns_sf100_json_zstd_hadoop` limit 10;
200038	1102768	1837067	63.98	5403	157322	1896.12	48	1171.92	3348.36	3448.81	71	1731.69	100.45	58555	2451142	739.98	1436.46	130	2368700
670399	463705	132207	5.0	652	19021	156.56	53	174.33	378.99	378.99	9	151.56	0.0	59886	2452100	59.35	145.31	277	2382371
915603	1042393	61317	59.22	1798	45401	203.67	30	121.47	1349.73	1349.73	27	144.45	0.0	48103	2452295	98.26	1130.0	364	542666
752398	436638	1417958	98.15	4918	134947	876.85	16	1920.9	2158.32	2352.56	34	584.46	194.24	57572	2451165	45.1	192.32	326	524442
83961	1729324	1785782	78.57	2019	199857	110.02	18	74.7	393.2	424.65	8	0.0	31.45	32499	2452376	238.87	79.63	352	327796
459038	143635	571813	42.99	3913	134895	827.09	51	118.41	910.89	929.1	29	765.89	18.21	35112	2452202	15.84	776.64	326	263544
NULL	131426	286224	55.17	7028	65756	NULL	NULL	16.44	NULL	80.71	NULL	22.22	NULL	NULL	2452014	41.97	16.33	NULL	711309
791122	1592878	1490347	50.16	5600	126566	1381.92	33	501.44	1567.0	1614.01	25	1284.75	47.01	61177	2451635	756.54	309.02	314	294891
929339	608874	1049152	64.23	1330	203665	386.4	17	163.98	341.64	345.05	26	318.76	3.41	39868	2452759	12.43	165.23	286	1726289
925732	1366375	1816722	43.06	5077	81288	3074.15	16	5977.47	6568.65	7094.14	45	2505.6	525.49	43954	2452651	254.2	336.98	211	1972510
Time taken: 10.731 seconds, Fetched 10 row(s)

Compatible with zstd CLI

Able to use zstd CLI to decompress the JSON file compressed by Hadoop ZstandardCodec part-00000-1f9622e4-f4ce-47dc-95ac-512fa5acfd6b-c000.json.zst, and vice versa.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (HADOOP-19855)?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

This PR contains content generated by Claude Code (Sonnet 4.6).

The PR includes the phrase "Contains content generated by "
where is the name of the AI tool used.
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

hadoop-yetus · 2026-04-02T16:23:58Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/1/console in case of problems.

hadoop-yetus · 2026-04-02T16:29:05Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/2/console in case of problems.

hadoop-yetus · 2026-04-03T07:15:47Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/3/console in case of problems.

hadoop-yetus · 2026-04-03T07:24:19Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/4/console in case of problems.

hadoop-yetus · 2026-04-03T08:41:35Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/5/console in case of problems.

pan3793 · 2026-04-03T09:06:49Z

hadoop-client-modules/hadoop-client-api/pom.xml

-         make the downstream pull it. -->
+    <!-- snappy-java/zstd-jni contains native library and cannot be relocated.
+         So we explicitly exclude it from shaded jar to prevent possible conflict.
+         Declare it as transitive dependency to make the downstream pull it. -->


in short, handling zst-jni in the same way as snappy-java

pan3793 · 2026-04-03T10:07:06Z

cc @sunchao @cnauroth @aajisaka @luben @dongjoon-hyun @cxzl25 @sandip-db

I know this change is many years overdue because almost all downstream projects that use the Hadoop Codec have opted to implement a zstd codec based on zstd-jni, resulting in a lot of repetitive work ... hope we can terminate such a situation.

aajisaka

Looks good to me, +1. Great improvement.

aajisaka · 2026-04-06T06:42:11Z

...common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/ZStandardCodec.java

-    }
-  }
-
-  public static boolean isNativeCodeLoaded() {


This class is not @Public, so I think it's okay to remove.

I saw 1 external usage in GitHub: https://github.com/trinodb/trino-hadoop-apache/blob/a1db88d0d14e4331034db0bcfb2dbc61d807db74/src/test/java/io/trino/hadoop/TestHadoopNative.java#L51. It's test only and it does not actually verify the zstd support.

trino-hadoop-apache shouldn't be an issue, it's a dedicated project used by trino to override some hadoop class implementation, so it's easy to adapt the hadoop upstream change.

Copilot

Pull request overview

This PR migrates Hadoop’s ZStandard (zstd) compression implementation from custom native libzstd C/JNI bindings to the com.github.luben:zstd-jni library, removing the need to build/install libzstd as part of Hadoop native builds while keeping ZStandardCodec as the integration point.

Changes:

Add zstd-jni as a managed dependency and wire it into relevant client/runtime modules.
Replace native zstd compressor/decompressor implementations with zstd-jni-backed Java implementations.
Remove zstd native build wiring (CMake, build properties, docker tooling/docs) and update unit tests to no longer require native zstd.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
hadoop-project/pom.xml	Adds `zstd-jni.version` and dependency management; removes zstd bundling/env setup.
hadoop-project-dist/pom.xml	Removes zstd native distribution bundling flags/args.
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml	Removes zstd-related native build properties/env vars; minor message formatting.
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zstd/TestZStandardCompressorDecompressor.java	Removes native-availability assumptions so tests run with zstd-jni.
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCompressionStreamReuse.java	Removes native-availability assumption for zstd reuse test.
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodec.java	Removes native-availability assumption for zstd sequence file test.
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util/NativeCodeLoader.c	Removes `buildSupportsZstd()` JNI implementation.
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/ZStandardDecompressor.c	Deletes native zstd decompressor implementation.
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.c	Deletes native zstd compressor implementation.
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/compress/zstd/org_apache_hadoop_io_compress_zstd.h	Deletes zstd native header.
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeLibraryChecker.java	Removes zstd native checks/output from `checknative`.
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java	Removes `buildSupportsZstd()` declaration.
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/ZStandardDecompressor.java	Reimplements decompression via `ZstdDecompressCtx` streaming API.
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.java	Reimplements compression via `ZstdCompressCtx` streaming API.
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/package-info.java	Updates zstd naming in package docs.
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/ZStandardCodec.java	Removes native-loading checks; reports library as `zstd-jni`.
hadoop-common-project/hadoop-common/src/config.h.cmake	Removes `HADOOP_ZSTD_LIBRARY` definition.
hadoop-common-project/hadoop-common/src/CMakeLists.txt	Removes zstd detection/build/linking from native build.
hadoop-common-project/hadoop-common/pom.xml	Adds `zstd-jni` dependency; removes zstd native build properties/args.
hadoop-client-modules/hadoop-client-runtime/pom.xml	Adds runtime dependency on `zstd-jni`; excludes from shading/relocation.
hadoop-client-modules/hadoop-client-minicluster/pom.xml	Adds runtime dependency on `zstd-jni`; excludes from shading/relocation.
hadoop-client-modules/hadoop-client-check-test-invariants/pom.xml	Updates invariant checks to exclude `zstd-jni` from relocation constraints.
hadoop-client-modules/hadoop-client-check-invariants/pom.xml	Updates invariant checks to exclude `zstd-jni` from relocation constraints.
hadoop-client-modules/hadoop-client-api/pom.xml	Adds transitive dependency on `zstd-jni`; excludes from shading/relocation.
dev-support/docker/pkg-resolver/packages.json	Removes `libzstd-dev` from base build package set.
dev-support/docker/pkg-resolver/install-zstandard.sh	Deletes zstd install helper script.
dev-support/docker/Dockerfile_rockylinux_8	Removes zstd install step from docker image build.
BUILDING.txt	Removes zstd native dependency/build option documentation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-06T08:00:23Z

...ject/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/ZStandardCompressor.java

+    boolean allConsumed = (uncompressedDirectBufLen - uncompressedDirectBufOff <= 0);
+    // Use END only when finish=true, no more user data, and all direct-buffer
+    // data consumed (mirrors ZSTD_endStream); otherwise FLUSH (mirrors
+    // ZSTD_compressStream + ZSTD_flushStream).
+    boolean shouldEnd = finish && userBufLen == 0 && allConsumed;
+    if (!allConsumed || shouldEnd) {
+      // Re-initialize the output direct buffer


In compress(), when allConsumed is true and shouldEnd is false, the method skips calling the zstd streaming API and returns 0. This can break streaming output draining: zstd may still have pending compressed bytes to flush even when no more input is available (especially with small output buffers), and the old native implementation would continue flushing in this state. Consider always invoking the zstd stream with FLUSH whenever there is no remaining compressedDirectBuf output, even if there is currently no input, and use the returned done/state to decide when it’s safe to report needsInput() or produce 0 bytes.

changed to always call FLUSH/END to drain any internally buffered bytes, even with no input, which mirrors how old native code always calls ZSTD_flushStream

Copilot · 2026-04-06T08:00:24Z

...ct/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/ZStandardDecompressor.java

+    // zstd-jni: use streaming decompression directly on the provided buffers
+    int origDstPos = dst.position();
+    boolean done = zstdJniCtx.decompressDirectByteBufferStream(dst, src);
+    if (done) {
+      finished = true;
+      remaining = 0;
    } else {
-      src.position(src.limit());
+      remaining = src.limit() - src.position();
    }


inflateDirect() sets finished=true and remaining=0 whenever done is true. If the source buffer contains concatenated frames or trailing bytes (i.e., src.position() < src.limit() after the call), this will misreport getRemaining() and can incorrectly signal end-of-stream while input bytes are still present. Align this logic with the non-direct decompress() path: compute remaining from src.limit() - src.position() and only set finished when done and remaining==0 (or otherwise explicitly document/handle concatenated frames).

aligned finished/remaining logic with decompress() - only set finished when done && remaining==0

Copilot · 2026-04-06T08:00:24Z

...ct/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zstd/ZStandardDecompressor.java

  @Override
  protected void finalize() {
    reset();
  }


finalize() currently calls reset() rather than end(). With the zstd-jni backend this risks leaking the native decompression context if callers forget to call end(), because reset() does not release the underlying native resources. If finalize() is retained, it should ensure the context is closed (and ideally delegate to super.finalize() as appropriate); alternatively, consider removing finalize() and relying on explicit end()/try-with-resources patterns if that matches project conventions.

replaced with end().

I would keep finalize() in this PR to mirror the existing behavior. Removing it might be evaluated in the future

hadoop-yetus · 2026-04-06T11:49:45Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 16s		#8399 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/6/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

…ensure release resource

…lly buffered bytes, even with no input (mirrors how native always calls ZSTD_flushStream) Removed the early return n=0 when input is exhausted. The streaming API is now always called (with FLUSH or END), so any internally buffered bytes from a previous call where the output buffer was too small will continue to be drained — matching how the native ZSTD_flushStream was always invoked unconditionally.

…ly set finished when done && remaining==0)

hadoop-yetus · 2026-04-06T11:50:46Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 16s		#8399 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/6/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 · 2026-04-06T11:51:18Z

follow yetus response to rebase on trunk (though it does not have conflicts)

hadoop-yetus · 2026-04-06T11:51:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 15s		#8399 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/6/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-04-06T11:52:38Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 14s		#8399 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/7/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

charlesconnell

zstd-jni can operator on byte arrays, if you wish to avoid copying to/from ByteBuffers. This does come at the expense of potental JNI critical section stalls though.

hadoop-yetus · 2026-04-06T11:53:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 16s		#8399 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/7/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-04-06T11:55:07Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 15s		#8399 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/7/console
versions	git=2.34.1
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 · 2026-04-06T12:04:33Z

@charlesconnell, thanks for your comments, I'd like to mirror the old behavior in this patch and post optimization or features (e.g., multi-threads) to later dedicated PRs.

Actually, I want to port the PARQUET-1866 to the Hadoop codebase in the next PR

PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD parquet-java#793

hadoop-yetus · 2026-04-07T01:58:11Z

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/8/console in case of problems.

hadoop-yetus · 2026-04-08T01:51:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+0 🆗	jsonlint	0m 1s		jsonlint was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 14s		trunk passed
+1 💚	compile	8m 22s		trunk passed
+1 💚	checkstyle	2m 44s		trunk passed
+1 💚	mvnsite	8m 53s		trunk passed
+1 💚	javadoc	6m 0s		trunk passed
+0 🆗	spotbugs	0m 16s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 15s		branch/hadoop-project-dist no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 17s		branch/hadoop-client-modules/hadoop-client-api no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 16s		branch/hadoop-client-modules/hadoop-client-runtime no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 15s		branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 16s		branch/hadoop-client-modules/hadoop-client-check-invariants no spotbugs output file (spotbugsXml.xml)
+0 🆗	spotbugs	0m 17s		branch/hadoop-client-modules/hadoop-client-check-test-invariants no spotbugs output file (spotbugsXml.xml)
-1 ❌	spotbugs	18m 37s	/branch-spotbugs-root-warnings.html	root in trunk has 92 extant spotbugs warnings.
+1 💚	shadedclient	33m 45s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s		Maven dependency ordering for patch
+1 💚	mvninstall	28m 23s		the patch passed
+1 💚	compile	8m 22s		the patch passed
+1 💚	cc	8m 22s		the patch passed
+1 💚	golang	8m 22s		the patch passed
+1 💚	javac	8m 22s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 53s	/results-checkstyle-root.txt	root: The patch generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56)
+1 💚	hadolint	0m 1s		No new issues.
+1 💚	mvnsite	8m 57s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	5m 58s		the patch passed
+0 🆗	spotbugs	0m 15s		hadoop-project has no data from spotbugs
+0 🆗	spotbugs	0m 15s		hadoop-project-dist has no data from spotbugs
+0 🆗	spotbugs	0m 16s		hadoop-client-modules/hadoop-client-api has no data from spotbugs
+0 🆗	spotbugs	0m 16s		hadoop-client-modules/hadoop-client-runtime has no data from spotbugs
+0 🆗	spotbugs	0m 16s		hadoop-client-modules/hadoop-client-minicluster has no data from spotbugs
+0 🆗	spotbugs	0m 16s		hadoop-client-modules/hadoop-client-check-invariants has no data from spotbugs
+0 🆗	spotbugs	0m 15s		hadoop-client-modules/hadoop-client-check-test-invariants has no data from spotbugs
+1 💚	shadedclient	33m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	742m 13s	/patch-unit-root.txt	root in the patch failed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		930m 58s

Reason	Tests
Failed junit tests	hadoop.hdfs.server.federation.router.async.TestRouterAsyncRpcClient
	hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor
	TEST-cetest
	hadoop.yarn.server.nodemanager.containermanager.resourceplugin.fpga.TestFpgaDiscoverer
	hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun
	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction
	hadoop.yarn.server.timelineservice.storage.TestTimelineReaderHBaseDown
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
	hadoop.yarn.server.timelineservice.storage.TestTimelineWriterHBaseDown
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain
	hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
	hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/8/artifact/out/Dockerfile
GITHUB PR	#8399
Optional Tests	dupname asflicense codespell detsecrets hadolint shellcheck shelldocs mvnsite unit jsonlint compile javac javadoc mvninstall shadedclient xmllint cc spotbugs checkstyle golang
uname	Linux 143b389add1d 5.15.0-141-generic #151-Ubuntu SMP Sun May 18 21:35:19 UTC 2025 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `7641c63`
Default Java	Debian-25.0.2+10-Debian-1deb13u2
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/8/testReport/
Max. process+thread count	4371 (vs. ulimit of 10000)
modules	C: hadoop-project hadoop-project-dist hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask hadoop-client-modules/hadoop-client-api hadoop-client-modules/hadoop-client-runtime hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-invariants hadoop-client-modules/hadoop-client-check-test-invariants . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8399/8/console
versions	git=2.47.3 maven=3.9.11 hadolint=1.11.1-0-g0e692dd spotbugs=4.9.7 shellcheck=0.10.0
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

pan3793 force-pushed the HADOOP-19855 branch from 9339972 to f76e4fc Compare April 2, 2026 16:27

github-actions bot added MapReduce build trunk Common Infra Native labels Apr 2, 2026

pan3793 force-pushed the HADOOP-19855 branch from f76e4fc to 7552cb8 Compare April 3, 2026 07:13

pan3793 force-pushed the HADOOP-19855 branch from 7552cb8 to 3a3143d Compare April 3, 2026 07:22

pan3793 force-pushed the HADOOP-19855 branch from 3a3143d to 14e941b Compare April 3, 2026 08:39

pan3793 commented Apr 3, 2026

View reviewed changes

pan3793 marked this pull request as ready for review April 3, 2026 09:58

aajisaka approved these changes Apr 6, 2026

View reviewed changes

slfan1989 requested review from cnauroth, Copilot and steveloughran April 6, 2026 07:55

Copilot started reviewing on behalf of slfan1989 April 6, 2026 07:56 View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

pan3793 added 5 commits April 6, 2026 19:50

HADOOP-19855. Replace native zstd C bindings with zstd-jni library

2be4279

ZStandardDecompressor call end() in finalize() instead of reset() to …

ffa8397

…ensure release resource

Remove unused LOG in ZStandardDecompressor

b840980

inflateDirect(): align finished/remaining logic with decompress() (on…

be66804

…ly set finished when done && remaining==0)

further review comments

7641c63

pan3793 force-pushed the HADOOP-19855 branch from 1f13fc0 to 7641c63 Compare April 6, 2026 11:50

charlesconnell reviewed Apr 6, 2026

View reviewed changes

slfan1989 approved these changes Apr 7, 2026

View reviewed changes

Conversation

pan3793 commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Apr 2, 2026

Uh oh!

hadoop-yetus commented Apr 2, 2026

Uh oh!

hadoop-yetus commented Apr 3, 2026

Uh oh!

hadoop-yetus commented Apr 3, 2026

Uh oh!

hadoop-yetus commented Apr 3, 2026

Uh oh!

pan3793 Apr 3, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Apr 3, 2026

Uh oh!

aajisaka left a comment

Choose a reason for hiding this comment

Uh oh!

aajisaka Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Apr 6, 2026

Uh oh!

hadoop-yetus commented Apr 6, 2026

Uh oh!

pan3793 commented Apr 6, 2026

Uh oh!

hadoop-yetus commented Apr 6, 2026

Uh oh!

hadoop-yetus commented Apr 6, 2026

Uh oh!

charlesconnell left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Apr 6, 2026

Uh oh!

hadoop-yetus commented Apr 6, 2026

Uh oh!

pan3793 commented Apr 6, 2026

Uh oh!

hadoop-yetus commented Apr 7, 2026

Uh oh!

hadoop-yetus commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

pan3793 commented Apr 2, 2026 •

edited

Loading