Skip to content

PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD#793

Merged
gszadovszky merged 7 commits intoapache:masterfrom
shangxinli:master_new
Jun 3, 2020
Merged

PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD#793
gszadovszky merged 7 commits intoapache:masterfrom
shangxinli:master_new

Conversation

@shangxinli
Copy link
Copy Markdown
Contributor

@shangxinli shangxinli commented May 21, 2020

Make sure you have checked all steps below.

Jira

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain Javadoc that explain what it does

@shangxinli shangxinli force-pushed the master_new branch 3 times, most recently from e5ebcac to 4c264fc Compare May 22, 2020 02:08
gszadovszky
gszadovszky requested changes May 22, 2020
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstdCodec.java Outdated
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/ZstdCodec.java
Comment thread parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestZstdCodec.java Outdated
@shangxinli
Copy link
Copy Markdown
Contributor Author

shangxinli commented May 26, 2020 •
edited
Loading

@luben, Do you have time to review the code?

@luben
Copy link
Copy Markdown

luben commented May 26, 2020

LGTM

@shangxinli
Copy link
Copy Markdown
Contributor Author

shangxinli commented May 27, 2020

@gszadovszky Do you have time for another look?

gszadovszky
gszadovszky requested changes May 28, 2020
View reviewed changes
Comment thread parquet-hadoop/README.md Outdated
Comment thread parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestZstandardCodec.java Outdated
Comment thread parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestZstandardCodec.java Outdated
@dongjoon-hyun
Copy link
Copy Markdown
Member

dongjoon-hyun commented May 29, 2020

Thank you, @shangxinli and all!
cc @dbtsai

@dbtsai
Copy link
Copy Markdown
Member

dbtsai commented Jun 2, 2020

+1 @shangxinli and thank you for this contribution.

This will allow users who are on order versions of hadoop that don't support native ZSTD to use ZSTD compression in Parquet, and also, users don't have to go through the very complicated hadoop native installation. For developers, we will be able to easily test this out in different local envs.

cc @rdblue

@dbtsai
Copy link
Copy Markdown
Member

dbtsai commented Jun 2, 2020

@shangxinli do we have benchmark comparing to native hadoop codec both in size and speed? Thanks.

gszadovszky
gszadovszky requested changes Jun 2, 2020
View reviewed changes
Comment thread parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestZstandardCodec.java Outdated
Comment thread parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestZstandardCodec.java Outdated
Comment thread parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestZstandardCodec.java
@gszadovszky gszadovszky merged commit dc40b59 into apache:master Jun 3, 2020
@shangxinli
Copy link
Copy Markdown
Contributor Author

shangxinli commented Jun 5, 2020

@shangxinli do we have benchmark comparing to native hadoop codec both in size and speed? Thanks.

Hi @dbtsai, I didn't because I don't have Hadoop host installed with ZSTD. @luben, did you ever compare it with Hadoop ZSTD?

@luben
Copy link
Copy Markdown

luben commented Jun 5, 2020

@shangxinli : I haven't benchmarked

shangxinli added a commit to shangxinli/parquet-mr that referenced this pull request Jun 8, 2020
@shangxinli
PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD (apache#793)
7a29b4c
@shangxinli shangxinli deleted the master_new branch June 11, 2020 18:58
shangxinli added a commit to shangxinli/parquet-mr that referenced this pull request Jun 15, 2020
@shangxinli
PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD (apache#793)
32ba400
shangxinli added a commit to shangxinli/parquet-mr that referenced this pull request Jun 15, 2020
@shangxinli
PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD (apache#793)
a15cc0f
emmanuelguerin pushed a commit to criteo-forks/parquet-mr that referenced this pull request Sep 1, 2020
Backport of PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD
f45c5bb 
Original pull request: apache#793

Change-Id: Iccb6a643f06664a4626b0f5e219de6c907743b94
emmanuelguerin pushed a commit to criteo-forks/parquet-mr that referenced this pull request Sep 1, 2020
Backport of PARQUET-1866: Replace Hadoop ZSTD with JNI-ZSTD
2a1b48b 
Original pull request: apache#793

Change-Id: Iccb6a643f06664a4626b0f5e219de6c907743b94
@zhongyujiang zhongyujiang mentioned this pull request Feb 9, 2022
Merged
bluesheeptoken added a commit to bluesheeptoken/kafka-connect-storage-common that referenced this pull request May 4, 2023
@bluesheeptoken
Update parquet version
3f5b3d5 
To add zstd-jni and support zstd-jni compression easily in connectors.
cf: apache/parquet-java#793
@bluesheeptoken bluesheeptoken mentioned this pull request May 4, 2023
Open
@pan3793 pan3793 mentioned this pull request Apr 3, 2026
Open
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gszadovszky gszadovszky gszadovszky approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@shangxinli @luben @dongjoon-hyun @dbtsai @gszadovszky