Skip to content

feat(rocksdb): add optional ToplingDB integration - #3134

Open
kenssa4eedfd wants to merge 63 commits into
apache:masterfrom
hugegraph:toplingdb
Open

feat(rocksdb): add optional ToplingDB integration#3134
kenssa4eedfd wants to merge 63 commits into
apache:masterfrom
hugegraph:toplingdb

Conversation

@kenssa4eedfd

Copy link
Copy Markdown
Contributor

Purpose of the PR

This PR introduces ToplingDB as an optional RocksDB-compatible storage engine
for HugeGraph.

ToplingDB is explicitly enabled through configuration. Standard RocksDB remains
the default, so existing deployments and data paths are unaffected unless
rocksdb.provider=topling is configured.

Main Changes

  • Add explicit storage engine selection through rocksdb.provider.
  • Integrate ToplingDB with HugeGraph Server, PD, and Store.
  • Use ToplingDB Easy Migrate YAML configuration while continuing to use the
    standard RocksDB Java API.
  • Add component-specific configurations for Server, PD, and Store, including
    separate HTTP ports and JRaft column-family settings.
  • Add startup scripts that:
    • Select and preload the ToplingDB JNI library.
    • Extract required native libraries and Web UI resources.
    • Configure LD_PRELOAD, LD_LIBRARY_PATH, and
      TOPLINGDB_EASY_MIGRATE_CONF.
    • Validate the operating system and architecture.
    • Verify downloaded dependencies using SHA-256.
    • Handle jemalloc and Ubuntu 24.04 libaio compatibility.
  • Align the RocksDB JNI version across Server, PD, and Store.
  • Update CI workflows to install and test the ToplingDB runtime dependencies.
  • Update third-party dependency metadata, LICENSE information, operational
    documentation, troubleshooting guides, and security notes.

Compatibility

  • Standard RocksDB remains the default provider.
  • Existing RocksDB data and APIs remain compatible.
  • ToplingDB is enabled only when rocksdb.provider=topling is configured.
  • No public HugeGraph API changes are introduced.
  • The bundled ToplingDB runtime currently targets Linux x86_64.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • Build all modules:
      mvn clean compile -Dmaven.javadoc.skip=true                                                                                                                                                        
    • Run HugeGraph Server tests with the RocksDB backend:
      mvn test -pl hugegraph-server/hugegraph-test -am \                                                                                                                                                 
        -P core-test,rocksdb                                                                                                                                                                             
    • Run PD tests:
      mvn install -pl hugegraph-struct -am -DskipTests                                                                                                                                                   
      mvn test -pl hugegraph-pd/hg-pd-test -am                                                                                                                                                           
    • Run Store tests:
      mvn test -pl hugegraph-store/hg-store-test -am                                                                                                                                                     
    • Verify Server, PD, and Store startup in both standard RocksDB mode and
      ToplingDB mode.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects: startup scripts, native library loading, and CI workflows
  • Nope

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

imbajin and others added 30 commits June 12, 2025 14:08
[pull] master from apache:master
[pull] master from apache:master
Co-authored-by: Hervé Boutemy <herve.boutemy@free.fr>
* refactor: Delete the package existing in java8

* chore(format): remove custom line breaks
syslucas and others added 25 commits October 15, 2025 07:53
…-detection to explicit `rocksdb.provider` configuration
# Conflicts:
#	.github/workflows/pd-store-ci.yml
#	hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh
#	install-dist/scripts/dependency/known-dependencies.txt
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. ci-cd Build or deploy dependencies Incompatible dependencies of package feature New feature labels Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd Build or deploy dependencies Incompatible dependencies of package feature New feature size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants