Improved File Versioning - #707
Conversation
|
Before you submit for review:
If you did not complete any of these, then please explain below. |
| private final boolean supportsMultiLayer; | ||
| private final boolean usesFooter; | ||
|
|
||
| /** A magic number to indicate the file footer */ |
There was a problem hiding this comment.
Can we get a slightly better readout on what the footer looks like here. These constants don't really explain it for the uninitiated.
jshook
left a comment
There was a problem hiding this comment.
I'll re-review when the convos are resolved
|
|
||
| @Test | ||
| public void testForVersionReturnsMatchingFormat() { | ||
| for (int version = 2; version <= OnDiskGraphIndex.CURRENT_VERSION; version++) { |
There was a problem hiding this comment.
It might be better to implement the version property as a decorator interface, so that callers which need to handle V1 look more like:
if (index instanceof VersionTaggedIndex vti) {
// normative code
} else {
// v1
}
As this allows us to avoid defining a method for a version which is effectively a false promise or actually undefined functionally.
| * on, so that a change to one version's format can't silently change another's behavior. | ||
| */ | ||
| @Test | ||
| public void testFormatCharacteristicsPerVersion() { |
There was a problem hiding this comment.
Like above, decorator interfaces for the features only implemented by a subset of indexes make for cleaner type system here. Each thing which you have to have implicit knowledge of based on a number to feature matrix can be properly represented as a decorator interface, as the call site logic for instanceof can be efficient when required.
Summary
This branch eliminates scattered version-comparison branching (
if version >= X,if version < Y,version == Z) from the on-disk graph index read/write path and replaces it with a Strategy + Factory pattern: eachsupported format version (2–6) gets its own
GraphIndexFormatimplementation, andGraphIndexFormatFactorycentralizes version→format dispatch and detection.What changed
New abstraction (
graph/diskpackage):GraphIndexFormat— strategy interface covering header/footer read+write, feature ordering, and full sequential/random-access write orchestration for a single format version.AbstractGraphIndexFormat— shared default implementation; versions override only what's actually different for them.GraphIndexFormatV2..GraphIndexFormatV6— one small class per version, built on a natural inheritance chain (V4extendsV3,V5extendsV4+ footer,V6extendsV5+ feature reordering + fused-PQsupport).
GraphIndexFormatFactory—forVersion(int),detectVersion(reader)(magic-number sniffing),getCurrentVersion().WriteContext— bundles the ~7 parameters that used to be threaded individually through every serializer method.Migrated call sites:
AbstractGraphIndexWriter(and itsOnDiskSequentialGraphIndexWriter/RandomAccessOnDiskGraphIndexWritersubclasses) now delegate all format-specific work to the resolvedGraphIndexFormatinstead ofbranching on
versioninline. Feature-support validation (Builder.build(), fused-PQ gating) now asks the format (supportsFeature) instead of hardcoding version thresholds.Header/CommonHeadernow delegate header size/read/write and common-header size/read/write to the per-version format, removing theversion >= 3/>= 4/>= 6chains that used to live in both classes.OnDiskGraphIndexload/footer dispatch goes throughGraphIndexFormat.loadOnDiskIndex, removing the inlineversion >= 5 && useFooterbranch; the two remaining fused-PQ-layout checks now askCompactWriter(streaming N:1 compaction writer) validates once, at construction, that fused PQ is actually supported by the target format; the three per-callversion == 6checks in its hot paths were simplifiedto the already-existing
fusedPQEnabledflag, since this writer only ever targets the current version. No added per-record overhead.Fixed along the way:
getInMemoryLayers/getInMemoryFeatures) for non-footer format versions — caught by aramBytesUsed()test failure,fixed by centralizing construction + priming in
OnDiskGraphIndex.construct().allFeatures()/nonFusedFeatures()helpers now return explicit, frozenEnumSet.of(...)listings instead of deriving fromEnumSet.allOf(...)/complementOf(...), so a futureFeatureIdaddition can'tsilently change what an already-shipped version claims to support.
Testing
TestGraphIndexFormatFactory: version→format dispatch, unsupported-version rejection, per-version characteristics (supportsMultiLayer,usesFooter,getSupportedFeatures), and magic-number versiondetection.
TestOnDiskGraphIndex.testVersionRoundTrip: explicit write/read round trips for versions 3, 4, and 5 (single-layer and, for 4/5, multi-layer), in addition to the existing version-2 and current-version (6)coverage.
jvector-testssuite passes (208 tests, 0 failures).