refactor: reflection-based search mapping + location geopoint

[dschmidt]

Builds on #3408 (the engine-parity fixes, merged): its pinned semantics are kept, the parity suite stays fully green. On top of its interim hand-written v3, the schema generation is unified into one constant and bumped to v4 (see services/search/MIGRATION.md).

Supersedes #2659. Moved to an upstream (opencloud-eu) branch and rebased cleanly onto latest main so the dependent search PRs can stack on top of it. The non-bot discussion from #2659 is carried over in the comments below (quoted, since it can't be reposted under the original authors).

Summary

Merges the bleve and OpenSearch index mappings into one reflection-based package (services/search/pkg/mapping) driven by the search.Resource struct + a small overrides map. Pulls services/graph's duplicated reflection walker onto the same helpers and, as a showcase of what the refactor enables, turns on location_geopoint indexing for spatial queries on both backends.

Adding a new facet (motionPhoto, ...) is now roughly: one struct field on content.Document, one line in service.go, one line in graph, one line per backend hit converter, plus whatever tika extraction logic the facet actually needs. Everything else falls out of reflection.

Behavior changes (deliberate)

A new schema generation works on a fresh index (opencloud-resource-v4, bleve-v4) and leaves the old one untouched; services/search/MIGRATION.md describes how to fill it.

• Favorites matches case-sensitively, like the other ids: the values are opaque user ids, so they carry no _lowercase sibling.

• An OpenSearch Purge is scoped to the resource's space; it matched the bare path before and could take same-path documents in other spaces with it (bleve was already scoped). Pinned as parity rootscope-04.

• OpenSearch Tags / Favorites: dynamic keyword → explicit keyword (unified with bleve), searched case-insensitively via the _lowercase siblings described below. No analyzer.

• OpenSearch facet sub-strings (audio.*, photo.*, image.*): dynamic text + keyword multi-field → explicit keyword plus the _lowercase sibling every keyword field gets by default (see below), so audio.artist:"iron maiden" finds "Iron Maiden" on both backends while aggregations keep case-preserving buckets. The tokenized path was never reachable from KQL anyway (no dot-syntax + pre-fix(search): preserve value case for non-lowercased bleve fields #2633 lowercasing), so no working query regresses.

• location: the libregraph {longitude, latitude, altitude} object is preserved at the location key on both backends (numeric sub-field queries like location.latitude:>49 keep working). A sibling location_geopoint is added for geo-distance / bounding-box / polygon queries.

• path: queries are case-sensitive on both backends; Path gets no _lowercase sibling. Paths act as references (location scoping, deep links) where /Foo and /foo are distinct siblings, and case-insensitive folder discovery is served by name:. This matches bleve on main, which always matched paths case-sensitively; only OpenSearch loses its case-insensitive path matching. Dropping the sibling also removes its costliest maintenance: Path is the one mutable sibling field, so the OpenSearch move script no longer has to rebuild a lowercased copy for a whole subtree. With paths case-sensitive, the ref path scope is applied at query level on both backends (term/prefix on bleve's keyword Path, term filter on OpenSearch's path_hierarchy tokens), replacing the old post-filter: totals and paging now respect the scope instead of being computed over the whole space, and a wrong-cased scope matches nothing.

• graph facet parsing: fail-soft per field. A malformed value drops only that field; the rest of the facet still populates.

• Audio facet is now shown whenever libre.graph.audio.* metadata is present: the read-side audio/ guard is dropped from all three readers (bleve, OpenSearch, graph), so the facet follows the metadata rather than re-checking the MIME type. Extraction still only produces audio metadata for audio/* files.

• Keyword fields are searchable by word on both backends, the way KQL matches text properties: a single word finds the values that contain it (report and name:report find Report.txt, Title:quarterly finds "quarterly report", audio.artist:iron finds "Iron Maiden"), which OpenSearch on main happened to do through its dynamic mapping and bleve never could. Modelled on SharePoint's NoWordBreaker with its default: every keyword field gets a search-only _words sibling (lowercased words, a dot is a word boundary, no stemming) next to _lowercase, unless it opts out because its values are labels or identifiers: Tags, Favorites (a tag is one label, tag:foo does not find foo-bar), ID/RootID/ParentID, MimeType. The base stays the whole value for returning and aggregating, wildcards and whole values keep using _lowercase, and quotes do not change the meaning (a phrase is a phrase either way). KQL's = (grammar from fix(search): make openSearch and bleve behave the same #3408) matches the whole value: a case-insensitive term on _lowercase. Web is unaffected either way, it always searches wildcarded (name:"*term*", see web useSearch.ts).

• Mtime is typed as a date on both backends, so mtime:>... ranges are chronological (was a keyword field / lexicographic compare on OpenSearch). Note: bleve has no sub-second date field, so a returned/re-indexed Mtime is second-precision (range queries stay exact). Previously the RFC3339Nano keyword round-tripped exactly.

• Resource.Hidden now survives Move/Delete/Restore on bleve. The old hand-rolled deserializer never read Hidden, so those ops silently reset it to false; the reflection deserializer reads every field, preserving it. Latent-bug fix.

• Case-insensitive search via per-field _lowercase siblings, on by default. KQL searches case-insensitively, so every keyword field indexes its case-preserved base (returned to clients, aggregated on, used for exact ops like the move/delete cascade) plus a lowercased sibling used only for matching, unless it opts out: ids are opaque (ID, RootID, ParentID), paths are POSIX (Path), the mime type is normalized already (MimeType). Which siblings a field carries is decided once, in mapping.SearchSiblings, and the mapping renderers, the document writer and the query lowering all read it from there. Queries route to the sibling and lowercase the value with the same Go strings.ToLower used at index time, so index and query stay consistent without an analyzer. The sibling is never read back, so it need not be stored: in bleve it is not stored, out of _all, no doc values; in OpenSearch it deliberately stays in _source, because excluding it would force every update-by-query script (move/delete/restore) to rebuild all siblings from the document via painless toLowerCase, which lowercases differently than Go and would drift from the query side. A lowercased copy of a name/path is negligible disk in a cluster. The OpenSearch Move script keeps base and sibling in sync via Go-lowercased params, so case-insensitive search still finds a file after it moves (the sibling used to go stale); bleve re-indexes whole documents on move/delete/restore and stays fresh for free. Also fixes a latent main bug: the Move/Delete cascade matched Path exactly against the lowercased index, so re-homing a mixed-case folder (e.g. /Photos) skipped its descendants; Path is now case-preserved. bleve path queries additionally match a folder and its descendants like OpenSearch's path_hierarchy.

• OpenSearch full-text (content:) search now analyzes the query. Single-term queries used an unanalyzed term query, so once this refactor dropped the old blanket query-value lowercasing, content:Foo (any uppercase) missed on OpenSearch (a regression introduced here; bleve was unaffected because its query analyzes). Fielded full-text queries now use an analyzed match query. Content is analyzed by the same words analyzer on both backends, without stemming, adopting the parity-pinned semantics from fix(search): make openSearch and bleve behave the same #3408 (bleve loses its porter stemmer): content:running and content:RUNNING match, content:run does not.

• To decide - content wildcards (content:foo*) are unanalyzed on both backends, so they match the stemmed, lowercased term dictionary literally: content:run* finds a document containing Running (indexed term run), but content:running* (past the stem) and content:Run* (uppercase) do not. This is inherent to a wildcard over an analyzed field and is now consistent across bleve and OpenSearch (previously OpenSearch degraded content:foo* to an exact match). Whether content wildcards should additionally be case-folded is left open.

• mediatype search is now case-insensitive on both backends: categories (mediatype:Folder, mediatype:IMAGE, ...) and literal MIME types are lowercased in the KQL lowering. Related behavior change: the raw MimeType field no longer expands category words. On main MimeType:folder / MimeType:file expanded to the folder / non-folder MIME set (a quirk of field-based expansion); now mediatype: is the way to query categories and MimeType: only matches a literal MIME type.

Upgrade note: the index name carries search.SchemaVersion, bumped to 3 here (OpenSearch <name>-v3, bleve bleve-v3). An upgraded instance starts on a fresh, empty index and leaves the old one in place; opencloud search index --all-spaces fills it, then the old index can be removed. #3197 adds the startup schema check on top.

Follow-ups (not in this PR)

• NOT a OR b on OpenSearch: per [MS-KQL] (2.1.13, NOT has highest precedence) this is (NOT a) OR b, but a flat OpenSearch bool query cannot OR-combine a must_not clause, so it collapses to NOT a AND b. bleve represents it correctly (a must_not sub-query can be a disjunct). Rare (a NOT directly followed by OR) and a flat-bool limitation, not the NOT binding, which is now spec-correct on both backends.

• Media-type category + operator precedence: X OR mediatype:<multi-type-category> AND Y (document/spreadsheet/presentation/archive) constrains only the last MIME type of the category, because the bleve compiler's mapBinary redistributes a left disjunction as an OR-chain. Pre-existing (main had the same), a pure query-compiler issue.

• []struct round-trip: a json-tagged slice-of-struct field is mapped as a nested object but not read back (fillStruct/casing/geo do not descend into slices), and its _lowercase/_geopoint siblings would never be written. Latent: no current facet is a []struct (facets are *struct, tags are []string); harden before adding one.

• bleve _all: unused (every query is fielded: the resolver always resolves a field, the compiler always emits field:value, and a bare term resolves to Name → Name_lowercase). Disabling it shrinks the bleve index and turns the per-field IncludeInAll handling into dead code to remove.

• KQL typed-value parity (booleans/numerics): Hidden:T/Size:>1000 are bleve query-string leakage: they parse as plain string restrictions and only work because the bleve compiler concatenates values into bleve's own query-string syntax; on OpenSearch the same literals hit typed fields and 400. The KQL-canonical hidden:true compiles to a query-string Hidden:true on bleve and silently never matches (bleve indexes booleans as the term "T"); it should compile to a typed bool-field query. Canonical numeric ranges (Size>1000) don't parse at all, the grammar has range operators only for datetime values. Consolidate in the query layer: typed BooleanNode compile on bleve, grammar-level numeric ranges, then decide whether the query-string extras stay.

• ? single-char wildcard parity: bleve treats ? as a wildcard, OpenSearch does not (the wildcard check only looks for *), so name:Fo? diverges. Pre-existing, not introduced here.

• Field-name notation: KQL vs graph sortProperties: the queryString exposes internal index field names (Name:foo*, Size:>1000, Tags:bar, Mtime:...), while the graph search endpoint's sortProperties deliberately accepts the property names a client sees on the hit resource (name, size, lastModifiedDateTime, mimeType, photo.takenDateTime, ...) and translates them to index fields via an alias table in pkg/search. Align the two surfaces: accept graph-notation field names in KQL (on top of that alias table) and decide whether the internal index names remain part of the public query surface.

[dschmidt]

Discussion carried over from #2659

Reposting the non-bot discussion from the original PR so it isn't lost with the move to this upstream branch. Quoted verbatim with attribution; the resolved bot review nits (codacy / copilot: embedded-pointer unwrap, IsNil on non-nilable kinds, a few doc/typo fixes) were all addressed there and aren't repeated here.

Why reflection? (design rationale)

@dragonchaser:

Why the extensive usage of reflections?

@dschmidt answered (click to expand)

Short version:
It's there so the search.Resource struct (plus a small overrides map) becomes the single source of truth for everything schema-shaped: the bleve DocumentMapping, the OpenSearch properties JSON, and the hit→struct deserialization on read. One struct, two backends, no parallel schemas to keep in sync.

Keeping all the mappings in sync for the existing facets and having to do a lot of copy paste for adding new facets, is tedious and error prone. The new code is not completely trivial to read, I agree, but it's basically write once and probably/hopefully never touch again. Basically it pulls together different pieces of reflection usage in a central location and concentrates it in the mapping package, so actually there's less reflection stuff spread around the code base.

Long version

1. No drift between bleve and OpenSearch. Right now there are two hand-maintained mapping definitions plus a third hand-maintained "read fields back out of hit.Fields" path. They have already drifted. With reflection both backends walk the same fields with the same json-tag names via walkFields (infer.go), so they can't disagree by accident.
2. Kills a duplicated walker. services/graph had its own reflection walker doing essentially the same thing; this PR collapses it onto the shared helpers in mapping/infer.go.
3. Adding a facet is now ~5 lines. One field on content.Document, one line in service.go, one in graph, one per backend hit converter. Deserialize[T] handles it from the json tags.
4. json tags are already the contract. Field names on the wire are already driven by struct tags, so reusing them via reflection for index field names just removes a second, hand-typed copy of the same names.
5. Cost is bounded. Reflection runs once at index-mapping build time (startup) and once per hit on read. Not on a hot inner loop, not in the query path.

The alternative would be either codegen (more machinery for the same outcome) or keeping the three parallel hand-written schemas (which is what already doesn't work right now and what motivated the refactor).

@dschmidt on the diff size:

By the way looking at the line counts is a bit misleading. It's not 1.8k lines plus for a simple refactor. [...] a lot of the new lines are also for tests we simply didn't have before and it also adds the geopoint feature (we can of course discuss to split it out of this PR if you prefer, it's basically a demonstrator for the concept).

Related: #2715

Review feedback

@fschade:

Okay, the PR is huge, but I can understand that it's hard to logically split up such a massive topic. The only thing bothering me right now is the schema update issue; I can't think of a clean way to handle that independently of the deployment [...] I think we just need to document clearly what needs to be done for schema updates (and we'd face that problem regardless of this PR). From my side: thumbs up.

Follow-up documentation issue opened by @fschade: #3092

@aduffeck (workflow):

it would be great if you would just add commits instead of force-pushing refinements, that makes it easier to figure out what changed after starting the review.

@dschmidt:

Yes, will do from now on [...] Will only add commits on top from now on!

Note for this new PR: it is a clean rebase of the same commits onto latest main (the earlier one-shot history rewrite was agreed with the maintainers). From here it's add-commits-on-top again.

Open item: breaking index change / re-indexing (unresolved)

@fschade:

Since the PR contains a breaking change, it's just taking a bit longer than it should! [...] we can only ship the PR as a major version because the index is change breaking! Our idea is that in the case of breaking index changes [...] we can create a new, fresh index and have the operator re-index the data. Downside: the search returns no results for some time, so we need beforehand: the operator must be able to send a message to the users (sticky, web-ui); and a function that allows us to automatically generate a fresh index in the case of breaking index changes.

@dschmidt:

I've worked on parts of that already, let's have a call about everything asap :)

@aduffeck (on services/search/pkg/opensearch/index.go):

Requiring another re-indexing of the whole tree right after the stable 7.2 release is unfortunate [...] But maybe this is the time to spend some time on making index upgrades less intrusive. It should be possible to add a read alias on the v2 index while building a v3 index and then flip the alias over [...]. At the very least we should use "v3" as the index though [...] so that the new index could be built up in a pre-deployment hook in kubernetes, for example.

@dschmidt:

True, but requiring it right before the release would have been risky on the other hand - there's never a perfect timing for this kind of change :) We'll discuss soft upgrade options with @fschade next week :)

Resolved: ginkgo test conversion

@aduffeck (on geo_verify_test.go):

Could you change the tests to ginkgo/gomega tests? That's (largely) what we settled on.

@dschmidt: Converted geo_verify + mtime and the new mapping package tests to ginkgo/gomega. (This carried into the rebase: the opensearch backend suite is now ginkgo too.)

Open item: Mtime typed as a date (unresolved)

@dschmidt (on services/search/pkg/content/content.go):

Mtime was the only Go string mapped as a date. Basic.Extract leaves it unset when the resource info carries no mtime [...] which serialised to "Mtime": "" -- and an empty string is not a date, so OpenSearch rejected the whole document. The bulk API reports that per item, so Batch.Push returned nil and the file silently vanished from the index (that swallowing is #3142). Typing it as *time.Time fixes the cause rather than the symptom [...] Reproduced against a real cluster: before this, upsert without mtime indexed 1 of 2 documents.

An earlier symptom of the same typing (@aduffeck spotted the fixture choking the unit tests with a mapper_parsing_exception on Mtime) was fixed by switching the fixture to RFC3339.

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 17 complexity · 15 duplication

Metric	Results
Complexity	17
Duplication	15

View in Codacy

🟢 Coverage 74.90% diff coverage · +0.54% coverage variation

Metric	Results
Coverage variation	✅ +0.54% coverage variation (-1.00%)
Diff coverage	✅ 74.90% diff coverage

View coverage diff in Codacy

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (0becbb8)	84824	19295	22.75%
Head commit (1bf8130)	85264 (+440)	19853 (+558)	23.28% (+0.54%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#3345)	1203	901	74.90%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[aduffeck]

Needs a rebase but the code looks good to me. I also tested with both opensearch and bleve and didn't find any issue (except for the favorite flag issue which was fixed in #3252 and thus should be fixed here after a rebase as well.