Skip to content

feat(search): graph search query endpoint with aggregations - #3211

Draft
dschmidt wants to merge 7 commits into
mainfrom
feat/graph-search-query
Draft

feat(search): graph search query endpoint with aggregations#3211
dschmidt wants to merge 7 commits into
mainfrom
feat/graph-search-query

Conversation

@dschmidt

@dschmidt dschmidt commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

This contains a huge libre-graph-api-go bump - so please don't be alarmed about the size of the diff

MS-Graph-style POST /graph/v1beta1/search/query with terms, range, metric and nested sub-aggregations, on both the bleve and OpenSearch backends.

Stacked on #3198.

Caveats

  • Vendored libre-graph-api-go is interim, generated from feat: add search query endpoint with request examples libre-graph-api#34 rebased onto main. Proper path: land that spec PR upstream (feat: add search query endpoint with request examples libre-graph-api#34) and pin a released version.
  • Aggregation behavior is pinned in the engine parity suite (AGG-01..09: term buckets, numeric and date ranges, open-ended bounds, metrics, malformed-bound errors), running against bleve and a real OpenSearch. That round also caught and fixed an OpenSearch bug: date range bounds were silently dropped, they now emit a date_range aggregation and malformed bounds are rejected on both engines. Nested sub-aggregations were smoke-tested end-to-end on bleve (artist -> album -> sum(duration)).

Note

Even once this PR and #3210 (honest has-preview + thumbnails relationship) are both merged, search hits still won't carry thumbnails — the $expand=thumbnails wiring for /search/query hits builds on both PRs and currently only exists on the consolidation branch (dschmidt/opencloud@feat/graph-search-full-2, commit 0d94199a: searchEntityThumbnailSet in searchquery.go, preview presence via thumbnail.HasPreviewForMimeType + indexed oc.preview/image-facet dimensions). It needs a follow-up PR once both are in.

@codacy-production

codacy-production Bot commented Jul 30, 2026

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 446 complexity

Metric Results
Complexity 446

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@dschmidt

dschmidt commented Aug 16, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: narrow Fields when scanning hits for metrics / sub-aggregations

Not blocking, results are correct. It's the cost profile.

needsSubAggScan widens the page to subAggScanSize (math.MaxInt) for any
aggregation with sub-aggregations or a metric, since neither has a native facet
representation in bleve. Together with bleveReq.Fields = []string{"*"}, a
mediatype:image request carrying one Size/sum metric materialises the whole
matching hit set with every stored field, to add up one number. OpenSearch nests
natively and returns no documents.

Easy to reach from a client: a dashboard asking for a few library metrics over
mediatype:image pulls the whole image library into memory per load. Flat term
and range facets are unaffected.

Options by payoff:

  1. Split scan from page. The scan needs only the metric and sub-agg fields; the
    rich Fields set only serves the PageSize window. One request cannot carry
    two field sets.
  2. Otherwise narrow Fields to the union of what hydration and aggregations
    read, instead of "*".
  3. Bound the scan, mark results approximate.
  4. For real library counters (total bytes of all images), maintain a value at
    index time instead.

@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from 1b61ef6 to fb1e41b Compare August 18, 2026 16:51
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch 4 times, most recently from c54ffe7 to cc3e6f4 Compare August 31, 2026 18:37
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from cc3e6f4 to af951f6 Compare September 1, 2026 05:48
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from 08eddc9 to ceb7df5 Compare September 1, 2026 07:11
@dschmidt
dschmidt changed the base branch from feat/search-tika-facets to main September 1, 2026 08:34
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch 3 times, most recently from 2513263 to 71ca404 Compare September 3, 2026 08:33
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from fef1116 to 7f7ecbf Compare September 11, 2026 14:30
Generated from opencloud-eu/libre-graph-api#34 rebased onto main: POST /search/query with hits, aggregations and metrics.
MS-Graph-style search query endpoint: hits from all accessible spaces, with from/size paging, remote items for hits from shared spaces, thumbnails on $expand, and the effective permission actions on every hit. The search proto gains the aggregation types of the graph API and the endpoint passes aggregations through to the search service, which does not evaluate them yet.
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch 2 times, most recently from beae60b to 654fba0 Compare September 12, 2026 15:38
@dschmidt

dschmidt commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up: narrow Fields when scanning hits for metrics / sub-aggregations

Outdated: metrics and sub-aggregations are folded from doc values by aggCollector now (bleve/aggregations.go), hooked into the document-match handler. No page widening, no stored fields loaded for the scan; Fields: "*" only serves the page window.

@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from 654fba0 to b2296c5 Compare September 12, 2026 16:23
Term buckets on both engines: bleve facets, OpenSearch terms aggregations, buckets merged across spaces and post-processed per BucketDefinition (minimum count, sort, size). Terms aggregations on numeric fields are rejected at the graph endpoint. Pinned in the parity suite as AGG-01 to AGG-03. Ranges, metrics and sub-aggregations are declined until the engines evaluate them.
Numeric and date ranges on both engines: bleve numeric and date-time range facets, OpenSearch range and date_range aggregations, a bound that is neither number nor date is rejected. A terms aggregation on a numeric field now points to ranges as the alternative. AGG-04, 05 and 07 to 09 in the parity suite.
Sum, min, max and avg on both engines. bleve facets cannot compute, so a collector hooked into the document-match handler folds the metric from doc values, one pass over every match, before the top-n cut. Avg travels as sum and count so the service can reduce it across spaces. AGG-06 in the parity suite.
Sub-aggregations on both engines. bleve folds child buckets from doc values through the same collector as the metrics, terms and ranges alike; the service unions nested buckets across spaces. AGG-10 to AGG-16 in the parity suite.
Every bucket carries an opaque aggregationFilterToken; a search request can pass tokens back as aggregationFilters. The service decodes them into KQL fragments once, the engines AND them into the query as exact, case-sensitive matches.
@dschmidt
dschmidt force-pushed the feat/graph-search-query branch from b2296c5 to be68afb Compare September 12, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant