test(search): engine parity suite for bleve and opensearch - #3418
Merged
Conversation
Move the OpenSearch test helpers to services/search/internal/opensearchtest so the parity package can import them, and drop the index generation and clock patch the suite was written against, neither exists on main yet.
Cold boots take well over the 5s startup timeout, and a full host disk tripped the flood-stage create-index block mid-run; test indexes are tiny.
KEEP_TEST_CONTAINER=true skips the terminate; the next run picks the container up by name instead of booting a fresh one.
Every case is one spec per engine, so one engine failing leaves the other's answer in the matrix. Query groups share their engines through an Ordered container, operations build their own. The README is assembled from report entries in ReportAfterSuite, which also works across ginkgo -p processes, and is only written once every planned row got an answer from every engine. A case can carry, per engine, what that engine answers today where it still differs from the expectation (engineOverrides): the spec asserts the override, the README marks the row as known, and the override fails once the engine answers as expected.
… suite The expectations are the desired state; where bleve or OpenSearch answer differently on main today the case records that answer as an engine override, so the suite is green and the README lists every divergence as known.
Everything the bleve and OpenSearch backend suites checked now runs against both engines in services/search/pkg/parity. The two cases without a twin there join it (a facet value keeps its case, batches stay apart), the one thing only OpenSearch can do, refuse an unhealthy cluster, stays in its package.
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 74 |
| Duplication | -133 |
🟢 Coverage 0.00% diff coverage · -0.83% coverage variation
Metric Results Coverage variation ✅ -0.83% coverage variation (-1.00%) Diff coverage ✅ 0.00% diff coverage Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (6ceedcf) 84480 19633 23.24% Head commit (fef2df8) 84501 (+21) 18937 (-696) 22.41% (-0.83%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#3418) 25 0 0.00% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Markdown lint has nothing to say about a table the test suite writes.
butonic
approved these changes
Aug 31, 2026
dschmidt
marked this pull request as ready for review
August 31, 2026 08:17
fschade
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Takes the engine parity suite from #3408 (the suite commit is @fschade's, cherry-picked as is) and runs it against main as it is today, without the behavior changes of that PR.
Every case runs against bleve and OpenSearch. The expectations are the desired state from #3408; where an engine answers differently today the case carries an
engineOverridesentry with that answer, so the suite is green and the generatedservices/search/pkg/parity/README.mdlists every divergence as❌ known(73 rows today). An override that stops holding fails, so fixing a divergence means removing its override in the same change: the diff of the overrides is the diff of the behavior.On top of that: the suite is ported to Ginkgo/Gomega (one spec per case and engine, matrix assembled from report entries so
ginkgo -pworks, README only written when every engine answered every row), the OpenSearch testcontainer waits on a health probe and can be kept between runs (KEEP_TEST_CONTAINER=true), fixtures are written in one batch on a 50ms refresh interval (a full run takes ~40s locally withginkgo -p), and the per-engine backend suites are replaced by the parity suite (the four cases without a twin were carried over).No production code changes. Meant as the base for #3408 and #3345, so their behavior changes show up as removed overrides.