Skip to content

SOLR-18624: Fix flaky ordering assertions in TestMultiWordSynonyms.testPf - #5294

Draft
nick-boss-tech wants to merge 2 commits into
apache:mainfrom
nick-boss-tech:solr-18624-flaky-multiword-synonyms
Draft

nick-boss-tech wants to merge 2 commits into
apache:mainfrom
nick-boss-tech:solr-18624-flaky-multiword-synonyms

Conversation

@nick-boss-tech

@nick-boss-tech nick-boss-tech commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖 (posted on behalf of Nick Shanin)

Draft status is due to Nick Shanin's open-PR cap; the change is complete and ready for review.

https://issues.apache.org/jira/browse/SOLR-18624

What happens today

TestMultiWordSynonyms.testPf is flaky because it expects a fixed order for documents with equal scores. The tropical cyclone foo request expects document 10 first, although all four matches can have the same score. Its two-word phrase boosts are tropical cyclone and cyclone foo. The first matches every document through the synonym bar; the second matches none. Document 10 has no score advantage, so a change in index document order can fail the assertion.

What this change does

The test checks scores and phrase boosts while allowing tied results in any order. The initial requests must show that bf=boost_d changes scores, and the phrase requests must show that a close phrase match can outweigh that boost. For reversed bar foo, slop zero must leave documents 10 and 13 tied, while slop two must make document 10 score higher. The final request checks all four IDs and both expected phrase boosts without requiring document 10 first. Test changes.

Proof

Linux verification at 0af4f5b

Verified November 5, 2026, with JDK 21 and Gradle 9.7.0:

  • TestMultiWordSynonyms.testPf: 25/25 repetitions, seed 18624C0DE.
  • Full TestMultiWordSynonyms: 6/6; neighboring TestExtendedDismaxParser.testPfPs: 1/1, seed 18624BEEF. Both runs had no failures, errors or skips.
  • Tidy, Error Prone compilation and root check -x test passed. The working tree stayed clean. A stale Gradle lock was cleared before the successful tidy run and before any tests ran. The root check used a normal clone at the same SHA for Antora.
  • With sort=score desc,id desc, document 10 won for bar foo, while tropical cyclone foo tied and returned 13,12,11,10.
  • Three deliberately broken versions of the production query parser each compiled and failed the intended test assertion: dropping the reversed bar foo phrase, ignoring bf, and dropping the cyclone foo phrase. The tests were unchanged, and all temporary production changes were reverted.

Verification record.

Related work

I also reviewed the surrounding phrase-query code. Separate PRs are forthcoming for explicit phrase slop and phrases following nested queries (SOLR-12532 and SOLR-15611). I am also preparing the eDisMax empty-phrase cleanup from Ian Springer's work in #4652 (SOLR-18314), keeping it separate from the DisMax ranking change. Linux verification of these follow-ups is pending. I am investigating synonym-dependent pf2/pf3 ranking separately under SOLR-12260. This PR preserves current scoring behavior.

Limits

The supplied Crave seed B3B4534A1A4D09F4 was replayed on the exact historical merge F, and testPf passed 1/1; it also passed 1/1 on F with only the H1 test file overlaid and on H1. The reported failure did not reproduce, and this replay does not establish that the test change fixes it. Replay ran on Linux 6.18.38+ / OpenJDK 21.0.12.1, versus the original Crave host’s Linux 6.8.0-51-generic / Adoptium 21.0.10. Replay receipt. Only one test file changes; production behavior is unchanged.

AI-generated description. Nick Shanin directed the work and takes responsibility for this contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant