Skip to content

SOLR-18622: Upgrade Lucene to 10.6.0 - #5293

Open
HoustonPutman wants to merge 4 commits into
apache:mainfrom
HoustonPutman:SOLR-18622-lucene-10.6
Open

HoustonPutman wants to merge 4 commits into
apache:mainfrom
HoustonPutman:SOLR-18622-lucene-10.6

Conversation

@HoustonPutman

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-18622

Only substantive change is inChildFieldValueSourceParser, which just utilizes the new Lucene inputs.

@nick-boss-tech

Copy link
Copy Markdown
Contributor

🤖 AI text below 🤖 (reviewed by Nick Shanin)

@HoustonPutman I have a Solr regression-test branch and a separate Lucene repair for two missing-value cases exposed by this PR:

  • With numeric sortMissingFirst, a sibling excluded by the child filter affects parent ordering as though it were a matching child with a missing value.
  • String sortMissingFirst with descending sorting throws during collection because a missing-value marker is treated as a real term ordinal in lookupOrd.

This PR starts passing the schema's missing-value setting to Lucene as childMissingValue, exposing these problems in ToParentDocValues. The Solr tests preserve your choice to keep the default parent missing-value setting.

A comparison using the same test patch with Lucene 10.4 passes the filtered-child test, confirming a regression after the upgrade. The string missing-first test returns the wrong order with 10.4 and throws with 10.6. The numeric missing-value test improves from failing on 10.4 to passing on 10.6; string missing-last passes on both.

The Solr test diff adds four tests using the existing test setup. On Linux/JDK 21, seed 18622C0DE, your original suite passes 11/11; two added tests pass and the two cases above fail. The branch still reproduces those failures and needs one formatting-only correction. Broader Solr checks were not run. To take the exact tested commit:

git switch SOLR-18622-lucene-10.6
git fetch https://github.com/nick-boss-tech/solr.git pr-5293-solr-18622-child-sort-missing
git merge --ff-only 769ca6e177d77acd3f1993a4a24bd3257ae243c0

The Lucene repair diff checks only matching children when deciding whether a value is missing. When a missing string wins selection, it reports no value so the comparator can apply the parent's missing-value setting. It changes one production file and two existing test files, including tests for paging and multi-valued numeric fields.

At that exact Lucene commit, on Windows/JDK 25 with seed 18622C0DE, both focused suites pass (10 tests); the full join module passes 104 tests with 3 existing nightly tests skipped. Four regression tests each fail at runtime with the original production source and pass with the repair. :lucene:join:tidy and root check -x test pass. Runtime tests for the rest of the repository were not run; Error Prone was disabled by the default local configuration and eclint was not configured.

In a separate Lucene checkout, the exact repair is available with:

git fetch https://github.com/nick-boss-tech/lucene.git fix-block-join-missing-values
git switch -c review-block-join-missing-values 55c3174bd154b5cc987d9b24e1c77043e23fe71e

This repair targets Lucene main. No Lucene PR has been opened, and a 10.6 maintenance backport or Solr run with patched Lucene has not been verified. Would you like to take the Lucene repair upstream and keep the Solr tests here as integration coverage once the dependency is repaired?

@HoustonPutman

Copy link
Copy Markdown
Contributor Author

@nick-boss-tech I have to say that is a very complicated issue, and not easy for me to wrap my head around. If you want to make a Lucene PR, following the project's AI Policy, I'll help out there. It's too much to do here, and not really the right place. I'll hold off on merging this for now, but we can assume there will be a delay until we release 10.2, so there's likely not a ton of harm upgrading right now. There are definitely other bugs that have been fixed in the meantime with 10.6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants