[SYSTEMDS-3951] MM Chain Optimization w/ Basic Average Estimator

[ywcb00]

Hi,
this PR changes the estimator for sparse mm chain rewrites from MNC to the metadata average estimator. Integration of MNC for these rewrites is missing the information about the underlying matrix histograms.
This PR also fixes the corresponding test, which previously only validated if there was any mm chain optimization.
Thank you and all the best,
David

[ywcb00]

Thank you very much @Baunsgaard. This was exactly what I was searching for and I changed the test validation accordingly. :)

Does anybody know why the builtin.part2 java test resulted in a timeout and is this behavior related to the current PR? Test output states: Our tests should run faster than 1000 sec each

If this is not related to the PR, I would like to merge the changes soon.

All the best,
David

[Baunsgaard]

Thank you very much @Baunsgaard. This was exactly what I was searching for and I changed the test validation accordingly. :)

Does anybody know why the builtin.part2 java test resulted in a timeout and is this behavior related to the current PR? Test output states: Our tests should run faster than 1000 sec each

If this is not related to the PR, I would like to merge the changes soon.

All the best, David

1. You are welcome.

2. The part2 seems like it got stuck with a threading error again, my hypothesis is Spark was closed by one thread, and another is waiting for response. **It is not related to your changes **. However, do you want to take a look at that suite to check if there is anywhere we close a spark session in an unsafe manner?

[Baunsgaard]

few Nits, otherwise LGTM

[codecov]

Codecov Report

❌ Patch coverage is 93.10345% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.61%. Comparing base (d6aabf6) to head (6aa1556).

Files with missing lines	Patch %	Lines
...rite/RewriteMatrixMultChainOptimizationSparse.java	90.00%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2525      +/-   ##
============================================
+ Coverage     71.54%   71.61%   +0.06%     
- Complexity    50240    50270      +30     
============================================
  Files          1623     1623              
  Lines        194317   194314       -3     
  Branches      37965    37965              
============================================
+ Hits         139032   139149     +117     
+ Misses        44366    44264     -102     
+ Partials      10919    10901      -18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.