perf: optimize spark_arrays_overlap (up to 18x faster)

[andygrove]

Which issue does this PR close?

N/A

Rationale for this change

Optimize existing expression.

What changes are included in this PR?

Replaced the per-probe-element Arrow eq kernel call (which allocated a BooleanArray per element) with typed scans over the flat values buffers plus a hash probe for large lists, cutting allocations and kernel dispatches to zero on flat element types.

How are these changes tested?

Existing tests.

Benchmark (criterion):

• spark_arrays_overlap_ int32 medium lists: 92.925% faster (base 11690342ns -> cand 827066ns)
• spark_arrays_overlap_ utf8 short lists: 77.473% faster (base 3281623ns -> cand 739251ns)
• spark_arrays_overlap_ utf8 long lists: 94.733% faster (base 11544605ns -> cand 608100ns)
• spark_arrays_overlap_ int32 long lists: 94.947% faster (base 12375951ns -> cand 625320ns)
• spark_arrays_overlap_ int32 short lists: 94.474% faster (base 3018371ns -> cand 166793ns)

Full criterion output:

spark_arrays_overlap: int32 short lists
                        time:   [166.19 µs 166.36 µs 166.55 µs]
                        change: [−94.489% −94.474% −94.460%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

spark_arrays_overlap: int32 medium lists
                        time:   [823.79 µs 825.75 µs 827.78 µs]
                        change: [−92.947% −92.925% −92.903%] (p = 0.00 < 0.05)
                        Performance has improved.

spark_arrays_overlap: int32 long lists
                        time:   [623.94 µs 625.77 µs 627.46 µs]
                        change: [−94.965% −94.947% −94.930%] (p = 0.00 < 0.05)
                        Performance has improved.

spark_arrays_overlap: utf8 short lists
                        time:   [728.82 µs 731.53 µs 734.73 µs]
                        change: [−77.607% −77.473% −77.335%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

spark_arrays_overlap: utf8 long lists
                        time:   [608.52 µs 609.75 µs 610.95 µs]
                        change: [−94.755% −94.733% −94.712%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) low mild
  1 (1.00%) high mild

[comphead]

Thnaks @andygrove we need to donate those optimizations to datafusion-spark one day