Repository navigation
Release DataFusion 47.0.0 (April 2025) #15072
Description
Is your feature request related to a problem or challenge?
Tracking ticket for next release, also a place to track desired inclusions
Previous release will be https://crates.io/crates/datafusion/45.0.0 (likely Feb 1, 2025) December 31, 2024 so next major release would be around March 1, 2025
Steps:
- Update version and changelog: Update version to 47.0.0, add CHANGELOG #15731
- Test with DataFusion Python:
- Test with DataFusion Comet - chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 datafusion-comet#1563
- Test with delta.rs: feat: upgrade to DataFusion 47.0.0 delta-io/delta-rs#3378
- Test with SailHQ: Test DataFusion 47 lakehq/sail#434
- Write upgrade guide: Upgrade guide for DataFusion 47.0.0 #15707
- Test with parquet viewer
- Voting Thread: https://lists.apache.org/thread/zrq9x9gf51r8b6m9qokf2q75kh251rm6
- Create ticket for next release: Release DataFusion
48.0.0(June 2025) #15771
Prior release tickets:
45.0.0: Release DataFusion45.0.0#1400846.0.0: Release DataFusion46.0.0#14123
Changes to add to upgrade guide
These PRs made changes that deserve a mention in the upgrade guide
- chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) #15123
- chore: remove ScalarUDFImpl::return_type_from_exprs #15130
- Fix type coercion for unsigned and signed integers (
Int64vsUInt64, etc) #15341 - Refactor: add
FileGroupstructure forVec<PartitionedFile>#15379 - Add
downcast_to_sourcemethod forDataSourceExec#15416 - Support computing statistics for FileGroup #15432
- chore: cleanup deprecated API since
version <= 40#15027 - Need to handle DisplayFormatType::TreeRender for execution plans.
- map_partial_batch is removed from schema mapper
- page_pruning_predicate is removed from public api -- parquet reader: move pruning predicate creation from ParquetSource to ParquetOpener #15561
Features to mention in the blog (if they make it)
- [EPIC] Complete
SQL EXPLAINTree Rendering #14914 - [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
- Change mapping of SQL
VARCHARfromUtf8toUtf8View#15096 - feat: introduce
JoinSetTracertrait for tracing context propagation in spawned tasks #14547 - Improve performance of
first_valueby implementing specialGroupsAccumulator#15266
Bugs that would be good to fix
- Regression: eager evaluation of expressions inside CASE conditional expression #15384
-
panicwhen evaluating trivial WHERE with a CTE #15386 - Fix: after repartitioning, the
PartitionedFileandFileGroupstatistics should be inexact/recomputed #15539 - Regression in
last_valuefunctionality #15676 - Remove waits from blocking threads reading spill files. #15654
Community Wishlist
- Comparison Operators for Decimals of Different Precisions and Scales #15174
- Format
Date32to string given timestamp specifiers #15361 - Feature: support cast
datetotimestampwith tz #14638 - fix: update group by columns for merge phase after spill #15531
- Add
statistics_by_partitionAPI toExecutionPlan#15495 - Add coerce int96 option for Parquet to support different TimeUnits, test int96_from_spark.parquet from parquet-testing #15537
Activity
xudong963 commentedon Mar 11, 2025
@alamb, I'll also be in charge of this release.
alamb commentedon Mar 11, 2025
@XiangpengHao also offered to test with the parquet viewer prior to 47: #15102 (comment)
xudong963 commentedon Mar 12, 2025
That's great, added it to release steps
shehabgamin commentedon Mar 19, 2025
I feel like this may be important enough to try to get into the release. Does anyone else have thoughts?
alamb commentedon Mar 19, 2025
Seems reasonable to me -- I have added it to the "good to get in " list
alamb commentedon Mar 19, 2025
I believe the voting deadline has passed and I will now will prmote / publish it.Update: wrong ticket (I was looking for 46.0.0
xudong963 commentedon Mar 26, 2025
The PR #15266 has significantly improved performance, so I added it to the blog section.
xudong963 commentedon Mar 28, 2025
@alamb I think we can start testing the 47.0.0 in the second week of April and begin the release process at the end of that week. What do you think?
alamb commentedon Mar 28, 2025
I think it sounds like a great idea -- thank you @xudong963
For your planning purposes I will be away the week of April 21 -- so perhaps we can start testing a week earlier (week of April 7 so we have time to complete / fix issues prior to April 14)
shehabgamin commentedon Mar 28, 2025
Happy to test whenever!
xudong963 commentedon Mar 29, 2025
That sounds good!
rluvaton commentedon Apr 2, 2025
Would really appreciate if could add the following PR to the release as well:
36 remaining items
andygrove commentedon Apr 14, 2025
@alamb I am hoping that we can merge #15537 for this release. It was just rebased now that the arrow-rs upgrade is merged.
gabotechs commentedon Apr 15, 2025
Thanks for putting this together! If we could additionally get #14412 in, that would be awesome 🙏
xudong963 commentedon Apr 15, 2025
@alamb It seems there are still one or two PRs that want to be included, so how about making release notes tomorrow(UTC+8, after I get up)
alamb commentedon Apr 15, 2025
Sorry @gabotechs -- I just merged that one!
SOunds like a good plan. I'll take a pass through the outstanding PRs again to see if there is anything else we can/should merge
Thank you all
alamb commentedon Apr 16, 2025
I just merged the version + changelog PR from @xudong963
I also created a
branch-47here for the release:@xudong963, given I think it is late in your timezone, I'll plan to make an RC in a few hours unless you let me know otherwise.
alamb commentedon Apr 16, 2025
I have made a release candidate and started a voting thread: https://lists.apache.org/thread/zrq9x9gf51r8b6m9qokf2q75kh251rm6
alamb commentedon Apr 16, 2025
Note that I will be away starting April 18, and so likely can not complete the vote / release process until April 26. @andygrove would it be possible for you to complete the voting process / publish the release to crates.io for this release?
andygrove commentedon Apr 16, 2025
Yes, I would be happy to do that. I will be offline on Saturday, but can take care of it on Sunday or Monday.
alamb commentedon Apr 16, 2025
Awesome -- we have a plan! I hope to work on the upgrade guide and maybe even a blog post about this release, but we'll see if I have the time
alamb commentedon Apr 17, 2025
Here is a draft upgrade guide:
alamb commentedon Apr 19, 2025
I filed the following ticket for the next release:
48.0.0(June 2025) #15771alamb commentedon Apr 28, 2025
DataFusion 47 is on crates.io https://crates.io/crates/datafusion/47.0.0
So closing this one down
Upgrade to datafusion 47 (#3016)
Bump arrow to 55 (#1038)