Repository navigation
Release DataFusion 56.0.0 (Oct/Nov 2026) #24461
Description
Is your feature request related to a problem or challenge?
Previous release is #22393 (Jul / Aug 2026) so next major release would be approximately Sep / Oct 2026.
Steps:
- Create a
branch-56branch: https://github.com/apache/datafusion/tree/branch-56 - Update version and changelog: [branch-56] Prepare for 56.0.0 release: Version and Changelog #27098
- Test with DataFusion Python - Upgrade DataFusion to version 56 datafusion-python#1826
- Test with DataFusion Comet - deps: upgrade to DataFusion 56.0.0 datafusion-comet#7249
- Test with DataFusion Ballista
- Test with DataFusion Federated - Update DataFusion to 56.0.0 and Arrow to 60.1 datafusion-contrib/datafusion-federation#191 (the build passes locally)
- Test with DataFusion Table Providers - Update DataFusion to 56.0 and Arrow to 60.1 datafusion-contrib/datafusion-table-providers#773 (the build passes locally)
- Test with delta.rs
- Test vortex - feat: update to DataFusion 56 upgrade vortex-data/vortex#10309
- Test with iceberg-rust
- Test with LakeSail
- Test with parquet viewer
- Test with datafusion-materialized-views: Upgrade to DataFusion branch-56 datafusion-contrib/datafusion-materialized-views#124
- Voting Thread
- Publish to crates.io
- Create ticket for next release: Release DataFusion
56.1.0(minor/patch) Release (Nov 2026) #24463 / Release DataFusion57.0.0(Dec 2026 / Jan 2027) #25901 - Create blog post: Blog post for the DataFusion
56.0.0release #27201
Features to mention in the blog (if they make it)
-
ASOFjoins from @Xuanwo : ASOF join support / Specialize Range Joins #318 - Push down topk through join #21621
- Add write support to Avro format #24925 from @pepijnve
- Add support for SQL based registration of external catalogs #25111 from @pepijnve
- Versioned documentation #17071
- Enable TopK per Partition (
datafusion.optimizer.enable_window_topn) by default #24608 - [EPIC] A Collection of Wrong Results Bugs - Q4 2026 #26672 from @alamb
- Add support for reading/writing ALP data
Bugs/issues that should be fixed
Before branch cut
- Side effects from
serde_json/preserve_orderfeature impact user applications #25883 @adriangb - [dbolympics] fix: cast outer projection expr to declared field type in SingleDistinctToGroupBy #26211 from @jayzhan211
- Rename
LogicalPlan::DescribeTabletoLogicalPlan::Showmove statement implementations out ofsqlmodule #26057 from @pepijnve (to reduce API churn) - refactor: register proto decoders with
T::register_protoinstead of a free function #26565 from @adriangb (so it is not a breaking change later) -
DISTINCT ONwith nested payloads can fail memory reservation after spilling #26744 from @shehabgamin - Regression:
ORDER BY col, (scalar subquery) LIMITover aRIGHT JOINfails inSanityCheckPlan#26572
After branch cut (requires backport to branch-56)
- TPC-H Q18 regression #27108 from @devanbenz (regression; revert feat: estimate Decimal filter selectivity with interval analysis #25790); Backport PR: [branch-56] Revert #25790 (Decimal filter selectivity) due to TPC-H Q18 regression (#27145) #27147
- Reusing a
limit=1FTS plan concurrently can return an empty result #27130 from @alamb (regression, reported by @Xuanwo); Backport PR: [branch-56] fix: give each SortExec reset_state copy its own TopK dynamic filter (backport #27142) #27174 (@raghavyadav01) - REGRESSION: Multi-key null-aware join silently defaults to one NOT IN value key #26569 from @jayzhan211 (regression); Backport PR: [branch-56] fix: reject a multi-key null-aware join without a NOT IN value key count (backport #27111) #27176
- SortExec panics in sort_run on FFI-imported sliced batches: assertion reservation.size() + shared.size() >= reserved_bytes #27069 from @andygrove (regression); Backport PR: [branch-56] fix: count the largest capacity seen for a shared buffer address (backport #27097) #27175
- REGRESSION: UNION: DataFrame::schema() reports the first input's type, but execution returns the coerced type #27023 from @mikewolfd (regression); Backport PR: [branch-56] fix: infer UNION output types before analysis (backport #27139) #27178
- REGRESSION: SanityCheckPlan error with sort-merge join when a filter makes a sort or join key constant (constant lost in
EquivalenceGroupprojection) #26624 from @alpaylan (regression); Fix PR: fix: preserve constant equivalence properties through projection #27123; Backport PR: [branch-56] fix: preserve constant equivalence properties through projection (backport #27123) #27179 - Regression: short floating-point IN lists mishandle signed zero #27121 from @lonless9 (regression); Fix PR: IN LIST: treat signed zeros as equal #25186 (@geoffreyclaude); Backport PR: [branch-56] IN LIST: treat signed zeros as equal (backport #25186) #27204
- REGRESSION: Window ordered by (total DESC, k) fails SanityCheckPlan when k is a function of a filtered constant column #26762 from @adriangb (regression); Fix PR: fix:
SanityCheckPlanerror when a per-partition constant leads an ordering #27191 (@jayzhan211)
Tickets @alamb thinks are not blockers (see issues for rationale)
- ExternalSorter memory accounting: fixes missing from branch-55 and remaining gaps on main #25804 (finding 9) from @andygrove (reported by @Xuanwo); Fix PR: fix: leave memory for others when a spilled sort merges its runs #26264
- Incorrect regex-to-LIKE simplification for literal backslashes #27120 from @lonless9
- REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193) #27065 from @andygrove (regression); Fix PR: fix: keep nested Parquet binary fields out of the binary-to-string coercion #27067
-
WITH ... AS MATERIALIZEDreturns wrong rows underORDER BY ... LIMIT: top-k aggregation limits the shared CTE body #27219 from @jayzhan211; Fix PR: fix: do not limit shared CTE body in top-k aggregation (#27219) #27230 (@Hrushikesh-ramilla)
Community Wishlist
Before branch cut
- Further improve performance of IN list evaluation #19241
- datafusion-physical-expr: add
DynamicFilterPhysicalExpr::update_from(Self)method #26534 @jayshrivastava - dev: Allow new release version to be automatically updated #26007 from @nuno-faria
- docs: add version picker and manual release snapshots #24792 from @yiaany
- fix: keep field metadata in the result of min and max #26939 from @cblp (requested by @alamb)
- fix: keep extension types when skip_metadata clears Parquet field metadata #26854 from @cblp (requested by @alamb)
After branch cut (requires backport to branch-56)
- feat: show extension type names in DESCRIBE and information_schema.columns #26893 from @cblp (requested by @alamb); Backport PR: [branch-56] feat: show extension type names in DESCRIBE and information_schema.columns (backport #26893) #27057
- perf: prune oversized dictionaries before IPC spilling #27015 from @brancz; Backport PR: [branch-56] perf: prune oversized dictionaries before IPC spilling (backport #27015) #27107
- feat: support ALP parquet encoding in writer options #27036 from @Surajsuthar (requested by @alamb); Backport PR: [branch-56] feat: support ALP parquet encoding in writer options (backport #27036) #27093
- fix: read Hive's null partition placeholder as NULL #26310 f4om @xanderbailey; Backport PR: [branch-56] fix: read Hive's null partition placeholder as NULL (backport #26310) #27171
- fix: size aggregate spill batches by bytes as well as rows #25853 from @andygrove; Backport PR: fix: [branch-56] size aggregate spill batches by bytes as well as rows (backport #25853) #27243
- Avoid eager merging of expanding-key ingestion spills #27043 from @brancz
- feat: store start/end bounds in
Partitioning::Range#26593 @jayshrivastava - fix(proto): keep FILTER, IGNORE NULLS and reversal state of window expressions in physical plans #27207 from @shehabgamin
Reacted by Adam Gutglick, Jens Holdgaard Pedersen, Nathan, Kosta Tarasov, Kumar Ujjawal, Matt Butrovich and Namgung Chan
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]Release DataFusion `56.0.0` (Sep / Oct 2026)[/-][+]Release DataFusion `56.0.0` (Oct 2026)[/+]xudong963 commentedon Aug 26, 2026
I can help with this release!
Xuanwo commentedon Sep 9, 2026
Really hope #318 can deliever in v56.0, let's rock!
alamb commentedon Sep 9, 2026
318 -- perhaps the lowest issue number left open that tracks something useful. A long standing request
Xuanwo commentedon Sep 14, 2026
One more question: what testing needs to be done for releases? Perhaps we could also include
lance. I'm willing to take on all the related work.alamb commentedon Sep 15, 2026
Thanks @Xuanwo -- I think what we typically do is to use the release ticket to coordinate testing with downstream crates that wish to help as we get close to making a release candidate. So for example, @AdamGS often tests with Vortex as part of that process.
Here is an example in the 55 release ticket where we were disucssing it
55.0.0(Jul / Aug 2026) #22393 (comment)alamb commentedon Sep 15, 2026
Since this release will have an update to arrow/parquet to version 60, perhaps this would be a good candidate to try out "LTS" / Long term support: #16622
kumarUjjawal commentedon Sep 15, 2026
Datafusion 56.0 will be in a good stable condition for downstream consumers and should be good starting point for new projects. So LTS makes sense.
alamb commentedon Sep 16, 2026
FYI @hareshkh who was asking about a more stable branch / backports in the call today
[-]Release DataFusion `56.0.0` (Oct 2026)[/-][+]Release DataFusion `56.0.0` (Oct/Nov 2026)[/+]alamb commentedon Sep 28, 2026
Adjusting date from Oct --> Oct/Nov as we are also working on a 55.2:
55.2.0(patch) Release (Oct 2026) #25758adriangb commentedon Sep 30, 2026
I would like to land a fix for #25883 before next release
[branch-55] fix: frame IPC messages when reading spill files so batch…
jayshrivastava commentedon Oct 22, 2026
👋🏽 Flagging a couple changes I would like to see in df56 :)
DynamicFilterPhysicalExpr::update_from(Self)method #26534Partitioning::Range#2659363 remaining items
[branch-56] fix: infer UNION output types before analysis (backport #…
fix: preserve constant equivalence properties through projection (apa…
[branch-56] fix: preserve constant equivalence properties through pro…
shehabgamin commentedon Nov 7, 2026
+1 for Sail
shehabgamin commentedon Nov 7, 2026
Probably too late, but figured I'd share just in case
#27207
[branch-56] fix: count the largest capacity seen for a shared buffer …
[branch-56] Prepare for 56.0.0 release: Version and Changelog (#27098)
jayzhan211 commentedon Nov 8, 2026
#27219:
ORDER BY ... LIMITabove aWITH ... AS MATERIALIZEDCTE can return wrong rows (none, in the repro), because theTopKAggregationrule limits the CTE body. The feature (#25580) is onbranch-56, and the repro gives the same wrong rows on abranch-56build. Mentioning it here so it's visible while 56.0.0 is being prepared.alamb commentedon Nov 8, 2026
Yeah I don't know why it hasn't merged yet.
andygrove commentedon Nov 8, 2026
It is in the merge queue now. I will create backport PR once it has merged.
andygrove commentedon Nov 8, 2026
back-port PR #27243
xudong963 commentedon Nov 9, 2026
FYI, I found that DF56 introduced additional copying overhead for grouped aggregates with
FILTERwhen #24444 fixed argument evaluation on rejected rows. #24869 reduced some overhead but still filtered every input column per aggregate. #27119 preserves the corrected semantics while filtering only the columns each aggregate needs. This overhead is new in DF56’s grouped path; ungrouped aggregation already had it in DF55.The influence range of this regression isn't large, only for some specific query pattern, so I'm okay with not waiting for it. We can introduce it for the next minor release if someone else raises it later.
xingyu-long commentedon Nov 9, 2026
Hi @alamb, for the test with DataFusion Python, i can help, any pointers on this task? thanks!
[branch-56] IN LIST: treat signed zeros as equal (backport #25186) (#…
nuno-faria commentedon Nov 9, 2026
@xingyu-long thank you but I already tested with Python: apache/datafusion-python#1826. I think there is a problem with the FFI crate but I'm waiting for @timsaucer to take a look.