[andygrove]

Which issue does this PR close?

No issue. This is preparation for the DataFusion 56 upgrade, following the same approach used for DataFusion 55 (#2241).

Rationale for this change

Tracking DataFusion's branch-56 ahead of the 56 release lets us absorb upstream breaking changes in small steps instead of all at once when 56 ships. Pinning to an exact commit keeps builds reproducible, and the pin can be bumped periodically until the release, when we switch back to crates.io versions.

What changes are included in this PR?

Dependencies:

• Pin datafusion, datafusion-cli, datafusion-functions-aggregate-common, datafusion-proto, datafusion-proto-common and datafusion-substrait to apache/datafusion@1635ed9.
• Bump the dependencies DataFusion branch-56 now requires: arrow / arrow-flight 60 (the flight-sql-experimental feature is renamed flight-sql), object_store 0.14.2, and rustyline 19 in ballista-cli.
• Raise the MSRV from 1.94 to 1.95 to match DataFusion.
• Re-sync the vendored datafusion.proto and datafusion_common.proto with dev/update_datafusion_proto.py.

Ports to upstream API changes:

• LogicalPlan::DescribeTable is replaced by LogicalPlan::Show, which now covers DESCRIBE, SHOW TABLES, SHOW COLUMNS, SHOW CREATE TABLE, SHOW ALL and SHOW FUNCTIONS. These all read the client's catalog and cannot be serialized, so BallistaQueryPlanner plans every Show locally. The Flight SQL frontend likewise runs Show statements on the scheduler, against the session catalog it holds. Previously SHOW TABLES and friends were rewritten into information_schema queries, which both already kept off the cluster.
• OptimizerRule::supports_rewrite was removed upstream.
• ExecutionPlan::with_preserve_order is deprecated and no longer called, so ChaosExec stops overriding it.
• TableProvider::scan takes Option<&[usize]> for the projection.

DataFusion's final physical plan check:

• DataFusion now runs SanityCheckPlan after the last physical optimizer rule of every pipeline, including custom ones, and offers no way to turn it off. Several AQE chains used to end with plans that only became valid on a later replan.
• End the logical-plan preparation rules in AdaptivePlanner::try_new with EnsureRequirements. A submitted physical plan skips it, because the client placed its repartitions on purpose and EnsureRequirements would drop the ones it finds unnecessary before they are pinned as exchanges.
• ParallelWindowRule leaves K partitions where the window had one, which broke any parent that required a single partition or a key partitioning. The rule now inserts a coalesce, an order-preserving merge or a hash repartition for such parents. A NULLS FIRST window also gets a per-partition sort to its own null placement, since the range sketch always sorts the key NULLS LAST. The replan used to add that sort.
• ChaosExec implements limit_behavior, because supports_limit_pushdown is deprecated.

Test updates:

• Two join-selection tests that build a sort-merge join now plan it with EnsureRequirements, because the unsorted form is rejected, and the nested-join test plans hash joins instead.
• DataFusion now rejects a null-aware HashJoinExec in Partitioned mode. Ballista's planner and AQE already lower null-aware joins to CollectLeft, so only two join-selection tests that built the Partitioned form needed to change.
• DataFusion now rejects RANGE offsets on a Utf8 order key, so the parallel-window "no encoding" test uses a CURRENT ROW frame instead. It still fails if the key is switched to an encodable type.
• DiskManager now creates its temp directories on the first spill file, so the spill-dir test creates one before checking the paths.
• Rebuilding an InterleaveExec with non-interleavable children no longer errors upstream, so a precondition check in the NormalizeInterleaveRule test is dropped. The rule itself may no longer be needed, which can be a follow-up.
• Refresh the TPC-H plan-stability golden files and insta snapshots. DataFusion now embeds projections into SortMergeJoinExec instead of placing a ProjectionExec above it, and the byte-size statistics estimates changed. The EXPLAIN ANALYZE expectation picks up new AggregateExec metrics.
• Cover the Flight SQL frontend's handling of SHOW statements with unit tests that need no socket: which statements run on the scheduler, and that SHOW TABLES lists the session's tables.

Python bindings:

• python/ stays on DataFusion 55, because it builds against a datafusion-python release. It takes the workspace crates by path, which would put the pinned DataFusion into its build next to DataFusion 55, so its ballista* dependencies now point at 72d68073 on main (from before this pin) and python/Cargo.lock is updated to match.
• The published 55.0.0 crates cannot stand in for that commit, because python/src uses core changes made after that release (feat(python): Plug into DataFusion Python natively through a custom FFI query planner  #2588).
• Restore the path dependencies, which are left commented in python/Cargo.toml, once DataFusion 56 and datafusion-python 56 are released.

Fixes for CI that are not part of the upgrade:

• Import ExecutionPlan in the execution_graph tests. fix(scheduler): keep AQE stage in the graph when a stage transition fails #2613 moved the FailingPlanRewriteExec test helper out of that test module and dropped the import with it, but the test that feat: support DataFrame::cache() materialize and read-back #2633 added still needs it, so the scheduler's test target does not compile on main. The same one-line fix is in feat: let custom ExecutionPlan nodes restrict themselves to a task's partitions #2721 and feat: let hash shuffles place rows by a client-computed partition id #2723, so whichever merges first makes the others no-ops.
• Run ruff format over benchmarks/cluster/ (from feat(bench): scripts to run the Ballista, Spark, Comet and Trino comparison on Kubernetes #2697). The Python workflow lints the whole repository and runs on any change under python/, so these unformatted scripts fail it for any PR that touches python/. Formatting only: the syntax tree of every file is unchanged.

Are there any user-facing changes?

Downstream crates now need arrow 60, object_store 0.14 and Rust 1.95, and see the DataFusion 56 API (for example LogicalPlan::Show and the new TableProvider::scan signature). EXPLAIN now shows the repartitions in its physical_plan row, which used to be omitted. Labelled api-change for that reason. The Python client is unaffected and stays on DataFusion 55.

[milenkovicm]

Would it be ok to postpone this until I get two python PRs merged if not very urgent ?

[andygrove]

Would it be ok to postpone this until I get two python PRs merged if not very urgent ?

Yes, of course.

[milenkovicm]

thanks for understanding, will try to address few of your comments later today