diff --git a/README.md b/README.md index 5d666f20cfb..4883403fefb 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ See the [Comet Benchmarking Guide](https://datafusion.apache.org/comet/contribut ## What Comet Accelerates -Comet accelerates Spark workloads by replacing Spark operators and expressions with high-performance implementations that process Apache Arrow columnar data directly. Most operators are powered by native Rust execution built on Apache DataFusion, while others run efficiently in the JVM on Arrow batches. This unified columnar execution model keeps processing within the Comet engine end-to-end, reducing overhead and delivering faster, more efficient query execution without reverting to Spark's traditional row-based engine. +Comet accelerates Spark workloads by replacing Spark operators and expressions with high-performance implementations that process Apache Arrow columnar data directly. Most operators are powered by native Rust code built on Apache DataFusion, while others run efficiently in the JVM on Arrow batches. This unified columnar execution model keeps processing within the Comet engine end-to-end, reducing overhead and delivering faster, more efficient query execution without reverting to Spark's traditional row-based engine. - **Parquet scans**: native Parquet reader integrated with Spark's query planner - **Apache Iceberg**: accelerated Parquet scans when reading Iceberg tables from Spark diff --git a/contrib/delta/src/main/scala/org/apache/comet/contrib/delta/DeltaConf.scala b/contrib/delta/src/main/scala/org/apache/comet/contrib/delta/DeltaConf.scala index bb8172c9b21..403c3e41fd7 100644 --- a/contrib/delta/src/main/scala/org/apache/comet/contrib/delta/DeltaConf.scala +++ b/contrib/delta/src/main/scala/org/apache/comet/contrib/delta/DeltaConf.scala @@ -75,7 +75,7 @@ object DeltaConf { "When true (default), the Delta contrib falls back to Spark's Delta reader on " + "any Delta protocol feature it doesn't yet support. When false, the contrib " + "raises an error instead -- useful for tests that want to assert the native " + - "path is reachable for a particular query.") + "Delta scan is reachable for a particular query.") .booleanConf .createWithDefault(true) diff --git a/docs/source/about/faq.md b/docs/source/about/faq.md index 9937e1ea277..e4c42d2c157 100644 --- a/docs/source/about/faq.md +++ b/docs/source/about/faq.md @@ -26,11 +26,11 @@ answered here, see [Where can I ask questions or report bugs?](#where-can-i-ask- ### How does Comet compare to other open-source Spark accelerators? -Several open-source projects speed up Spark by running query plans in native code. They differ in -the engine they build on and the hardware they need. +Several open-source projects speed up Spark by running query plans in engines outside the JVM. They +differ in the engine they build on and the hardware they need. - [Apache Gluten](https://gluten.apache.org/) is the closest to Comet in design: a Spark plugin that - passes the parts of a plan it supports to a native engine, such as the C++ engine Velox. See + passes the parts of a plan it supports to a C++ engine, such as Velox. See [Comparison of Comet and Gluten](gluten_comparison.md). - [Apache Auron (incubating)](https://auron.apache.org/), formerly Blaze, is also a Spark plugin built on DataFusion. Comet is developed within the DataFusion project itself. @@ -103,10 +103,10 @@ driver or an executor. ### How much memory does Comet need? Comet needs Spark's off-heap memory: without `spark.memory.offHeap.enabled=true`, Comet disables -itself and logs a warning. Comet's native operators share the off-heap pool, sized by -`spark.memory.offHeap.size`, with Spark. When the pool runs out, operators that can spill, such as -sorts, aggregations, and shuffle writes, spill to disk, and operators that cannot spill fail the -task. +itself and logs a warning. Comet's Rust-implemented operators and its JVM shuffle share the +off-heap pool, sized by `spark.memory.offHeap.size`, with Spark. When the pool runs out, operators +that can spill, such as sorts, aggregations, and shuffle writes, spill to disk, and operators that +cannot spill fail the task. Comet also uses memory that the pool does not track, which has to fit in `spark.executor.memoryOverhead`. If the cluster manager kills executors for exceeding their memory @@ -131,7 +131,7 @@ Common causes of a small speedup are: manager, Comet disables itself and logs a warning, unless `spark.comet.shuffle.enabled=false`, in which case Comet runs but every shuffle runs in Spark. See [Shuffle Tuning](../user-guide/latest/tuning/shuffle.md). -- **Too little memory.** Native operators spill to disk when the off-heap pool is too small. See +- **Too little memory.** Comet's operators spill to disk when the off-heap pool is too small. See [Memory Tuning](../user-guide/latest/tuning/memory.md). - **Little computation to accelerate.** Queries that spend most of their time reading from storage, listing files, or planning, or that process little data, have less to gain. @@ -173,7 +173,7 @@ the code generator instead. If you have a use case that a function of one row handles poorly, such as working on the UTF-8 bytes of strings or calling a library that processes whole batches, please describe it in -[#6694](https://github.com/apache/datafusion-comet/issues/6694). For native speed, see the +[#6694](https://github.com/apache/datafusion-comet/issues/6694). To run a function in Rust, see the experimental [Rust UDF API](../user-guide/latest/rust_udfs.md). ### Does Comet support Structured Streaming? diff --git a/docs/source/about/versioning_policy.md b/docs/source/about/versioning_policy.md index 11e9ce935b6..58dfc595d86 100644 --- a/docs/source/about/versioning_policy.md +++ b/docs/source/about/versioning_policy.md @@ -75,10 +75,10 @@ change in any release: - Every Scala, Java, and Rust type not on that list, including the internal structure of the classes that are on it. See [Everything Else Is Internal](#everything-else-is-internal). - `EXPLAIN` output, and the shape of the plans Comet produces. Which operators are fused, how a - native block is partitioned, and how a plan is rendered may all change between releases. + Rust plan is partitioned, and how a plan is rendered may all change between releases. - Metric names and log output. -- Performance characteristics, including which expressions and operators run natively and which - fall back to Spark. An expression that ran natively in one release may fall back in the next, +- Performance characteristics, including which expressions and operators run in Comet and which + fall back to Spark. An expression that ran in Comet in one release may fall back in the next, and vice versa. The results stay the same; only the speed changes. ### Testing and Internal Configurations Are Exempt @@ -307,7 +307,7 @@ full vendor contract. ### Everything Else Is Internal Every other Scala, Java, and Rust type Comet ships is internal. This includes the rest of -`org.apache.comet.*`, everything Comet contributes to `org.apache.spark.*`, and all of the native +`org.apache.comet.*`, everything Comet contributes to `org.apache.spark.*`, and all of the Rust crates. These types exist to make the plugin work, not to be programmed against. They may be renamed, changed, or removed in any release, including a patch release, with no deprecation cycle and no upgrade guide entry. diff --git a/docs/source/contributor-guide/adding_a_new_expression.md b/docs/source/contributor-guide/adding_a_new_expression.md index 5a0c5ffdad7..403b4665c60 100644 --- a/docs/source/contributor-guide/adding_a_new_expression.md +++ b/docs/source/contributor-guide/adding_a_new_expression.md @@ -115,7 +115,7 @@ Follow these rules when wiring scalar expressions: `scalarFunctionExprToProtoWithReturnType(..., failOnError, ...)`. The function must also have a dedicated match arm in `create_comet_physical_fun_with_eval_mode` that actually consumes the flag; - it must not fall through to registry lookup. Existing native consumers + it must not fall through to registry lookup. Existing Rust consumers include `make_decimal`, `make_date`, `make_time`, and `next_day`. #### When to set the return type explicitly @@ -129,7 +129,7 @@ org.apache.comet.CometNativeException: Error from DataFusion: Function 'levenshtein' expects 2 arguments but received 3. ``` -The classic case is `levenshtein`. Spark accepts an optional 3rd `threshold` argument, DataFusion's built-in is 2-arg only, so the 3-arg form fails native execution unless the serde sets the return type explicitly. Other names that exist in both engines with potentially different signatures include `concat`, `coalesce`, `sha2`, and `regexp_replace`. If you are adding a function whose name is shared with `datafusion-functions`, check the upstream signature before deciding how to serialize. +The classic case is `levenshtein`. Spark accepts an optional 3rd `threshold` argument, DataFusion's built-in is 2-arg only, so the 3-arg form fails at execution time unless the serde sets the return type explicitly. Other names that exist in both engines with potentially different signatures include `concat`, `coalesce`, `sha2`, and `regexp_replace`. If you are adding a function whose name is shared with `datafusion-functions`, check the upstream signature before deciding how to serialize. To avoid the registry lookup, write a custom `CometExpressionSerde` and use `scalarFunctionExprToProtoWithReturnType`, passing the Spark expression's declared `dataType`: @@ -154,7 +154,7 @@ When the return type is set on the proto, the native planner skips the registry #### Arguments that Spark skips after a NULL -A null-intolerant `BinaryExpression` or `TernaryExpression` returns NULL as soon as one of its arguments is NULL, without evaluating the arguments after it. Native execution evaluates every argument of a `ScalarFunc` over the whole batch, so an argument that can fail, such as an ANSI cast of a malformed string, would fail on rows that Spark skips. Wrap the serialized call of such an expression in `withNullShortCircuit`, as `CometArrayContains` and `CometSlice` do, after checking the conditions in its Scaladoc. +A null-intolerant `BinaryExpression` or `TernaryExpression` returns NULL as soon as one of its arguments is NULL, without evaluating the arguments after it. Comet's Rust code evaluates every argument of a `ScalarFunc` over the whole batch, so an argument that can fail, such as an ANSI cast of a malformed string, would fail on rows that Spark skips. Wrap the serialized call of such an expression in `withNullShortCircuit`, as `CometArrayContains` and `CometSlice` do, after checking the conditions in its Scaladoc. #### Registering the Expression Handler @@ -181,7 +181,7 @@ The `SupportLevel` sealed trait has three possible values: - **`Compatible(notes: Option[String] = None)`** - Comet supports this expression with full compatibility with Spark, or may have known differences in specific edge cases unlikely to affect most users. This is the default if you don't override `getSupportLevel`. - **`Incompatible(notes: Option[String] = None)`** - Comet supports this expression but results can differ from Spark. The expression will only be used if the expression-specific config `spark.comet.expression..allowIncompatible=true` is set. -- **`Unsupported(notes: Option[String] = None)`** - Comet does not support this expression under the current conditions. Spark will fall back to its native execution. +- **`Unsupported(notes: Option[String] = None)`** - Comet does not support this expression under the current conditions. The operator that contains it falls back to Spark, unless a `CodegenDispatchFallback` serde routes it through codegen dispatch. All three accept an optional `notes` parameter to provide additional context that is logged for debugging. @@ -366,7 +366,7 @@ That run rewrites every Implementation cell in place, mapping SQL function names #### Adding Spark-side Tests for the New Expression -It is important to verify that the new expression is correctly recognized by the native execution engine and matches the expected Spark behavior. The preferred way to add test coverage is to write a Comet SQL Test. This approach is simpler than writing Comet Scala Tests and makes it easy to cover many input combinations and edge cases. +It is important to verify that the new expression is correctly recognized by Comet and matches the expected Spark behavior. The preferred way to add test coverage is to write a Comet SQL Test. This approach is simpler than writing Comet Scala Tests and makes it easy to cover many input combinations and edge cases. ##### Writing a Comet SQL Test @@ -398,7 +398,7 @@ query SELECT unhex('48656C6C6F'), unhex(''), unhex(NULL) ``` -Each `query` block automatically runs the SQL through both Spark and Comet and compares results, and also verifies that Comet executes the expression natively (not falling back to Spark). +Each `query` block automatically runs the SQL through both Spark and Comet and compares results, and also verifies that the expression runs in Comet (not falling back to Spark). Run the test with: @@ -410,11 +410,11 @@ For full documentation on the test file format, including directives like `Confi ##### Tips -- **Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated natively. +- **Cover both column references and literals.** Comet often uses different code paths for each. The Comet SQL Tests suite automatically disables constant folding, so all-literal queries are evaluated by Comet. - **Include edge cases** such as `NULL`, empty strings, boundary values, `NaN`, and multibyte UTF-8 characters. - **For `FLOAT` and `DOUBLE` input, include `-0.0` and a NaN with the sign bit set.** [Floating-Point Semantics](floating_point.md) explains which Spark rule an expression follows and how to test it. - **Keep one file per expression** to make failures easy to locate. -- **Pin the mechanism when it depends on the input.** If your serde routes some input types natively and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md). +- **Pin the mechanism when it depends on the input.** If your serde routes some input types to its native implementation and others through the JVM codegen dispatcher, say so with `expect_native(...)` and `expect_dispatch(...)`. A plain `query` cannot tell the two apart, so without these a later change that swaps one for the other passes silently. See [Comet SQL Tests](sql-file-tests.md). ##### Comet Scala Tests (alternative) @@ -444,7 +444,7 @@ test("unhex") { `checkSparkAnswerAndOperator` verifies that results match Spark and that Comet did not fall back for the operator. It does not distinguish _how_ Comet evaluated the expression. Comet has two -accelerated paths, a native DataFusion expression and the JVM codegen dispatcher (Spark's own +accelerated paths, a DataFusion expression in Rust and the JVM codegen dispatcher (Spark's own `doGenCode` compiled into an Arrow batch kernel), and both produce Spark-matching results by construction. @@ -466,9 +466,9 @@ test("bit_length dispatches on binary and stays native on string") { Names are the expression's `prettyName` lowercased. Naming an expression asserts both that it ran through the expected mechanism and that it did not run through the other one, so a serde that -later gains a native path (or loses its dispatcher route) fails here instead of passing unnoticed. -Prefer the SQL file equivalents, `expect_native(...)` and `expect_dispatch(...)`, when the test -fits in a fixture. +later gains a native implementation (or loses its dispatcher route) fails here instead of passing +unnoticed. Prefer the SQL file equivalents, `expect_native(...)` and `expect_dispatch(...)`, when +the test fits in a fixture. When writing Comet Scala Tests with literal values (e.g., `SELECT my_func('literal')`), Spark's constant folding optimizer may evaluate the expression at planning time, bypassing Comet. To prevent this, disable constant folding: @@ -516,8 +516,8 @@ How this works is somewhat dependent on the type of expression you're adding. Ex If you're adding a new expression that requires custom protobuf serialization, you may need to: 1. Add a new message to the protobuf definition in `native/proto/src/proto/expr.proto` -2. Add a native expression handler in `expression_registry.rs` to deserialize the new protobuf message type and - create a native expression +2. Add an expression handler in `expression_registry.rs` to deserialize the new protobuf message type and + create a DataFusion `PhysicalExpr` For most expressions, you can skip this step if you're using the existing scalar function infrastructure. @@ -584,7 +584,7 @@ Use `datafusion_comet_common::decode_utf8_spark_lossy` (defined in `native/commo replaces each ill-formed sequence with `U+FFFD` exactly as the JVM's `new String(bytes, UTF_8)` does, so Comet's rendered output matches Spark (including the surrogate-range cases where `str::from_utf8_lossy` would differ). This is the decoder used by both `CAST(binary AS string)` and -the native columnar shuffle. See the +JVM shuffle's Rust conversion of Spark rows to Arrow. See the [strings with non-UTF-8 bytes](../user-guide/latest/compatibility/index.md) note in the compatibility guide for the user-visible behavior, including the cases where decoding diverges from Spark. diff --git a/docs/source/contributor-guide/adding_a_new_operator.md b/docs/source/contributor-guide/adding_a_new_operator.md index b2beae1740b..05f5e9de813 100644 --- a/docs/source/contributor-guide/adding_a_new_operator.md +++ b/docs/source/contributor-guide/adding_a_new_operator.md @@ -24,36 +24,37 @@ This guide explains how to add support for a new Spark physical operator in Apac ## Overview `CometExecRule` is responsible for replacing Spark operators with Comet operators. There are different approaches to -implementing Comet operators depending on where they execute and how they integrate with the native execution engine. +implementing Comet operators depending on where they execute and how they integrate with Comet's Rust code. ### Types of Comet Operators `CometExecRule` maintains two distinct maps of operators: -#### 1. Native Operators (`nativeExecs` map) +#### 1. Operators in the `nativeExecs` Map -These operators run entirely in native Rust code and are the primary way to accelerate Spark workloads. Native -operators are registered in the `nativeExecs` map in `CometExecRule.scala`. +Most of these operators run entirely in native Rust code and are the primary way to accelerate Spark workloads. They +are registered in the `nativeExecs` map in `CometExecRule.scala`. `LocalTableScanExec`, `InMemoryTableScanExec` and +`EmptyRelationExec` are the exceptions: they become JVM-side Comet scans that feed Arrow batches into a native plan. -Key characteristics of native operators: +Key characteristics of the Rust-implemented operators: -- They are converted to their corresponding native protobuf representation -- They execute as DataFusion operators in the native engine +- They are converted to their corresponding protobuf representation +- They execute as DataFusion operators in Rust - The `CometOperatorSerde` implementation handles enable/disable checks, support validation, and protobuf serialization Examples: `ProjectExec`, `FilterExec`, `SortExec`, `HashAggregateExec`, `SortMergeJoinExec`, `ExpandExec`, `WindowExec` #### 2. Sink Operators (`sinks` map) -Sink operators serve as entry points (data sources) for native execution blocks. They are registered in the `sinks` +Sink operators serve as entry points (data sources) for native plans. They are registered in the `sinks` map in `CometExecRule.scala`. Key characteristics of sinks: - They become `ScanExec` operators in the native plan (see `operator2Proto` in `CometExecRule.scala`) -- They can be leaf nodes that feed data into native execution blocks +- They can be leaf nodes that feed data into native plans - They are wrapped with `CometScanWrapper` or `CometSinkPlaceHolder` during plan transformation -- Examples include operators that bring data from various sources into native execution +- Examples include operators that bring data from various sources into a native plan Examples: `UnionExec`, `CoalesceExec`, `CollectLimitExec`, `TakeOrderedAndProjectExec` @@ -74,7 +75,7 @@ Examples: `CometBroadcastExchangeExec`, `CometShuffleExchangeExec` `CometTakeOrderedAndProjectExec` owns local candidate selection, the optional shuffle, and final selection with offset and projection. When `spark.comet.exec.topK.fusion.enabled` is enabled for an -eligible native Parquet scan, its conversion inserts `CometLocalTopKExec` before native blocks are +eligible native Parquet scan, its conversion inserts `CometLocalTopKExec` before native plans are serialized. The local node extends `CometUnaryExec` and serializes a bounded sort above the native scan, so they execute in one block. It preserves the scan's output and partitioning and advertises the local sort order. @@ -82,7 +83,7 @@ the local sort order. The inserted node keeps the original Spark TopK for plan bookkeeping, but only the outer TopK owns Spark's offset and projection. Both transition reversion and aggregate-buffer restoration therefore remove the inserted local node when restoring Spark execution. Restoring its `originalPlan` would -apply the global TopK twice. A single input partition uses a final native limit and projection; +apply the global TopK twice. A single input partition uses a final limit and projection in Rust; multiple input partitions still require a final TopK after the shuffle. With `spark.comet.exec.topK.dynamicFilter.enabled`, only the local sort sets the protobuf @@ -97,26 +98,26 @@ attachment counters belong to the local TopK. When adding a new operator, choose based on these criteria: -**Use Native Operators when:** +**Use Rust-implemented operators when:** - The operator transforms data (e.g., project, filter, sort, aggregate, join) - The operator has a direct DataFusion equivalent or custom implementation -- The operator consumes native child operators and produces native output +- The operator consumes and produces data within a native plan - The operator is in the middle of an execution pipeline **Use Sink Operators when:** -- The operator serves as a data source for native execution (becomes a `ScanExec`) -- The operator brings data from non-native sources (e.g., `UnionExec` combining multiple inputs) +- The operator serves as a data source for a native plan (becomes a `ScanExec`) +- The operator brings data from sources outside the native plan (e.g., `UnionExec` combining multiple inputs) - The operator is typically a leaf or near-leaf node in the execution tree -- The operator needs special handling to interface with the native engine +- The operator needs special handling at the JVM/native boundary **Implementation Note for Sinks:** Sink operators are handled specially in `CometExecRule.operator2Proto`. Instead of converting to their own operator type, they are converted to `ScanExec` in the native plan. This allows them to serve as entry points for native -execution blocks. The original Spark operator is wrapped with `CometScanWrapper` or `CometSinkPlaceHolder` which -manages the boundary between JVM and native execution. +plans. The original Spark operator is wrapped with `CometScanWrapper` or `CometSinkPlaceHolder` which +manages the JVM/native boundary. ### Operators That Should Not Be Converted @@ -131,18 +132,18 @@ stages, mark whole-stage code generation boundaries, choose which shuffle blocks or point at a plan that runs elsewhere. AQE creates the query stages itself, after Comet's rules have run on the plan inside them, and depends on their exact class. For example, it casts the root of the final plan to `ResultQueryStageExec`. When a query stage wraps a Comet shuffle, broadcast, -or cached relation, `CometExecRule` reads from it as a native input through `CometExchangeSink` -and leaves the stage itself in place. +or cached relation, `CometExecRule` reads from it as an input to a native plan through +`CometExchangeSink` and leaves the stage itself in place. **Operators that run user JVM code on JVM objects.** The typed `Dataset` API plans `DeserializeToObjectExec`, `SerializeFromObjectExec`, `MapElementsExec`, `MapPartitionsExec`, `AppendColumnsExec`, `AppendColumnsWithObjectExec`, `MapGroupsExec`, and `CoGroupExec`. They convert rows to JVM objects, run an arbitrary user function on those objects, or convert them back, and most of them pass the objects to the next operator as an `ObjectType` column, which has no -Arrow representation. None of this can run natively, so these operators stay on Spark. Every typed +Arrow representation. None of this can run in Comet, so these operators stay on Spark. Every typed operation ends in `SerializeFromObjectExec`, though, whose output is ordinary rows. With `spark.comet.convert.typedDataset.enabled`, `CometExecRule` puts a `CometSparkToColumnarExec` above -it, so the operators above the typed operation can run natively. Spark inserts no columnar +it, so the operators above the typed operation can run in Comet. Spark inserts no columnar transitions below a `RowToColumnarTransition`, so the rule inserts them for the typed operation's own operators itself. Spark computes a typed operation's rows one at a time, as they are read, while the conversion fills a whole Arrow batch first. So the rule leaves the output unconverted where a @@ -168,9 +169,9 @@ summary does not count it as a Spark operator. If `CometExecRule` visits the nod the operators it leaves in place without recording a fallback reason. `ExtendedExplainInfo` already skips every `QueryStageExec`, so a new query stage type needs no change there. -## Implementing a Native Operator +## Implementing an Operator in Rust -This section focuses on adding a native operator, which is the most common and complex case. +This section focuses on adding a Rust-implemented operator, which is the most common and complex case. ### Step 1: Define the Protobuf Message @@ -364,25 +365,25 @@ An operator that overrides `outputOrdering`, as `CometFilterExec` does above wit the order satisfies. Report what the Spark operator reports and nothing more. `SampleExec`, for example, reports no order, so `CometSampleExec` must not report its child's. -The claim has to hold at run time as well. A native hash join declares the order Spark reports for -its streamed side to DataFusion, and a native window the order of its partition keys. An aggregate +The claim has to hold at run time as well. A Comet hash join declares the order Spark reports for +its streamed side to DataFusion, and a Comet window the order of its partition keys. An aggregate above may then run in sorted mode, or a top-k sort may stop reading early, on the strength of that order. If the order is false, the result is wrong groups or wrong rows, not only rows out of order. -So an operator that reports an ordering must keep it natively, and the native operator must say so +So an operator that reports an ordering must keep it in Rust, and its Rust operator must say so by returning `true` from `maintains_input_order` for that child, as DataFusion's filter and projection do. The planner declares the order through an operator only when the operator says it -keeps it. An operator that cannot keep the order natively must not report it, or has to be added to +keeps it. An operator that cannot keep the order in Rust must not report it, or has to be added to `mayReportFalseOrder` in `operators.scala`. The hash join and window serdes then send no order over it, an outer hash join above it that has to keep that order falls back to Spark, and a window that runs in Spark above it gets its input sorted again. Comet's hash joins are the exception within one -native plan: an outer hash join over one of them stays native, even over one that declared no order, -since DataFusion keeps the order it can see there. +native plan: an outer hash join over one of them stays in Comet, even over one that declared no +order, since DataFusion keeps the order it can see there. A leaf that reports an ordering is where the order starts, so the planner has to know that the -native operator produces its rows in that order. It knows that of an input from the JVM and of a -scan of files, which reads the files of each partition one after the other, and the native range -declares its order itself through `SortedInputExec`. A new leaf that reports an ordering must -declare it natively the same way. +leaf's Rust operator produces its rows in that order. It knows that of an input from the JVM and of +a scan of files, which reads the files of each partition one after the other, and the Rust range +operator declares its order itself through `SortedInputExec`. A new leaf that reports an ordering +must declare it in Rust the same way. #### Plan Identity and Exchange Reuse @@ -403,9 +404,9 @@ The examples above follow the implementations in - `nativeOp` is the protobuf representation and `originalPlan` is the source Spark plan; neither is compared by these operators. `CometNativeExec.canonicalizePlans` clears the non-child Spark plan references, including their `originalPlan`. -- `serializedPlanOpt` holds the bytes for a native execution block. These operators compare it in - `equals`, but omit it from `hashCode`; `CometNativeExec.doCanonicalize` clears the block's serialized - plan. The bytes therefore do not distinguish canonicalized plans. +- `serializedPlanOpt` holds the bytes for the native plan of an execution block. These operators + compare it in `equals`, but omit it from `hashCode`; `CometNativeExec.doCanonicalize` clears the + block's serialized plan. The bytes therefore do not distinguish canonicalized plans. - `stringArgs` controls the plan's displayed arguments. Show the semantic fields and children rather than serialization state; this method does not define equality. @@ -456,7 +457,7 @@ Note that Comet will treat an operator as incompatible if any of the child expre Add your operator to the appropriate map in `CometExecRule.scala`: -#### For Native Operators +#### For Operators in the `nativeExecs` Map Add to the `nativeExecs` map (`CometExecRule.scala`): @@ -500,7 +501,7 @@ val COMET_EXEC_YOUR_OPERATOR_ENABLED: ConfigEntry[Boolean] = Run `make` to update the user guide. The new configuration option will be added to `docs/source/user-guide/latest/configs.md`. -### Step 5: Implement the Native Operator in Rust +### Step 5: Implement the Operator in Rust #### Update the Planner @@ -634,7 +635,7 @@ test("your operator") { The `checkSparkAnswerAndOperator` helper verifies: -1. Results match Spark's native execution +1. Results match Spark without Comet 2. Your operator is actually being used (not falling back) #### Plan Identity Regression Tests @@ -650,7 +651,7 @@ Use the "aggregate canonicalization preserves result expressions and equivalent as a model. Inspect the executed plans: an optimizer rewrite can make the branches differ for an unrelated reason, allowing a test to pass even when the intended field is missing from `equals`. Choose inputs and query shapes that preserve the parameter difference, and traverse adaptive and -query-stage wrappers when checking the native operator and reused exchange. +query-stage wrappers when checking the Comet operator and reused exchange. #### Rust Unit Tests @@ -674,7 +675,7 @@ Add your operator to the supported operators list in `docs/source/user-guide/lat ## Implementing a Sink Operator -Sink operators are converted to `ScanExec` in the native plan and serve as entry points for native execution. The implementation is simpler than native operators because sink operators extend the `CometSink` base class which provides the conversion logic. +Sink operators are converted to `ScanExec` in the native plan and serve as entry points for native plans. The implementation is simpler than for Rust-implemented operators because sink operators extend the `CometSink` base class which provides the conversion logic. ### Step 1: Create a CometOperatorSerde Implementation @@ -772,7 +773,7 @@ val COMET_EXEC_YOUR_SINK_ENABLED: ConfigEntry[Boolean] = ### Step 4: Add Tests -Test that your sink operator correctly feeds data into native execution: +Test that your sink operator correctly feeds data into a native plan: ```scala test("your sink operator") { diff --git a/docs/source/contributor-guide/benchmarking_micro_ec2.md b/docs/source/contributor-guide/benchmarking_micro_ec2.md index a848b2d849d..d6ec9959dcd 100644 --- a/docs/source/contributor-guide/benchmarking_micro_ec2.md +++ b/docs/source/contributor-guide/benchmarking_micro_ec2.md @@ -107,7 +107,7 @@ What each is for: - `git` clones the repository - `make` drives the build through the top level `Makefile` -- `gcc`, `gcc-c++`, `cmake` build the native dependencies of the Rust crates +- `gcc`, `gcc-c++`, `cmake` build the C and C++ dependencies of the Rust crates - `protobuf-compiler` provides `protoc`, needed to generate the plan serialization code - `java-17-amazon-corretto-devel` provides the JDK. JDK 17 or later is required for Spark 4.0 and above, which is the default build profile @@ -115,7 +115,7 @@ What each is for: ### Protobuf Compiler -The native build compiles the Comet protobuf definitions with `prost-build`, which needs a `protoc` +The Rust build compiles the Comet protobuf definitions with `prost-build`, which needs a `protoc` binary on the `PATH`. Without one, `make release` fails with: ``` @@ -345,7 +345,7 @@ compiler. Install one and rerun `make release`: sudo dnf install -y gcc gcc-c++ make ``` -**`Could not find protoc` during the native build.** See +**`Could not find protoc` during the Rust build.** See [Protobuf Compiler](#protobuf-compiler) above, then rerun `make release`. **`JAVA_HOME could not be determined`.** Install a JDK and export `JAVA_HOME`, or pass a checkout diff --git a/docs/source/contributor-guide/bug_triage.md b/docs/source/contributor-guide/bug_triage.md index b2bacd7567c..878dc255ca9 100644 --- a/docs/source/contributor-guide/bug_triage.md +++ b/docs/source/contributor-guide/bug_triage.md @@ -50,7 +50,7 @@ appropriate label from the table below. | Label | Color | Description | Examples | | ------------------- | ------ | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | `priority:critical` | Red | Data corruption, silent wrong results, security vulnerabilities | Wrong aggregation results, FFI data corruption, incorrect cast output | -| `priority:high` | Orange | Crashes, panics, segfaults, major functional breakage affecting production workloads | Native engine panic, JVM segfault, NPE on supported code path | +| `priority:high` | Orange | Crashes, panics, segfaults, major functional breakage affecting production workloads | Rust panic, JVM segfault, NPE on supported code path | | `priority:medium` | Yellow | Functional bugs, performance regressions, broken features that have workarounds | Missing expression support, writer feature gaps, excessive spilling | | `priority:low` | Green | Minor issues, test-only failures, tooling, CI flakes, cosmetic issues | Flaky CI test, build script edge case, documentation generator bug | @@ -60,7 +60,7 @@ Use this decision tree: 1. **Can this bug cause silent wrong results?** If yes → `priority:critical`. These are the most dangerous bugs because users may not notice the incorrect output. -2. **Does this bug crash the JVM or native engine?** If yes → `priority:high`. Crashes are +2. **Does this bug crash the JVM or native code?** If yes → `priority:high`. Crashes are disruptive but at least visible to the user. 3. **Does this bug break a feature or cause significant performance degradation?** If yes → `priority:medium`. The user can work around it (e.g., falling back to Spark) but it impacts @@ -85,7 +85,7 @@ help contributors find bugs in their area of expertise. | Label | Description | | ------------------ | ----------------------------------------- | | `area:writer` | Native writer (Parquet and other formats) | -| `area:shuffle` | Shuffle (JVM and native) | +| `area:shuffle` | Shuffle (JVM and native shuffle) | | `area:aggregation` | Hash aggregates, aggregate expressions | | `area:scan` | Data source scan (Parquet, CSV, Iceberg) | | `area:expressions` | Expression evaluation | @@ -150,7 +150,7 @@ When fixing correctness bugs, always add a regression test that verifies the out ### Crash Bugs (`priority:high`) -These are bugs where the native engine panics, segfaults, or throws an unhandled exception. +These are bugs where native code panics, segfaults, or throws an unhandled exception. Common patterns include: - **All-scalar inputs:** Some expressions assume at least one columnar input and panic when all diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md index ca9b6d28dd6..2f8eca319d3 100644 --- a/docs/source/contributor-guide/ci.md +++ b/docs/source/contributor-guide/ci.md @@ -78,15 +78,15 @@ on push to `main`, because the queue already tested the exact tree that landed. the Linux and macOS builds, which also run on push so that the dependency caches on `main` stay fresh: a pull request can only restore caches saved on its own branch or on `main`, and the queue's temporary branch takes its caches with it when it is deleted. On push the macOS build runs only its -native build, and only when `main` has no cache entry yet for the current dependency set. +native library build, and only when `main` has no cache entry yet for the current dependency set. That push run is for the caches and nothing else, so it runs in **cache-refresh-only** mode: only -the three jobs that own a cache entry (the native CI build and the two TPC-H/TPC-DS jobs, the last -two stopping before their query passes), plus the short `Lint` job the native build depends on. The -lints, the Rust tests and the Comet test matrix are skipped, which is the difference between 587 -runner-minutes a push and about 40. If you add a job to `pr_build_linux.yml`, give it -`if: ${{ !inputs.cache-refresh-only }}` unless it writes a cache that `main` needs; -`dev/ci/check-ci-config.py` fails the build if you forget. +the three jobs that own a cache entry (the native library build and the two TPC-H/TPC-DS jobs, the +last two stopping before their query passes), plus the short `Lint` job the native library build +depends on. The lints, the Rust tests and the Comet test matrix are skipped, which is the +difference between 587 runner-minutes a push and about 40. If you add a job to +`pr_build_linux.yml`, give it `if: ${{ !inputs.cache-refresh-only }}` unless it writes a cache +that `main` needs; `dev/ci/check-ci-config.py` fails the build if you forget. For a pull request from a fork, the pull request column's "yes" rows run on your fork rather than here; see [Pull requests from forks](#pull-requests-from-forks). @@ -146,11 +146,11 @@ Some things to know: fork's default branch, and a run on your fork's `main` saves them only when it builds. A push to `main` that adds nothing to upstream, such as syncing your fork, selects no suite, so it warms nothing, and a branch run rebuilds the native library. A run of the PR tier on a fork with cold - caches took about an hour of wall time and 240 runner-minutes, of which the native build alone was - 23 minutes. The same run with warm caches took 50 minutes and 213 runner-minutes. A run that also - routes to the Spark SQL and Iceberg suites took 89 minutes and about 650 runner-minutes. Their - shards run in parallel with the Linux build's test matrix, but the Spark SQL job first builds its - own native library and test classes, which took 45 minutes. + caches took about an hour of wall time and 240 runner-minutes, of which the native library build + alone was 23 minutes. The same run with warm caches took 50 minutes and 213 runner-minutes. A run + that also routes to the Spark SQL and Iceberg suites took 89 minutes and about 650 + runner-minutes. Their shards run in parallel with the Linux build's test matrix, but the Spark SQL + job first builds its own native library and test classes, which took 45 minutes. - **Scheduled runs stay upstream.** The nightly run, Miri, the stale-PR closer and the weekly CodeQL scan run in `apache/datafusion-comet` only. If your fork has scheduled workflows enabled they start no job. Miri can still be run on your fork by dispatching it. @@ -219,9 +219,9 @@ re-run the failed jobs from the Actions page, or remove and re-apply the label. Use a label when a change is likely to behave differently on a version or platform the PR tier does not cover. Some examples: -- a change to the serde, the planner, or a native operator, where the Comet test suites pass but - Spark's own SQL suite is the thing that would catch a behavior difference; the PR tier runs it - against the default profile, and `run-spark-3.5-tests` / `run-spark-4.0-tests` / +- a change to the serde, the planner, or a Rust-implemented operator, where the Comet test suites + pass but Spark's own SQL suite is the thing that would catch a behavior difference; the PR tier + runs it against the default profile, and `run-spark-3.5-tests` / `run-spark-4.0-tests` / `run-spark-4.2-tests` run it against the others - code under `spark/src/main/spark-3.5/`, `spark-4.0/` or the shared `spark-3.x/` directory, or any change to `CometExprShim` and friends; `run-all-spark-profiles` runs the Comet @@ -455,9 +455,9 @@ dev/local-ci.sh --print-config # what it read from ci.yml and dev/ci The version defaults to the one the merge queue gates on. That, the matrix rows and the shard count are read from the workflow files and from `dev/ci/`, so a local shard runs what the CI shard of the -same name runs. It prepares first (native build, Comet install, patched Spark or Iceberg source -under `$COMET_LOCAL_CI_HOME`, default `/tmp/comet-local-ci`), then runs the tests, compiling only -the sbt projects the selected rows need. +same name runs. It prepares first (native library build, Comet install, patched Spark or Iceberg +source under `$COMET_LOCAL_CI_HOME`, default `/tmp/comet-local-ci`), then runs the tests, compiling +only the sbt projects the selected rows need. `SKIP_PREPARE=1` runs the tests only. It skips the **Comet install** too, so do not use it after changing Comet or the run tests the previously installed JAR and goes green regardless. @@ -486,7 +486,7 @@ Four caveats: additionally sweeps the **whole** repository for POMs with no sibling JAR. That is a shared cache, so other projects will re-download. `--print-config` touches nothing. - CI is x86_64 Linux built with `-Ctarget-cpu=x86-64-v3`, so a pass elsewhere covers Scala, serde - and planner behavior but not x86-specific native codegen. + and planner behavior but not x86-specific code in the native library. The underlying steps are documented in [Spark SQL Tests](spark-sql-tests.md) and [Iceberg Spark Tests](iceberg-spark-tests.md), which are also where the diff-regeneration workflow diff --git a/docs/source/contributor-guide/config_conventions.md b/docs/source/contributor-guide/config_conventions.md index 6c198f740c9..3ddfbd2fedb 100644 --- a/docs/source/contributor-guide/config_conventions.md +++ b/docs/source/contributor-guide/config_conventions.md @@ -197,8 +197,8 @@ Two cases do **not** need a legacy config: - **Correctness fixes.** When a `Compatible` expression or operator does not match Spark, that is a bug. Fixing it is a bug fix and may ship in any release, including a patch release. Add a legacy config only if the fix has an unusually wide blast radius, and say so in the PR description. -- **Changes to which expressions and operators run natively.** Falling back to Spark, or ceasing - to, changes performance rather than results. +- **Changes to which expressions and operators run in the Comet pipeline.** Falling back to + Spark, or ceasing to, changes performance rather than results. Nor does a change to a key in the `testing` category or a key marked `internal()`, both [exempt from the versioning policy](../about/versioning_policy.md#testing-and-internal-configurations-are-exempt): diff --git a/docs/source/contributor-guide/debugging.md b/docs/source/contributor-guide/debugging.md index 51d74a1ea94..63b12940881 100644 --- a/docs/source/contributor-guide/debugging.md +++ b/docs/source/contributor-guide/debugging.md @@ -22,9 +22,9 @@ under the License. This HOWTO describes how to debug JVM code and Native code concurrently. The guide assumes you have: 1. IntelliJ as the Java IDE -2. CLion as the Native IDE. For Rust code, the CLion Rust language plugin is required. Note that the +2. CLion as the Rust IDE. For Rust code, the CLion Rust language plugin is required. Note that the IntelliJ Rust plugin is not sufficient. -3. CLion/LLDB as the native debugger. CLion ships with a bundled LLDB and the Rust community has +3. CLion/LLDB as the Rust debugger. CLion ships with a bundled LLDB and the Rust community has its own packaging of LLDB (`lldb-rust`). Both provide a better display of Rust symbols than plain LLDB or the LLDB that is bundled with XCode. We will use the LLDB packaged with CLion for this guide. 4. We will use a Comet _unit_ test as the canonical use case. @@ -106,9 +106,9 @@ Detecting the debugger ## Verbose debug -### Enabling Native Backtraces +### Enabling Rust Backtraces -By default, Comet does not show native backtraces when exceptions happen in native code: +By default, Comet does not show Rust backtraces when exceptions happen in native code: ```scala scala> spark.sql("my_failing_query").show(false) @@ -123,7 +123,7 @@ This was likely caused by a bug in DataFusion's code and we would welcome that y ``` -Comet can be built with DataFusion's [backtrace] feature enabled, which will include native back traces in `CometNativeException`. +Comet can be built with DataFusion's [backtrace] feature enabled, which will include Rust backtraces in `CometNativeException`. [backtrace]: https://arrow.apache.org/datafusion/user-guide/example-usage.html#enable-backtraces @@ -174,13 +174,13 @@ stderr of the executor where it happened, on a line starting with `Comet native `CometNativeException` it surfaces as carries the panic message only, because a panic can reach the JVM on a different thread than the one that raised it. -### Native log configuration +### Native-side log configuration By default, Comet emits native-side logs at the `INFO` level to `stderr`. You can use the `COMET_LOG_LEVEL` environment variable to specify the log level. Supported values are: `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`. -For example, to configure native logs at the `DEBUG` level on spark executor: +For example, to configure native-side logs at the `DEBUG` level on spark executor: ``` spark.executorEnv.COMET_LOG_LEVEL=DEBUG diff --git a/docs/source/contributor-guide/development.md b/docs/source/contributor-guide/development.md index 16a458d180b..e9fe87a5912 100644 --- a/docs/source/contributor-guide/development.md +++ b/docs/source/contributor-guide/development.md @@ -30,8 +30,8 @@ under the License. ## Threading Architecture -Comet's native execution runs on a shared tokio multi-threaded runtime. Understanding this -architecture is important because it affects how you write native operators and JVM callbacks. +Comet's native plans run on a shared tokio multi-threaded runtime. Understanding this +architecture is important because it affects how you write Rust operators and JVM callbacks. ### How execution works @@ -51,7 +51,7 @@ the Spark task instead of ending its output early. `pull_input_batches()` to feed data from the JVM into the ScanExec and ShuffleScanExec operators, whose streams register the poll's waker and are woken by the refill. The stream is polled with a waker that also sets a flag. If the flag is still clear when the pull returns, the stream is -waiting on native I/O, and the thread parks until a waker fires instead of busy-polling. If it is +waiting on I/O in Rust, and the thread parks until a waker fires instead of busy-polling. If it is set, the loop polls again. It checks the flag rather than trusting the thread's parker because the pull can run another Comet plan on the same thread, and that plan's `block_on()` shares the parker and can consume the wake-up meant for the outer loop. @@ -428,8 +428,8 @@ from a clean IntelliJ configuration: On Apple Silicon, the selected JDK architecture must match the native Rust target. Verify that `file "$JAVA_HOME/lib/server/libjvm.dylib"` reports `arm64`. -5. Build the profile once from the command line so generated sources and native artifacts are in - place: +5. Build the profile once from the command line so generated sources and the native library are + in place: ```sh PROFILES="-Pspark-4.0" make release @@ -438,7 +438,7 @@ from a clean IntelliJ configuration: The `spark-4.0` profile sets the Scala 2.13 properties, and every profile targets Java 17. If you need to be explicit, use `PROFILES="-Pspark-4.0 -Pscala-2.13" make release`. - If the native build previously used a different JDK, clear Cargo's cached JNI link path before + If the Rust build previously used a different JDK, clear Cargo's cached JNI link path before rebuilding: ```sh @@ -715,8 +715,8 @@ If you add a new Comet Scala Test suite, you must add it to the `suite` matrix i - `.github/workflows/pr_build_linux.yml` - `.github/workflows/pr_build_macos.yml` -Each file contains a `suite` matrix with named groups such as `fuzz`, `shuffle`, `parquet`, `csv`, -`exec`, `expressions`, and `sql`. Add your new suite's fully qualified class name to the +Each file contains a `suite` matrix with four named groups: `scans`, `shuffle`, `exec`, and +`expressions`. Add your new suite's fully qualified class name to the appropriate group. For example, if you add a new expression test suite, add it to the `expressions` group: @@ -730,15 +730,12 @@ appropriate group. For example, if you add a new expression test suite, add it t Choose the group that best matches the area your test covers: -| Group | Covers | -| ------------- | ---------------------------------------------------------- | -| `fuzz` | Fuzz testing and data generation | -| `shuffle` | Shuffle operators and related exchange behavior | -| `parquet` | Parquet read/write and native reader tests | -| `csv` | CSV native read tests | -| `exec` | Execution operators, joins, aggregates, plan rules, TPC-\* | -| `expressions` | Expression evaluation, casts, and Comet SQL Tests | -| `sql` | SQL-level behavior tests | +| Group | Covers | +| ------------- | ----------------------------------------------------------- | +| `scans` | Parquet, CSV and Iceberg reads and writes, and fuzz testing | +| `shuffle` | Shuffle operators and related exchange behavior | +| `exec` | Execution operators, joins, aggregates, plan rules, TPC-\* | +| `expressions` | Expression evaluation, casts, and Comet SQL Tests | **Important:** The suite lists in both workflow files must stay in sync. The `preflight` job in `.github/workflows/ci.yml` runs `dev/ci/check-suites.py` on every pull request. It scans for all diff --git a/docs/source/contributor-guide/ffi.md b/docs/source/contributor-guide/ffi.md index 64db891347c..4a955aa02b6 100644 --- a/docs/source/contributor-guide/ffi.md +++ b/docs/source/contributor-guide/ffi.md @@ -25,7 +25,8 @@ Comet transfers Arrow data across the JVM/native boundary in two directions: 1. **JVM → Native**: Native code pulls batches from the JVM over the [Arrow C Stream Interface](https://arrow.apache.org/docs/format/CStreamInterface.html). The JVM exports each - per-partition iterator once as an `ArrowArrayStream`, and native pulls every batch through a single C callback. + per-partition iterator once as an `ArrowArrayStream`, and the Rust side pulls every batch through a single C + callback. 2. **Native → JVM**: JVM pulls batches from native code using `CometExecIterator`, via the [Arrow C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html) (one `ArrowArray`/`ArrowSchema` pair per column of each batch). @@ -75,11 +76,11 @@ exports once per partition: │ (C struct) │ └────────┬────────┘ │ Arrow C Stream Interface - │ (native pulls each batch via the get_next callback) + │ (the Rust side pulls each batch via the get_next callback) ▼ ┌─────────────────┐ │ ScanExec │ ── owns an ArrowArrayStreamReader -│ (Rust/native) │ +│ (Rust) │ └────────┬────────┘ │ ▼ @@ -100,9 +101,9 @@ implementation depends on the source of the data: - `ColumnarBatchArrowReader`: an Arrow-backed `ColumnarBatch` (retains its buffers before closing the input batch) The exported `ArrowArrayStream`s are boxed into the `Array[Object]` that `CometExecIterator` / `CometExecRDD` pass -to native `createPlan`, one slot per scan input. +to `Native.createPlan`, one slot per scan input. -Not every slot is an Arrow stream. When a native operator consumes Comet shuffle output and +Not every slot is an Arrow stream. When a native plan consumes Comet shuffle output and `spark.comet.shuffle.directRead.enabled` is set, that slot carries a `CometShuffleBlockIterator` instead, and the compressed shuffle blocks are decoded inside the native plan by `ShuffleScanExec` rather than crossing this FFI boundary at all. `CometExecRDD.resolveInputObjects` classifies the slots, driven by which scan slots the serialized @@ -121,18 +122,18 @@ panic ([apache/arrow-rs#10028](https://github.com/apache/arrow-rs/issues/10028)) ### Errors from the JVM Producer -Arrow Java's exported stream catches whatever the reader throws in `get_next` and hands native only its text, so -native fails the plan with a `CometNativeException` built from that text. `CometArrowStream.stream` wraps every -reader to keep the throwable itself until the task completes, and `CometExecIterator` rethrows it in place of the -native error. The task then fails with the exception Spark would have thrown, such as a `SparkArithmeticException` -from an upstream plan. For an input of Arrow-backed `ColumnarBatch`es, the first batch never takes this path: schema -reconciliation reads it on the JVM before the stream is exported, so what it throws propagates directly. A test of -this path therefore has to fail a later batch. +Arrow Java's exported stream catches whatever the reader throws in `get_next` and hands the Rust side only its text, +so the Rust side fails the plan with a `CometNativeException` built from that text. `CometArrowStream.stream` wraps +every reader to keep the throwable itself until the task completes, and `CometExecIterator` rethrows it in place of +the `CometNativeException`. The task then fails with the exception Spark would have thrown, such as a +`SparkArithmeticException` from an upstream plan. For an input of Arrow-backed `ColumnarBatch`es, the first batch +never takes this path: schema reconciliation reads it on the JVM before the stream is exported, so what it throws +propagates directly. A test of this path therefore has to fail a later batch. ### Schema Reconciliation `CometArrowStream.reconcileStreamSchema` advertises the stream's schema from the actual `CometVector` types in the -first batch rather than the consumer's Spark-declared types. Native `ScanExec` already casts its input to the +first batch rather than the consumer's Spark-declared types. The Rust `ScanExec` already casts its input to the declared scan-input schema in `build_record_batch`, so the truthful first-batch schema lets that cast fire; if the two differ, it logs one deduplicated warning naming the operator, column, and type drift. @@ -142,7 +143,7 @@ No input stream carries a dictionary. `RowArrowReader` and `SparkColumnarArrowRe ### Memory Layout -When a batch is transferred from JVM to native: +When a batch is transferred from the JVM to the Rust side: ``` JVM Heap: Native Memory: @@ -162,7 +163,7 @@ Off-heap Memory: │ └──────────────────┘ ``` -**Key Point**: The actual data buffers are shared zero-copy; native only takes pointers to the off-heap buffers. +**Key Point**: The actual data buffers are shared zero-copy; the Rust side only takes pointers to the off-heap buffers. ### Ownership and Lifecycle @@ -170,37 +171,37 @@ For JVM columnar consumers, `CometNativeArrowSource` uses `CometArrowStream.read with independent vectors that retain the reader's buffers without copying the data. Closing a batch or its columns leaves the source reader's reusable vector structure intact for its next batch. The iterator releases the batch vectors when it advances, and closes the final batch, reader and allocator at task completion. A later -native consumer's `ColumnarBatchArrowReader` retains the buffers before closing its input batch. +native plan's `ColumnarBatchArrowReader` retains the buffers before closing its input batch. -The Arrow C Stream Interface transfers ownership by reference count: native takes ownership of each imported batch, -so it is safe to buffer batches in operators such as `SortExec` or `ShuffleWriterExec` without a deep copy. +The Arrow C Stream Interface transfers ownership by reference count: the Rust side takes ownership of each imported +batch, so it is safe to buffer batches in operators such as `SortExec` or `ShuffleWriterExec` without a deep copy. The whole per-partition stream is exported once, so the JVM allocates one `ArrowArrayStream` per partition rather than a per-batch, per-column `ArrowArray`/`ArrowSchema` wrapper object pair. Lifecycle is anchored at the stream: when `ScanExec` drops its `ArrowArrayStreamReader`, the stream's release callback fires synchronously back into the JVM -and closes the `ArrowReader` and its `VectorSchemaRoot`, releasing the off-heap buffers. Because native holds those -buffers until the reader drops, an operator that buffers many batches keeps the corresponding JVM-side data alive -until then. +and closes the `ArrowReader` and its `VectorSchemaRoot`, releasing the off-heap buffers. Because the Rust side holds +those buffers until the reader drops, an operator that buffers many batches keeps the corresponding JVM-side data +alive until then. ## Native → JVM Data Flow (CometExecIterator) ### Architecture -When the JVM needs results from native execution: +When the JVM needs results from a native plan: ``` ┌─────────────────┐ │ DataFusion Plan │ -│ (native) │ +│ (Rust) │ └────────┬────────┘ │ produces RecordBatch ▼ ┌─────────────────┐ │ prepare_output │ ── fills one ArrowArray/ArrowSchema pair per column -│ (Rust/native) │ +│ (Rust) │ └────────┬────────┘ │ Arrow C Data Interface - │ (structs allocated by the JVM, filled by native) + │ (structs allocated by the JVM, filled by the Rust side) ▼ ┌─────────────────┐ │ NativeUtil │ ◄─── CometExecIterator calls Native.executePlan @@ -240,10 +241,10 @@ numbers dictionaries from 0, so two dictionary-encoded columns would collide in Arrow Java's C Data import ignores `ArrowArray.offset` at every level ([apache/arrow-java#88](https://github.com/apache/arrow-java/issues/88)) and reads each buffer from its start. arrow-rs folds a slice into the buffers for almost every type, but a sliced `BooleanArray` keeps its bit offset, and a struct -exports offset 0 even when its children are sliced. So every array native exports to the JVM first goes through +exports offset 0 even when its children are sliced. So every array the Rust side exports to the JVM first goes through `zero_offsets` (in `native/common/src/ffi_offsets.rs`), which re-slices boolean bitmaps to start at bit 0 at every level and shares every other buffer. `move_to_spark` applies it to executed batches and decoded shuffle blocks, and -`JvmScalarUdfExpr` applies it to the inputs of the JVM UDF bridge. A new native to JVM export path has to call it too, +`JvmScalarUdfExpr` applies it to the inputs of the JVM UDF bridge. A new Rust-to-JVM export path has to call it too, or sliced booleans reach the JVM misaligned ([#6288](https://github.com/apache/datafusion-comet/issues/6288)). @@ -262,32 +263,32 @@ Python runner rebases the same way before it sends a batch to the worker. All th ### Ownership and Lifecycle -Native allocates the data, and the JVM references it without copying: +The Rust side allocates the data, and the JVM references it without copying: -- The `FFI_ArrowArray` that `move_to_spark` writes holds a reference to the column's native buffers, and its release +- The `FFI_ArrowArray` that `move_to_spark` writes holds a reference to the column's arrow-rs buffers, and its release callback drops that reference. -- On import, Arrow Java wraps each native buffer in an `ArrowBuf`. Closing the last `ArrowBuf` over an imported - column runs its release callback, and native frees the buffers once no Rust reference to them remains. +- On import, Arrow Java wraps each arrow-rs buffer in an `ArrowBuf`. Closing the last `ArrowBuf` over an imported + column runs its release callback, and the buffers are freed once no Rust reference to them remains. - `CometExecIterator` closes the batch it returned when the consumer next calls `hasNext` or `next`, and on `close()`. A batch is therefore valid only until the consumer asks for the next one, and a consumer that keeps the data longer has to copy it. -By the time the JVM receives a batch, native has usually stopped reserving it in the memory pool, but it stays +By the time the JVM receives a batch, the Rust side has usually stopped reserving it in the memory pool, but it stays resident until the JVM closes it. See [Crossing the FFI boundary](memory_management.md#crossing-the-ffi-boundary). ## Memory Ownership Rules ### JVM → Native -| Scenario | Ownership | Action Required | -| --------- | ----------- | --------------------------------------------------------------------------------------------------------- | -| All cases | Native owns | None; the C Stream transfers ownership by reference count. Dropping the reader releases the JVM-side data | +| Scenario | Ownership | Action Required | +| --------- | -------------- | --------------------------------------------------------------------------------------------------------- | +| All cases | Rust side owns | None; the C Stream transfers ownership by reference count. Dropping the reader releases the JVM-side data | ### Native → JVM -| Scenario | Ownership | Action Required | -| --------- | -------------------------------- | ---------------------------------------------------------- | -| All cases | Native allocates, JVM references | JVM must call `close()` to trigger native release callback | +| Scenario | Ownership | Action Required | +| --------- | ----------------------------------- | ------------------------------------------------------------ | +| All cases | Rust side allocates, JVM references | JVM must call `close()` to trigger the Rust release callback | ## Further Reading diff --git a/docs/source/contributor-guide/floating_point.md b/docs/source/contributor-guide/floating_point.md index a650ed0aaa8..cdaf42b1beb 100644 --- a/docs/source/contributor-guide/floating_point.md +++ b/docs/source/contributor-guide/floating_point.md @@ -44,8 +44,8 @@ include a NaN with the sign bit set. | Scala `==`, for boxed values (`BoxesRunTime.equals`) | Equal | A NaN equals nothing, not even another NaN | Scala sets and maps of `Any`, such as `collect_set`'s buffer before Spark 4.2 | | `java.lang.Double.compare` | `-0.0` sorts before `0.0` | As in SQL ordering | An ascending `sort_array` of elements that cannot be null, in Spark's generated code | -Some functions changed rules in a Spark release, so a native path has to follow the Spark version -it runs against: +Some functions changed rules in a Spark release, so a native implementation has to follow the +Spark version it runs against: - `collect_set` keys its buffer with Scala's `==` before Spark 4.2, so `-0.0` and `0.0` are one value and every NaN is a value of its own. Spark 4.2 normalizes NaN and `-0.0` first and keys the @@ -84,9 +84,9 @@ normalize the values, or compare them with the helpers below. NaNs with the sign bit set are the normal case on x86-64, not a corner case. Every NaN that arithmetic produces at run time, such as `sqrt(-1)` or `Infinity - Infinity`, is `0xfff8000000000000` in both Rust and the JVM on x86-64, while aarch64 produces -`0x7ff8000000000000`. Spark hides the difference through `doubleToLongBits`. A native path that -compares raw values puts those NaNs below every other value, so a query that passes on an Apple -Silicon laptop can fail on a Linux x86 cluster. +`0x7ff8000000000000`. Spark hides the difference through `doubleToLongBits`. A native +implementation that compares raw values puts those NaNs below every other value, so a query that +passes on an Apple Silicon laptop can fail on a Linux x86 cluster. ## Where Comet applies the rules @@ -141,10 +141,11 @@ normalized and the output keeps the original values. ### Spark version differences -Rules that depend on the Spark version are decided in Scala. The serde either chooses the native -path or passes a flag to native code, as `Mode`'s `normalize_neg_zero` does. Where a patch release -changed the rule, add a predicate to `SparkFixes` that checks the version down to the patch with -`Utils.majorMinorPatchVersion`, as `SparkFixes.arraySetFunctionsNormalizeFloats` does. +Rules that depend on the Spark version are decided in Scala. The serde either decides whether to use +the native implementation or passes a flag to native code, as `Mode`'s `normalize_neg_zero` does. +Where a patch release changed the rule, add a predicate to `SparkFixes` that checks the version down +to the patch with `Utils.majorMinorPatchVersion`, as `SparkFixes.arraySetFunctionsNormalizeFloats` +does. Until an expression has a native implementation that follows its rule, report it `Incompatible` for floating-point input so that Spark evaluates it. @@ -201,7 +202,7 @@ and only for a type that contains a `FLOAT` or `DOUBLE`. - Nest the values in arrays and structs, two levels deep, and make the nested NaN a sign-bit one. - When arithmetic produces the NaN, run with ANSI mode both on and off (`-- ConfigMatrix: spark.sql.ansi.enabled=false,true`), because division and remainder take - different native paths. + different code paths in Rust. - Where the rule changed in a Spark release, gate the fixtures with `MinSparkVersion` or `MaxSparkVersion`, and add the `run-all-spark-profiles` label to the pull request. CI pins one patch release per Spark line, so when a patch release changed the rule, also test the routing in diff --git a/docs/source/contributor-guide/iceberg-scheduler-failures.md b/docs/source/contributor-guide/iceberg-scheduler-failures.md index b8c9aee0e2c..7b9f8c1cf8d 100644 --- a/docs/source/contributor-guide/iceberg-scheduler-failures.md +++ b/docs/source/contributor-guide/iceberg-scheduler-failures.md @@ -22,17 +22,17 @@ under the License. `CometIcebergSchedulerFailureSuite` tests speculative execution and executor loss during multi-task native Iceberg writes. It covers the scheduler scenarios in [#5646](https://github.com/apache/datafusion-comet/issues/5646), complementing the mid-write, -post-native handoff and job/commit failure tests in `CometIcebergWriteActionSuite`. +post-handoff and job/commit failure tests in `CometIcebergWriteActionSuite`. The suite adds test instrumentation without changing Iceberg commit or cleanup semantics. ## Test scenarios -| Scenario | Required scheduler evidence | Storage expectation | -| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | -| Original attempt wins speculation | Two attempts on distinct hosts finish native handoff; the original succeeds and its message is accepted | Only accepted winner files are referenced; no orphan or missing files | -| Speculative attempt wins | Two attempts on distinct hosts finish native handoff; the speculative attempt succeeds and its message is accepted | Only accepted winner files are referenced; no orphan or missing files | -| Executor-loss task replacement | Executor removal, `ExecutorLostFailure`, and successful replacement of the affected partition on another executor | Only accepted winner files are referenced; every orphan belongs to a known rejected attempt; no missing files | -| Executor loss with shuffle-output loss | The task-replacement evidence, plus `FetchFailed` and a higher writer `stageAttemptId` | The same storage assertions as executor-loss task replacement | +| Scenario | Required scheduler evidence | Storage expectation | +| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| Original attempt wins speculation | Two attempts on distinct hosts finish handoff; the original succeeds and its message is accepted | Only accepted winner files are referenced; no orphan or missing files | +| Speculative attempt wins | Two attempts on distinct hosts finish handoff; the speculative attempt succeeds and its message is accepted | Only accepted winner files are referenced; no orphan or missing files | +| Executor-loss task replacement | Executor removal, `ExecutorLostFailure`, and successful replacement of the affected partition on another executor | Only accepted winner files are referenced; every orphan belongs to a known rejected attempt; no missing files | +| Executor loss with shuffle-output loss | The task-replacement evidence, plus `FetchFailed` and a higher writer `stageAttemptId` | The same storage assertions as executor-loss task replacement | An increased task attempt number alone does not prove stage re-execution. The shuffle-output-loss case fails if the cluster only replaces the task without the required stage and fetch-failure events. @@ -45,7 +45,7 @@ Every scenario writes to a new empty table and asserts: complete expected ID multiset, rather than relying on counts alone. - Exactly one new snapshot, one commit operator, and one successful commit marker. - One driver-accepted message per logical writer partition. Each message's complete file set - identifies exactly one native handoff attempt with a scheduler `Success` event. + identifies exactly one handoff attempt with a scheduler `Success` event. - Manifest references equal the complete union of accepted winner files. Files reported by rejected attempts must not be referenced. - Every referenced file exists under the data location. @@ -60,7 +60,7 @@ missing = manifest-referenced files - physical files For speculation, both sets must be empty after the losing attempt terminates and cleanup finishes. For executor loss, `missing` must be empty and `orphan` must be a subset of `rejectedFiles`, the -paths observed through attempt-unique file-name prefixes or native handoff, belonging to +paths observed through attempt-unique file-name prefixes or handoff, belonging to attempts whose messages were not accepted. Unknown orphan files fail the test. This checks the storage state even when readers see the correct table rows. `SIGKILL` bypasses executor-side cleanup, so known rejected-attempt orphans do not by themselves @@ -78,8 +78,8 @@ Use a dedicated Spark Standalone cluster with: - At least two Linux worker hosts, as seen by the Spark scheduler, with at least two cores per worker and executor replacement enabled. `local[...]` and `local-cluster[...]` are rejected. -- Matching Spark/Scala profiles, JDK, operating system and native architecture on the driver and - executors. A macOS native build cannot serve Linux executors. +- Matching Spark/Scala profiles, JDK, operating system and CPU architecture on the driver and + executors. A native library built on macOS cannot serve Linux executors. - A driver address reachable from both workers, configured through the usual Spark settings. - A shared POSIX filesystem mounted at the same absolute path on the driver and all workers. It holds the warehouse, source files, attempt markers and executor classpath snapshot. @@ -111,25 +111,25 @@ captured in task closures, so executors do not depend on a driver-installed JVM `IcebergCommitExec` records messages actually accepted by Spark's `runJob` result handler and records the successful commit. Duplicate accepted messages or commit markers fail the probe. -`CometIcebergWriteExec` installs per-attempt gates before parent input iteration and after native -payload handoff. `IcebergSchedulerProbeRDD` wraps the JVM input iterator and pauses before +`CometIcebergWriteExec` installs per-attempt gates before parent input iteration and after the +native writer's handoff. `IcebergSchedulerProbeRDD` wraps the JVM input iterator and pauses before pulling another batch after the configured row threshold. The native writer pulls this input -through `CometArrowStream.inputObjects`. It can prefetch a batch while a write awaits native I/O, +through `CometArrowStream.inputObjects`. It can prefetch a batch while a write awaits I/O, so the row count measures supplied input rather than completed writes. The pause occurs before input EOF and writer close; the physical-file checks establish actual write progress. The probe inventories files under the data location using the attempt-unique `file_name_prefix` (partition ID, task attempt ID and operation ID), recording -paths and their current byte sizes in `native-progress.json`. There is no native or protobuf test +paths and their current byte sizes in `native-progress.json`. There is no Rust or protobuf test hook. JSON markers are published atomically under unique run/task-attempt directories; all gate waits have bounded timeouts and diagnostics. ### Speculative attempts Four source Parquet files contain disjoint ID ranges. The original attempt for writer partition 0 -is held after native handoff while other partitions finish, allowing Spark to launch a speculative +is held after handoff while other partitions finish, allowing Spark to launch a speculative attempt on another host. Both attempts must report nonempty, disjoint file sets for the same stage, stage attempt and partition. The tests control each winner direction by releasing only the selected attempt and requiring its -scheduler `Success` and accepted message. Both attempts completing native handoff does not mean +scheduler `Success` and accepted message. Both attempts completing handoff does not mean both succeed in the scheduler: Spark cancels the loser after accepting the winner. ### Executor process loss @@ -138,10 +138,10 @@ Initial writer attempts pause after supplying 4,000 rows, using 1,000-row batche target file size and `write.parquet.page-row-limit=1000`. The native writer holds back each partition's first data page before opening a file; its default 20,000-row page exceeds this fixture's 12,000 rows per task. The smaller page permits early file rolling. The 4,000-row input threshold leaves -room for native input prefetch and asynchronous file creation before the JVM iterator pauses. +room for the writer's input prefetch and asynchronous file creation before the JVM iterator pauses. The row threshold alone is not evidence of physical write progress. -Before termination, the test independently requires physical bytes at a native-owned path, no -native handoff or terminal task event for the target, and progress on another writer partition +Before termination, the test independently requires physical bytes at a writer-owned path, no +handoff or terminal task event for the target, and progress on another writer partition in a different executor process. The ordinary loss case targets partition 0. The shuffle-output-loss case selects an active writer @@ -149,7 +149,7 @@ executor that also owns completed upstream shuffle output. Two other reducers ar constructing their parent iterators, preventing eager shuffle fetches. After executor removal, releasing those reducers must cause `FetchFailed` and writer-stage re-execution. -The suite records the target's exact host, PID, executor ID and native-owned paths before invoking +The suite records the target's exact host, PID, executor ID and writer-owned paths before invoking the termination helper. `dev/iceberg-scheduler-kill-executor.py` is a Linux/SSH implementation that checks `/proc/PID/cmdline` for `CoarseGrainedExecutorBackend` and the expected executor ID before sending `SIGKILL`, then waits for process exit. It terminates the executor, not the worker or driver. @@ -206,7 +206,7 @@ COMET_SCHEDULER_SPECULATION_ENABLED=false ./mvnw test -Pspark-4.1 -Dtest=none \ By default the suite snapshots the actual Maven test JVM's unshaded classpath to the shared root before creating SparkContext. It includes current reactor classes, test helpers, dependencies and -native resources. A shaded Comet jar alone is insufficient for the unshaded task closures. +the native library. A shaded Comet jar alone is insufficient for the unshaded task closures. The snapshot is removed after SparkContext stops. To use a pre-provisioned matching unshaded classpath instead, set `COMET_SCHEDULER_EXECUTOR_CLASSPATH` to paths visible on every worker. diff --git a/docs/source/contributor-guide/iceberg-writes.md b/docs/source/contributor-guide/iceberg-writes.md index 0fad3c35370..6577430f304 100644 --- a/docs/source/contributor-guide/iceberg-writes.md +++ b/docs/source/contributor-guide/iceberg-writes.md @@ -46,7 +46,7 @@ criteria for turning the flag on by default are tracked in [#5644](https://github.com/apache/datafusion-comet/issues/5644) under the epic [#5649](https://github.com/apache/datafusion-comet/issues/5649). -One rule runs through the whole native path: **the native writer must produce the outcome +One rule runs through the whole native writer: **the native writer must produce the outcome iceberg-java would have produced, or decline.** iceberg-java is the reference for every data file, manifest entry, partition value and failure behaviour. Where Comet cannot show that it reproduces iceberg-java for some table or setting, the write falls back to iceberg-java at plan time. The @@ -106,8 +106,8 @@ Things to know before changing this layer: On Spark 4.1+, `IcebergWriteSummaryShim` finds either Spark's `MergeRowsExec` or `CometMergeRowsExec` in the executed query and forwards its eight action counters to -`BatchWrite.commit(messages, summary)`. Native instructions carry the context of each `Keep`; -`Discard` counts a deletion and `Split` counts one update. Spark 4.2 uses last-attempt +`BatchWrite.commit(messages, summary)`. Serialized `MergeInstruction`s carry the context of each +`Keep`; `Discard` counts a deletion and `Split` counts one update. Spark 4.2 uses last-attempt accumulators, read through `IcebergSemanticMetricsShim`, so summary values come from the tasks that produced the committed output. Stock V2 writers still require the concrete Spark node, so `IcebergWriteStrategy` tags a copy of its logical MERGE query before planning and AQE. The @@ -121,8 +121,8 @@ conversion by calling `convertToComet`. The converted node keeps the `IcebergWri `originalPlan`, so AQE re-plans the write from that node. `CometIcebergNativeWrite.requiresNativeChildren` is `true`. The native writer consumes Arrow -batches from its child over FFI, so the conversion is declined unless the child is already a Comet -native operator. This is why a write fed by a `LocalTableScanExec` (`INSERT ... VALUES`, a local +batches from its child over FFI, so the conversion is declined unless the child is already a +`CometNativeExec`. This is why a write fed by a `LocalTableScanExec` (`INSERT ... VALUES`, a local DataFrame) stays on the JVM writer unless `spark.comet.exec.localTableScan.enabled` is also set. `CometIcebergWriteExec` is row-based (it emits the commit message row) over a columnar child, so @@ -135,7 +135,7 @@ below it and never inserts the transitions the rest of the plan needs `CometIcebergWriteExec` carries its own serialized plan (`serializedPlanOpt` serializes `nativeOp` on demand), so `CometExecRule` resets `firstNativeOp` at the write and lets its child start a -separate native block. +separate native plan. ## The Eligibility Gate @@ -179,7 +179,7 @@ change. Whether each existing restriction is permanent is tracked in ## What Crosses the Boundary -### JVM to native: the `IcebergWrite` proto +### JVM to Rust: the `IcebergWrite` proto `buildIcebergWriteProto` builds an `IcebergWrite` message (`native/proto/src/proto/operator.proto`) on the driver. Almost all of it is the per-write `IcebergWriteCommon`: @@ -222,7 +222,7 @@ and `_last_updated_sequence_number` to the write schema of copy-on-write DML and `requireNoMetadataColumns` declines any write schema with a field id in the range the spec reserves for metadata columns, so those writes stay on iceberg-java. -### Native execution +### Execution in Rust The planner builds `IcebergWriteExec` (`native/core/src/execution/operators/iceberg_write.rs`) over the FFI scan of the child's batches. `run_write_task` builds iceberg-rust's partitioning writer once @@ -344,7 +344,7 @@ allowlists aligned when adding storage support. Reaching `FileIOBuilder.with_prop` is not proof of support: the builder accepts unknown properties and the backend ignores them. A fallback reason must contain property names only, never values. -### Native to JVM: the task payload +### Rust to JVM: the task payload Each task emits exactly one batch with one row and two `BINARY` columns (`build_output_schema`): @@ -383,8 +383,8 @@ when the JVM polls the output stream to EOF: | Phase | Owner | Mechanism | | ------------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | -| Writing, closing writers, encoding the manifest, building the batch | Native | `TrackingLocationGenerator` records every location. `AbortOnDrop` deletes on error or plan drop and stays armed after yielding the batch. | -| JVM has recorded locations, before its final EOF poll | Native + JVM task | `WrittenFileCleanup` owns the decoded locations while `AbortOnDrop` is still armed. A failure may make both sides attempt best-effort deletion. | +| Writing, closing writers, encoding the manifest, building the batch | Rust | `TrackingLocationGenerator` records every location. `AbortOnDrop` deletes on error or plan drop and stays armed after yielding the batch. | +| JVM has recorded locations, before its final EOF poll | Rust + JVM task | `WrittenFileCleanup` owns the decoded locations while `AbortOnDrop` is still armed. A failure may make both sides attempt best-effort deletion. | | After the JVM's EOF poll, until the task succeeds | JVM task | The EOF poll disarms `AbortOnDrop`; `WrittenFileCleanup` remains registered for the rest of the task. | | Job failure after some tasks completed | Driver (`IcebergCommitExec`) | Calls `BatchWrite.abort` with the completed messages, then deletes their files through the table's `FileIO`. | | Commit failure | Iceberg | `SparkWrite.abort` on the genuine `TaskCommit` messages, the same as the stock path. | @@ -402,8 +402,8 @@ rejected something the gate admitted is a bug in the gate, even if cleanup works ## Matching iceberg-java -Changes to the native path are measured against iceberg-java's output in three tiers, which are the -same tiers the user guide's "accepted divergences" section uses: +Changes to the native writer are measured against iceberg-java's output in three tiers, which are +the same tiers the user guide's "accepted divergences" section uses: 1. **Logical content and manifest metadata must match.** Row data, partition values, record counts, metrics, spec ids and sort order ids. A difference here outlives the write, because later @@ -445,13 +445,13 @@ the writer: run the write suites and the Iceberg Spark tests. The pin policy is | `CometIcebergWriteActionSuite` | End-to-end writes through the split plan and the native writer: parity with iceberg-java, row-level DML, partition evolution, file order, cleanup on task and job failure, writes outgrowing the memory pool, AQE re-planning. | | [`CometIcebergSchedulerFailureSuite`](iceberg-scheduler-failures.md) | Manual multi-host speculation and executor-loss recovery, with scheduler evidence and a manifest-to-storage audit. | | `CometIcebergWriteDetectionSuite` | Most eligibility rules, each accepted and declined. | -| `CometIcebergSystemFunctionSuite` | Native `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort native end to end. | +| `CometIcebergSystemFunctionSuite` | Rust-implemented `bucket`, `truncate`, `years`/`months`/`days`/`hours`, which keep a partitioned write's hash distribution and sort in Comet end to end. | | `CometIcebergRewriteActionSuite` | Iceberg's `rewrite_data_files` with the split plan and the native writer. | | `IcebergWriteProtoTranslationSuite` | Translation of properties into `IcebergParquetWriteSettings` and the writer mode. | | Rust tests in `iceberg_write.rs` and in `iceberg_partition_*.rs` | File rolling on the 1000-row grid, fanout order, clustered input checks, cleanup guard, manifest round trip, memory reservation, partition path rendering, partition values past `chrono`'s calendar. | | Rust tests in `iceberg_dictionary.rs` | The per-column dictionary choice against answers recorded from parquet-mr, including columns either side of its cut-off. | | `iceberg_dictionary_chooser` benchmark in `native/core/benches` | The time one partition's dictionary choice takes, over primitive, struct, array, map and nested columns. | -| `CometIcebergWriteBenchmark` | Native versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. | +| `CometIcebergWriteBenchmark` | The native writer versus iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. It checks each arm's plan before timing it. | | `CometIcebergEndToEndBenchmark` | An Iceberg read, hash repartition and Iceberg write against stock Spark, over primitive, struct, array, map and nested columns. It checks each arm's plan and the rows it wrote before timing it. | The Comet suites run against the Iceberg version each Spark profile pins in `spark/pom.xml`: 1.8.1 @@ -466,7 +466,7 @@ When writing a native-write test: - **Assert that the native writer ran.** Every fallback is silent to the query, so a passing comparison can mean both sides used iceberg-java. Use `assertNativeWriteEngages`, or collect `CometIcebergWriteExec` from the captured plans. Enable the writer with `withNativeEnabled`, which - also turns on `localTableScan` so `INSERT ... VALUES` input is native. + also turns on `localTableScan` so `INSERT ... VALUES` input runs in Comet. - **Compare against iceberg-java, not against expected literals.** Write the same rows through the JVM writer into a sibling table and compare. - **Run plan-shape and transition changes with AQE off as well as on.** The suites run with AQE on diff --git a/docs/source/contributor-guide/memory_management.md b/docs/source/contributor-guide/memory_management.md index 4e7248f2ba3..2d1b044a43c 100644 --- a/docs/source/contributor-guide/memory_management.md +++ b/docs/source/contributor-guide/memory_management.md @@ -80,7 +80,7 @@ page: | -------------------------------------------------- | ----------- | ------------------------------------------------------------- | ----------------- | | Spark execution + storage (on-heap) | JVM heap | `spark.executor.memory` and the unified memory manager | Yes | | Spark Tungsten (off-heap) | Off-heap | `spark.memory.offHeap.size` via `TaskMemoryManager` | Yes | -| Comet native (Rust global allocator) | Native heap | `memory_limit` (see below), enforced only via the memory pool | Reservations only | +| Comet Rust code (global allocator) | Native heap | `memory_limit` (see below), enforced only via the memory pool | Reservations only | | Comet JVM Arrow (`CometArrowAllocator`) | Off-heap | **Nothing**: a `RootAllocator(Long.MaxValue)` | No | | Comet JVM shuffle pages | Off-heap | `spark.memory.offHeap.size` via `TaskMemoryManager` | Yes | | Embedded Python and PyArrow (`python-udf` feature) | Native heap | **Nothing**: CPython's and PyArrow's own allocators | No | @@ -120,13 +120,13 @@ the tasks of an executor that read it at the same time share one such copy. The usage log reports the allocator's total, and the part of it charged to the import allocator described below, so that the overhead can be sized for it. -The opt-in broadcast producer batching path also retains native decoded shuffle arrays outside +The opt-in broadcast producer batching path also retains shuffle arrays decoded in Rust outside the execution pool. Each group is bounded by `min(spark.comet.batchSize, 65536)` rows, 8 MiB of retained/copy estimates, and 1,024 input batches, then concatenated once before Arrow export. Inputs and output coexist during that copy, and one decoded lookahead can be retained. The grouping limit therefore does not cap total reader RSS, and oversized input batches pass through unchanged. -Sharing lowers the bytes that are resident, not the bytes that are reserved. Each task's native +Sharing lowers the bytes that are resident, not the bytes that are reserved. Each task's Comet join reserves its whole build side, because DataFusion skips only the buffers that the same plan has already counted, so with N tasks sharing one copy the pool holds N times the bytes that are resident, and spilling behaves as it did before. The memory usage log's untracked figure, @@ -138,10 +138,10 @@ limit warning, or clamp it to zero. Charging these buffers to Spark's off-heap pool instead was tried in [#5998](https://github.com/apache/datafusion-comet/pull/5998) and dropped. Spark cannot make -Comet's native consumer release anything, and native operators fill a task's share before they -spill, so the JVM allocation that decodes their next input batch is refused first. Refusing it -fails the task where native would have spilled; not refusing it bounds nothing when the pool is -full. It needs native reclaim first; see +`NativeMemoryConsumer` release anything, and Rust-implemented operators fill a task's share before +they spill, so the JVM allocation that decodes their next input batch is refused first. Refusing it +fails the task where those operators would have spilled; not refusing it bounds nothing when the +pool is full. It needs Spark to be able to reclaim memory from the Rust side first; see [#3873](https://github.com/apache/datafusion-comet/issues/3873). One further child, `CometArrowImportAllocator` (`comet-ffi-imports`), is what the Arrow C Data @@ -216,7 +216,7 @@ three ways and `NonFatal` catches only the first. It runs other consumers' `spil `SparkOutOfMemoryError`, which extends `OutOfMemoryError` and is therefore an `Error`. Separately, `ExecutionMemoryPool.acquireMemory` parks in `lock.wait()` when a task is below its fair share, so killing a task raises a plain `InterruptedException`, which `NonFatal` excludes by name. Comet -reaches this method from native over JNI, so whatever escapes crosses the JNI boundary. +reaches this method from the Rust side over JNI, so whatever escapes crosses the JNI boundary. **Spark exposes no per-consumer usage figure.** `TaskMemoryManager.getMemoryConsumptionForThisTask` is task-wide, and a consumer that reaches `acquireExecutionMemory` directly rather than through @@ -321,9 +321,10 @@ JNI, which goes through Spark's ordinary `TaskMemoryManager`. That means: - Spark can force _Spark's_ consumers in the same task to spill to satisfy Comet's request. - The reverse does not hold. Comet registers a `NativeMemoryConsumer` with the `TaskMemoryManager` so that Spark has something to charge, but its `spill()` always returns `0` - (`CometTaskMemoryManager.java`). A Spark allocation can therefore never make a native sorter or - aggregate release its reservations. Native operators spill only when their _own_ `try_grow` - fails, so a JVM consumer that is blocked behind native reservations has no way to reclaim them. + (`CometTaskMemoryManager.java`). A Spark allocation can therefore never make a Comet sorter or + aggregate release its reservations. Rust-implemented operators spill only when their _own_ + `try_grow` fails, so a JVM consumer that is blocked behind Comet's reservations has no way to + reclaim them. - In `try_grow`, a partial grant (`acquired < additional`) is released immediately and reported as `ResourcesExhausted`, which is the signal DataFusion uses to spill. - `grow` cannot fail, so when Spark grants less than asked the pool records the full amount anyway @@ -333,11 +334,11 @@ JNI, which goes through Spark's ordinary `TaskMemoryManager`. That means: is refused unless Spark can cover both, so operators spill until the debt is repaid. Both pools' `Display` output and their `try_grow` errors report the current overcommit. -`CometFairMemoryPool` exists because Spark cannot ask a native operator to spill. Spark limits each -task to an even share of its off-heap pool among the tasks that hold memory from it. Once one -operator holds its task's share, Spark refuses the task's other operators, and one that cannot -spill, such as a sort's merge or the build side of a hash join, fails the task. The pool keeps -memory for those operators by limiting the ones that can spill to a share of the task's, as +`CometFairMemoryPool` exists because Spark cannot ask a Rust-implemented operator to spill. Spark +limits each task to an even share of its off-heap pool among the tasks that hold memory from it. +Once one operator holds its task's share, Spark refuses the task's other operators, and one that +cannot spill, such as a sort's merge or the build side of a hash join, fails the task. The pool +keeps memory for those operators by limiting the ones that can spill to a share of the task's, as DataFusion's `FairSpillPool` does over its whole pool ([issue #7088](https://github.com/apache/datafusion-comet/issues/7088)). @@ -364,8 +365,8 @@ differences go in these directions: tasks than Spark does, the task's share is more than Spark will grant it, and the shares of its consumers that can spill can add up to more than Spark grants. That is [issue #7088](https://github.com/apache/datafusion-comet/issues/7088) again, so look here first - when a stage that mixes JVM and native operators still fails a sort's merge. -- A task that holds a native pool but no memory, such as a native scan that feeds a JVM shuffle, + when a stage that mixes JVM and Rust-implemented operators still fails a sort's merge. +- A task that holds a Comet pool but no memory, such as a native scan that feeds a JVM shuffle, is in Comet's count and not Spark's. Comet then divides by more tasks than Spark does, and its consumers that can spill spill earlier than Spark requires. Look here when a mixed workload spills more under `fair_unified` than under `greedy_unified`. @@ -398,7 +399,7 @@ The pool goes by what a consumer declares, with one exception. DataFusion 55.2 r `OrderedFinalAggregateStream` as able to spill even when its input is sorted on every grouping key, as a workaround for [apache/datafusion#17334](https://github.com/apache/datafusion/issues/17334), but builds a spill path only for partially sorted input. Fully sorted, as in a `CometSortAggregateExec` -over a native sort, a refusal fails the query, so the pool checks its requests only against the +over a Comet sort, a refusal fails the query, so the pool checks its requests only against the pool, as it does those of a consumer that cannot spill (`limited_to_its_share` in `fair_pool.rs`). It still counts as a consumer that can spill when the pool divides the task's share, so a sort below it stays at half of the share and leaves the aggregate the other half. The partially sorted @@ -434,8 +435,8 @@ Three details matter for tuning: ### Task-shared pools and their lifetime A single Spark task can run more than one native plan at a time. A native shuffle is not one of -these cases, because its writer is planned together with the native operators that feed it. These -operators do split a task's native work into separate plans: +these cases, because its writer is planned together with the operators that feed it. These +operators do split a task's work into separate native plans: - `CometUnionExec` and `CometCoalesceExec` read their children through the JVM, so the native plan above them and the native plans below them are separate. @@ -477,7 +478,7 @@ removing it. ## How DataFusion consumes the pool -Native operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservation` API: +Rust-implemented operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservation` API: - `try_grow(n)` may fail. Spillable operators (`ExternalSorter`, the grouped hash aggregate, sort-merge join) respond to a `ResourcesExhausted` error by spilling to disk and retrying. This is @@ -538,7 +539,7 @@ out of the share that the task's operators that can spill divide. ## Crossing the FFI boundary -Batches move between the JVM and native over the Arrow C Data and C Stream interfaces, which are +Batches move between the JVM and Rust over the Arrow C Data and C Stream interfaces, which are zero-copy. Nothing is copied, so the _allocator_ that produced a batch and the _runtime_ that decides when it dies can be on opposite sides of the boundary. See [Arrow FFI](ffi.md) for the mechanics; what matters here is who is charged and who controls the lifetime. @@ -548,11 +549,11 @@ accounting (if any) saw the bytes appear. Whether the buffer is _reserved_ in Co separate decision, made by whichever operator holds it, and that operator neither knows nor cares which side of the boundary the bytes came from. -**JVM → native (`ScanExec`).** The JVM allocates the Arrow buffers from a child of +**JVM → Rust (`ScanExec`).** The JVM allocates the Arrow buffers from a child of `CometArrowAllocator` and exports the whole per-partition iterator once as an `ArrowArrayStream`. `ScanExec` imports each batch through arrow-rs's `ArrowArrayStreamReader` and keeps every column as an `Arc` clone of the imported buffers; the JVM decodes dictionaries before export, so there is -nothing to unpack. Those bytes stay where Java Arrow put them and are pinned for as long as any native +nothing to unpack. Those bytes stay where Java Arrow put them and are pinned for as long as any Rust reference survives. They are invisible to Spark's `TaskMemoryManager`, and `CometArrowAllocator` is unbounded, so nobody charged for them at allocation time. Whether they are charged _later_ depends on who holds them. DataFusion's `ExternalSorter` reserves `get_reserved_bytes_for_record_batch` for @@ -562,7 +563,7 @@ buffers. So an imported batch that a sort or join has buffered _is_ reserved in through a unified pool is charged to Spark. The same batch held by an operator that does not reserve (a projection or filter, or `ScanExec` itself between polls) is charged to nothing. -**Native → JVM (`CometExecIterator`).** DataFusion produces the batch in Rust, and whether it is +**Rust → JVM (`CometExecIterator`).** DataFusion produces the batch in Rust, and whether it is still reserved when it reaches the boundary depends on the producing operator. The sort's output is wrapped in a `ReservationStream` that shrinks the reservation by the batch's size as each batch is emitted, so by the time `prepare_output` sees a sorted batch its bytes are no longer reserved. @@ -728,9 +729,9 @@ much they matter: - **Buffer and reservation lifetimes are independent across the FFI boundary.** A batch can be resident on either side with no reservation covering it, because reservations are made and withdrawn by individual operators while the bytes outlive them. -- **Spark cannot trigger native spilling.** `NativeMemoryConsumer.spill()` returns `0`, so native - reservations are released only when a native operator decides to spill on its own failed - `try_grow`. JVM consumers in the same task can be starved behind them. +- **Spark cannot make Rust-implemented operators spill.** `NativeMemoryConsumer.spill()` returns + `0`, so Comet's reservations are released only when such an operator decides to spill on its own + failed `try_grow`. JVM consumers in the same task can be starved behind them. [Issue #4576](https://github.com/apache/datafusion-comet/issues/4576) tracks work on the first two. diff --git a/docs/source/contributor-guide/native_shuffle.md b/docs/source/contributor-guide/native_shuffle.md index db2eb0d7fb3..01d69afd5da 100644 --- a/docs/source/contributor-guide/native_shuffle.md +++ b/docs/source/contributor-guide/native_shuffle.md @@ -225,7 +225,7 @@ blocks. `DecodedBatchCoalescer` groups eligible small decoded batches before FFI `Utils.serializeBatches`. The read RDD is copied with the same dependency and AQE partition specifications, so other consumers of a reused shuffle keep their original reader. Grouping is bounded as described in [Memory Management](memory_management.md). Task completion releases -pending native batches, exported Arrow batches, and the raw stream. +batches still pending in the Rust coalescer, exported Arrow batches, and the raw stream. **JVM decode**, used when direct read does not apply: diff --git a/docs/source/contributor-guide/plugin_overview.md b/docs/source/contributor-guide/plugin_overview.md index a522f5fa6b2..212636fd14d 100644 --- a/docs/source/contributor-guide/plugin_overview.md +++ b/docs/source/contributor-guide/plugin_overview.md @@ -89,12 +89,12 @@ conversion produces. Each phase is described below. ### Phase 1: CometScanRule -`CometScanRule` replaces any Parquet scans with Comet operators. There are different paths for Spark v1 and v2 data sources. +`CometScanRule` replaces supported scans with Comet operators. There are different paths for Spark v1 and v2 data sources. -When reading from Parquet v1 data sources, Comet replaces `FileSourceScanExec` with a `CometScanExec`, and for v2 -data sources, `BatchScanExec` is replaced with `CometBatchScanExec`. In both cases, Comet replaces Spark's Parquet -reader with a custom vectorized Parquet reader. This is similar to Spark's vectorized Parquet reader used by the v2 -Parquet data source but leverages native code for decoding Parquet row groups directly into Arrow format. +When reading from Parquet v1 data sources, Comet replaces `FileSourceScanExec` with a `CometScanExec`, and for CSV +and Iceberg v2 data sources, `BatchScanExec` is replaced with `CometBatchScanExec`. Operator conversion then turns +these into native scans. The native Parquet scan reads files with DataFusion's Parquet reader, which decodes Parquet +row groups directly into Arrow format. Comet only supports a subset of data types and will fall back to Spark's scan if unsupported types exist. Comet can still accelerate the rest of the query execution in this case because `CometSparkToColumnarExec` will diff --git a/docs/source/contributor-guide/profiling.md b/docs/source/contributor-guide/profiling.md index 1c4faa5a869..5316e276bb3 100644 --- a/docs/source/contributor-guide/profiling.md +++ b/docs/source/contributor-guide/profiling.md @@ -260,7 +260,7 @@ Here is the flamegraph for this example: ### Use wall-clock profiling to spot JNI boundary overhead When profiling Comet with async-profiler, `wall` mode is often more revealing than `cpu` -because it captures time spent crossing the JNI boundary, waiting for native results, +because it captures time spent crossing the JNI boundary, waiting for results from the Rust side, and blocked on I/O — none of which show up in CPU-only profiles. ```shell @@ -276,7 +276,7 @@ objects are created during data transfer between JVM and native code: $ASYNC_PROFILER_HOME/bin/asprof -e alloc -d 60 -f alloc-profile.html ``` -Look for allocations in `CometExecIterator` (native to JVM output), `CometArrowStream` (JVM to native +Look for allocations in `CometExecIterator` (Rust-to-JVM output), `CometArrowStream` (JVM-to-Rust input export), and Arrow vector classes. ### Isolate Rust-only performance issues @@ -293,4 +293,4 @@ cargo flamegraph --root --bench In async-profiler flame graphs, native Rust frames appear below JNI entry points like `Java_org_apache_comet_Native_*`. Look for these transition points to understand how -time is split between Spark's JVM code and Comet's native execution. +time is split between Spark's JVM code and Comet's Rust code. diff --git a/docs/source/contributor-guide/release_process.md b/docs/source/contributor-guide/release_process.md index e0820d2fead..b00158c14ce 100644 --- a/docs/source/contributor-guide/release_process.md +++ b/docs/source/contributor-guide/release_process.md @@ -349,7 +349,7 @@ git push apache 0.13.0-rc1 #### A note on workspace cleanliness The `build-release-comet.sh` script does not build Comet from your local checkout. It resolves the requested -branch to a single commit when it starts, and builds the Docker images, the native binaries (inside the Docker +branch to a single commit when it starts, and builds the Docker images, the native libraries (inside the Docker containers) and the jars from fresh clones of the repository at that commit. The script itself is the only file it uses from your checkout, so uncommitted changes and stale build output elsewhere in your working tree cannot end up in the release artifacts (see [#2232](https://github.com/apache/datafusion-comet/issues/2232) for how a stale @@ -376,7 +376,7 @@ The `build-release-comet.sh` script will create a docker image for each architec to build the platform specific binaries. These builder images are created every time this script is run. The branch or tag to build is required, and the script optionally allows overriding the repository. For a release, always build from the release candidate tag created in the previous step. The tag is resolved to a single commit when -the script starts, and the builder images, the native binaries and the final uber jar are all built from that commit. +the script starts, and the builder images, the native libraries and the final uber jar are all built from that commit. Only the script itself comes from your local checkout, so the branch you have checked out does not matter. ```shell diff --git a/docs/source/contributor-guide/roadmap.md b/docs/source/contributor-guide/roadmap.md index 2724910e85f..2adccb4fe7a 100644 --- a/docs/source/contributor-guide/roadmap.md +++ b/docs/source/contributor-guide/roadmap.md @@ -24,7 +24,7 @@ helpful to have a roadmap for some of the major items that require coordination ## Window Expressions -Native window execution runs by default (`spark.comet.exec.window.enabled`). The ranking functions (`rank`, +Comet runs window operators by default (`spark.comet.exec.window.enabled`). The ranking functions (`rank`, `dense_rank`, `row_number`, `percent_rank`, `cume_dist`, `ntile`), value functions (`lag`, `lead`, `nth_value`, `first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated. Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, `variance`, @@ -39,16 +39,16 @@ supported functions, frames, and fallback cases. [#4834]: https://github.com/apache/datafusion-comet/issues/4834 [#4835]: https://github.com/apache/datafusion-comet/issues/4835 -## Native Lambda Evaluation +## Rust Lambda Evaluation Spark supports higher-order functions on arrays and maps that take a lambda, including `transform`, `exists`, `forall`, `aggregate`, `zip_with`, `map_filter`, and `map_zip_with`. Comet evaluates these today through a JVM codegen-dispatch bridge (`CometScalaUDF`, `CometBatchKernelCodegen`) instead of falling back to Spark, but the -lambda body is still interpreted row-at-a-time on the JVM rather than natively in DataFusion. DataFusion added -native higher-order function support (`array_transform`, `array_filter`, `array_any_match`, etc.) that Comet +lambda body is still interpreted row-at-a-time on the JVM rather than in DataFusion. DataFusion added +higher-order function support (`array_transform`, `array_filter`, `array_any_match`, etc.) that Comet does not yet use; it's not yet known whether their semantics are Spark-compatible. We'll explore whether DataFusion's implementations can replace the JVM codegen-dispatch bridge to remove that round-trip and let -these expressions benefit from vectorized native execution. +these expressions benefit from vectorized execution in Rust. ## Native Coverage for Codegen-Dispatched Expressions @@ -57,7 +57,7 @@ functions, for example) route through the same JVM codegen-dispatch bridge by de DataFusion or `datafusion-spark` implementation has known semantic differences from Spark, or because no native implementation exists yet. The [expression reference] records which expressions are codegen-dispatched today. The bridge covers only scalar expressions, so an aggregate function with no Spark-compatible native implementation falls -back to Spark instead. We're exploring closing these gaps so that more expressions run natively by default, which +back to Spark instead. We're exploring closing these gaps so that more expressions run in Rust by default, which would remove JVM round-trips beyond those the lambda work above addresses. [expression reference]: ../user-guide/latest/expressions.md @@ -100,9 +100,9 @@ core DataFusion operators (joins, aggregates, window) made in support of Comet a ## Spillable Hash Join -Comet's native hash join currently requires the build side to fit entirely in memory. Adding spill-to-disk +Comet's Rust-implemented hash join currently requires the build side to fit entirely in memory. Adding spill-to-disk support ([#2545]) will allow Comet to handle larger joins without falling back to Spark, improving both reliability -and performance for memory-intensive workloads. Comet's native hash join uses DataFusion's `HashJoinExec`, and a +and performance for memory-intensive workloads. Comet's hash join uses DataFusion's `HashJoinExec`, and a design for spilling in that operator, behind a flag that is off by default, is proposed upstream in [datafusion #24768]. @@ -136,7 +136,7 @@ enabled by default ([#1625]). ## Iceberg Table Writes -Comet can write Iceberg tables natively, controlled by one setting, `spark.comet.write.iceberg.enabled`, which is +Comet can write Iceberg tables in Rust, controlled by one setting, `spark.comet.write.iceberg.enabled`, which is off by default. Comet splits Spark's Iceberg V2 write operator into separate writer and committer operators, so the query feeding the write becomes visible to AQE and to Comet's columnar rules ([#4658]), and delegates each task's Parquet write to `iceberg-rust` when the write passes an eligibility check ([#5361]); writes that don't pass keep diff --git a/docs/source/contributor-guide/s3-credential-provider-design.md b/docs/source/contributor-guide/s3-credential-provider-design.md index 416e2f8e50f..2c4ce7a735d 100644 --- a/docs/source/contributor-guide/s3-credential-provider-design.md +++ b/docs/source/contributor-guide/s3-credential-provider-design.md @@ -23,7 +23,7 @@ This page captures why the `org.apache.comet.cloud.s3.CometS3CredentialProvider` ## The gap the SPI fills -Comet's native scan paths (`object_store` for raw Parquet, `opendal` via `iceberg-rust` for Iceberg) bypass Spark's Hadoop S3A code path. That means credentials cannot flow through any of the contracts that vendors typically wire into for S3A: +Comet's native scans (`object_store` for raw Parquet, `opendal` via `iceberg-rust` for Iceberg) bypass Spark's Hadoop S3A code path. That means credentials cannot flow through any of the contracts that vendors typically wire into for S3A: - `org.apache.spark.deploy.security.cloud.CloudCredentialsProvider` yields a single JWT per service name. No path argument, no AWS credential. - Hadoop S3A custom signers hide path-aware logic inside `Signer.sign(request, credentials)`. The credential never leaves the signing pipeline, and the underlying secret is an HMAC key that is not present in the signed output, so running the signer against a synthesized request cannot recover it. @@ -51,7 +51,7 @@ Comet caches one provider instance per `(FQCN, dispatchKey, catalogProperties)` - `catalogProperties` enters the key to handle multi-tenant JVMs (Spark Connect, Thrift Server, `SparkSession.newSession()`) where two sessions can configure the same provider class against the same `dispatchKey` but with different REST endpoints, OAuth tokens, or vendor keys. Without it the second session would silently use the first session's credentials. - Keying solely by FQCN would force vendors to encode multi-tenant routing in static state. The triple-key keeps each call site independent. -`ensureInitialized` returns a `long` handle that the native bridge stashes and replays on every per-request call. Routing per-request lookups by handle avoids re-sending the property bag across JNI on the hot path and unambiguously selects the right provider when the same `(FQCN, dispatchKey)` pair maps to multiple instances. +`ensureInitialized` returns a `long` handle that the Rust bridge stashes and replays on every per-request call. Routing per-request lookups by handle avoids re-sending the property bag across JNI on the hot path and unambiguously selects the right provider when the same `(FQCN, dispatchKey)` pair maps to multiple instances. ## Why fresh construction in `initialize`, not probing a JVM-wide static @@ -59,7 +59,7 @@ A provider implementation might be tempted to probe an existing static populated - The driver JVM hits `S3AFileSystem.initialize` during analysis (raw `s3a://` paths) or during Hadoop catalog manifest reads (Iceberg with Hadoop catalog), so the static is populated there. - The driver may not hit `S3AFileSystem` at all under Iceberg with REST catalog plus `S3FileIO`, because `S3FileIO` calls AWS SDK directly without going through the Hadoop layer. The static stays null. -- Executors with Comet-only reads never instantiate `S3AFileSystem`. The data path is `object_store` (raw Parquet) or `opendal` via `iceberg-rust` (Iceberg native scan). Neither touches Hadoop S3A. The static stays null on every executor. +- Executors with Comet-only reads never instantiate `S3AFileSystem`. The data path is `object_store` (raw Parquet) or `opendal` via `iceberg-rust` (native Iceberg scan). Neither touches Hadoop S3A. The static stays null on every executor. Constructing a fresh provider from `catalogProperties` plus `SparkEnv` is the only strategy that works across all four cases. The trade-off is that on the driver (and any JVM where Hadoop S3A is also active), two credential caches now exist for the same identity: one inside the Hadoop signer's provider, one inside the SPI implementation's. The vendor pays for this with a small number of extra AS round-trips on cold starts and TTL boundaries. A future optional optimization could probe the static first and reuse if non-null, falling back to fresh construction otherwise. @@ -78,7 +78,7 @@ The bridge used to forward every call, which left each path with a different and - object_store asks for a credential on every request and has no notion of expiry, so the Parquet path called the provider once per HTTP request and used a credential however close it was to expiring. object_store also signs a request once and sends that signature again on every retry, for up to 3 minutes by default, so a retry after backoff could arrive after the session token expired, and a 403 is not retried. - iceberg-rust builds a new opendal operator, and with it a new reqsign signer, for every storage call, so reqsign's own cache lasts one operation, and a scan called the provider at least once per data and delete file, even with one-hour tokens. -The reuse window is not a tuning knob. It follows the expiry the vendor reports, and a vendor that wants every request to reach it reports `0`. Five minutes is the refresh-ahead of Comet's other credential caches, the native Parquet credential chain and the IRSA web-identity provider, and it covers object_store's retries. A vendor whose credentials can be revoked before the expiry it reports should report an earlier one. +The reuse window is not a tuning knob. It follows the expiry the vendor reports, and a vendor that wants every request to reach it reports `0`. Five minutes is the refresh-ahead of Comet's other credential caches, the native Parquet scan's credential chain and the IRSA web-identity provider, and it covers object_store's retries. A vendor whose credentials can be revoked before the expiry it reports should report an earlier one. ## Location-scoped credentials on the Parquet path @@ -154,7 +154,7 @@ The cache holds the `FileIO` and its bridge. The bridge reuses its credential un ## Property-bag handling on the Parquet path -The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI for the non-adapter native path (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects the object store prefixes of the schemes the scan reads (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present. +The Parquet path forwards the full `fs.s3a.*` config subset as `catalog_properties`, so an SPI provider sees the same `fs.s3a.*` config Spark would. `forward_catalog_properties` in `native/core/src/parquet/objectstore/s3.rs` keeps every `fs.s3a.*` key, including the static credentials (`*.access.key`, `*.secret.key`, `*.session.token`). Forwarding them is deliberate: `AWSCredentialProviderList` skips a provider that throws `NoAwsCredentialsException` and moves to the next entry, so stripping the static keys would let a chain such as `SimpleAWSCredentialsProvider,customProvider` silently resolve through a different provider than Spark, reading data as a different principal. These keys already cross JNI when Comet's Rust code builds the credential chain itself (`build_credential_provider` reads them), so this matches existing behavior rather than widening exposure. Because `NativeConfig.extractObjectStoreOptions` only collects the object store prefixes of the schemes the scan reads (plus the `fs.comet.*` scheme keys), config a provider reads that is not under `fs.s3a.*` -- notably `hadoop.security.credential.provider.path` -- never crosses JNI. `AdapterSupport.toConfiguration` therefore seeds the Hadoop `Configuration` from the executor's own Spark-derived Hadoop conf (`spark.hadoop.*`) and overlays the forwarded `fs.s3a.*` on top, so those keys are present. ## Built-in adapters diff --git a/docs/source/contributor-guide/spark_configs_support.md b/docs/source/contributor-guide/spark_configs_support.md index 92107975baf..9f887fdb7de 100644 --- a/docs/source/contributor-guide/spark_configs_support.md +++ b/docs/source/contributor-guide/spark_configs_support.md @@ -28,12 +28,13 @@ verification has been performed, and any known gaps. The status column uses these values: - **Supported** -- Comet honors every value of the config within the Comet pipeline and - produces results matching Spark. The execution path may be native or use the JVM codegen - dispatcher, as noted for each config. -- **Partial** -- Comet runs natively for some values of the config but falls back to - Spark for others, or runs natively but with documented incompatibilities. -- **Falls back** -- Comet does not run the affected expressions natively under this - config and always defers to Spark. + produces results matching Spark. The execution path may be Rust-implemented or use the JVM + codegen dispatcher, as noted for each config. +- **Partial** -- Comet keeps the affected expressions in the Comet pipeline for some values + of the config but falls back to Spark for others, or keeps them in the Comet pipeline but + with documented incompatibilities. +- **Falls back** -- Comet does not run the affected expressions in the Comet pipeline under + this config and always defers to Spark. - **Unaudited** -- the config's interaction with Comet has not yet been verified. ## Audited Configurations @@ -112,9 +113,9 @@ policy-independent cases and routes policy-sensitive expressions through Spark's expression code via the Arrow-direct codegen dispatcher. In particular, `to_timestamp`, `to_timestamp_ntz`, `to_date`, `try_to_date`, and `try_to_timestamp` are rewritten by Spark to `Cast` or `GetTimestamp` before Comet -sees the plan. Supported casts run natively, while `GetTimestamp` runs through the +sees the plan. Supported casts run in Rust, while `GetTimestamp` runs through the codegen dispatcher and therefore preserves Spark's selected parser policy inside the -Comet pipeline. Other affected expressions use a mixture of native, codegen-dispatch, +Comet pipeline. Other affected expressions use a mixture of Rust-implemented, codegen-dispatch, and fallback paths: - `date_format` is `Compatible` only for a small whitelist of formats under UTC; the @@ -128,14 +129,14 @@ and fallback paths: selected parser policy. - `to_unix_timestamp` routes through the codegen dispatcher. -If a Comet contributor adds native string-format parsing or extends the date_format +If a Comet contributor adds Rust string-format parsing or extends the date_format whitelist, this audit should be revisited and the policy must be honored explicitly. **Test coverage.** `spark/src/test/resources/sql-tests/expressions/datetime/`: - One ConfigMatrix file per originally audited expression covering convergent inputs under `LEGACY,CORRECTED,EXCEPTION` (`*_time_parser_policy.sql`). - `try_to_date`, added in Spark 4.1, has native-coverage tests in + `try_to_date`, added in Spark 4.1, is tested in `try_datetime.sql` but has not yet been added to the parser-policy matrix. - Per-policy files locking in divergent behavior: - `_legacy.sql` -- lenient inputs (single-digit fields, out-of-range values, diff --git a/docs/source/contributor-guide/sql-file-tests.md b/docs/source/contributor-guide/sql-file-tests.md index 78bb8806c95..6fb41ee3b27 100644 --- a/docs/source/contributor-guide/sql-file-tests.md +++ b/docs/source/contributor-guide/sql-file-tests.md @@ -233,8 +233,8 @@ mode controls how results are validated. #### `query` (default mode) -Checks that the query runs natively on Comet (not falling back to Spark) and that results -match Spark exactly. +Checks that the query runs in Comet (not falling back to Spark) and that results match Spark +exactly. ```sql query @@ -243,7 +243,7 @@ SELECT abs(v) FROM test_abs #### `query spark_answer_only` -Only checks that Comet results match Spark. Does not assert that the query runs natively. +Only checks that Comet results match Spark. Does not assert that the query runs in Comet. Use this for expressions that Comet may not fully support yet but should still produce correct results. @@ -285,18 +285,18 @@ SELECT unsupported_func(v) FROM test_table Checks results and coverage like a plain `query`, and additionally asserts how Comet evaluated the named expressions. -Comet runs an expression either natively (a DataFusion expression) or through the JVM codegen +Comet runs an expression either in Rust (a DataFusion expression) or through the JVM codegen dispatcher (Spark's own `doGenCode` compiled into an Arrow batch kernel). Both produce Spark-matching results, so a plain `query` cannot tell them apart. Use these modes on fixtures where the mechanism is the point of the test, typically an expression whose support depends on its argument type. ```sql --- BinaryType has no native path and must route through the dispatcher -query expect_dispatch(bit_length) -SELECT bit_length(b) FROM test_bit_length_binary +-- lower routes through the dispatcher unless spark.comet.caseConversion.enabled=true +query expect_dispatch(lower) +SELECT lower(s) FROM test_lower --- StringType must stay on the native path +-- bit_length of a string runs in Rust query expect_native(bit_length) SELECT bit_length(s) FROM test_bit_length ``` @@ -306,8 +306,8 @@ Names are comma-separated. A name is the expression's `prettyName` lowercased (` expression asserts both that it ran through the expected mechanism and that it did not run through the other one. -A query carries one mode, so a query mixing a native and a dispatched expression has to be split -into two queries, one per mode. +A query carries one mode, so a query mixing a Rust-implemented and a dispatched expression has to +be split into two queries, one per mode. #### `query ignore()` @@ -396,10 +396,10 @@ operators and needs no additional successful query. `NULL`, boundary values, and negative numbers. 5. Add `query` blocks for each expression or behavior to test. Use the default `query` mode - when you expect Comet to run the expression natively. Use `query spark_answer_only` when - native execution is not yet expected. + when you expect the query to run in Comet. Use `query spark_answer_only` when Comet is not + yet expected to run it. - If the expression's serde routes some input types to a native DataFusion expression and + If the expression's serde routes some input types to a DataFusion expression in Rust and others through the JVM codegen dispatcher, use `expect_native(...)` and `expect_dispatch(...)` for those queries. A plain `query` cannot tell the two mechanisms apart, so the split is otherwise untested. @@ -417,8 +417,8 @@ operators and needs no additional successful query. Reach for `expect_native(...)` / `expect_dispatch(...)` whenever the fixture's own comments explain which path an input takes. That comment is a claim about behavior, and these modes are what turn it into a test. Expressions worth annotating are the ones whose support level depends -on argument type or on a config: `round` (float and double dispatch, decimal and integral stay -native), `lower` / `upper` (dispatch by default), and anything mixing in `CodegenDispatchFallback`. +on argument type or on a config: `round` (float and double dispatch, decimal and integral run in +Rust), `lower` / `upper` (dispatch by default), and anything mixing in `CodegenDispatchFallback`. A query carries a single mode, so a query that mixes both mechanisms has to be split. That split is usually worth doing on its own: it forces you to say which argument takes which path. @@ -459,12 +459,11 @@ SELECT concat_ws(',', 'hello', 'world') **Note on constant folding:** Normally Spark constant-folds all-literal expressions during planning, so Comet would never see them. However, `CometSqlFileTestSuite` automatically -disables constant folding (by excluding `ConstantFolding` from the optimizer rules), so -all-literal queries are evaluated by Comet's native engine. This means you can use the -default `query` mode for all-literal cases and they will be tested natively just like -column-based queries, subject to other optimizer rewrites such as `NullPropagation` and -Comet's own plan-time literal handling. `ConstantFolding: enabled` opts out of the default -exclusion for fixtures that deliberately test folded literals. +disables constant folding (by excluding `ConstantFolding` from the optimizer rules), so that +rule does not remove all-literal expressions before Comet sees them. Other optimizer rules, such +as `NullPropagation`, and Comet's own plan-time literal handling can still remove them, so an +all-literal query is not guaranteed to exercise the expression in Comet. `ConstantFolding: +enabled` opts out of the default exclusion for fixtures that deliberately test folded literals. #### Cover edge cases diff --git a/docs/source/contributor-guide/sql_error_propagation.md b/docs/source/contributor-guide/sql_error_propagation.md index eb253466b64..71ee167b95b 100644 --- a/docs/source/contributor-guide/sql_error_propagation.md +++ b/docs/source/contributor-guide/sql_error_propagation.md @@ -21,7 +21,7 @@ ## Overview -Apache Comet is a native query accelerator for Apache Spark. It runs SQL expressions in +Apache Comet is a query accelerator for Apache Spark. It runs SQL expressions in **Rust** (via Apache DataFusion) instead of the JVM, which is much faster. But there's a catch: when something goes wrong — say, a divide-by-zero or a type cast failure — the error needs to travel from Rust code all the way back to the Spark/Scala/Java world as a proper @@ -45,7 +45,7 @@ SQL Query (Spark/Scala) │ │ 2. Rust deserializes the plan and registers query contexts ▼ - Native execution (DataFusion / Rust) + Rust execution (DataFusion) │ │ 3. An error occurs (e.g. divide by zero) ▼ @@ -207,7 +207,7 @@ pub(crate) fn create_expr(&self, spark_expr: &Expr, ...) { --- -## Step 3: An Error Occurs During Native Execution +## Step 3: An Error Occurs During Rust Execution During query execution, a Rust expression might encounter something like division by zero. @@ -359,8 +359,8 @@ public final class CometQueryExecutionException extends CometNativeException { ## Step 7: Scala Converts JSON Back to a Real Spark Exception -`CometExecIterator.scala` is the Scala code that drives the native execution. Every time it -calls into the native engine for the next batch of data, it catches +`CometExecIterator.scala` is the Scala code that drives the native plan. Every time it +calls into the Rust side for the next batch of data, it catches `CometQueryExecutionException` and converts it: ```scala @@ -457,7 +457,7 @@ case "BinaryArithmeticOverflow" => ## Step 8: The User Sees a Proper Spark Error -The final exception that propagates out of Spark looks exactly like what native Spark would +The final exception that propagates out of Spark looks exactly like what Spark on its own would produce for the same error, including the ANSI error code and the SQL pointer: ``` diff --git a/docs/source/contributor-guide/timezones.md b/docs/source/contributor-guide/timezones.md index 951b8b93ee5..1f047f793e7 100644 --- a/docs/source/contributor-guide/timezones.md +++ b/docs/source/contributor-guide/timezones.md @@ -24,8 +24,9 @@ aimed at contributors working on datetime expressions, casts, scans, or any nati produces or consumes timestamp columns. For user-facing differences from Spark, see the datetime and cast sections of the [Expression Compatibility](../user-guide/latest/compatibility/expressions/index.md) pages, and the -[Parquet](../user-guide/latest/parquet.md) notes. Known bugs are tracked in -[#6335](https://github.com/apache/datafusion-comet/issues/6335). +[Parquet](../user-guide/latest/parquet.md) notes. The closed epic +[#6335](https://github.com/apache/datafusion-comet/issues/6335) lists the timezone bugs an audit +found and links the related issues, some of which are still open. The short version: Comet never converts timestamp values at the JVM/native boundary. The session timezone is not part of a value. It travels with each timezone-aware expression and is applied @@ -106,7 +107,8 @@ partitions, so a wrong label disappears at the next stage boundary. `CASE`, `COA cast a branch whose Arrow type differs from the others to a common type (`coerce_branch` in `native/spark-expr/src/conditional_funcs/case_when.rs`). For a timestamp that cast only changes the label, so a mislabelled branch does not fail there either. A test that only projects the result -passes. The label only matters when the result is compared or feeds another native expression. +passes. The label only matters when the result is compared or feeds another Rust-implemented +expression. ## How the session timezone reaches native code @@ -141,8 +143,10 @@ timezone and then return the result labelled `"UTC"`. That applies both to the d DataFusion's own datetime functions take their timezone from the argument's label, or from `datafusion.execution.time_zone`, which Comet leaves unset. Wiring one of them in for a timezone-aware Spark expression therefore evaluates it in UTC, unless the session timezone is -passed explicitly. For example, `to_char` formats in the input's label, which is why `date_format` -runs natively only in UTC sessions. +passed explicitly. For example, `to_char` formats in the input's label, which is why Comet uses it +for `date_format` only in UTC sessions. With +`spark.comet.expression.DateFormatClass.allowIncompatible=true` it uses `to_char` in other sessions +too, and the result is formatted in UTC. ### Parsing timezone IDs @@ -154,8 +158,8 @@ such as `America/Los_Angeles` and fixed offsets written as `+HH`, `+HHMM` or `+H `CometTimeZone.nativeId` in `spark/src/main/scala/org/apache/comet/serde/CometTimeZone.scala` bridges the two. It normalizes the zone first, so an ID whose offset never changes becomes `UTC` for a zero offset, which covers `Z`, `GMT` and `Etc/UTC`, or `+HH:MM` otherwise. A short ID becomes -its region. An offset with seconds, such as `+05:45:30`, has no native spelling, so `nativeId` -returns `None`. The serde then reports the expression as unsupported through +its region. An offset with seconds, such as `+05:45:30`, has no spelling that `Tz::from_str` +accepts, so `nativeId` returns `None`. The serde then reports the expression as unsupported through `CometTimeZone.supportLevel`, and it runs in the codegen dispatcher or falls back to Spark. Code that takes a fast path for UTC should still compare the ID against a fixed list of UTC @@ -168,9 +172,9 @@ Spark converts between instants and local time using the JVM's timezone rules (` code uses chrono-tz, which compiles its own copy of the IANA database into libcomet (`chrono_tz::IANA_TZDB_VERSION`), and its precomputed DST transitions end around 2100. The two can disagree, both for zones whose rules changed between the two database versions and for far-future -timestamps. When the library loads, `NativeBase` compares the native version (`getTzdataVersion`) -with the JVM's and logs a warning if they differ. The user guide describes the effect under -"Timezone Database Versions" on the datetime +timestamps. When the library loads, `NativeBase` compares the native library's IANA database +version (`getTzdataVersion`) with the JVM's and logs a warning if they differ. The user guide +describes the effect under "Timezone Database Versions" on the datetime [expression compatibility](../user-guide/latest/compatibility/expressions/index.md) page. ## Scans @@ -195,7 +199,7 @@ The timestamp-to-timestamp adaptations go through `CometCastColumnExpr` and between `TimestampType` and `TimestampNTZType` would apply the session timezone, and the Parquet reader does not. -Two Spark settings are not handled natively: +Comet's native Parquet scan does not handle two Spark settings: - Comet disables itself for a session with `spark.sql.parquet.int96TimestampConversion=true`, which shifts `INT96` values written by Impala. @@ -216,11 +220,12 @@ not UTC. Several timezone-dependent expressions have no native implementation that is compatible in every session, including `date_trunc`, `date_format`, `from_unixtime`, `from_utc_timestamp`, -`to_utc_timestamp`, `make_timestamp` and `to_timestamp`. Their serdes route the cases the native -path cannot handle through the JVM codegen dispatcher, as they do for a session timezone that -`CometTimeZone.nativeId` cannot express. The dispatcher runs Spark's generated code with the -`timeZoneId` stamped on the expression, so the results match Spark, and its Arrow output is -labelled `"UTC"` like everything else. +`to_utc_timestamp`, `make_timestamp` and `to_timestamp`. Their serdes route the cases that Comet's +Rust code cannot handle through the JVM codegen dispatcher, as they do for a session timezone that +`CometTimeZone.nativeId` cannot express. For `make_timestamp`, and for `to_timestamp` with a +format, that is every case. The dispatcher runs Spark's generated code with the `timeZoneId` +stamped on the expression, so the results match Spark, and its Arrow output is labelled `"UTC"` +like everything else. ## Guidelines @@ -232,7 +237,7 @@ labelled `"UTC"` like everything else. - Never label a `TimestampType` value with the session timezone, and never return `Timestamp(_, None)` for a `TimestampType` result. - Don't assume `"UTC"` is the only UTC session timezone. `Etc/UTC` is the common default, and a - native path gated on "the session is UTC" must still return `"UTC"`-labelled output there. + Rust fast path gated on "the session is UTC" must still return `"UTC"`-labelled output there. - Don't apply a timezone to a `TimestampNTZType` value, except when converting it to `TimestampType`. @@ -255,7 +260,7 @@ labelled `"UTC"` like everything else. - When collecting timestamps in Scala tests, set `spark.sql.datetime.java8API.enabled=true`, or cast to strings. `java.sql.Timestamp` conversion goes through the JVM's default timezone and the hybrid calendar. -- Check that the expression ran natively, not only that the plan has Comet operators. The codegen +- Check that the expression ran in Rust, not only that the plan has Comet operators. The codegen dispatcher returns Spark's answer from inside a Comet operator, and a dispatched expression takes its whole subtree with it, including children that have a native implementation. `expect_native` in SQL file tests and `checkSparkAnswerAndImpl` in Scala tests check which one ran. diff --git a/docs/source/contributor-guide/tracing.md b/docs/source/contributor-guide/tracing.md index 8acdbe841ae..efc2bd4e65d 100644 --- a/docs/source/contributor-guide/tracing.md +++ b/docs/source/contributor-guide/tracing.md @@ -131,7 +131,7 @@ excess that grows over the run, as it does from 26 MB to 171 MB above, is the si memory the pool is not accounting for. Arrow memory on the JVM side is reported separately, because it is off-heap and so invisible to -`jvm_heap_used`. Comet imports batches from native over the Arrow C Data Interface, and Arrow +`jvm_heap_used`. Comet imports batches from the Rust side over the Arrow C Data Interface, and Arrow charges a buffer to whichever allocator owns it, so those imports are taken against a dedicated child allocator and reported as `jvm_arrow_imported`, within the `jvm_arrow_allocated` total. The executor's periodic memory usage log reports the same two figures. @@ -150,8 +150,8 @@ Ownership and allocation come apart in both directions: For the same reason neither counter is a count of unique physical bytes, and `jvm_arrow_imported` is not guaranteed to be included in `native_allocated`. Usually the producer is Rust and the bytes -are counted in both, but a buffer that Comet exported to native and that native passed back by -reference is imported without the Rust allocator ever having handed it out. Finally, the two +are counted in both, but a buffer that Comet exported to the Rust side and that the Rust side passed +back by reference is imported without the Rust allocator ever having handed it out. Finally, the two counters are separate reads of process-wide state, so concurrent tasks can change them between samples: they are not an atomic per-query balance, and neither is a measure of RSS. @@ -161,7 +161,7 @@ samples: they are not an atomic per-query balance, and neither is a measure of R | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | jvm_heap_used | JVM heap memory usage of live objects for the executor process | | jemalloc_allocated | Native memory usage for the executor process (requires `jemalloc` feature) | -| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator tree on the JVM, including buffers imported from native over the Arrow C Data Interface | +| jvm_arrow_allocated | Bytes charged to Comet's Arrow allocator tree on the JVM, including buffers imported from the Rust side over the Arrow C Data Interface | | jvm_arrow_imported | Bytes charged to the Arrow C Data Interface import allocator, a subset of `jvm_arrow_allocated`. An allocator charge, not a measure of where the bytes were allocated; see above. | | native_allocated | Bytes handed out by the Rust global allocator, process-wide. Approximate to within 64 KiB of un-flushed delta per live thread. | | comet_memory_reserved_total | Total memory reserved across every live Comet memory pool, process-wide, whatever the configured pool type. Counts a pool shared between execution contexts once, so unlike the per-thread counters it can be compared directly against an allocation counter. | diff --git a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md index 5cc435c6b09..ce00930f00e 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md +++ b/docs/source/user-guide/latest/compatibility/expressions/_category_template/cast.md @@ -22,12 +22,12 @@ under the License. Cast operations in Comet fall into three levels of support: - **C (Compatible)**: The results match Apache Spark -- **I (Incompatible)**: Comet's native cast may produce different results from Apache Spark for some inputs. By +- **I (Incompatible)**: Comet's Rust cast may produce different results from Apache Spark for some inputs. By default, Comet instead runs Spark's own code-generated cast inside the Comet pipeline, so the results match Spark. - Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the native cast, but this is not recommended + Setting `spark.comet.expression.Cast.allowIncompatible=true` selects the Rust cast, but this is not recommended for production use. -- **U (Unsupported)**: Comet does not provide a native version of this cast, so it runs Spark's own code-generated - cast inside the Comet pipeline. +- **U (Unsupported)**: Comet does not provide a native implementation of this cast, so it runs Spark's own + code-generated cast inside the Comet pipeline. - **N/A**: Spark does not support this cast. Incompatible and unsupported casts fall back to Spark only when Comet's codegen dispatcher cannot handle them (for @@ -98,16 +98,16 @@ Supported input formats match Spark exactly: ## Date to Timestamp -Comet's native `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast interprets each -date as midnight in the session timezone and converts to a UTC epoch value. DST transitions -are handled correctly, including spring-forward gaps (where midnight may not exist) and -fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's +Comet's native implementation of `CAST(date AS TIMESTAMP)` is compatible with Spark. The cast +interprets each date as midnight in the session timezone and converts to a UTC epoch value. DST +transitions are handled correctly, including spring-forward gaps (where midnight may not exist) +and fall-back ambiguity (where Comet picks the earlier/DST occurrence, matching Spark's `LocalDate.atStartOfDay(zoneId)` behavior). ## Date to TimestampNTZ -Comet's native `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The cast is -timezone-independent: each date is converted to midnight as pure arithmetic +Comet's native implementation of `CAST(date AS TIMESTAMP_NTZ)` is compatible with Spark. The +cast is timezone-independent: each date is converted to midnight as pure arithmetic (`days * 86,400,000,000` microseconds) with no session timezone offset applied. The result is the same regardless of the session timezone setting. @@ -115,8 +115,8 @@ is the same regardless of the session timezone setting. In Legacy mode, `CAST(date AS INT)`, `CAST(date AS LONG)`, and casts to all other numeric types (Boolean, Byte, Short, Float, Double, Decimal) always return `NULL`. Comet handles -this by short-circuiting to a null literal during query planning, so no native execution -is needed. In ANSI and Try modes, Spark rejects these casts at analysis time (before +this by short-circuiting to a null literal during query planning, so no cast runs at +execution time. In ANSI and Try modes, Spark rejects these casts at analysis time (before execution reaches Comet). The short-circuit applies only to a top-level cast. See [Complex Types](#complex-types) for `DATE` values inside structs and maps. @@ -154,7 +154,7 @@ The result is always a wall-clock timestamp with no timezone conversion or DST a ## TimestampNTZ Casts -Comet supports the following `TIMESTAMP_NTZ` casts natively: +Comet supports the following `TIMESTAMP_NTZ` casts in Rust: | Cast | Compatible | Notes | | ---------------------------------- | ---------- | ----------------------------------------------------------------- | @@ -171,8 +171,8 @@ the same result regardless of the session timezone. ## Date to String -Comet's native `CAST(date AS STRING)` is compatible with Spark. Years below 1000 are -zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are +Comet's native implementation of `CAST(date AS STRING)` is compatible with Spark. Years below +1000 are zero-padded to four digits (e.g. year 999 renders as `0999-01-01`). Years above 9999 are rendered without truncation. The cast is timezone-independent. ## Decimal with Negative Scale to String @@ -180,7 +180,7 @@ rendered without truncation. The cast is timezone-independent. Casting a `DecimalType` with a negative scale to `StringType` is marked as incompatible when `spark.sql.legacy.allowNegativeScaleOfDecimal` is `false` (the default). When that config is disabled, Spark cannot create negative-scale decimals, so by default Comet runs Spark's -code-generated cast instead of the native one. +code-generated cast instead of the Rust one. When `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, the cast is compatible. Comet matches Spark's behavior of using Java `BigDecimal.toString()` semantics, which produces scientific @@ -195,11 +195,11 @@ integer types to a negative-scale `DecimalType`. Comet reports these as unsuppor routes them through the JVM codegen dispatcher (Spark's own generated code) inside the Comet pipeline, so the results match Spark exactly. -The native kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which +The Rust kernels align scale by computing `10^|scale|` in fixed-point arithmetic, which overflows when the scale is negative. See [#5013](https://github.com/apache/datafusion-comet/issues/5013). -All other cast directions to and from negative-scale decimals run natively: `FloatType` and +All other cast directions to and from negative-scale decimals run in Rust: `FloatType` and `DoubleType`, `StringType` in both directions (subject to the config gate described above), casts from a negative-scale decimal to `BooleanType`, and widening to another `DecimalType`. Casts from `BooleanType` to a `DecimalType` route through the dispatcher at any scale, which is @@ -226,12 +226,12 @@ support level as casting `INT` to `BIGINT`. The following casts are exceptions to these rules: -- `ARRAY` can only be cast natively to `ARRAY` or `ARRAY`. Casts to other array +- `ARRAY` can only be cast in Rust to `ARRAY` or `ARRAY`. Casts to other array types are unsupported. - A cast from a `DATE` struct field or map value to a numeric or boolean type is unsupported. In Legacy mode these casts always return `NULL` (see [Date to Numeric Types](#date-to-numeric-types)). - When `spark.sql.legacy.castComplexTypesToString.enabled=true`, casts from arrays, structs, and - maps to `STRING` are unsupported, because Comet's native cast only produces Spark's default + maps to `STRING` are unsupported, because Comet's Rust cast only produces Spark's default string format. In Try mode, a cast between maps fails with an error when a key cannot be cast, where Spark returns @@ -243,8 +243,9 @@ casts that Spark accepts under `TRY_CAST`, such as narrowing `BIGINT` to `INT`. Casting a `DecimalType` with a negative scale, which Spark only creates when `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, to `ByteType`, `ShortType`, `IntegerType` -or `LongType` is not supported natively in any eval mode. The native kernels divide by -`10^scale`, which overflows for a negative scale, so Comet leaves the cast to Spark. +or `LongType` is not supported in Rust in any eval mode. The Rust kernels divide by +`10^scale`, which overflows for a negative scale, so Comet runs Spark's own code-generated cast +instead. ## Legacy Mode diff --git a/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md b/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md index 410652c9489..3edc8694b87 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md +++ b/docs/source/user-guide/latest/compatibility/expressions/_category_template/datetime.md @@ -19,10 +19,10 @@ under the License. # Date/Time Expressions -- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native path is marked Incompatible and - routes through the JVM codegen dispatcher by default, producing Spark-identical results. The - native path is itself correct for dates within chrono-tz's DST horizon (approximately year 2100; - see "Date and Time Functions" below) and can be enabled by setting +- **TruncTimestamp (date_trunc)**: In non-UTC sessions the native implementation is marked + Incompatible and routes through the JVM codegen dispatcher by default, producing Spark-identical + results. The native implementation is itself correct for dates within chrono-tz's DST horizon + (approximately year 2100; see "Date and Time Functions" below) and can be enabled by setting `spark.comet.expression.TruncTimestamp.allowIncompatible=true`. TimestampNTZ inputs are handled correctly regardless of session timezone (timezone-independent truncation). @@ -44,10 +44,10 @@ If you need to process dates far in the future with accurate timezone handling, ### Timezone Database Versions -Comet's native code converts between instants and local time with the IANA timezone database that +Comet's Rust code converts between instants and local time with the IANA timezone database that chrono-tz compiles into the Comet library. Spark uses the JVM's timezone database (`tzdb.dat`), whose version depends on the JDK build and on whether its timezone data has been updated. When the two versions -have different rules for a timezone, local times that Comet computes natively for that timezone can differ +have different rules for a timezone, local times that Comet computes in Rust for that timezone can differ from Spark's for the affected dates. That covers `hour`, casts between timestamps and strings or dates, `date_trunc`, and parsing strings as timestamps. Comet logs a warning at startup when the two versions differ. diff --git a/docs/source/user-guide/latest/compatibility/expressions/_category_template/map.md b/docs/source/user-guide/latest/compatibility/expressions/_category_template/map.md index 2acbfa88a63..aff3f997dc6 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/_category_template/map.md +++ b/docs/source/user-guide/latest/compatibility/expressions/_category_template/map.md @@ -23,7 +23,7 @@ under the License. Spark 4.0 inserts `MapSort` to normalize map values when they appear in shuffle hash partitioning keys, in `try_element_at`, and in other contexts where map ordering must be deterministic. Comet -runs `MapSort` natively, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. +runs `MapSort` in Rust, so map shuffle and group-by-on-map stay on Comet under Spark 4.0. When `spark.comet.exec.strictFloatingPoint=true`, `MapSort` falls back to Spark for maps whose keys contain `Float` or `Double`. Arrow's sort uses diff --git a/docs/source/user-guide/latest/compatibility/expressions/index.md b/docs/source/user-guide/latest/compatibility/expressions/index.md index dbfe346b2f8..defc6ba5690 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/index.md @@ -33,7 +33,7 @@ Spark 4.2 ``` Expressions that are not 100% Spark-compatible fall back to Spark by default, except those -with a JVM codegen-dispatch path, which stay in Comet's native pipeline and match Spark +with a JVM codegen-dispatch path, which stay in the Comet pipeline and match Spark exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name, to run Comet's native implementation despite its differences from Spark. See the diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md index db11f250837..f244565c31b 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md @@ -21,7 +21,7 @@ under the License. Compatibility notes for Comet running on Apache Spark 3.5. Expressions that are not 100% Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch -path, which stay in Comet's native pipeline and match Spark exactly. Set +path, which stay in the Comet pipeline and match Spark exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name, to run Comet's native implementation despite its differences from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md) diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md index 371da9a49fb..6fe06eb2fdc 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md @@ -21,7 +21,7 @@ under the License. Compatibility notes for Comet running on Apache Spark 4.0. Expressions that are not 100% Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch -path, which stay in Comet's native pipeline and match Spark exactly. Set +path, which stay in the Comet pipeline and match Spark exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name, to run Comet's native implementation despite its differences from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md) diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md index aaca6b485b6..c722983d383 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md @@ -21,7 +21,7 @@ under the License. Compatibility notes for Comet running on Apache Spark 4.1. Expressions that are not 100% Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch -path, which stay in Comet's native pipeline and match Spark exactly. Set +path, which stay in the Comet pipeline and match Spark exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name, to run Comet's native implementation despite its differences from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md) diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/index.md index 2fe5d698654..fd3e93c5e84 100644 --- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/index.md +++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.2/index.md @@ -21,7 +21,7 @@ under the License. Compatibility notes for Comet running on Apache Spark 4.2. Expressions that are not 100% Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch -path, which stay in Comet's native pipeline and match Spark exactly. Set +path, which stay in the Comet pipeline and match Spark exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name, to run Comet's native implementation despite its differences from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md) diff --git a/docs/source/user-guide/latest/compatibility/floating-point.md b/docs/source/user-guide/latest/compatibility/floating-point.md index e43f602ea9a..f211d97dfe1 100644 --- a/docs/source/user-guide/latest/compatibility/floating-point.md +++ b/docs/source/user-guide/latest/compatibility/floating-point.md @@ -24,7 +24,7 @@ However, one exception is comparison. Spark does not normalize NaN and zero when because they are handled well in Spark (e.g., `SQLOrderingUtil.compareFloats`). But the comparison functions of arrow-rs used by DataFusion do not normalize NaN and zero (e.g., [arrow::compute::kernels::cmp::eq](https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html#)). For `FLOAT` and `DOUBLE` comparisons (`=`, `<>`, `<=>`, `<`, `<=`, `>` and `>=`), Comet -compares the operands in Spark's order natively, including noncanonical NaNs: `-0.0` equals +compares the operands in Spark's order in Rust, including noncanonical NaNs: `-0.0` equals `0.0`, all NaNs are equal, and NaN sorts above every other value. This applies wherever a comparison appears: projections, filters, aggregate arguments and `FILTER` clauses, join conditions, sort keys, and generator arguments. @@ -56,7 +56,7 @@ or structs; see [#6019](https://github.com/apache/datafusion-comet/issues/6019). ## Nested comparisons and membership -For arrays and structs containing `FLOAT` or `DOUBLE`, native `=`, `<>`, `IN`, and `NOT IN` +For arrays and structs containing `FLOAT` or `DOUBLE`, Comet's `=`, `<>`, `IN`, and `NOT IN` compare signed zeros as equal and all NaN representations as equal, matching Spark. This also covers single-candidate membership that Spark rewrites into equality. `<=>`, `<`, `<=`, `>`, and `>=` follow Spark's order too, in which NaN sorts above every other value. @@ -73,25 +73,25 @@ Spark's `ORDER BY`, `RANK`, `DENSE_RANK`, and window frame comparisons route thr define `-0.0 == 0.0`. NaN sorts above every non-NaN value. For `FLOAT` and `DOUBLE` keys, and for keys that nest them in arrays and structs at any depth, -Comet normalizes NaNs and signed zeros before native sorting, window peer comparisons, and -`WindowGroupLimitExec` rank comparisons. Native range partitioning normalizes its keys and +Comet normalizes NaNs and signed zeros before sorting, window peer comparisons, and +`WindowGroupLimitExec` rank comparisons. Native shuffle's range partitioning normalizes its keys and sampled boundaries in the same way; it only accepts scalar keys. Only comparison keys are normalized; returned values retain their original NaN representations and zero signs. Because those comparison keys match Spark, `spark.comet.exec.strictFloatingPoint=true` does not force a fallback for them: sort keys, window and rank order keys, and range partitioning keys all -stay native under strict mode, whether the floats in them are scalar or nested. +stay in Comet under strict mode, whether the floats in them are scalar or nested. That includes a key that nests floats in an array or struct whose type can hold a null element or field. Spark orders such a null below every other value, whatever the key's `NULLS FIRST` or `NULLS LAST`. Comet keeps the key's root null placement separate from its nested comparison, so -sorts and ranking windows stay native under either null order +sorts and ranking windows stay in Comet under either null order ([#6476](https://github.com/apache/datafusion-comet/issues/6476)). A `RANGE` window frame that has to find a row's peers over such a key still falls back to Spark ([#6477](https://github.com/apache/datafusion-comet/issues/6477)). The [operator compatibility notes](operators.md) describe both. -`array_min` and `array_max` use Spark-compatible native comparisons in both strict and non-strict +`array_min` and `array_max` use Spark-compatible Rust comparisons in both strict and non-strict floating-point modes. Signed zeros compare equal, and all NaN representations compare equal and greater than non-NaN values. The original first equal element is retained: for example, `array_min(array(0.0D, -0.0D))` returns `0.0`, while reversing those elements returns `-0.0`. @@ -104,7 +104,7 @@ The `min` and `max` aggregates, including over window frames, and `greatest` and `FLOAT` and `DOUBLE` values in the same order: NaN is larger than every other value and `-0.0` equals `0.0`. Of equal values the first one wins, as in Spark: `greatest(-0.0D, 0.0D)` returns `-0.0`, and `max` returns whichever zero it reads first. `greatest` and `least` follow the same -order inside arrays and structs. These run natively in strict floating-point mode too. +order inside arrays and structs. These run in Comet in strict floating-point mode too. Spark treats `greatest` and `least` as commutative when it matches expressions, so a projection that holds both `greatest(a, b)` and `greatest(b, a)` evaluates only one of them and returns the @@ -114,18 +114,18 @@ the two can differ. ## Array distinct and union When their element type contains `FLOAT` or `DOUBLE`, `array_distinct` and `array_union` run -natively on Spark 4.0.5+, 4.1.4+, and 4.2+. These releases treat `-0.0` and `0.0`, and every NaN +in Rust on Spark 4.0.5+, 4.1.4+, and 4.2+. These releases treat `-0.0` and `0.0`, and every NaN representation, as one value in these functions, including inside nested arrays and structs, and return the normalized value. Spark 4.2.0 normalizes the arguments in the plan (SPARK-54918), and 4.0.5, 4.1.4, and 4.2.1 normalize while these functions evaluate (SPARK-59602). Comet normalizes the elements the same way. Spark 3.5, 4.0.0 to 4.0.4, and 4.1.0 to 4.1.3 keep positive and negative zero distinct in flat arrays, so on those versions these functions fall back to Spark. -Other element types remain native on every version. +Other element types run in Rust on every version. Elements that hold a string with a non-`UTF8_BINARY` collation fall back on every version, including a struct with a `DOUBLE` and a `STRING COLLATE UTF8_LCASE` field. Spark compares such strings under their collation, so `'a'` and `'A'` are one value under `UTF8_LCASE`, while the -native kernels compare their bytes. +Rust kernels compare their bytes. Where these functions fall back, the check is based on the element type, not the values. It also applies to NULL or empty floating-point arrays and columns that never contain negative zero. The @@ -135,22 +135,22 @@ unrelated expressions in the same projection out of Comet. For example, This can have a substantial cost. A local Spark 4.1.3 benchmark of `sum(cardinality(array_distinct(d)))` over two million `array` rows found the default -projection fallback about 15 times slower than native opt-in (best of five runs). -The slowdown depends on the workload. +projection fallback about 15 times slower than opting in to the native implementation (best of +five runs). The slowdown depends on the workload. Setting `spark.comet.expression.ArrayDistinct.allowIncompatible=true` or -`spark.comet.expression.ArrayUnion.allowIncompatible=true` restores native execution on the older -versions, but signed-zero results may differ from Spark. Native execution treats `-0.0` and `0.0` -as one value and returns `0.0`. The older versions keep both zeros in a flat floating-point array, -and inside nested arrays or structs they return whichever zero came first. Only opt in if these -differences are acceptable for your data. +`spark.comet.expression.ArrayUnion.allowIncompatible=true` restores the native implementation on the +older versions, but signed-zero results may differ from Spark. The native implementation treats +`-0.0` and `0.0` as one value and returns `0.0`. The older versions keep both zeros in a flat +floating-point array, and inside nested arrays or structs they return whichever zero came first. +Only opt in if these differences are acceptable for your data. The check uses the Spark version number, not the changes a build contains. A vendor build that reports an older version but includes SPARK-54918 or SPARK-59602 still falls back. ## `arrays_overlap` -`arrays_overlap` runs natively on every Spark version and follows that version's rule for `FLOAT` +`arrays_overlap` runs in Rust on every Spark version and follows that version's rule for `FLOAT` and `DOUBLE` elements in a flat array. Spark 4.0.5+, 4.1.4+, and 4.2+ treat `-0.0` and `0.0` as one value (SPARK-54918, SPARK-59602), and earlier releases keep them apart. Every NaN representation is one value on every version, and nested arrays and structs compare their elements in Spark's order, @@ -165,7 +165,7 @@ is a zero of opposite sign, where Spark returns `true`. `array_contains` compares `FLOAT` and `DOUBLE` elements as Spark does: `-0.0` equals `0.0`, and all NaN representations are equal, inside nested arrays and structs too. The result keeps Spark's three-valued form: null when nothing matches and the array holds a null element. Flat arrays run a -native kernel; nested float elements go through the codegen dispatcher. +Rust kernel; nested float elements go through the codegen dispatcher. `array_remove` compares `FLOAT` and `DOUBLE` elements as Spark does: `-0.0` equals `0.0`, and all NaN representations are equal, inside nested arrays too. The elements it keeps retain their @@ -175,11 +175,11 @@ original NaN representations and zero signs. representations tie, and keeps equal elements in their original order. In an array whose elements can be null, such as one built from nullable columns, `-0.0` and `0.0` therefore tie. Sorting an array whose elements cannot be null in ascending order, Spark's generated code puts `-0.0` before -`0.0`, and so does Comet. All three expressions run natively in strict floating-point mode. +`0.0`, and so does Comet. All three expressions run in Comet in strict floating-point mode. ## Map construction -`map_from_arrays` and `map_from_entries` don't build a map natively when its key type holds a +`map_from_arrays` and `map_from_entries` don't build a map in Rust when its key type holds a `FLOAT` or `DOUBLE` at any depth. They run Spark's code through the codegen dispatcher instead, so duplicate keys, including NaNs and zeros of either sign, behave as in Spark. `FLOAT` and `DOUBLE` values don't affect either function. The expressions' entries in the diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md index dca663d9228..734cf67b1bb 100644 --- a/docs/source/user-guide/latest/compatibility/index.md +++ b/docs/source/user-guide/latest/compatibility/index.md @@ -30,10 +30,10 @@ This guide documents areas where Comet's behavior is known to differ from Spark. - **Regular expressions**: differences between the Rust regexp crate and Java's regex engine. - **Operators**: operator-level compatibility notes, including window functions and round-robin partitioning. - **Expressions**: per-expression compatibility notes, including cast. -- **JSON**: choosing between the native and Spark-compatible engines for JSON expressions. +- **JSON**: choosing between the Rust and Spark-compatible engines for JSON expressions. - **Spark versions**: version-specific known issues and limitations. -## Compatible by default, opt in to native +## Compatible by default, opt in to native implementations Comet runs a Spark-compatible implementation of every supported expression by default. Some expressions also have a native implementation that can differ from Spark for certain @@ -41,17 +41,17 @@ inputs. These are not used unless you opt in by setting the relevant `spark.comet.expression..allowIncompatible=true` config (a few use a dedicated config, noted per expression below), after which you accept the documented differences. -You can discover where a native opt-in is available for a specific query in the verbose extended -explain output. A `[COMET-INFO: ...]` segment points at an available native path and does not mean -the operator falls back to Spark. This is distinct from `[COMET: ...]`, which records a reason an -operator did fall back. +You can discover where an opt-in native implementation is available for a specific query in the +verbose extended explain output. A `[COMET-INFO: ...]` segment points at an available native +implementation and does not mean the operator falls back to Spark. This is distinct from +`[COMET: ...]`, which records a reason an operator did fall back. ## Native and codegen-dispatch implementations Some Spark expressions have two implementations in Comet: - A **codegen-dispatch** implementation that runs Spark's own generated code for the - expression inside Comet's native pipeline (via the Arrow-direct codegen dispatcher). This + expression inside the Comet pipeline (via the Arrow-direct codegen dispatcher). This produces byte-exact Spark results at the cost of one JNI round-trip per batch. It is gated globally by `spark.comet.exec.scalaUDF.codegen.enabled` (enabled by default); when the dispatcher is disabled, these expressions fall back to Spark. @@ -59,10 +59,10 @@ Some Spark expressions have two implementations in Comet: known semantic differences from Spark for some inputs or patterns. Because the codegen-dispatch path matches Spark exactly, Comet uses it by **default**. The -native path is **opt-in per expression** via that expression's +native implementation is **opt-in per expression** via that expression's `spark.comet.expression..allowIncompatible=true` flag, which declares that you -accept its differences from Spark. There is no global opt-in. When the native path is enabled -but a specific input or pattern has no native implementation, Comet routes that case back +accept its differences from Spark. There is no global opt-in. When the native implementation is +enabled but a specific input or pattern has no native implementation, Comet routes that case back through the codegen dispatcher rather than running something incompatible. This is the model behind the [regular expression](regex.md) and [JSON](json.md) families, @@ -70,15 +70,15 @@ which document their per-expression configs and the specific differences to expe This is distinct from expressions that have **no** codegen-dispatch path: there, the incompatible cases fall back to Spark by default, and `allowIncompatible=true` runs the native -(incompatible) path instead. Aggregate functions such as `mode` are the main example, because the -codegen dispatcher covers only scalar expressions; see the +(incompatible) implementation instead. Aggregate functions such as `mode` are the main example, +because the codegen dispatcher covers only scalar expressions; see the [expression reference](../expressions.md) for which expressions have incompatible cases. ## Strings with non-UTF-8 bytes Spark's `StringType` can hold arbitrary bytes, including sequences that are not valid UTF-8 (for example `CAST(X'FF' AS STRING)`). Arrow's string type requires valid UTF-8, so Comet cannot store -the raw bytes natively. When Comet produces a string from arbitrary bytes (such as +the raw bytes in an Arrow string. When Comet produces a string from arbitrary bytes (such as `CAST(binary AS string)`, a columnar shuffle, or a nested Parquet BINARY value read as a string), it decodes them the same way the JVM does (`new String(bytes, UTF_8)`), replacing each ill-formed sequence with the Unicode replacement @@ -126,7 +126,7 @@ a result: repartitioning a Latin-1 Text table can return `U+FFFD` where Spark keeps the original bytes. Setting `spark.comet.shuffle.mode=native` keeps such shuffles on Spark. - **`CAST(binary AS STRING)`.** Comet decodes the bytes where Spark keeps them. Because `min` and `max` - over strings run natively, a cast below them is decoded before the comparison, and the two can swap. + over strings run in Comet, a cast below them is decoded before the comparison, and the two can swap. With `b` holding `X'80'` and `X'C3A9'`, Spark returns `min(CAST(b AS STRING))` = the `0x80` string and `max` = `é`, whereas Comet returns `min` = `é` and `max` = `U+FFFD`. This is the ordering difference above. If the cast input can hold bytes that are not valid UTF-8, disable Comet for the @@ -135,10 +135,10 @@ a result: `spark.comet.shuffle.mode=native`. All three are needed: with only the first two, the aggregate runs in Spark, but Comet's columnar shuffle still decodes its partial results. These settings are not enough when the cast is in a projection below the aggregate, for example - `SELECT min(s) FROM (SELECT CAST(b AS STRING) AS s FROM t)`, because Comet runs that projection - natively and decodes the bytes before the aggregate sees them. + `SELECT min(s) FROM (SELECT CAST(b AS STRING) AS s FROM t)`, because that projection runs in + Comet and decodes the bytes before the aggregate sees them. -Consistent handling of invalid UTF-8 across all native string paths is tracked by +Consistent handling of invalid UTF-8 across all of Comet's string paths is tracked by [#4764](https://github.com/apache/datafusion-comet/issues/4764). Separately, Comet's native Parquet scan currently rejects string columns whose stored bytes are not @@ -147,7 +147,7 @@ valid UTF-8 rather than reading them like Spark ## ANSI errors from rows skipped by LIMIT -Native operators evaluate batches of rows. With `spark.sql.ansi.enabled=true`, some `LIMIT` +Comet operators evaluate batches of rows. With `spark.sql.ansi.enabled=true`, some `LIMIT` plan shapes can therefore raise an expression error from a row that Spark stops before evaluating. For example, with a Parquet batch read in the order `(id, value) = (0, '0'), (1, '1')`: @@ -173,7 +173,7 @@ evaluate that row, for example to sort the input or process an `OFFSET` prefix. ## ANSI-mode error classes and messages -Under `spark.sql.ansi.enabled=true`, several native error paths raise the error at the correct +Under `spark.sql.ansi.enabled=true`, several Rust error paths raise the error at the correct input but with a different exception type, error class, SQLSTATE, or message text than Spark. Code that catches `SparkException` and only asserts on message substrings is unaffected; code that inspects the exception class, `getCondition()`, or the parameterised error class will observe @@ -184,18 +184,18 @@ divergence: ## Known result-value divergences -The following native paths can return different values or raise errors that Spark does not raise -for edge-case inputs. +The following Rust-implemented paths can return different values or raise errors that Spark does +not raise for edge-case inputs. Most also have entries in the per-category expression pages linked above; they are collected here so users hunting an unexpected value have a single place to check. Parquet and Iceberg reads and writes have their own lists, under Known Issues on the [Parquet](../parquet.md#known-issues) and [Iceberg](../iceberg.md#known-issues) pages: -- Native `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from - Spark when the boundary arithmetic overflows for `DATE` or `DECIMAL` `ORDER BY` columns +- Comet's `RANGE` window frames with an explicit `PRECEDING` / `FOLLOWING` offset diverge from + Spark when the boundary arithmetic overflows for a `DATE` `ORDER BY` column ([#5022](https://github.com/apache/datafusion-comet/issues/5022)). - Grouped ANSI decimal `AVG` can evaluate an overflowing group in a batch even when Spark would - stop before that group at `LIMIT`. It remains natively enabled with this error-timing difference; + stop before that group at `LIMIT`. It still runs in Comet with this error-timing difference; disabling `spark.comet.exec.aggregate.enabled` restores Spark's aggregate evaluation. On final result overflow, Comet's error message prints the average rounded to the result scale; Spark 4.0 prints the quotient truncated to 39 fractional digits. diff --git a/docs/source/user-guide/latest/compatibility/json.md b/docs/source/user-guide/latest/compatibility/json.md index 2dc79c27a89..5be0569b756 100644 --- a/docs/source/user-guide/latest/compatibility/json.md +++ b/docs/source/user-guide/latest/compatibility/json.md @@ -28,29 +28,29 @@ Comet can evaluate JSON expressions (`get_json_object`, `from_json`, `to_json`, batch. This rides the codegen dispatcher (`spark.comet.exec.scalaUDF.codegen.enabled`, enabled by default); if the dispatcher is disabled, the operator falls back to Spark. -- **Native (rust) path:** the native DataFusion implementation. Faster, but has +- **Rust path:** Comet's Rust implementation. Faster, but has known compatibility gaps with Spark on certain inputs, so it is **opt-in per expression** via the expression's `allowIncompatible` config. Any expression or - input case with no native implementation falls back to the codegen dispatcher. + input case with no native implementation routes through the codegen dispatcher. ## Expression coverage -| SQL | Native (rust) path | Opt-in config | +| SQL | Rust path | Opt-in config | | ------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | | `get_json_object` | Supported, with gaps on single-quoted JSON and unescaped control characters | `spark.comet.expression.GetJsonObject.allowIncompatible` | | `from_json` | Supported with restrictions (PERMISSIVE mode only, simple schema types only) | `spark.comet.expression.JsonToStructs.allowIncompatible` | | `to_json` | Supported for struct inputs only, no options | `spark.comet.expression.StructsToJson.allowIncompatible` | | `json_array_length` | Supported, with gaps on single-quoted JSON, unescaped control characters, and trailing content | `spark.comet.expression.LengthOfJsonArray.allowIncompatible` | -When the native path is enabled but an expression or input case has no native +When the Rust path is enabled but an expression or input case has no native implementation (for example `to_json` with map or array inputs, or `from_json` -with an unsupported schema), Comet falls back to the codegen dispatcher for that -case. +with an unsupported schema), Comet routes that case through the codegen +dispatcher. -## When to use the native path +## When to use the Rust path -- You want the faster native path and your inputs avoid the known compatibility +- You want the faster Rust path and your inputs avoid the known compatibility gaps above. - Enable it per expression, for example - `spark.comet.expression.GetJsonObject.allowIncompatible=true`. Cases the native path - does not cover still fall back to the codegen dispatcher. + `spark.comet.expression.GetJsonObject.allowIncompatible=true`. Cases the Rust path + does not cover still run through the codegen dispatcher. diff --git a/docs/source/user-guide/latest/compatibility/operators.md b/docs/source/user-guide/latest/compatibility/operators.md index e54594bac66..1d899dd2949 100644 --- a/docs/source/user-guide/latest/compatibility/operators.md +++ b/docs/source/user-guide/latest/compatibility/operators.md @@ -22,56 +22,56 @@ under the License. ## Observations `CollectMetricsExec`, used by `DataFrame.observe`, preserves Spark's observation counters while -passing the original Arrow batches between native operators. Metric expressions still execute +passing the original Arrow batches between Comet operators. Metric expressions still execute in Spark over row views of each batch; the data is not copied into a row-based pipeline. Spark retains ownership of accumulator publication, observation listeners, resets and snapshots. -This path requires native input and is enabled by `spark.comet.exec.collectMetrics.enabled`. -Observations retain Spark execution when a consumer can stop reading mid-batch. This includes -limits, joins, `Dataset.mapPartitions`, replacement sampling at a nonpositive rate, and external -actions such as `foreachPartition` or `rdd.take`, which do not introduce a physical-plan node. -The final row iterator is therefore +This path requires input from the Comet pipeline and is enabled by +`spark.comet.exec.collectMetrics.enabled`. Observations retain Spark execution when a consumer can +stop reading mid-batch. This includes limits, joins, `Dataset.mapPartitions`, replacement sampling +at a nonpositive rate, and external actions such as `foreachPartition` or `rdd.take`, which do not +introduce a physical-plan node. The final row iterator is therefore treated as potentially partial, even when the eventual action reads all rows. A local sort or aggregate does not remove this restriction: an external consumer may never request its output, -so Spark may leave its input unread while native stream setup fetches a batch. +so Spark may leave its input unread while stream setup in Rust fetches a batch. A materializing shuffle or broadcast exchange allows columnar observations in its producer -subtree, unless another short-circuiting consumer intervenes. This preserves native observation +subtree, unless another short-circuiting consumer intervenes. This preserves columnar observation in aggregation and shuffle pipelines while keeping counters aligned with Spark's consumption. Direct observations and write pipelines without such an exchange retain Spark collection; writer nodes alone do not enable columnar observations. ## Empty Relations -On Spark 4.0 and later, Comet supports `EmptyRelationExec` as a native input. It is enabled by -default and can be disabled with `spark.comet.exec.emptyRelation.enabled=false`. The operator -preserves Spark's output attributes and zero partitions; the eliminated logical subtree is not -executed. +On Spark 4.0 and later, Comet supports `EmptyRelationExec` as an input to the Comet pipeline. It is +enabled by default and can be disabled with `spark.comet.exec.emptyRelation.enabled=false`. The +operator preserves Spark's output attributes and zero partitions; the eliminated logical subtree +is not executed. -Supported parent joins and aggregates remain eligible for native execution. Global aggregates +Supported parent joins and aggregates can still run in Comet. Global aggregates still return one row (`COUNT = 0`, `SUM = NULL`), and grouped aggregates return no rows. Independent operator restrictions and aggregate buffer compatibility checks still apply. -A native Parquet write over a native empty relation stays native. Like Spark's writer, it runs one -task for the empty input, so the output still gets a schema-only Parquet file that readers can -infer the schema from. +A native Parquet write over a Comet empty relation stays in the Comet pipeline. Like Spark's writer, +it runs one task for the empty input, so the output still gets a schema-only Parquet file that +readers can infer the schema from. ## In-Memory Cache Comet can store cached relations (`df.cache()`, `CACHE TABLE`) in Arrow format and scan them -natively. This is experimental and disabled by default; see [In-Memory Cache](../in-memory-cache.md) -for how to enable it. Comet does not replace a `spark.sql.cache.serializer` that the application -has already set. Relations whose schema Comet's Arrow writer does not support are cached in -Spark's default format, and their scans fall back to Spark. Reads that feed Spark operators rather -than Comet operators can be slower than Spark's cache. +in the Comet pipeline. This is experimental and disabled by default; see +[In-Memory Cache](../in-memory-cache.md) for how to enable it. Comet does not replace a +`spark.sql.cache.serializer` that the application has already set. Relations whose schema Comet's +Arrow writer does not support are cached in Spark's default format, and their scans fall back to +Spark. Reads that feed Spark operators rather than Comet operators can be slower than Spark's cache. With Kryo and `spark.kryo.registrationRequired=true`, Comet needs its Kryo registrator whether or not the cache is enabled; see [Kryo serialization](../installation.md#kryo-serialization). ## Sampling -Comet runs `SampleExec` natively when sampling is performed without replacement, which covers -`DataFrame.sample`, SQL `TABLESAMPLE`, and `DataFrame.randomSplit`. The native implementation +Comet runs `SampleExec` in Rust when sampling is performed without replacement, which covers +`DataFrame.sample`, SQL `TABLESAMPLE`, and `DataFrame.randomSplit`. The Rust implementation reproduces Spark's per-row `XORShiftRandom` draw sequence, so for a given seed it selects the same rows as Spark. @@ -81,7 +81,7 @@ order than Spark, such as a join or an aggregate, the result is still a valid sa expected size, but not necessarily the same rows. Sampling with replacement (`df.sample(withReplacement = true, ...)`) falls back to Spark, because -it draws from a Poisson distribution that Comet does not implement natively +it draws from a Poisson distribution that Comet does not implement in Rust ([#5109](https://github.com/apache/datafusion-comet/issues/5109)). ## Sort @@ -89,7 +89,7 @@ it draws from a Poisson distribution that Comet does not implement natively Spark orders a null element of an array sort key, or a null field of a struct sort key, below every other value, whatever the root key's `NULLS FIRST` or `NULLS LAST`. Comet compares nested values by Spark's order and applies the requested null placement only to the root key. Sort, -TopK, and ranking window order keys run natively under both default and nondefault null orders +TopK, and ranking window order keys run in Comet under both default and nondefault null orders ([#6476](https://github.com/apache/datafusion-comet/issues/6476)). A `RANGE` window frame that searches boundaries over a nullable nested key still runs on Spark ([#6477](https://github.com/apache/datafusion-comet/issues/6477)). @@ -106,19 +106,19 @@ accumulator, following the decimal SUM implementation, is tracked in This fallback applies to the whole aggregate operator: a single wide-decimal `AVG` or `TRY_AVG` also moves sibling aggregates such as `MIN`, `MAX`, `COUNT`, and `SUM` to Spark. Both partial and final aggregate operators stay in Spark, including when a shuffle separates them. Narrower -ungrouped decimal averages remain eligible for native execution. +ungrouped decimal averages can still run in Comet. -Grouped decimal `AVG` remains eligible for native execution with hash aggregation. With Spark's +Grouped decimal `AVG` can still run in Comet with hash aggregation. With Spark's object aggregation, decimal `AVG` and `TRY_AVG` fall back when their intermediate sum has precision 38, because Spark's object buffer can recover after an intermediate overflow. As with the ungrouped -case, the whole aggregate operator falls back. Native finalization can raise an +case, the whole aggregate operator falls back. Comet's finalization can raise an ANSI overflow for a group that Spark would never consume after `LIMIT`; this error-timing difference is documented under [known divergences](index.md#known-result-value-divergences). ## Sort Aggregation -Comet runs `SortAggregateExec` natively when Comet shuffle is enabled and Comet supports every -aggregate in it. The native aggregate keeps the grouping-key output order that Spark relies on. +Comet runs `SortAggregateExec` in Rust when Comet shuffle is enabled and Comet supports every +aggregate in it. Comet's aggregate keeps the grouping-key output order that Spark relies on. Decimal `sum` and `avg` over input precision 28 or more keep a running sum at precision 38, which can overflow even when the final sum fits. Whether Spark's sort aggregation recovers from such an @@ -134,7 +134,7 @@ Both results are valid under Spark's semantics for these functions. ## Hash Join -Spark plans `NOT IN` as a null-aware anti join, and the native hash join builds it on Spark's +Spark plans `NOT IN` as a null-aware anti join, and Comet's hash join builds it on Spark's streamed side without keeping that side's order. When Spark reports the streamed side as ordered, for example the output of a window, a sort-merge join, or a cached table sorted within its partitions, Comet falls back to Spark for the join, so that a sort Spark removed above it on the @@ -143,12 +143,12 @@ strength of that order is not lost. ### Output Order Spark reports the sort order of a hash join's streamed side as the join's own, and drops a sort -above the join that this order satisfies. Comet's native hash joins keep that order where Comet can +above the join that this order satisfies. Comet's hash joins keep that order where Comet can show that it holds. That includes a left or right outer join over a streamed side that was sorted -outside the native plan, such as a sorted relation read from the [in-memory cache](#in-memory-cache). +outside the Rust plan, such as a sorted relation read from the [in-memory cache](#in-memory-cache). -Comet passes the order to the native join as sort keys, and an outer join needs at least the first -key to be one that Comet can express natively. Declaring any leading part of the order keeps the +Comet passes the order to its Rust join as sort keys, and an outer join needs at least the first +key to be one that Comet can express. Declaring any leading part of the order keeps the rows in place, so the join does not need the rest. An outer join whose first sort key Comet cannot express falls back to Spark and records a fallback reason (see [Understanding Comet Plans](../understanding-comet-plans.md)), because Comet cannot tell whether @@ -156,7 +156,7 @@ an operator above the join relies on the order. An outer join falls back to Spark when Comet cannot check the order Spark reports and that order reaches the join through an input from the JVM or through a nested loop join. Spark keeps the rows -in the order they arrive. One such input is an RDD checkpoint. It reaches a native join only with +in the order they arrive. One such input is an RDD checkpoint. It reaches a Comet join only with `spark.comet.convert.rdd.enabled=true`, which defaults to `false`. Without that setting the checkpoint stays a Spark input and the join stays in Spark anyway. Another is a cache written by an outer hash join whose own probe input was not a scan @@ -164,17 +164,17 @@ outer hash join whose own probe input was not a scan join can already be out of order when it reaches the join ([#6898](https://github.com/apache/datafusion-comet/issues/6898)). -When the order Comet cannot check comes only from another native hash join in the same plan, the -join does not fall back. It runs natively and declares no order, since DataFusion keeps the order +When the order Comet cannot check comes only from another Comet hash join in the same plan, the +join does not fall back. It runs in Comet and declares no order, since DataFusion keeps the order it can see there. ## Window Functions -Comet runs `WindowExec` natively and it is enabled by default (`spark.comet.exec.window.enabled`). A broad set of +Comet runs `WindowExec` in Rust and it is enabled by default (`spark.comet.exec.window.enabled`). A broad set of window functions is accelerated, and any shape Comet does not support falls back to Spark rather than producing an incorrect result. When any single window expression in a `WindowExec` falls back, the entire operator runs on Spark. -**Accelerated natively:** +**Accelerated in Comet:** - Ranking functions: `row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`, `ntile`. - Value functions: `lag`, `lead`, `nth_value`, `first_value` (`first`), `last_value` (`last`). `IGNORE NULLS` is @@ -187,15 +187,15 @@ incorrect result. When any single window expression in a `WindowExec` falls back - Aggregate window functions other than the ones listed above, including the statistical aggregates (`stddev`, `stddev_pop`, `stddev_samp`, `var_pop`, `var_samp`, `corr`, `covar_pop`, `covar_samp`). These run - natively as plain aggregations but not as window functions + in Comet as plain aggregations but not as window functions ([#4766](https://github.com/apache/datafusion-comet/issues/4766)). - `min` / `max` on timestamp-without-time-zone, interval, or array input types, and `sum` / `avg` on year-month or day-time interval input types. Windowed aggregates inherit the same input-type support as the batch aggregates, so these fall back in both contexts. -- `min` / `max` on string or binary input types. These run natively as plain aggregations but not as window +- `min` / `max` on string or binary input types. These run in Comet as plain aggregations but not as window functions. - `min` / `max` on struct input types. A struct whose fields are all supported `min` / `max` types other than - `float` and `double` runs natively as a plain aggregation, but not as a window function. Other structs fall + `float` and `double` runs in Comet as a plain aggregation, but not as a window function. Other structs fall back in both contexts. - `sum` or `avg` on `DECIMAL` with a sliding (non ever-expanding) frame, because the sliding path would wrap on overflow instead of returning Spark's `NULL`. @@ -205,39 +205,39 @@ incorrect result. When any single window expression in a `WindowExec` falls back - `RANGE` frame with an explicit offset when the `ORDER BY` column is `DECIMAL` ([#4834](https://github.com/apache/datafusion-comet/issues/4834)). - `RANGE` frame with an offset above 2144550751 days when the `ORDER BY` column is `DATE`. Spark adds the offset - to the date as an `Int` that wraps on overflow, while the native frame treats an overflowing bound as the edge of + to the date as an `Int` that wraps on overflow, while Comet's frame treats an overflowing bound as the edge of the partition. Smaller offsets cannot wrap for any date between 0001-01-01 and 9999-12-31 ([#5022](https://github.com/apache/datafusion-comet/issues/5022)). - `RANGE` frame bounded by `CURRENT ROW` when an `ORDER BY` key is an array of arrays or structs, or a struct holding an array, such as `array(named_struct('x', x))`. DataFusion cannot compare those values to find the frame's bounds ([apache/datafusion#24937](https://github.com/apache/datafusion/issues/24937)). Ranking functions - and `ROWS` frames over the same keys run natively. + and `ROWS` frames over the same keys run in Comet. - `RANGE` frame bounded by `CURRENT ROW` when an `ORDER BY` key is an array or struct whose type can hold a null element or field, such as `array(x)` over a nullable `x`. DataFusion orders such a null above every other value when it looks for the frame's bounds, while the sort puts it first as Spark does, so a frame could run to the end of the partition ([#6477](https://github.com/apache/datafusion-comet/issues/6477)). Ranking functions, `ROWS` - frames, and a key that cannot hold a null element or field, such as `array(coalesce(x, 0))`, run natively. + frames, and a key that cannot hold a null element or field, such as `array(coalesce(x, 0))`, run in Comet. - `first_value` / `last_value` on a `RANGE` frame with a literal offset ([#4835](https://github.com/apache/datafusion-comet/issues/4835)). - `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)). - A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric. - Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize. - Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation (e.g. - `UTF8_LCASE`). The native window finds partitions and peers by comparing keys byte by byte, which splits values + `UTF8_LCASE`). Comet's window finds partitions and peers by comparing keys byte by byte, which splits values that the collation ties. -A `RANGE` frame with an explicit offset on a `DATE` `ORDER BY` runs natively, but over a date outside Spark's +A `RANGE` frame with an explicit offset on a `DATE` `ORDER BY` runs in Comet, but over a date outside Spark's supported range of 0001-01-01 to 9999-12-31, such as one produced by `date_add` with a very large day count, the -native frame can treat the bound as the edge of the partition where Spark compares against it +Comet frame can treat the bound as the edge of the partition where Spark compares against it ([#5022](https://github.com/apache/datafusion-comet/issues/5022)). `WindowGroupLimitExec` (window-based limit pushdown for `ROW_NUMBER`, `RANK`, and `DENSE_RANK`) -runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (default: true). +runs in Rust; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (default: true). **Falls back to Spark:** - Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation - (e.g. `UTF8_LCASE`). The native operator detects partitions and order-key peer groups by + (e.g. `UTF8_LCASE`). Comet's operator detects partitions and order-key peer groups by comparing Arrow row-encoded keys for byte equality, which splits peers that Spark ties. Floating-point `ORDER BY` keys, including floats nested in arrays and structs, are normalized @@ -246,26 +246,26 @@ strict floating-point mode. ## MERGE INTO (MergeRowsExec) -Spark `MergeRowsExec` appears as `CometMergeRows` when native execution is enabled. +Spark `MergeRowsExec` appears as `CometMergeRows` when it runs in Comet. -Comet can run `MergeRowsExec` (Spark's row-level `MERGE INTO` dispatch operator) natively on +Comet can run `MergeRowsExec` (Spark's row-level `MERGE INTO` dispatch operator) in Rust on Spark 3.5+, but it is disabled by default. Enable it with `spark.comet.exec.mergeRows.enabled=true`. On Spark 4.1+, stock V2 writers discover the concrete Spark `MergeRowsExec` to build `MergeSummary`. When a write remains on Spark's V2 writer, Comet therefore keeps that JVM node -even when native MergeRows is enabled. Comet's split Iceberg write path can run MergeRows natively: +even when Comet's MergeRows is enabled. Comet's split Iceberg write path can run MergeRows in Rust: its `IcebergCommit` collects the same eight semantic action counters and forwards them through the summary-aware `BatchWrite.commit` contract. Spark 4.2 uses last-attempt metrics for these counters, matching Spark's retry-aware summary semantics. -**Cardinality validation memory use can exceed Spark's:** native MERGE cardinality validation +**Cardinality validation memory use can exceed Spark's:** Comet's MERGE cardinality validation currently stores matched target row IDs in an unspillable hash set. For MERGEs with many matched rows per task, this can use more memory than Spark's compressed bitmap and may reach the native memory limit earlier than Spark. See [#6608](https://github.com/apache/datafusion-comet/issues/6608). -**Undeclared physical output order can differ from Spark:** native execution is set-at-a-time. Within +**Undeclared physical output order can differ from Spark:** Comet's version is set-at-a-time. Within an input batch it emits rows grouped by the MERGE instruction that produced them, and it processes the MATCHED, NOT MATCHED, then NOT MATCHED BY SOURCE groups. Spark's row-at-a-time implementation emits rows in input order. This is not a MERGE row-value semantic difference: an unordered table @@ -287,7 +287,7 @@ across the whole group and report `DIVIDE_BY_ZERO` first. The same ordering diff between different MERGE groups, between the two projections of a `Split`, or between a cardinality violation and an unrelated clause-evaluation error. In these cases both engines reject the query, but the surfaced Spark error condition can differ. This limitation only applies to Spark versions -where native `MergeRowsExec` is enabled. +where Comet runs `MergeRowsExec`. ## Round-Robin Partitioning diff --git a/docs/source/user-guide/latest/compatibility/regex.md b/docs/source/user-guide/latest/compatibility/regex.md index 6e5bb5dd38a..2bc2e9fba1d 100644 --- a/docs/source/user-guide/latest/compatibility/regex.md +++ b/docs/source/user-guide/latest/compatibility/regex.md @@ -28,16 +28,16 @@ Comet evaluates Spark regular-expression expressions (`rlike`, `regexp_replace`, default (`spark.comet.exec.scalaUDF.codegen.enabled=true`); if the dispatcher is disabled, regex expressions fall back to Spark. This is the default for every regex expression except an in-subset `rlike` literal (see below). -- **Native (rust) engine** — the Rust [`regex`] crate, run natively with no JNI overhead. It is +- **Rust engine** — the Rust [`regex`] crate, which runs with no JNI overhead. It is faster but has different semantics from Java regex (see below). For `rlike`, a plan-time - analyzer admits a conservative subset of `UTF8_BINARY` literal patterns and runs those natively + analyzer admits a conservative subset of `UTF8_BINARY` literal patterns and runs those in Rust **by default**. Out-of-subset default-collation literals still require `spark.comet.expression.RLike.allowIncompatible`. Non-literal patterns, NULL patterns, and non-default collations stay on the codegen dispatcher even with that flag. Every other regex expression still requires its own `allowIncompatible` flag. `regexp_instr` has no native implementation and always runs through the codegen dispatcher. -| SQL | Native (rust) opt-in config | +| SQL | Rust engine opt-in config | | -------------------- | ----------------------------------------------------------- | | `rlike` | `spark.comet.expression.RLike.allowIncompatible` | | `regexp_replace` | `spark.comet.expression.RegExpReplace.allowIncompatible` | @@ -45,12 +45,12 @@ Comet evaluates Spark regular-expression expressions (`rlike`, `regexp_replace`, | `regexp_extract_all` | `spark.comet.expression.RegExpExtractAll.allowIncompatible` | | `split` | `spark.comet.expression.StringSplit.allowIncompatible` | -`allowIncompatible` can force native execution for a non-null default-collation -literal. Non-default collations remain on the JVM dispatcher because the native +`allowIncompatible` can force the Rust engine for a non-null default-collation +literal. Non-default collations remain on the JVM dispatcher because the Rust kernel does not implement Spark collation semantics. It is not needed for in-subset literals, and it does not apply to non-literal or NULL patterns. -When the native path is selected but a case has no native implementation (for example a +When the Rust engine is selected but a case has no native implementation (for example a non-scalar `rlike` pattern, `regexp_replace` with a non-1 offset, or `regexp_extract` with a non-literal pattern or idx), Comet routes that case through the codegen dispatcher. @@ -75,16 +75,16 @@ the engine selector: | | Rust engine | Codegen dispatcher | | -------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | **Compatibility** | In-subset `rlike` literals match Java regex; other patterns may differ (see below) | 100% compatible with Spark | -| **Feature coverage** | `rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all` natively; `regexp_instr` via fallthrough | All regexp expressions (`rlike`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`, `regexp_replace`, `split`) | -| **Performance** | Fully native, no JNI overhead | One JNI round-trip per batch (Arrow vectors stay columnar) | +| **Feature coverage** | `rlike`, `regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all` via Rust; `regexp_instr` via fallthrough | All regexp expressions (`rlike`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`, `regexp_replace`, `split`) | +| **Performance** | Fully in Rust, no JNI overhead | One JNI round-trip per batch (Arrow vectors stay columnar) | | **Pattern support** | Linear-time subset only | All Java regex features (backreferences, lookaround, etc.) | | **`rlike` default** | Used automatically for analyzer-admitted `UTF8_BINARY` literals | Used for every other `rlike` pattern | The **Rust engine** is faster but cannot match Java regex semantics for every pattern. For `rlike`, -Comet therefore runs only the analyzer-admitted subset natively by default. Setting -`spark.comet.expression.RLike.allowIncompatible=true` can force native execution for a non-null +Comet therefore runs only the analyzer-admitted subset in Rust by default. Setting +`spark.comet.expression.RLike.allowIncompatible=true` can force the Rust engine for a non-null default-collation literal that the analyzer cannot prove equivalent, and declares acceptance of -any remaining differences. Non-default collations remain on the JVM dispatcher because the native +any remaining differences. Non-default collations remain on the JVM dispatcher because the Rust kernel does not implement Spark collation semantics. The other regex expressions still require their own `allowIncompatible` flag. @@ -154,11 +154,11 @@ endpoints, and counted, aggregate, nested-group, or nested-quantifier patterns t conservative compile-size budget stay on the Java engine. Any unrecognized construct also stays on the Java engine. -For `regexp_replace`, `split`, `regexp_extract`, and `regexp_extract_all`, the native path is +For `regexp_replace`, `split`, `regexp_extract`, and `regexp_extract_all`, the Rust engine is still opt-in via `allowIncompatible`. If you are confident those patterns fit the same ASCII, non-anchored shape, opting in is generally safe. -Native `rlike` operates on Arrow string values and assumes well-formed UTF-8. +The Rust engine's `rlike` operates on Arrow string values and assumes well-formed UTF-8. Behavior for malformed string data supplied outside Arrow's StringArray invariants has not been established. diff --git a/docs/source/user-guide/latest/datasources.md b/docs/source/user-guide/latest/datasources.md index fdde7d9cd17..99e8b272da5 100644 --- a/docs/source/user-guide/latest/datasources.md +++ b/docs/source/user-guide/latest/datasources.md @@ -30,7 +30,7 @@ issues. ### Apache Iceberg Comet accelerates Iceberg scans of Parquet files and can write the data files of eligible Iceberg -writes natively, an opt-in feature. See the [Iceberg](iceberg.md) page for more information. +writes in Rust, an opt-in feature. See the [Iceberg](iceberg.md) page for more information. ### CSV @@ -65,8 +65,8 @@ above them can run in Comet. Only `spark.comet.convert.oneRowRelation.enabled` i because the row it converts has no columns. - `spark.comet.convert.range.enabled`: `spark.range` and SQL `range()`, for ranges that Comet does - not generate natively with `spark.comet.exec.range.enabled`. -- `spark.comet.convert.inMemoryCache.enabled`: in-memory cached tables that Comet's native cache + not generate in Rust with `spark.comet.exec.range.enabled`. +- `spark.comet.convert.inMemoryCache.enabled`: in-memory cached tables that Comet's cache scan does not read, such as tables cached in Spark's default format. - `spark.comet.convert.rdd.enabled`: a DataFrame created from an RDD of rows, for example with `spark.createDataFrame(rdd, schema)`. @@ -84,9 +84,9 @@ works but is deprecated, like naming `Range`, `InMemoryTableScan`, `RDDScan`, `O `spark.comet.convert.text.enabled` instead. One case is converted automatically: when the build (broadcast) side of a broadcast join is a file -or V2 scan Comet cannot read natively - for example a small ORC lookup table - and every scan on -the probe side is already native, Comet converts that build-side scan to Arrow so the whole join -runs natively over the large probe. This is controlled by +or V2 scan that Comet has no native scan for - for example a small ORC lookup table - and every +scan on the probe side is already a native scan, Comet converts that build-side scan to Arrow so +the whole join runs in Comet over the large probe. This is controlled by `spark.comet.convert.broadcastBuildSide.enabled`, which is **on by default**, unlike the switches above. The build side is usually small, but an explicit `BROADCAST` hint or a selective filter above the scan can make the copied volume larger, and with dynamic partition pruning the dimension is read @@ -440,7 +440,7 @@ Comet's S3 support has the following limitations: 2. **Custom credential providers**: Custom credential provider classes named in `fs.s3a.aws.credentials.provider` are not supported; only the standard providers listed in the table above are. To route credential requests through your own Java code, implement Comet's `CometS3CredentialProvider` SPI; see [S3 Credential Providers](s3-credential-providers.md). Broader Hadoop S3A integration is tracked in [#1829](https://github.com/apache/datafusion-comet/issues/1829). -3. **Settings that Hadoop's S3A client resolves differently**: The native scans resolve storage settings with their own client rather than with Hadoop's. Where the two disagree, a native read can use a different identity or route than Spark and fail where Spark succeeds. Examples are JCEKS-backed secrets and the precedence of per-bucket settings, which the native Parquet scan forwards without checking for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). +3. **Settings that Hadoop's S3A client resolves differently**: The native scans resolve storage settings with their own client rather than with Hadoop's. Where the two disagree, a native scan can use a different identity or route than Spark and fail where Spark succeeds. Examples are JCEKS-backed secrets and the precedence of per-bucket settings, which the native Parquet scan forwards without checking for a difference ([#5662](https://github.com/apache/datafusion-comet/issues/5662)). 4. **AWS profile credential providers**: The native scans ignore `fs.s3a.auth.profile.name` and `fs.s3a.auth.profile.file` ([#4245](https://github.com/apache/datafusion-comet/issues/4245)), and resolve some profiles differently from Hadoop 3.4 and later, for example profiles that use `credential_process` or chains of assumed roles ([#6575](https://github.com/apache/datafusion-comet/issues/6575)). @@ -479,9 +479,9 @@ hadoop-azure must be on the driver classpath. It already is whenever Spark can l | `WorkloadIdentityTokenProvider` with a `fs.azure.account.oauth2.client.assertion.provider.type` (hadoop-azure 3.5.0 and later) | Not supported | | | `UserboundSASWithOAuth` (hadoop-azure 3.5.0 and later) | Not supported | | -Custom provider classes hand each token request to Java code the native scan cannot call, and the refresh token and user password flows have no `object_store` counterpart. When Hadoop selects one of these, the scan falls back to Spark's own scan, which authenticates through hadoop-azure, and the fallback reason names the auth type and, for SAS, OAuth and Workload Identity, the provider class (`Custom` names only the auth type). Configure a supported mechanism for that account to read it natively. Values arrive as Hadoop resolved them, trimmed and defaulted where the ABFS driver trims and defaults them, so the store sees what the driver would have used. +Custom provider classes hand each token request to Java code the native scan cannot call, and the refresh token and user password flows have no `object_store` counterpart. When Hadoop selects one of these, the scan falls back to Spark's own scan, which authenticates through hadoop-azure, and the fallback reason names the auth type and, for SAS, OAuth and Workload Identity, the provider class (`Custom` names only the auth type). Configure a supported mechanism for that account to read it with the native scan. Values arrive as Hadoop resolved them, trimmed and defaulted where the ABFS driver trims and defaults them, so the store sees what the driver would have used. -Client credentials and Workload Identity token requests go over HTTPS only. The managed identity request goes to the IMDS endpoint over HTTP, as Hadoop's does. For client credentials, Hadoop posts to `fs.azure.account.oauth2.client.endpoint` as given, while `object_store` posts to `//oauth2/v2.0/token`. Comet takes the tenant from the path segment before the last `oauth2` and everything before that segment as the authority host, so a v1 `/oauth2/token` endpoint and a proxy with a path prefix both resolve. The token request path is always the v2.0 one. An `http://` endpoint, or one with no tenant segment, makes the scan fall back to Spark's own scan. So does an `http://` Workload Identity authority (`fs.azure.account.oauth2.msi.authority`). The driver decides this by asking the native store whether it can be built from the resolved values, so the scan falls back exactly when the native store would reject them. A native write to such an account fails with an error naming the key. +Client credentials and Workload Identity token requests go over HTTPS only. The managed identity request goes to the IMDS endpoint over HTTP, as Hadoop's does. For client credentials, Hadoop posts to `fs.azure.account.oauth2.client.endpoint` as given, while `object_store` posts to `//oauth2/v2.0/token`. Comet takes the tenant from the path segment before the last `oauth2` and everything before that segment as the authority host, so a v1 `/oauth2/token` endpoint and a proxy with a path prefix both resolve. The token request path is always the v2.0 one. An `http://` endpoint, or one with no tenant segment, makes the scan fall back to Spark's own scan. So does an `http://` Workload Identity authority (`fs.azure.account.oauth2.msi.authority`). The driver decides this by asking the Rust side whether it can build the store from the resolved values, so the scan falls back exactly when the store would reject them. A native write to such an account fails with an error naming the key. ### Environment variables @@ -491,7 +491,7 @@ Once Hadoop names anything, only the resolved values reach the store. A resoluti ### Credentials in transit -The resolved values travel in the native plan from the driver to the executors, as the static `fs.s3a.access.key` and `fs.s3a.secret.key` values of an S3 configuration do. Spark's RPC channel is plaintext by default. `spark.network.crypto.enabled` or `spark.ssl.rpc.enabled` protects it. See [Wire encryption](s3-credential-providers.md#wire-encryption). +The resolved values travel in the serialized plan from the driver to the executors, as the static `fs.s3a.access.key` and `fs.s3a.secret.key` values of an S3 configuration do. Spark's RPC channel is plaintext by default. `spark.network.crypto.enabled` or `spark.ssl.rpc.enabled` protects it. See [Wire encryption](s3-credential-providers.md#wire-encryption). ### Examples @@ -561,12 +561,12 @@ Comet 1.1 started from the environment and overlaid the Hadoop keys it found by ### Azure Limitations -1. **Authentication keys only**: Only the keys in the table above reach the native store. Other `fs.azure.*` settings (read-ahead, retries, HTTP tuning) do not apply to the native scan. +1. **Authentication keys only**: Only the keys in the table above reach the `object_store` client. Other `fs.azure.*` settings (read-ahead, retries, HTTP tuning) do not apply to the native scan. -2. **Supported schemes**: Only `abfs` and `abfss` are routed to the native Azure store. `wasb[s]`, `az`, `azure`, and `adl` are not supported. `wasb[s]` is not recognised by `object_store` at all; `az`, `azure`, and `adl` are recognised by `object_store` but treat the URL host as the _container_ rather than the _account_, which is incompatible with Hadoop's account-scoped configuration keys. +2. **Supported schemes**: Only `abfs` and `abfss` are routed to `object_store`'s Azure store. `wasb[s]`, `az`, `azure`, and `adl` are not supported. `wasb[s]` is not recognised by `object_store` at all; `az`, `azure`, and `adl` are recognised by `object_store` but treat the URL host as the _container_ rather than the _account_, which is incompatible with Hadoop's account-scoped configuration keys. 3. **URL shape**: URLs must include the account in the host, i.e. `abfss://@.dfs.core.windows.net/`. Bare `abfs:///` (fsspec-style, no account in the URL) is not supported because Comet cannot resolve the storage account name. 4. **Endpoint hosts**: `object_store` builds an Azure store only for hosts under `dfs.core.windows.net`, `blob.core.windows.net`, `dfs.fabric.microsoft.com` and `blob.fabric.microsoft.com`. Sovereign-cloud endpoints such as `dfs.core.chinacloudapi.cn` are not supported, whatever keys are configured for them. -5. **Credentials that resolve differently from Hadoop's ABFS driver**: Container-scoped OAuth settings, Fabric hosts and a few other configurations resolve differently from Hadoop's ABFS driver, so a native read can fail where Spark succeeds ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). +5. **Credentials that resolve differently from Hadoop's ABFS driver**: Container-scoped OAuth settings, Fabric hosts and a few other configurations resolve differently from Hadoop's ABFS driver, so a native scan can fail where Spark succeeds ([#6605](https://github.com/apache/datafusion-comet/issues/6605)). diff --git a/docs/source/user-guide/latest/datatypes.md b/docs/source/user-guide/latest/datatypes.md index 061d7ae46bf..710e21ddf83 100644 --- a/docs/source/user-guide/latest/datatypes.md +++ b/docs/source/user-guide/latest/datatypes.md @@ -19,9 +19,9 @@ # Spark Data Type Support -This page is the complete reference for how Apache Comet handles each Spark data type. Comet's -native execution path is built on Apache Arrow, so the set of types Comet can express natively -is constrained by Arrow's type system. When a query references a type Comet does not support, +This page is the complete reference for how Apache Comet handles each Spark data type. The Comet +pipeline is built on Apache Arrow, so the set of types Comet can express is constrained by Arrow's +type system. When a query references a type Comet does not support, the relevant operator falls back to Spark; results are unaffected. For per-scan and per-operator type caveats (for example, Parquet read-time conversions or @@ -31,7 +31,7 @@ hash-aggregate group-key restrictions), see the [Compatibility Guide](compatibil | Status | Meaning | | ---------------------- | --------------------------------------------------------------------------------------- | -| ✅ Supported | Native support; enabled by default. | +| ✅ Supported | Handled in the Comet pipeline; enabled by default. | | ⚠️ Supported (caveats) | Works, but with limits: certain values, contexts, or configurations fall back to Spark. | | 🔜 Planned | Intended; tracked by an open issue or pull request. | @@ -40,7 +40,7 @@ hash-aggregate group-key restrictions), see the [Compatibility Guide](compatibil The following types fall back to Spark and are not on the current roadmap. They are omitted from the tables below and may be reconsidered based on demand: -- **`UserDefinedType`**: user-defined types are application-specific and outside the scope of native acceleration; queries referencing UDTs fall back to Spark. +- **`UserDefinedType`**: user-defined types are application-specific and outside the scope of Comet acceleration; queries referencing UDTs fall back to Spark. ## Numeric @@ -71,17 +71,17 @@ the tables below and may be reconsidered based on demand: ## Datetime -| Type | Status | Notes | -| ------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `DateType` | ✅ | | -| `TimestampType` | ✅ | | -| `TimestampNTZType` | ✅ | | -| `TimeType` | ⚠️ | Spark 4.1+. Native serialization is in place. Parquet scans of `TIME` columns fall back, and some operators (sort, min/max) are still being wired up ([#4288](https://github.com/apache/datafusion-comet/issues/4288)). | +| Type | Status | Notes | +| ------------------ | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `DateType` | ✅ | | +| `TimestampType` | ✅ | | +| `TimestampNTZType` | ✅ | | +| `TimeType` | ⚠️ | Spark 4.1+. Serde support is in place. Parquet scans of `TIME` columns fall back, and some operators (sort, min/max) are still being wired up ([#4288](https://github.com/apache/datafusion-comet/issues/4288)). | ## Interval All three interval types are mapped to Arrow and flow through serde, native shuffle, and the -codegen dispatcher, so interval columns and the interval-producing expressions run natively. +codegen dispatcher, so interval columns and the interval-producing expressions run in Comet. Several operators still gate on the type and fall back: Parquet scans of ANSI interval columns, single-column sorts, hash aggregates (`min` / `max` / `sum` / `avg`), `GROUP BY`, window functions. Remaining work is tracked by @@ -114,7 +114,7 @@ Parquet timestamp inference settings. Support for Spark's whole-value pushdown r by [#5519](https://github.com/apache/datafusion-comet/issues/5519). Nested Variant columns, pushed-down Variant field extraction, expressions, writes, shuffle and spill, Python operators, encrypted files, and Iceberg scans that read a Variant column fall back to Spark. Iceberg scans of tables -whose Variant columns the query does not read run natively. Spark also handles columnar-to-row +whose Variant columns the query does not read run in Comet. Spark also handles columnar-to-row conversion of the native scan output and strict reads with `allowReadingShredded=false`. Broader support is tracked by [#4295](https://github.com/apache/datafusion-comet/issues/4295) and [#3983](https://github.com/apache/datafusion-comet/issues/3983). diff --git a/docs/source/user-guide/latest/delta.md b/docs/source/user-guide/latest/delta.md index 2d8d7872b30..44db6420c46 100644 --- a/docs/source/user-guide/latest/delta.md +++ b/docs/source/user-guide/latest/delta.md @@ -19,7 +19,7 @@ under the License. # Delta Lake (experimental) -Comet can execute DSv1 Delta Lake table scans natively. Reads planned by +Comet can execute DSv1 Delta Lake table scans in Rust. Reads planned by delta-spark run through Comet's native Parquet scan, inheriting row-group pruning, page-index pruning, and filter pushdown, with deletion vectors applied inside the scan. @@ -48,7 +48,7 @@ as `TIMESTAMP_NTZ` keeps the date spec), and a column read as `TIMESTAMP` takes the datetime spec even when the file marks it as not adjusted to UTC. Dates written with the legacy hybrid Julian/Gregorian calendar are rebased exactly, timestamps are rebased exactly when the file records a fixed UTC writer time zone, and ancient values whose calendar cannot be -applied natively (non-UTC legacy writer zones, or files that do not declare a +applied in Rust (non-UTC legacy writer zones, or files that do not declare a policy under the `EXCEPTION` read mode) raise an error rather than silently returning shifted values. Modern values are unaffected: dates from 1582-10-15 onward, and timestamps from 1900-01-01T00:00:00Z onward (Spark's own diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md index 58ba641a2f8..a8bf5888887 100644 --- a/docs/source/user-guide/latest/expressions.md +++ b/docs/source/user-guide/latest/expressions.md @@ -53,7 +53,7 @@ The **Implementation** column records how Comet executes each expression when it | Implementation | Meaning | | --- | --- | | Native | Comet's Rust engine evaluates the expression end to end. | -| Codegen dispatch | Spark's own generated JVM code is evaluated inside the Comet pipeline. Used when a byte-exact match to Spark matters more than the native speedup, or when no native path exists. | +| Codegen dispatch | Spark's own generated JVM code is evaluated inside the Comet pipeline. Used when a byte-exact match to Spark matters more than the speedup of a native implementation, or when no native implementation exists. | | Hybrid | Both paths exist. Comet picks between them based on input, and the user can override with `spark.comet.expression.EXPRNAME.allowIncompatible=true` — see [Native and codegen-dispatch implementations](compatibility/index.md#native-and-codegen-dispatch-implementations). | | — | No direct wire-up: the row is either planned, is rewritten to another expression before Comet sees it (for example `to_date` → `Cast`), or is handled at the operator level rather than the expression level (for example `explode`, window functions). | @@ -62,20 +62,20 @@ The Implementation column is auto-generated from the serde definitions in `Query ## Not currently planned Comet focuses acceleration on mainstream relational, string, datetime, math, and collection -expressions. The following function families are **not currently planned** for native acceleration (they are not on the current roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand: +expressions. The following function families are **not currently planned** for acceleration in Comet (they are not on the current roadmap): specialized functionality with narrow real-world analytics use and high implementation cost. They fall back to Spark and may be reconsidered based on demand: - **Probabilistic sketches and approximate top-k** (`kll_sketch_*`, `theta_*`, `count_min_sketch`, `bitmap_*`, `approx_top_k*`): specialized data structures with exact-correctness traps. - **Geospatial** (`st_*`): brand-new Spark 4.1 functionality, specialized. - **Avro / Protobuf codecs** (`from_avro`, `to_avro`, `from_protobuf`, `to_protobuf`, `schema_of_avro`): format conversion belongs at the IO layer, not expression evaluation. - **JVM reflection** (`java_method`, `reflect`, `try_reflect`): niche, and they invoke arbitrary JVM methods (a security concern). - **UTF-8 validation** (`is_valid_utf8`, `make_valid_utf8`, `validate_utf8`, `try_validate_utf8`): niche Spark 4.x string-validation helpers. -- **Miscellaneous niche** (`histogram_numeric`, `version`, `sentences`, `quote`): low-value or specialized functions with little benefit from native acceleration. +- **Miscellaneous niche** (`histogram_numeric`, `version`, `sentences`, `quote`): low-value or specialized functions with little benefit from acceleration in Comet. The file-metadata functions `input_file_name`, `input_file_block_start`, and `input_file_block_length` depend on scan-internal per-row file information rather than the expression layer; their support status is covered in the [Parquet page](parquet.md#reading). -Note that `median` and `mode` are supported: they are mainstream exact aggregates. `mode` runs natively only with `spark.comet.expression.Mode.allowIncompatible=true`. `approx_count_distinct` is supported because Comet ports Spark's `HyperLogLogPlusPlus` exactly, so its result is bit-identical to Spark. +Note that `median` and `mode` are supported: they are mainstream exact aggregates. `mode` runs in Comet only with `spark.comet.expression.Mode.allowIncompatible=true`. `approx_count_distinct` is supported because Comet ports Spark's `HyperLogLogPlusPlus` exactly, so its result is bit-identical to Spark. -The Apache DataSketches HLL functions (`hll_sketch_agg`, `hll_union_agg`, `hll_sketch_estimate`, `hll_union`) are the one sketch family Comet does accelerate, on Spark 4.0+. The sketches are mutually readable with Spark's, but the point estimate can differ slightly after a merge, so the native path is off by default and opt-in per expression via `allowIncompatible` — see the rows below. +The Apache DataSketches HLL functions (`hll_sketch_agg`, `hll_union_agg`, `hll_sketch_estimate`, `hll_union`) are the one sketch family Comet does accelerate, on Spark 4.0+. The sketches are mutually readable with Spark's, but the point estimate can differ slightly after a merge, so the native implementation is off by default and opt-in per expression via `allowIncompatible` — see the rows below. The tables below list every Spark built-in expression with its current status. @@ -106,34 +106,34 @@ The tables below list every Spark built-in expression with its current status. | `first_value` | ✅ | Native | | | `grouping` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS | | `grouping_id` | ✅ | — | Grouping indicator for ROLLUP/CUBE/GROUPING SETS | -| `hll_sketch_agg` | ✅ | Native | Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible ([details](compatibility/expressions/aggregate.md)) | -| `hll_union_agg` | ✅ | Native | Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible ([details](compatibility/expressions/aggregate.md)) | -| `kurtosis` | 🔜 | — | Not yet implemented natively | +| `hll_sketch_agg` | ✅ | Native | Spark 4.0+ only; falls back by default, the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/aggregate.md)) | +| `hll_union_agg` | ✅ | Native | Spark 4.0+ only; falls back by default, the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/aggregate.md)) | +| `kurtosis` | 🔜 | — | Not yet implemented in Comet | | `last` | ✅ | Native | | | `last_value` | ✅ | Native | | | `listagg` | ✅ | Native | Spark 4.0+. `StringType` input with a literal delimiter; `WITHIN GROUP (ORDER BY ...)` and `BinaryType` inputs fall back to Spark. Without `WITHIN GROUP`, concatenation order is the (non-deterministic) group arrival order, matching Spark. | | `max` | ✅ | Native | | | `max_by` | ✅ | Native | Value and ordering must be fixed-length types | | `mean` | ✅ | Native | Interval types fall back. For `DECIMAL` input with precision 28 or more, fallback depends on the aggregate operator; see [Hash Aggregation](compatibility/operators.md#hash-aggregation) | -| `median` | ✅ | — | Rewrites to `percentile(col, 0.5)` and runs natively for supported percentile inputs | +| `median` | ✅ | — | Rewrites to `percentile(col, 0.5)` and runs in Comet for supported percentile inputs | | `min` | ✅ | Native | | | `min_by` | ✅ | Native | Value and ordering must be fixed-length types | | `mode` | ✅ | Native | `mode(col)` only; Spark breaks ties non-deterministically, so Comet returns the smallest tied value and falls back by default, opt-in via allowIncompatible | -| `percentile` | ✅ | Native | Single literal percentage on numeric input runs natively; array of percentages and a frequency argument fall back to Spark | +| `percentile` | ✅ | Native | Single literal percentage on numeric input runs in Comet; array of percentages and a frequency argument fall back to Spark | | `percentile_approx` | ✅ | Native | Alias of `approx_percentile`; same restrictions apply | -| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark | +| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs in Comet, `DESC` falls back to Spark | | `percentile_disc` | 🔜 | — | Percentile aggregate | | `pivot_first` | ✅ | Native | Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark's standard filtered-aggregate path. Binary, array and struct pivot columns (including pivots on more than one column), non-default collations and pivot lists with duplicate values fall back to Spark | -| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | -| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | -| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | +| `regr_avgx` | ✅ | — | Spark rewrites to `Average`, which runs in Rust (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | +| `regr_avgy` | ✅ | — | Spark rewrites to `Average`, which runs in Rust (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | +| `regr_count` | ✅ | — | Spark rewrites to `Count`, which runs in Rust (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) | | `regr_intercept` | ✅ | Native | | | `regr_r2` | ✅ | Native | | | `regr_slope` | ✅ | Native | | | `regr_sxx` | ✅ | Native | | | `regr_sxy` | ✅ | Native | | | `regr_syy` | ✅ | Native | | -| `skewness` | 🔜 | — | Not yet implemented natively | +| `skewness` | 🔜 | — | Not yet implemented in Comet | | `some` | ✅ | — | | | `std` | ✅ | Native | | | `stddev` | ✅ | Native | | @@ -156,28 +156,28 @@ The tables below list every Spark built-in expression with its current status. | `array` | ✅ | Native | | | `array_append` | ✅ | Native | | | `array_compact` | ✅ | — | | -| `array_contains` | ✅ | Hybrid | Flat float/double element arrays use a native kernel with Spark's equality; nested float elements, non-default string collations, and a value that is not a literal or column over a nullable array, route through the JVM codegen dispatcher | -| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back | -| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) | +| `array_contains` | ✅ | Hybrid | Flat float/double element arrays use a Rust kernel with Spark's equality; nested float elements, non-default string collations, and a value that is not a literal or column over a nullable array, route through the JVM codegen dispatcher | +| `array_distinct` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ if you opt in to the native implementation ([details](compatibility/floating-point.md)). Non-default string collations fall back | +| `array_except` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) | | `array_insert` | ✅ | Native | | -| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) | -| `array_join` | ✅ | Hybrid | Native for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) | -| `array_max` | ✅ | Hybrid | Native Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) | -| `array_min` | ✅ | Hybrid | Native Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) | +| `array_intersect` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) | +| `array_join` | ✅ | Hybrid | Runs in Rust for literal or column delimiter and null replacement; other cases and non-UTF8_BINARY collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) | +| `array_max` | ✅ | Hybrid | Runs in Rust with Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) | +| `array_min` | ✅ | Hybrid | Runs in Rust with Spark-compatible floating-point and nested ordering; non-default string collations use the JVM codegen dispatcher ([details](compatibility/expressions/array.md)) | | `array_position` | ✅ | Native | Binary/struct/map/null elements fall back | | `array_prepend` | ✅ | — | | | `array_remove` | ✅ | Native | | | `array_repeat` | ✅ | Native | | -| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back | +| `array_union` | ✅ | Native | Floating-point elements fall back on Spark versions before 4.0.5, 4.1.4, and 4.2; signed-zero results may differ if you opt in to the native implementation ([details](compatibility/floating-point.md)). Non-default string collations fall back | | `arrays_overlap` | ✅ | Hybrid | Non-default string collations use the JVM codegen dispatcher | | `arrays_zip` | ✅ | Native | | | `element_at` | ✅ | Native | | | `flatten` | ✅ | Native | Binary/struct/map elements fall back | | `get` | ✅ | — | | -| `sequence` | ✅ | Hybrid | Integral types run natively; date/timestamp sequences use codegen dispatch | +| `sequence` | ✅ | Hybrid | Integral types run in Rust; date/timestamp sequences use codegen dispatch | | `shuffle` | ✅ | Native | Binary/struct/map elements fall back | -| `slice` | ✅ | Native | Native ([#4149](https://github.com/apache/datafusion-comet/pull/4149)) | -| `sort_array` | ✅ | Hybrid | Struct, nested-array, floating-point, and null elements run natively; other element types (for example intervals) route through the JVM codegen dispatcher | +| `slice` | ✅ | Native | Runs in Rust ([#4149](https://github.com/apache/datafusion-comet/pull/4149)) | +| `sort_array` | ✅ | Hybrid | Struct, nested-array, floating-point, and null elements run in Rust; other element types (for example intervals) route through the JVM codegen dispatcher | --- @@ -235,7 +235,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | Function | Status | Implementation | Notes | | --- | --- | --- | --- | -| `cast` | ✅ | Native | Casts without a native path (for example boolean to decimal) route through the JVM codegen dispatcher ([details](compatibility/expressions/cast.md)) | +| `cast` | ✅ | Native | Casts without a native implementation (for example boolean to decimal) route through the JVM codegen dispatcher ([details](compatibility/expressions/cast.md)) | --- @@ -245,7 +245,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | --- | --- | --- | --- | | `from_csv` | ✅ | Codegen dispatch | | | `schema_of_csv` | ✅ | Codegen dispatch | | -| `to_csv` | ✅ | Hybrid | Codegen dispatch by default; the native path is opt-in via allowIncompatible | +| `to_csv` | ✅ | Hybrid | Codegen dispatch by default; the native implementation is opt-in via allowIncompatible | --- @@ -254,37 +254,37 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | Function | Status | Implementation | Notes | | --- | --- | --- | --- | | `add_months` | ✅ | Codegen dispatch | | -| `convert_timezone` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | +| `convert_timezone` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | | `curdate` | ✅ | — | Constant-folded to a literal (alias of `current_date`) | | `current_date` | ✅ | — | Constant-folded to a literal before Comet sees the plan | | `current_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Constant-folded to a literal before Comet sees the plan | | `current_timestamp` | ✅ | — | Constant-folded to a literal before Comet sees the plan | | `current_timezone` | ✅ | — | | -| `date_add` | ✅ | Native | The 2-argument form is native; the `date_add(UNIT, n, ts)` form (Spark 3.5+) parses to `timestampadd` and runs through codegen dispatch | -| `date_diff` | ✅ | Native | The 2-argument form is native; the `date_diff(UNIT, start, end)` form (Spark 3.5+) parses to `timestampdiff` and runs through codegen dispatch | +| `date_add` | ✅ | Native | The 2-argument form runs in Rust; the `date_add(UNIT, n, ts)` form (Spark 3.5+) parses to `timestampadd` and runs through codegen dispatch | +| `date_diff` | ✅ | Native | The 2-argument form runs in Rust; the `date_diff(UNIT, start, end)` form (Spark 3.5+) parses to `timestampdiff` and runs through codegen dispatch | | `date_format` | ✅ | Hybrid | | | `date_from_unix_date` | ✅ | Native | | | `date_part` | ✅ | — | | | `date_sub` | ✅ | Native | | | `date_trunc` | ✅ | Hybrid | | -| `dateadd` | ✅ | Native | The 2-argument form is native; the `dateadd(UNIT, n, ts)` form parses to `timestampadd` and runs through codegen dispatch | -| `datediff` | ✅ | Native | The 2-argument form is native; the `datediff(UNIT, start, end)` form parses to `timestampdiff` and runs through codegen dispatch | +| `dateadd` | ✅ | Native | The 2-argument form runs in Rust; the `dateadd(UNIT, n, ts)` form parses to `timestampadd` and runs through codegen dispatch | +| `datediff` | ✅ | Native | The 2-argument form runs in Rust; the `datediff(UNIT, start, end)` form parses to `timestampdiff` and runs through codegen dispatch | | `datepart` | ✅ | — | | | `day` | ✅ | Native | | | `dayname` | ✅ | — | Abbreviated day name (Spark 4.0+) | | `dayofmonth` | ✅ | Native | | | `dayofweek` | ✅ | Native | | | `dayofyear` | ✅ | Native | | -| `extract` | ✅ | — | `SECOND FROM TIME` is native for Spark 4.1+ with `Decimal(8,6)` output | +| `extract` | ✅ | — | `SECOND FROM TIME` runs in Rust for Spark 4.1+ with `Decimal(8,6)` output | | `from_unixtime` | ✅ | Hybrid | | -| `from_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | +| `from_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | | `hour` | ✅ | Native | | | `last_day` | ✅ | Native | | | `localtimestamp` | ✅ | — | | -| `make_date` | ✅ | Native | Computed arguments preserve year → month → day NULL checks. Expressions fall back when shared subexpressions or a parent could change which arguments Spark evaluates. IN probes stay native for literal/column candidates and array/struct constructors of those values; other computed candidates require Spark evaluation. Operator-level safeguards and their opt-out are described in [make_date evaluation](#make_date-evaluation). | +| `make_date` | ✅ | Native | Computed arguments preserve year → month → day NULL checks. Expressions fall back when shared subexpressions or a parent could change which arguments Spark evaluates. IN probes stay in the Comet pipeline for literal/column candidates and array/struct constructors of those values; other computed candidates require Spark evaluation. Operator-level safeguards and their opt-out are described in [make_date evaluation](#make_date-evaluation). | | `make_dt_interval` | ✅ | Codegen dispatch | | -| `make_interval` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; intervals outside Arrow's nanosecond range are tracked by [#5279](https://github.com/apache/datafusion-comet/issues/5279); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | -| `make_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Runs natively; remaining TIME type work is tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) | +| `make_interval` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; intervals outside Arrow's nanosecond range are tracked by [#5279](https://github.com/apache/datafusion-comet/issues/5279); the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | +| `make_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. Runs in Rust; remaining TIME type work is tracked by [#4288](https://github.com/apache/datafusion-comet/issues/4288) | | `make_timestamp` | ✅ | Hybrid | | | `make_timestamp_ltz` | ✅ | — | The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch | | `make_timestamp_ntz` | ✅ | — | The 2-argument `(date, time)` form (Spark 4.1+) runs through codegen dispatch | @@ -293,26 +293,26 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | `month` | ✅ | Native | | | `monthname` | ✅ | — | Abbreviated month name (Spark 4.0+) | | `months_between` | ✅ | Codegen dispatch | | -| `next_day` | ✅ | Hybrid | Non-UTF8_BINARY collated `dayOfWeek` routes through the JVM codegen dispatcher; other input runs natively | +| `next_day` | ✅ | Hybrid | Non-UTF8_BINARY collated `dayOfWeek` routes through the JVM codegen dispatcher; other input runs in Rust | | `now` | ✅ | — | Constant-folded to a literal (alias of `current_timestamp`) | | `quarter` | ✅ | Native | | | `second` | ✅ | Native | | -| `session_window` | 🔜 | — | Batch session-window grouping falls back (`UpdatingSessionsExec` is not yet native); tracked by [#4785](https://github.com/apache/datafusion-comet/issues/4785) | +| `session_window` | 🔜 | — | Batch session-window grouping falls back (Comet does not yet support `UpdatingSessionsExec`); tracked by [#4785](https://github.com/apache/datafusion-comet/issues/4785) | | `time_diff` | ✅ | Codegen dispatch | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default | | `time_trunc` | ✅ | Codegen dispatch | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default | | `timediff` | ✅ | — | Spark 4.0+ grammar alias that parses to `timestampdiff`; runs through codegen dispatch | | `timestamp_micros` | ✅ | Codegen dispatch | | | `timestamp_millis` | ✅ | Codegen dispatch | | -| `timestamp_seconds` | ✅ | Hybrid | Integer, long, float and double inputs run natively; decimal, byte and short inputs route through the JVM codegen dispatcher | +| `timestamp_seconds` | ✅ | Hybrid | Integer, long, float and double inputs run in Rust; decimal, byte and short inputs route through the JVM codegen dispatcher | | `timestampadd` | ✅ | — | Reached through the grammar rather than the function registry; runs through codegen dispatch | | `timestampdiff` | ✅ | — | Reached through the grammar rather than the function registry; runs through codegen dispatch | | `to_date` | ✅ | — | Rewrites to `Cast` (or `Cast(GetTimestamp)` with a format) before Comet sees the plan | -| `to_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. The one-argument form runs natively, and the form with a format argument runs through codegen dispatch | +| `to_time` | ✅ | — | Spark 4.1+; requires `spark.sql.timeType.enabled=true`, which Spark leaves off by default. The one-argument form runs in Rust, and the form with a format argument runs through codegen dispatch | | `to_timestamp` | ✅ | — | Rewrites to `Cast` (or `GetTimestamp` with a format) before Comet sees the plan | | `to_timestamp_ltz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampType`) | | `to_timestamp_ntz` | ✅ | — | Rewrites to `to_timestamp` (`TimestampNTZType`) | | `to_unix_timestamp` | ✅ | Hybrid | | -| `to_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native path is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | +| `to_utc_timestamp` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default (handles all timezone forms); the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/datetime.md)) | | `trunc` | ✅ | Hybrid | | | `try_make_interval` | ✅ | — | Rewrites to `MakeInterval`; same support as `make_interval` (Spark 4.0+) | | `try_make_timestamp` | ✅ | — | | @@ -325,11 +325,11 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci | `unix_micros` | ✅ | Codegen dispatch | | | `unix_millis` | ✅ | Codegen dispatch | | | `unix_seconds` | ✅ | Codegen dispatch | | -| `unix_timestamp` | ✅ | Hybrid | String parsing uses Spark's codegen and honors the time parser policy, ANSI mode, and session time zone. Date and timestamp inputs ignore the format and use native execution. | +| `unix_timestamp` | ✅ | Hybrid | String parsing uses Spark's codegen and honors the time parser policy, ANSI mode, and session time zone. Date and timestamp inputs ignore the format and run in Rust. | | `weekday` | ✅ | Native | | | `weekofyear` | ✅ | Native | | -| `window` | ✅ | — | Batch tumbling and sliding time-window grouping runs natively | -| `window_time` | ✅ | — | Batch time-window grouping runs natively | +| `window` | ✅ | — | Batch tumbling and sliding time-window grouping runs in the Comet pipeline | +| `window_time` | ✅ | — | Batch time-window grouping runs in the Comet pipeline | | `year` | ✅ | Native | | --- @@ -371,13 +371,13 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | Function | Status | Implementation | Notes | | --- | --- | --- | --- | -| `from_json` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native path (supported schemas only) is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) | -| `get_json_object` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object)) | -| `json_array_length` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native path, which differs for single-quoted JSON, unescaped control characters, and trailing content, is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#json_array_length)) | +| `from_json` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native implementation (supported schemas only) is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#from_json)) | +| `get_json_object` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native implementation is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#get_json_object)) | +| `json_array_length` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the native implementation, which differs for single-quoted JSON, unescaped control characters, and trailing content, is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#json_array_length)) | | `json_object_keys` | ✅ | Codegen dispatch | | | `json_tuple` | 🔜 | — | [#3160](https://github.com/apache/datafusion-comet/issues/3160) | | `schema_of_json` | ✅ | Codegen dispatch | | -| `to_json` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default, including options and map/array inputs; the native path is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) | +| `to_json` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default, including options and map/array inputs; the native implementation is opt-in via allowIncompatible ([audit](../../contributor-guide/expression-audits/json_funcs.md#to_json)) | --- @@ -386,9 +386,9 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | Function | Status | Implementation | Notes | | --- | --- | --- | --- | | `aggregate` | ✅ | Codegen dispatch | | -| `array_sort` | ✅ | Native | The default comparator runs natively; custom comparators and arrays with non-default string collations use the JVM codegen dispatcher | +| `array_sort` | ✅ | Native | The default comparator runs in Rust; custom comparators and arrays with non-default string collations use the JVM codegen dispatcher | | `exists` | ✅ | Codegen dispatch | | -| `filter` | ✅ | Native | General lambda routed through the JVM codegen dispatcher; the `array_compact` form runs natively | +| `filter` | ✅ | Native | General lambda routed through the JVM codegen dispatcher; the `array_compact` form runs in Rust | | `forall` | ✅ | Codegen dispatch | | | `map_filter` | ✅ | Codegen dispatch | | | `map_zip_with` | ✅ | Codegen dispatch | | @@ -427,7 +427,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `+` | ✅ | Native | | | `-` | ✅ | Native | | | `/` | ✅ | Native | DayTime interval division routes through the JVM codegen dispatcher; YearMonth and Calendar interval division fall back | -| `abs` | ✅ | Hybrid | Interval types route through the JVM codegen dispatcher; numeric types run natively | +| `abs` | ✅ | Hybrid | Interval types route through the JVM codegen dispatcher; numeric types run in Rust | | `acos` | ✅ | Native | | | `acosh` | ✅ | Native | | | `asin` | ✅ | Native | | @@ -452,10 +452,10 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `expm1` | ✅ | Native | | | `factorial` | ✅ | Native | | | `floor` | ✅ | — | Two-arg form falls back | -| `greatest` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively | +| `greatest` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs in Rust | | `hex` | ✅ | Native | | | `hypot` | ✅ | Codegen dispatch | | -| `least` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively | +| `least` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs in Rust | | `ln` | ✅ | Native | | | `log` | ✅ | Native | | | `log10` | ✅ | Native | | @@ -474,7 +474,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `random` | ✅ | Native | Alias for `rand`; seed must be a literal | | `randstr` | ✅ | Native | Random string (Spark 4.0+); length and seed must be literals | | `rint` | ✅ | Native | | -| `round` | ✅ | Hybrid | Float/double inputs route through the JVM codegen dispatcher; other types run natively | +| `round` | ✅ | Hybrid | Float/double inputs route through the JVM codegen dispatcher; other types run in Rust | | `sec` | ✅ | Native | | | `shiftleft` | ✅ | Native | | | `sign` | ✅ | Native | | @@ -507,8 +507,8 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `current_schema` | ✅ | — | Alias of `current_database`; resolved to a literal by the analyzer | | `current_user` | ✅ | — | Resolved to a literal by the analyzer; same as `user` | | `equal_null` | ✅ | — | Lowers to `<=>` (`EqualNullSafe`) | -| `hll_sketch_estimate` | ✅ | Native | Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible ([details](compatibility/expressions/misc.md)) | -| `hll_union` | ✅ | Native | Spark 4.0+ only; falls back by default, the native path is opt-in via allowIncompatible ([details](compatibility/expressions/misc.md)) | +| `hll_sketch_estimate` | ✅ | Native | Spark 4.0+ only; falls back by default, the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/misc.md)) | +| `hll_union` | ✅ | Native | Spark 4.0+ only; falls back by default, the native implementation is opt-in via allowIncompatible ([details](compatibility/expressions/misc.md)) | | `is_variant_null` | 🔜 | — | Requires `VariantType` support | | `monotonically_increasing_id` | ✅ | Native | | | `parse_json` | 🔜 | — | Requires `VariantType` support | @@ -552,9 +552,9 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `like` | ✅ | Hybrid | | | `not` | ✅ | Native | | | `or` | ✅ | Native | | -| `regexp` | ✅ | Hybrid | In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) | -| `regexp_like` | ✅ | Hybrid | In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) | -| `rlike` | ✅ | Hybrid | In-subset literal patterns run natively; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) | +| `regexp` | ✅ | Hybrid | In-subset literal patterns run in Rust; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) | +| `regexp_like` | ✅ | Hybrid | In-subset literal patterns run in Rust; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) | +| `rlike` | ✅ | Hybrid | In-subset literal patterns run in Rust; other patterns route through the JVM codegen dispatcher ([details](compatibility/regex.md)) | --- @@ -572,7 +572,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `chr` | ✅ | Native | | | `collate` | 🔜 | — | Spark collation (umbrella [#2190](https://github.com/apache/datafusion-comet/issues/2190)) | | `collation` | ✅ | — | Constant-folded to a literal (Spark 4.0+) | -| `concat_ws` | ✅ | Hybrid | Mixed string and array arguments run natively; all-foldable arguments use codegen dispatch | +| `concat_ws` | ✅ | Hybrid | Mixed string and array arguments run in Rust; all-foldable arguments use codegen dispatch | | `contains` | ✅ | — | | | `decode` | ✅ | — | | | `elt` | ✅ | Codegen dispatch | | @@ -582,17 +582,17 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `format_number` | ✅ | Codegen dispatch | | | `format_string` | ✅ | Codegen dispatch | | | `initcap` | ✅ | Hybrid | | -| `instr` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively | +| `instr` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs in Rust | | `lcase` | ✅ | Hybrid | | | `left` | ✅ | Native | | | `len` | ✅ | Native | | | `length` | ✅ | Native | | -| `levenshtein` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively | +| `levenshtein` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs in Rust | | `locate` | ✅ | Codegen dispatch | | | `lower` | ✅ | Hybrid | | -| `lpad` | ✅ | Hybrid | String inputs use the native kernel with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch. | -| `ltrim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively | -| `luhn_check` | ✅ | — | Native via `StaticInvoke` (tests: luhn_check.sql) | +| `lpad` | ✅ | Hybrid | String inputs run in Rust with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch. | +| `ltrim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs in Rust | +| `luhn_check` | ✅ | — | Runs in Rust via `StaticInvoke` (tests: luhn_check.sql) | | `mask` | ✅ | — | Routed through the JVM codegen dispatcher | | `octet_length` | ✅ | Native | | | `overlay` | ✅ | Codegen dispatch | | @@ -605,10 +605,10 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `regexp_replace` | ✅ | Hybrid | | | `regexp_substr` | ✅ | — | Rewrites to `nullif(regexp_extract(...), '')`; `regexp_extract` routes through the JVM codegen dispatcher by default | | `repeat` | ✅ | Native | | -| `replace` | ✅ | Hybrid | Native by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation; other cases use the JVM codegen dispatcher by default | +| `replace` | ✅ | Hybrid | Runs in Rust by default for a direct string column with a non-empty, valid UTF-8 search literal and either a replacement literal (both at most 64 KiB) or a replacement column (search at most 256 bytes and at most 8 MiB when broadcast per batch), under UTF8_BINARY collation; other cases use the JVM codegen dispatcher by default | | `right` | ✅ | Native | | -| `rpad` | ✅ | Hybrid | String inputs use the native kernel with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch. | -| `rtrim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively | +| `rpad` | ✅ | Hybrid | String inputs run in Rust with a column string and literal padding; literal strings and column padding use codegen dispatch. Binary inputs use codegen dispatch. | +| `rtrim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs in Rust | | `soundex` | ✅ | Native | | | `space` | ✅ | Native | | | `split` | ✅ | Hybrid | | @@ -616,17 +616,17 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | `startswith` | ✅ | — | | | `substr` | ✅ | Native | | | `substring` | ✅ | Native | | -| `substring_index` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs natively | -| `to_binary` | ✅ | — | The hex form runs natively; the base64 and `utf-8` forms route through the JVM codegen dispatcher | +| `substring_index` | ✅ | Hybrid | Non-UTF8_BINARY collated input routes through the JVM codegen dispatcher; other input runs in Rust | +| `to_binary` | ✅ | — | The hex form runs in Rust; the base64 and `utf-8` forms route through the JVM codegen dispatcher | | `to_char` | ✅ | Codegen dispatch | | | `to_number` | ✅ | Codegen dispatch | | | `to_varchar` | ✅ | Codegen dispatch | | -| `translate` | ✅ | Hybrid | Codegen dispatch by default: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible | -| `trim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs natively | +| `translate` | ✅ | Hybrid | Codegen dispatch by default: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native implementation is opt-in via allowIncompatible | +| `trim` | ✅ | Hybrid | Non-UTF8_BINARY collated input with a trim string routes through the JVM codegen dispatcher; other input runs in Rust | | `try_to_binary` | ✅ | — | Rewrites to `try_eval(to_binary(...))`, which routes through the JVM codegen dispatcher | | `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher | | `ucase` | ✅ | Hybrid | | -| `unbase64` | ✅ | Codegen dispatch | Column or literal input runs natively; other child expressions, and the strict form used by `to_binary(str, 'base64')`, route through the JVM codegen dispatcher | +| `unbase64` | ✅ | Codegen dispatch | Column or literal input runs in Rust; other child expressions, and the strict form used by `to_binary(str, 'base64')`, route through the JVM codegen dispatcher | | `upper` | ✅ | Hybrid | | --- @@ -645,7 +645,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab | Function | Status | Implementation | Notes | | --- | --- | --- | --- | | `parse_url` | ✅ | Native | | -| `try_parse_url` | ✅ | — | Spark 4.0+; runs natively and returns NULL for an invalid URL | +| `try_parse_url` | ✅ | — | Spark 4.0+; runs in Rust and returns NULL for an invalid URL | | `try_url_decode` | ✅ | — | | | `url_decode` | ✅ | — | | | `url_encode` | ✅ | — | | @@ -659,8 +659,8 @@ Aggregate window functions (`count`, `min`, `max`, `sum`, `avg`, `first_value`, `last_value`), ranking functions (`row_number`, `rank`, `dense_rank`, `percent_rank`, `cume_dist`, `ntile`), and value-shift functions (`lag`, `lead`, `nth_value`) are all wired in the window serde -and execute natively. Statistical aggregates such as `stddev`, `var_pop`, -`corr`, and `covar_pop` run natively as plain aggregations but fall back +and run in Comet. Statistical aggregates such as `stddev`, `var_pop`, +`corr`, and `covar_pop` run in Comet as plain aggregations but fall back to Spark when used as window functions. A handful of frame shapes also fall back. See [window function compatibility](compatibility/operators.md) for the full list of supported functions, frames, and fallback cases. @@ -705,10 +705,10 @@ Comet also accelerates a number of Catalyst expressions that have no Spark SQL f - **Operator and optimizer-injected expressions:** runtime bloom-filter join probes (`BloomFilterMightContain`, `BloomFilterAggregate`), optimized `IN` sets (`InSet`), scalar subqueries (`ScalarSubquery`), and floating-point normalization (`KnownFloatingPointNormalized`). - **Accessor expressions (subscript and field access, not functions):** struct field access (`col.field`), array element access (`arr[i]`), and map value access (`map[key]`). - **Internal decimal arithmetic:** `CheckOverflow`, `MakeDecimal`, and `UnscaledValue`, which the analyzer inserts around decimal operations. -- **Missing-data predicates:** `AtLeastNNonNulls`, used by `DataFrame.na.drop`, counts non-null and non-NaN values natively. +- **Missing-data predicates:** `AtLeastNNonNulls`, used by `DataFrame.na.drop`, counts non-null and non-NaN values in Rust. - **User-defined functions:** Scala UDFs registered through the DataFrame or SQL API. -- **DataSource V2 catalog functions:** Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (for example `system.bucket(16, id)`) run natively; see [Iceberg system functions](iceberg.md#iceberg-system-functions). Other DataSource V2 catalog functions run in Spark. -- **Lowered built-ins:** Spark lowers some built-in functions to `StaticInvoke` or `Invoke` calls. Those without a native mapping run through the JVM codegen dispatcher when their input and output types are supported. +- **DataSource V2 catalog functions:** Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (for example `system.bucket(16, id)`) run in Rust; see [Iceberg system functions](iceberg.md#iceberg-system-functions). Other DataSource V2 catalog functions run in Spark. +- **Lowered built-ins:** Spark lowers some built-in functions to `StaticInvoke` or `Invoke` calls. Those without a native implementation run through the JVM codegen dispatcher when their input and output types are supported. - **Structural expressions:** aliases, attribute references, literals, sort orders, and `CASE WHEN`. This list is illustrative, not exhaustive: the per-function tables are not the complete set of expressions Comet can accelerate. @@ -717,7 +717,7 @@ This list is illustrative, not exhaustive: the per-function tables are not the c With ANSI mode enabled, `make_date(2024, 13, 1)` raises an invalid-date error. If an input iterator yields a valid row followed by that invalid row, a `LIMIT 1` can stop -before evaluating the second date. A native projection can evaluate a whole batch +before evaluating the second date. A Comet projection can evaluate a whole batch before the consuming operator decides which rows it needs. By default, Comet keeps affected operators in Spark to preserve skipped date evaluation. @@ -726,7 +726,7 @@ This applies to date projections below `LIMIT`, `SAMPLE`, `FILTER`, joins, and ordered limits; and projections below a window group limit. A parent projection can also defer a date output when an `IF`, `COALESCE`, or NULL-skipping operand does not need it. Comet preserves that evaluation across projections while allowing eager date consumers, -such as `YEAR(dt)`, to remain native when their operands are safe to evaluate. An enclosing +such as `YEAR(dt)`, to run in Comet when their operands are safe to evaluate. An enclosing conditional can still defer the extractor's date input. These safeguards also preserve NULL short-circuiting when a date argument is computed in an earlier projection, including across the streamed side of a broadcast join. The materialized build side remains independent. @@ -735,7 +735,7 @@ the join rejects before producing an output row. Selecting the date eagerly does not force conditionally used sibling expressions; those computations also retain Spark evaluation when needed. -With ANSI disabled, a direct date projection can remain native below these consumers when +With ANSI disabled, a direct date projection can run in Comet below these consumers when all its computations are safe to evaluate early. This includes `make_date(2024, month, day)` over input columns and years widened from `TINYINT` or `SMALLINT`. An unrestricted integer year still needs the safeguard: Spark's conversion to Date32 can overflow independently of @@ -743,23 +743,23 @@ ANSI mode. Computed arguments, their deferred producers, and throwing sibling ex retain their safeguards. Expression-level parent guards remain in effect. For example, if `dt` is a nullable date alias, `unix_date(dt) + element_at(values, i)` keeps its consuming operator in Spark when a NULL date can skip an invalid array index. -The safe date-producing projection can remain native. +The safe date-producing projection can run in Comet. -An unfiltered `COUNT(make_date(...))` can remain native when its sole argument is nullable, +An unfiltered `COUNT(make_date(...))` can run in Comet when its sole argument is nullable, because Spark checks that argument on every input row. Filtered counts and counts with multiple arguments retain the safeguard when they can skip date evaluation. -Only affected parts of the plan fall back. An ordinary native scan can remain native. -An adjacent collect limit and date projection can also stay native: Comet caps the projection's +Only affected parts of the plan fall back. An ordinary native scan still runs in Comet. +An adjacent collect limit and date projection can also run in Comet: Comet caps the projection's input before evaluating dates, so unread rows cannot raise errors. This requires a supported -native local limit; otherwise the Spark safeguard remains. The cap does not bypass safeguards +Comet local limit; otherwise the Spark safeguard remains. The cap does not bypass safeguards for a lower argument-producing projection or intervening filters and joins. Exchanges materialize their inputs and partition keys independently, so an outer limit does not force an eager date partition key to Spark. Local sorts and aggregates can also separate date evaluation from an outer limit. Users who accept eagerly evaluating dates, projected arguments, and sibling expressions can set -`spark.comet.exec.makeDate.operatorFallback.enabled=false` to allow native operators for +`spark.comet.exec.makeDate.operatorFallback.enabled=false` to allow Comet operators for these shapes. The default is `true`. Disabling it can expose invalid-date or argument errors that Spark would skip under ANSI mode. This setting does not change `make_date`'s invalid-date handling or the expression-level safeguards for skipped diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md index 4c603f79b86..ac2a0373221 100644 --- a/docs/source/user-guide/latest/iceberg.md +++ b/docs/source/user-guide/latest/iceberg.md @@ -24,7 +24,7 @@ Comet accelerates Iceberg tables whose data files are Parquet, in two ways: - **[Reading](#native-reader)**: the native Iceberg reader scans the table's data and delete files in Rust. It is enabled by default. - **[Writing](#writes)**: with `spark.comet.write.iceberg.enabled=true`, Comet writes the data - files of eligible writes natively. It is disabled by default. + files of eligible writes in Rust. It is disabled by default. This page covers both, followed by the [known issues](#known-issues). Behavior that applies to Parquet files in general is documented on the [Parquet](parquet.md) page. @@ -32,7 +32,7 @@ Parquet files in general is documented on the [Parquet](parquet.md) page. ## Native Reader Comet's native Iceberg reader relies on reflection to extract `FileScanTask`s from Iceberg, which are -then serialized to Comet's native execution engine (see +then serialized and passed to Comet's Rust code (see [PR #2528](https://github.com/apache/datafusion-comet/pull/2528)). The example below uses Spark's package downloader to retrieve Comet $COMET_VERSION and Iceberg @@ -91,7 +91,7 @@ The native Iceberg reader supports the following features: - Encrypted v3 tables using 128-bit or 256-bit AES-GCM data keys (requires Iceberg 1.11 or newer). Iceberg-Java unwraps the key envelope on the driver during planning and stores the plaintext data key in each file's `key_metadata`, which the native reader uses directly, so no KMS integration is - needed on the native side. Iceberg-Java also permits 192-bit data keys; those tables fall back to + needed on the Rust side. Iceberg-Java also permits 192-bit data keys; those tables fall back to Spark (no AES-192-GCM in the underlying crypto). **Schema and data types:** @@ -122,8 +122,9 @@ The native Iceberg reader supports the following features: NULL checks on struct, array, and map columns still use native scans and return correct results, but are not pushed into iceberg-rust, which binds accessors only for primitive fields. -These residuals provide no native row-group pruning, nor do conjunctions containing them; -safe partial pruning is tracked in [#5883](https://github.com/apache/datafusion-comet/issues/5883). +These residuals provide no row-group pruning in the native scan, nor do conjunctions containing +them; safe partial pruning is tracked in +[#5883](https://github.com/apache/datafusion-comet/issues/5883). **Partitioning:** @@ -242,7 +243,7 @@ The following scenarios will fall back to the JVM Iceberg reader: - v3 tables with columns that declare an initial default value - v3 column types the native reader cannot read (`geometry`, `geography`, `unknown`), and `variant` columns the query reads (on Spark 4.0+, a table whose `variant` columns are not - projected is read natively) + projected is read by the native reader) - Encrypted tables with 192-bit data keys (no AES-192-GCM in the underlying crypto) - Delete files in a format other than Parquet or Puffin (Avro or ORC positional/equality deletes) - Equality deletes keyed on a field nested in a struct @@ -271,7 +272,7 @@ The following scenarios will fall back to the JVM Iceberg reader: `schema.name-mapping.default`, including tables imported with `snapshot`, `migrate`, or `add_files` ([#6790](https://github.com/apache/datafusion-comet/issues/6790)). The native reader does not apply name mappings to nested fields. Queries that read only primitive columns can - still run natively. The fallback remains while the property is set, even if the data files + still run in Comet. The fallback remains while the property is set, even if the data files have since been rewritten with field IDs. Iceberg's nullness rules are separate from the Parquet legacy setting; [#6934](https://github.com/apache/datafusion-comet/issues/6934) tracks the reader update and review of this fallback @@ -284,7 +285,7 @@ succeeds, without falling back are listed under [Known Issues](#known-issues). Writes are not covered by this list. By default Spark plans Iceberg writes with its own V2 write operator, and iceberg-java writes the data files. With `spark.comet.write.iceberg.enabled=true`, -Comet plans an Iceberg write with its split-operator plan and writes eligible data files natively; +Comet plans an Iceberg write with its split-operator plan and writes eligible data files in Rust; see [Writes](#writes) for the plan, the native writer, and when each applies. ### Iceberg UDFs @@ -299,7 +300,7 @@ Iceberg ships several `ScalaUDF`s that surface in user queries and maintenance a during compaction. [Scala UDF and Java UDF Support](scala_java_udfs.md) is enabled by default -(`spark.comet.exec.scalaUDF.codegen.enabled=true`), so these UDFs run through native execution and +(`spark.comet.exec.scalaUDF.codegen.enabled=true`), so these UDFs run through codegen dispatch and the project, exchange, and sort operators around them stay on the Comet path end-to-end. Setting `spark.comet.exec.scalaUDF.codegen.enabled=false` causes the enclosing operator to fall back to Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuffle from @@ -308,14 +309,14 @@ Spark, which forces a columnar-to-row roundtrip and demotes the surrounding shuf ### Iceberg system functions Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (the SQL -form of its partition transforms, for example `SELECT system.bucket(16, id) FROM t`) run natively. +form of its partition transforms, for example `SELECT system.bucket(16, id) FROM t`) run in Rust. Spark binds them as static invocations of Iceberg's per-type implementations under `org.apache.iceberg.spark.functions`, and Comet recognizes those classes wherever the expression appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle. The exception is a call nested inside an expression that Comet runs through the JVM codegen dispatcher, such as `map(...)`. That makes the operator fall back to Spark. -The native kernels reproduce Iceberg's Java semantics exactly rather than approximately: +The Rust kernels reproduce Iceberg's Java semantics exactly rather than approximately: - `bucket` hashes the spec's byte encoding of each value (8-byte little-endian for integers, dates, and timestamps; UTF-8 for strings; raw bytes for binary; the minimal big-endian two's complement @@ -332,13 +333,13 @@ magnitude, so the result can need one more digit than the column's precision all `truncate(10, v)` on a `decimal(18,4)` value of `-99999999999999.9999` is `-100000000000000.0000`, which has 19 digits. Iceberg's `TruncateDecimal` hands that oversized value back unchanged and Spark turns it into null only when the row is materialized. An Arrow -`Decimal128(precision, scale)` array has no encoding for that intermediate, so a native kernel +`Decimal128(precision, scale)` array has no encoding for that intermediate, so a Rust kernel would have to null it during evaluation, which changes what an enclosing predicate or hash sees. -Every other `truncate` input type, and `bucket` on decimals, runs natively. +Every other `truncate` input type, and `bucket` on decimals, runs in Rust. This matters most for writes. A partitioned table with the default `write.distribution-mode` (`hash`) is planned with a shuffle and a local sort keyed on the partition transforms, and with -these functions native the whole sub-plan feeding the [native Iceberg writer](#writes) +these functions Rust-implemented, the whole sub-plan feeding the [native Iceberg writer](#writes) stays in Comet. A `numBuckets` or `width` argument that is not a positive integer literal makes the expression fall back to Spark. @@ -351,8 +352,8 @@ Iceberg Java does not explicitly report `bytesRead` to Spark's task input metric ## Writes With `spark.comet.write.iceberg.enabled=true`, Comet plans Iceberg writes with the split-operator -plan described below, and writes the data files of each eligible write natively with iceberg-rust. -A write that is not eligible falls back to iceberg-java's writer. Natively written files differ +plan described below, and writes the data files of each eligible write with iceberg-rust. +A write that is not eligible falls back to iceberg-java's writer. The native writer's files differ from iceberg-java's in the ways listed under [Accepted divergences](#accepted-divergences). The setting is off by default, so Spark plans its own write operator and iceberg-java writes every @@ -382,7 +383,7 @@ operators: The split moves data-file writing inside AQE and separates it from the commit, which gives Comet a step to replace: when the write passes the eligibility check below, the `IcebergWrite` operator's per-task Parquet write is delegated to -[iceberg-rust](https://github.com/apache/iceberg-rust) via Comet's native execution pipeline +[iceberg-rust](https://github.com/apache/iceberg-rust) via Comet's Rust code ([#5361](https://github.com/apache/datafusion-comet/pull/5361)). A write that does not pass keeps `IcebergWrite`, and iceberg-java writes its data files; only the plan shape changes. @@ -395,7 +396,7 @@ configuration (the table's `FileIO` properties, e.g. REST-vended credentials, me `fs.s3a.*` settings translated from the effective Hadoop configuration carried by the table's `FileIO`, including catalog-specific `hadoop.*` overrides, since `HadoopFileIO` carries its S3 configuration in the Hadoop Configuration rather than in `FileIO` properties), and per-task IDs — -into the serialized native plan. On each task, iceberg-rust writes the Parquet files and +into the serialized Rust plan. On each task, iceberg-rust writes the Parquet files and returns its `DataFile` metadata packed as a single in-memory Iceberg V2 data manifest; the JVM decodes those bytes with Iceberg's own `ManifestFiles.read`, re-derives each file's manifest metrics from the written Parquet footer with Iceberg's `MetricsConfig` logic (so metrics modes, @@ -406,8 +407,8 @@ is untouched: `IcebergCommit` performs the normal `BatchWrite.commit`. For `ResolvingFileIO`, Hadoop settings are taken from the delegate opening the data location. An S3 location handled by `S3FileIO` uses its initialized FileIO properties, so Hadoop options -on the wrapper do not change the native endpoint or encryption settings. If that delegate cannot -be resolved, the write falls back to iceberg-java. +on the wrapper do not change the native writer's endpoint or encryption settings. If that delegate +cannot be resolved, the write falls back to iceberg-java. Before a task opens a partition's first file, it holds the partition's initial rows in memory to choose which columns to dictionary-encode using parquet-mr's size accounting (see the @@ -451,9 +452,9 @@ supports: For an unpartitioned copy-on-write `MERGE`, the native Iceberg writer is reachable on Spark 3.5+ when `spark.comet.exec.mergeRows.enabled=true`. With the flag disabled, the JVM -`MergeRowsExec` breaks the fully-native child chain required by `CometIcebergWriteExec`. -On Spark 4.1+, the native MergeRows path also preserves the semantic counters required by the -summary-aware writer commit contract. +`MergeRowsExec` breaks the chain of Comet operators required below `CometIcebergWriteExec`. +On Spark 4.1+, Comet's Rust-implemented MergeRows also preserves the semantic counters required by +the summary-aware writer commit contract. For copy-on-write row-level DML, the mechanism differs by Spark version: on Spark 4.0+ the analyzer emits operation-coded rows that Comet's writer dispatches through `ReplaceData`'s @@ -477,7 +478,7 @@ The rewrite is skipped — and the write runs through Spark's stock combined ope - `spark.comet.write.iceberg.enabled` is `false` (the default) and the testing setting `spark.comet.write.iceberg.splitOperator.enabled`, which Comet 1.1.0 documented as a user setting, is not on; -- Comet is disabled (`spark.comet.enabled=false`), or its native execution is +- Comet is disabled (`spark.comet.enabled=false`), or its operators are (`spark.comet.exec.enabled=false`); - Comet is in plan-only mode (`spark.comet.explain.planOnly.enabled=true`); - the write is neither an Iceberg `SparkWrite` nor a supported Iceberg position-delta write; @@ -500,13 +501,13 @@ When `spark.comet.write.iceberg.enabled=true` per-task Parquet write is delegated to [iceberg-rust](https://github.com/apache/iceberg-rust). The native writer must produce the same outcome as iceberg-java — the same Parquet features, statistics, and manifest metadata — so a write is only eligible when every table property it -depends on is one the native path reproduces exactly, and additionally only when the plan -feeding the write is fully Comet-native. For a partitioned table that plan includes the hash -distribution and local sort Iceberg requests on its partition transforms; those stay native +depends on is one the native writer reproduces exactly, and additionally only when the plan +feeding the write runs entirely in Comet. For a partitioned table that plan includes the hash +distribution and local sort Iceberg requests on its partition transforms; those stay in Comet because the transforms themselves have native implementations (see [Iceberg system functions](#iceberg-system-functions)). Ineligible writes run through iceberg-java unchanged, with the reason reported as a fall-back reason in Comet's extended EXPLAIN -output. A write that runs natively shows `CometIcebergWrite` under `IcebergCommit` in the physical +output. A native write shows `CometIcebergWrite` under `IcebergCommit` in the physical plan; an ineligible write keeps `IcebergWrite`. The native writer reads its input as Arrow batches from a Comet operator, so the write's input @@ -530,7 +531,7 @@ A write is eligible only when ALL of the following hold: | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | resolved write format (`write-format` option overlaid on `write.format.default`) | `parquet` | | `format-version` | `1`, `2` or `3` (on `3`, writes that carry row lineage fall back; see below) | -| `write.parquet.compression-codec` / `compression-level` / `row-group-size-bytes` / `page-size-bytes` / `page-row-limit` / `dict-size-bytes` | the sizes/limit must be positive Java ints (`Integer.parseInt` semantics — no trimming, no values past `Int.MaxValue` — matching iceberg-java, whose writer fails on anything else); `compression-level` must be a Java int within the native writer's per-codec range (zstd 1–22, gzip 0–9, brotli 0–11; ignored by both writers for snappy/lz4/none) — iceberg-java never validates the level, so an out-of-range value falls back rather than becoming a native task failure | +| `write.parquet.compression-codec` / `compression-level` / `row-group-size-bytes` / `page-size-bytes` / `page-row-limit` / `dict-size-bytes` | the sizes/limit must be positive Java ints (`Integer.parseInt` semantics — no trimming, no values past `Int.MaxValue` — matching iceberg-java, whose writer fails on anything else); `compression-level` must be a Java int within the native writer's per-codec range (zstd 1–22, gzip 0–9, brotli 0–11; ignored by both writers for snappy/lz4/none) — iceberg-java never validates the level, so an out-of-range value falls back rather than becoming a failed native write | | `write.parquet.row-group-check-min-record-count` | unset or `100` (the default) | | `write.parquet.row-group-check-max-record-count` | unset or `10000` (the default) | | `write.parquet.page-version` | unset or `v1` | @@ -568,7 +569,7 @@ A `gs` data location additionally requires that the `FileIO` actually opening it `GCSFileIO` (for a `ResolvingFileIO`, the delegate it instantiates for that location, which is a `HadoopFileIO` when the GCS `FileIO` cannot be loaded or initialized). A `HadoopFileIO` takes its GCS credentials, endpoint and project from `fs.gs.*` in the Hadoop -Configuration, and only `fs.s3a.*` is translated into the native `FileIO`, so the native writer +Configuration, and only `fs.s3a.*` is translated into iceberg-rust's `FileIO`, so the native writer could resolve a different storage identity or endpoint than the JVM writer would. That combination falls back; a `GCSFileIO` carries its `gcs.*` settings in `FileIO.properties()`, which are forwarded. @@ -577,7 +578,7 @@ For an `s3` or `s3a` data location, the gate also inspects both the table FileIO configuration and `table.io().properties()`. These are separate allowlists because Hadoop S3A keys are translated before they reach iceberg-rust, while Iceberg `FileIO` keys are forwarded directly. When the FileIO exposes a Hadoop configuration, its initialized values govern both the -gate and native translation, including after session or catalog options change. FileIO +gate and the translation, including after session or catalog options change. FileIO implementations without a Hadoop configuration, such as `S3FileIO`, use only their initialized properties; session and catalog Hadoop options are neither checked nor forwarded. Hadoop's built-in `core-default.xml` values are not treated as explicit settings, but @@ -590,10 +591,10 @@ none of them can alter an Iceberg data-file write request. Spark 4.1 sets `commi matched by their plain `fs.s3a.` keys, so a per-bucket `fs.s3a.bucket..` form still falls back. Unknown explicit `fs.s3a.*`, `s3.*`, or `client.*` settings therefore fall back at planning time instead of being -silently ignored by the native storage backend. The exception is a vendor-owned `s3.*` / +silently ignored by the native writer's storage backend. The exception is a vendor-owned `s3.*` / `client.*` property when `s3.comet.credential.provider.class` is configured: the provider receives the unfiltered FileIO -bag and can consume that property. Iceberg-defined settings that the native storage path cannot +bag and can consume that property. Iceberg-defined settings that the native writer cannot honour still fall back even with a provider. A per-bucket Hadoop setting counts only for the exact data-bucket name, so configuration for a longer dotted bucket does not by itself disable the native write. If Iceberg's AWS property classes cannot be loaded, vendor `s3.*` / `client.*` keys @@ -605,8 +606,8 @@ back. Copy-on-write `DELETE`, `UPDATE` and `MERGE` and `rewrite_data_files` writ columns `_row_id` and `_last_updated_sequence_number` into the new data files, so that rewritten rows keep their ids, and the native writer does not write those columns. Appends, overwrites, `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` write the data columns only, and -Iceberg assigns the new rows their ids when it commits, so those run natively. Earlier Iceberg -versions write no lineage columns, so the rule never applies to them. +Iceberg assigns the new rows their ids when it commits, so those use the native writer. Earlier +Iceberg versions write no lineage columns, so the rule never applies to them. A format-version-1 table keeps a dropped partition field as a `void` field, and its source column can be dropped afterwards. A spec that mixes such a field with a live one falls back: iceberg-java @@ -619,7 +620,7 @@ writer produce different data files: distribution and ordering settings shape th identically on both paths, WAP / branch / snapshot properties act on the JVM committer, `write.avro.*` / `write.orc.*` apply only to formats already excluded, and merge-on-read settings route the write through `WriteDelta`, whose task writer remains iceberg-java rather -than the native writer. Most native eligibility rules are pinned by +than the native writer. Most native write eligibility rules are pinned by `CometIcebergWriteDetectionSuite`. Manifest `DataFile` metrics are assembled on the JVM before commit: each written file's @@ -631,19 +632,19 @@ bounds carried over from the native writer's tracked state. iceberg-java's metad suppression — are therefore applied by iceberg-java's own code regardless of what the native writer reports. -iceberg-java's writer takes these metrics from the footer it still holds in memory, but the native -path reads each footer back from storage: two small reads per written file, one for the footer's -length and one for the footer itself, made one file after another before the task finishes. On S3 -or GCS each read is a GET request, so the task waits two request round trips per file. That is -small next to uploading a large file, but a write that produces many small files, such as a fanout -write over many partitions, pays it for every one of them. +iceberg-java's writer takes these metrics from the footer it still holds in memory, but with the +native writer each footer is read back from storage: two small reads per written file, one for the +footer's length and one for the footer itself, made one file after another before the task +finishes. On S3 or GCS each read is a GET request, so the task waits two request round trips per +file. That is small next to uploading a large file, but a write that produces many small files, +such as a fanout write over many partitions, pays it for every one of them. ### Failure handling Eligibility is decided entirely at plan time. That includes the reflection surface: every iceberg-java class, method, and constructor the executor-side commit-message assembly uses is eagerly resolved by the eligibility gate on the driver, so an Iceberg release that moves any -of them declines the native path with a fall-back reason instead of failing tasks mid-write. +of them declines the native write with a fall-back reason instead of failing tasks mid-write. Once planned, the physical plan is fixed — there is no per-task re-decision or runtime switch back to the JVM writer. @@ -653,7 +654,7 @@ re-executes it — through the native writer again. Retries cannot collide: each attempt id is embedded in its data file names. The native writer's buffers are charged to Comet's memory pool, the off-heap budget Comet's other -native operators draw on, where iceberg-java's buffers sit on the JVM heap. A fanout write keeps a +Rust operators draw on, where iceberg-java's buffers sit on the JVM heap. A fanout write keeps a data file open for every partition a task writes to. Each open file holds the row group it is writing in memory, up to `write.parquet.row-group-size-bytes`, and on S3 or GCS also the last row group it flushed, which is uploaded once the next one is complete or the file closes. Each @@ -679,17 +680,17 @@ Partial results are never committed. The commit set is exactly the commit messag successful tasks — a failed task contributes none — and if the job fails, the driver-side commit operator aborts without committing anything. A failed task attempt also deletes the data files it created, as iceberg-java's writer abort does. The native writer records every -location it hands to a file writer, and cleanup has no ownership gap. Native keeps its cleanup -guard armed after yielding the output batch. The JVM reads those locations into a task failure -listener first, then polls the native output to EOF; that EOF is the acknowledgement that lets -native disarm. A failure in this narrow handoff window can therefore trigger best-effort deletion -on both sides, which is harmless. The native guard still covers a failed write, a task torn down -before the write completed — for example because the operator feeding it threw — and a failure -encoding the manifest or building the output batch. The handoff does not depend on decoding the -manifest: the locations are read before the manifest is decoded, so a failure in that decode still -cleans up. Cleanup never masks the original failure; anything it misses is invisible to every -reader, since readers resolve files through committed manifests only, and is reclaimed by Iceberg's -normal `remove_orphan_files` maintenance. +location it hands to a file writer, and cleanup has no ownership gap. The native writer keeps its +cleanup guard armed after yielding the output batch. The JVM reads those locations into a task +failure listener first, then polls the native writer's output to EOF; that EOF is the +acknowledgement that lets the native writer disarm. A failure in this narrow handoff window can +therefore trigger best-effort deletion on both sides, which is harmless. The native writer's guard +still covers a failed write, a task torn down before the write completed — for example because the +operator feeding it threw — and a failure encoding the manifest or building the output batch. The +handoff does not depend on decoding the manifest: the locations are read before the manifest is +decoded, so a failure in that decode still cleans up. Cleanup never masks the original failure; +anything it misses is invisible to every reader, since readers resolve files through committed +manifests only, and is reclaimed by Iceberg's normal `remove_orphan_files` maintenance. When one task fails, the tasks that had already completed leave committed-nothing data files too. The committer collects each task's commit message as that task finishes, so on a job @@ -716,8 +717,8 @@ individually below). No Iceberg reader bases a planning or correctness decision on these; they change the bytes of a data file but not what any reader computes from it: -- Footer key-value metadata differs: native files carry an `ARROW:schema` entry and no - `iceberg.schema` entry; iceberg-java files are the opposite. +- Footer key-value metadata differs: files from the native writer carry an `ARROW:schema` entry + and no `iceberg.schema` entry; iceberg-java files are the opposite. - The Parquet root schema element is named `arrow_schema` (iceberg-java: `table`). - `created_by` reads `Apache Iceberg (Comet)`, where iceberg-java's files carry parquet-mr's version string. @@ -736,8 +737,8 @@ a data file but not what any reader computes from it: each partition still buffering can make its choice before it has a full first page. Later rows could have changed parquet-mr's decision, so this difference is not limited to columns close to the size cut-off. Where the page size rather than `write.parquet.page-row-limit` ends a - column's first page, the native page ends at the first row past parquet-mr's size threshold, - while parquet-mr only ends it at its next periodic size check, so a column close to the + column's first page, the native writer's page ends at the first row past parquet-mr's size + threshold, while parquet-mr only ends it at its next periodic size check, so a column close to the cut-off can be decided the other way. Close to the cut-off both encodings take about the same space. A column that keeps its dictionary and later fills it falls back to plain on both writers, but parquet-rs's dictionary page then holds every entry, where parquet-mr's holds @@ -777,8 +778,8 @@ a data file but not what any reader computes from it: order of an unordered `SELECT *`. On a format-version 3 table it also decides which row ids the commit gives each file's rows, so the same rows can get different `_row_id` values from the two writers. The ids are unique either way, and across tasks iceberg-java's own assignment already - depends on the order in which the tasks finish. Only the sorted order is reproducible on the - native path: the order the native writer closes its files in depends on the order rows arrive in + depends on the order in which the tasks finish. Only the sorted order is reproducible with the + native writer: the order the native writer closes its files in depends on the order rows arrive in and on which partitions it closes early to free memory. Clustered and unpartitioned writes append in creation order on both paths and are unaffected. - Compressed page bytes are implementation-defined: the codec and any explicit level are @@ -806,16 +807,16 @@ iceberg-java's _writer-tracked_ state would have recorded, and both are analyzed - Float/double bounds involving zero may differ in sign: parquet-rs normalises footer statistics to min `-0.0` / max `+0.0` (the parquet-format recommendation), while - iceberg-java's writer-tracked bounds preserve the exact sign it saw. The native path's + iceberg-java's writer-tracked bounds preserve the exact sign it saw. The footer-derived manifest bounds inherit the normalised values — a strictly conservative widening that cannot change pruning decisions. - On Iceberg 1.9+, manifest `value_counts` / `null_value_counts` for float/double columns nested under a nullable struct count rows whose parent struct is null (they come from the - parquet footer), while iceberg-java's writer-tracked counts do not. The native counts are the + parquet footer), while iceberg-java's writer-tracked counts do not. The footer counts are the correct ones. iceberg-java's counts leave out rows whose struct is null. When every non-null struct in a file holds a value, iceberg-java records a null count of 0, and `IS NULL` prunes a file that contains matching rows - ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The native counts + ([apache/iceberg#17561](https://github.com/apache/iceberg/issues/17561)). The footer counts never prune such a file. `IS NOT NULL` pruning is the same for both. All content not listed above — the logical data, statistics values, and manifest metadata — @@ -832,9 +833,9 @@ its tracking issue. The cases where Comet falls back to Spark are listed under operator](#when-comet-falls-back-to-sparks-write-operator). Storage-specific issues are listed under the [S3](datasources.md#s3-limitations) and [Azure](datasources.md#azure-limitations) limitations. -Which entries apply depends on which native paths are enabled: +Which entries apply depends on which of these features are enabled: -| Native path | Setting | Default | +| Feature | Setting | Default | | -------------------- | --------------------------------------------- | ------- | | Iceberg scan | `spark.comet.scan.icebergNative.enabled` | `true` | | Iceberg scan of HDFS | `spark.comet.scan.icebergNative.hdfs.enabled` | `true` | @@ -868,8 +869,8 @@ The native Iceberg writer is disabled by default. With `spark.comet.write.iceber that iceberg-java reads as written ([#7008](https://github.com/apache/datafusion-comet/issues/7008)). -The physical differences that every native data file has, and why they are safe, are listed in -[Accepted divergences](#accepted-divergences). +The physical differences that every data file from the native writer has, and why they are safe, +are listed in [Accepted divergences](#accepted-divergences). ### Reads that fail where Spark succeeds diff --git a/docs/source/user-guide/latest/in-memory-cache.md b/docs/source/user-guide/latest/in-memory-cache.md index ef73632b925..8c37225af0a 100644 --- a/docs/source/user-guide/latest/in-memory-cache.md +++ b/docs/source/user-guide/latest/in-memory-cache.md @@ -38,10 +38,10 @@ It has to be set before the `SparkContext` starts. Comet's driver plugin chooses `spark.sql.cache.serializer` once, while the context is initializing, so an application keeps the cache format it started with however the config is set afterwards. The plugin installs Comet's serializer only if `spark.comet.enabled` and `spark.comet.exec.enabled` are enabled at that point -too, because an application that starts without native execution could not scan Comet's format -natively. It also keeps Spark's format when Comet shuffle is enabled but `spark.shuffle.manager` is -not one of Comet's shuffle managers, since Comet then disables itself, and when Kryo would reject -Comet's format; see [Kryo](#kryo). +too, because an application that starts with Comet operators disabled would store its caches in +Comet's format with only Spark operators to read them. It also keeps Spark's format when Comet +shuffle is enabled but `spark.shuffle.manager` is not one of Comet's shuffle managers, since Comet +then disables itself, and when Kryo would reject Comet's format; see [Kryo](#kryo). ## What changes when it is enabled @@ -175,7 +175,7 @@ on Comet either way and only the cache-scan boundary moves: on the left, Spark's `InMemoryTableScanExec` feeds those same Comet operators through a `CometSparkColumnarToColumnar` bridge, which the benchmark turns on with `spark.comet.convert.inMemoryCache.enabled`; on the right, `CometInMemoryTableScan` feeds them directly. Both columns read the same -Comet-written `CometCachedBatch`. These numbers are therefore "keep the cached scan native" against +Comet-written `CometCachedBatch`. These numbers are therefore "keep the cache scan in Comet" against "fall back to a Spark cache scan and convert", not Comet against Spark execution, and not a comparison with Spark's own cache format, which follows. @@ -196,7 +196,7 @@ Ryzen 9 7950X3D (JDK 17, Spark 4.1, release build): | Full projection (6 of 6) | 299 ms | 224 ms | 1.3x | A Spark operator above the cache scan, standing in for any operator Comet does not support, is -measured the same way, with Comet's aggregate turned off. With Comet's format, the native scan feeds +measured the same way, with Comet's aggregate turned off. With Comet's format, Comet's scan feeds that operator through a columnar-to-row transition: | Query shape | Spark's cache format | Comet's cache format | Relative | @@ -253,8 +253,8 @@ Comet's cached batch, Comet's driver plugin does not install Comet's serializer, Spark's format. Registrations made another way, through a registrator of the application's own or `spark.kryo.classesToRegister`, count as well. The plugin warns at startup when Kryo requires registration and has not registered every class `CometKryoRegistrator` registers. An application -that sets `spark.sql.cache.serializer` to Comet's serializer itself gets the error instead. Native -broadcast needs the same registrator even when the cache is disabled; see +that sets `spark.sql.cache.serializer` to Comet's serializer itself gets the error instead. Comet's +broadcast exchange needs the same registrator even when the cache is disabled; see [Kryo serialization](installation.md#kryo-serialization). Spark registers its own cached batch with Kryo only from Spark 4.1, so on earlier versions caching @@ -264,7 +264,7 @@ Spark's cached batch too. ## Limitations A relation cached after a sort, such as `sortWithinPartitions`, reports that order to the queries -that read it, and Spark removes the sorts it makes redundant. Some native joins over such a +that read it, and Spark removes the sorts it makes redundant. Some Comet joins over such a relation do not keep the order, so partitions above them can come back unsorted, and an operator that relies on the order, such as a sort-merge join, can return wrong rows. These cases are tracked in [#6787](https://github.com/apache/datafusion-comet/issues/6787), @@ -276,10 +276,10 @@ Reads that feed **Spark** operators rather than Comet ones can be slower than Sp format, because every cached batch is decoded from Arrow before Spark reads it. How a Spark operator reads a relation cached in Comet's format depends on the scan below it: -- With native execution enabled (`spark.comet.exec.enabled=true`), the cache is scanned by +- With Comet operators enabled (`spark.comet.exec.enabled=true`), the cache is scanned by `CometInMemoryTableScan`, and a Spark operator above it reads the scan's batches through `CometColumnarToRow`, as it would above any other Comet operator. -- With Comet enabled but native execution disabled, the cache is scanned by Spark's +- With Comet enabled but Comet operators disabled, the cache is scanned by Spark's `InMemoryTableScanExec`. When the operator directly above the scan takes part in whole-stage code generation, as filters, projections and aggregates do, Comet puts Spark's `ColumnarToRowExec` between the two, and the generated code reads the cached Arrow vectors directly, with no @@ -297,7 +297,7 @@ operator reads a relation cached in Comet's format depends on the scan below it: Whenever Spark's scan reads Comet's format, which also happens for a relation whose cached plan records `Dataset.observe` metrics, Comet records a fallback reason on the scan. -Measured by the same benchmark over the same 5M-row relation, with native execution off so that +Measured by the same benchmark over the same 5M-row relation, with Comet operators off so that Spark operators consume the cached data, Comet disabled for the row reader and enabled for the fused reader (Apple M4, JDK 17, Spark 4.1; the average of two runs): @@ -312,7 +312,7 @@ The fused reader is faster than Spark's own format for the narrowest reads and w the others. The row reader takes up to 1.6 times as long, and it is the only reader for relations wider than `spark.sql.codegen.maxFields`: reading every column of relations of 100, 200 and 1500 nullable `bigint` columns took 2.2 to 2.5 times as long as from Spark's format. A Spark operator -above Comet's native cache scan does not pay this; see [Performance](#performance). +above Comet's cache scan does not pay this; see [Performance](#performance). Most of what the row reader pays is decompression, so `lz4` narrows these gaps. Measured on an Apple M3 Ultra, the row reader takes up to 1.25 times as long as Spark's format with `lz4`, against diff --git a/docs/source/user-guide/latest/installation.md b/docs/source/user-guide/latest/installation.md index 05e17ef325b..85ba4b212bf 100644 --- a/docs/source/user-guide/latest/installation.md +++ b/docs/source/user-guide/latest/installation.md @@ -211,7 +211,7 @@ names `DataWritingCommandExec`. Query the data from the test source and check: -- INFO message shows the native Comet library has been initialized. +- INFO message shows Comet's native library has been initialized. - The query plan reflects Comet operators being used for this query instead of Spark ones ```scala @@ -270,7 +270,7 @@ Some cluster managers may require additional configuration, see ` (for example `com.amazonaws.auth.DefaultAWSCredentialsProviderChain`), the class you named in `fs.s3a.aws.credentials.provider` is one that plain Spark/Hadoop accepts but Comet's native reader does not reimplement. Comet ships two built-in `CometS3CredentialProvider` adapters that fix this with a one-line config change; you leave your existing `fs.s3a.aws.credentials.provider` untouched. -These adapters cover the Parquet native scan path only. Enabling one is opt-in: naming it is what activates it. Note the native side forwards the `fs.s3a.*` config to `initialize()` for _any_ provider class named on the Parquet path, not just these two adapters: a vendor `CometS3CredentialProvider` that received an empty map in Comet 1.0 now receives the `fs.s3a.*` subset (including static keys), and one cached instance per distinct `fs.s3a.*` config rather than one per bucket. This is additive, but a provider that logs the map or treats an empty map as "the Parquet path" should be aware of it. +These adapters cover the native Parquet scan only. Enabling one is opt-in: naming it is what activates it. Note the Rust side forwards the `fs.s3a.*` config to `initialize()` for _any_ provider class named on the Parquet path, not just these two adapters: a vendor `CometS3CredentialProvider` that received an empty map in Comet 1.0 now receives the `fs.s3a.*` subset (including static keys), and one cached instance per distinct `fs.s3a.*` config rather than one per bucket. This is additive, but a provider that logs the map or treats an empty map as "the Parquet path" should be aware of it. The adapters and the AWS SDK are loaded through the class loader that loaded Comet, so `hadoop-aws` and the matching AWS SDK must be visible from there — put them on the same classpath as Comet (`spark.executor.extraClassPath` / `spark.driver.extraClassPath`, or `$SPARK_HOME/jars`), not only via `--packages`. If they are only on the user-jar loader, credential resolution fails at planning with `NoClassDefFoundError` before the adapter can report anything useful. @@ -149,7 +149,7 @@ Without the config set, no credential-related log lines appear at startup; nativ This is automatic; there is nothing to configure to get the protection, and it does not involve a bridge class. It applies to native Iceberg **reads and writes** (both build their `FileIO` through the same path). The raw-Parquet path is unaffected: it uses the AWS SDK default chain, which already retries and stops on a provider error rather than downgrading. -On EKS with [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), executors assume the application role by calling STS `AssumeRoleWithWebIdentity`. Under a large concurrent startup burst (many executors times many cores, all fetching credentials at once), STS can throttle that call. On the Iceberg path, opendal's default credential chain does not retry the throttle and falls through to the EKS node instance role, which usually lacks bucket access, so every native read then fails with a hard `403 AccessDenied` even though the throttle was transient. +On EKS with [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), executors assume the application role by calling STS `AssumeRoleWithWebIdentity`. Under a large concurrent startup burst (many executors times many cores, all fetching credentials at once), STS can throttle that call. On the Iceberg path, opendal's default credential chain does not retry the throttle and falls through to the EKS node instance role, which usually lacks bucket access, so every read then fails with a hard `403 AccessDenied` even though the throttle was transient. When Comet detects IRSA (both `AWS_WEB_IDENTITY_TOKEN_FILE` and `AWS_ROLE_ARN` are set), a region is set, and the catalog configures no explicit credentials, the native Iceberg reader resolves web-identity credentials itself instead of using opendal's default chain. It builds the STS client from the AWS SDK's fully-resolved config, so region, `AWS_USE_FIPS_ENDPOINT`, `AWS_USE_DUALSTACK_ENDPOINT`, and any profile/custom STS endpoint are honored. The provider: @@ -191,7 +191,7 @@ spark.sql.catalog..s3.path-style-access = true (path-style endpoin ## Writing a bridge -Comet's native scan paths (`object_store` for raw Parquet, `opendal` via `iceberg-rust` for Iceberg) bypass Hadoop S3A entirely. The standard `AWSCredentialsProvider.getCredentials()` has no path argument, so vendors that issue per-path STS credentials cannot expose them through it. The `CometS3CredentialProvider` SPI fills that gap. +Comet's native scans (`object_store` for raw Parquet, `opendal` via `iceberg-rust` for Iceberg) bypass Hadoop S3A entirely. The standard `AWSCredentialsProvider.getCredentials()` has no path argument, so vendors that issue per-path STS credentials cannot expose them through it. The `CometS3CredentialProvider` SPI fills that gap. Implement `org.apache.comet.cloud.s3.CometS3CredentialProvider`: @@ -209,7 +209,7 @@ public interface CometS3CredentialProvider extends AutoCloseable { } ``` -The class must have a public no-arg constructor. `getCredentialsForPath` may be invoked concurrently from many native tokio worker threads; the implementation must be thread-safe. The supplied `CometS3CredentialContext` exposes `getBucket()`, `getPath()`, and `getMode()`; future Comet releases may add accessors here without changing the method signature, so vendors compiled against today's API stay binary-compatible. +The class must have a public no-arg constructor. `getCredentialsForPath` may be invoked concurrently from many Rust tokio worker threads; the implementation must be thread-safe. The supplied `CometS3CredentialContext` exposes `getBucket()`, `getPath()`, and `getMode()`; future Comet releases may add accessors here without changing the method signature, so vendors compiled against today's API stay binary-compatible. ### Lifecycle @@ -387,7 +387,7 @@ public final class IcebergRESTVendedS3Provider implements CometS3CredentialProvi | Value | Used for | | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `READ` | All native scan paths (raw Parquet, Iceberg). | +| `READ` | All native scans (raw Parquet, Iceberg). | | `WRITE` | The native Iceberg writer (see [Iceberg Writes](iceberg.md#writes)). If the configured provider fails to initialize, the write fails rather than falling back to the default credential chain. | A `WRITE` credential is not implicitly read-capable. Vendors that need read-during-write workflows include the required read permissions in the IAM policy attached to their `WRITE` credentials. diff --git a/docs/source/user-guide/latest/tuning.md b/docs/source/user-guide/latest/tuning.md index 9b22839b568..3a4d6f50000 100644 --- a/docs/source/user-guide/latest/tuning.md +++ b/docs/source/user-guide/latest/tuning.md @@ -43,12 +43,12 @@ guide is split into the following pages: Rows with many columns, long strings, or nested columns such as arrays of structs make every batch larger, so queries over such data spill more. For these workloads: -- Choose the codec for spill files from native sorts, aggregations, and joins, which Comet +- Choose the codec for spill files from Rust-implemented sorts, aggregations, and joins, which Comet compresses with `lz4` by default. See [Compressing Spill Files](tuning/memory.md#compressing-spill-files). - Lower the batch size if wide batches cause spilling or out-of-memory errors. See [Batch Size](tuning/memory.md#batch-size). -- Check which shuffles can run natively when partitioning keys are nested. See +- Check which shuffles can use native shuffle when partitioning keys are nested. See [Native Shuffle](tuning/shuffle.md#native-shuffle). - Watch for row/columnar conversions, which are expensive for such schemas. See [Wide or Deeply Nested Schemas](tuning/transitions.md#wide-or-deeply-nested-schemas). @@ -69,10 +69,11 @@ cores from the first of these that applies: On any other cluster manager, such as Mesos, Comet cannot tell how many cores the executor has when `spark.executor.cores` is not set. It starts one worker thread for each processor available to the executor's JVM, and -logs a warning unless `COMET_WORKER_THREADS` is set. Too few worker threads cost more than a few idle ones: native plans -that read no input from the JVM, such as a native Parquet scan feeding a sort, run entirely on the worker threads, so -tasks share them when an executor runs more tasks at once than it has worker threads. The worker and blocking thread -counts can be overridden using the environment variables `COMET_WORKER_THREADS` and `COMET_MAX_BLOCKING_THREADS`. +logs a warning unless `COMET_WORKER_THREADS` is set. Too few worker threads cost more than a few idle ones: +Rust-implemented plans that read no input from the JVM, such as a native Parquet scan feeding a sort, run entirely +on the worker threads, so tasks share them when an executor runs more tasks at once than it has worker threads. The +worker and blocking thread counts can be overridden using the environment variables `COMET_WORKER_THREADS` and +`COMET_MAX_BLOCKING_THREADS`. ## Metrics Overhead diff --git a/docs/source/user-guide/latest/tuning/celeborn.md b/docs/source/user-guide/latest/tuning/celeborn.md index 48249ec7dd1..9e82b6b31c4 100644 --- a/docs/source/user-guide/latest/tuning/celeborn.md +++ b/docs/source/user-guide/latest/tuning/celeborn.md @@ -29,7 +29,7 @@ implementation. | `spark.comet.shuffle.rss.maxFrameBytes` | 64 MiB | Maximum size of one complete encoded frame. The admission budget can reduce the effective limit. | | `spark.comet.shuffle.rss.maxInFlightBytes` | 512 MiB | Shared memory admission budget for map attempts using the same executor-side remote shuffle client. | -Admission includes Arrow encoding workspace and overlapping native, JNI, and client frame +Admission includes Arrow encoding workspace and overlapping Rust, JNI, and client frame copies. An ordinary uncompressed frame needs roughly seven times its size plus schema and codec overhead. The default 512 MiB budget accommodates ordinary frames up to the default 64 MiB frame limit. Compression reduces transmitted bytes but still needs workspace for the @@ -43,6 +43,6 @@ encoding workspace as well as the encoded frame. Raising only `maxFrameBytes` ma when `maxInFlightBytes` is the limiting budget. Larger budgets also increase potential executor memory use. -Native frames use Comet's [shuffle compression settings](shuffle.md#shuffle-compression). +Native shuffle frames use Comet's [shuffle compression settings](shuffle.md#shuffle-compression). The raw Celeborn path bypasses Celeborn's additional row compression and decompression, so frames are not compressed twice. diff --git a/docs/source/user-guide/latest/tuning/memory.md b/docs/source/user-guide/latest/tuning/memory.md index 25510c8e9c3..cba70ea1c78 100644 --- a/docs/source/user-guide/latest/tuning/memory.md +++ b/docs/source/user-guide/latest/tuning/memory.md @@ -33,9 +33,9 @@ overhead to cover the part of its footprint that no pool tracks. See [Configurin Comet shares an off-heap memory pool with Spark. The size of the pool is specified by `spark.memory.offHeap.size`. The pool is a shared _budget_ rather than a shared -allocator: Comet's native operators allocate from the Rust heap rather than from JVM off-heap -memory, but every reservation they make is charged against this same pool, so Comet and Spark's own -off-heap consumers draw down one number. +allocator: Comet's Rust-implemented operators allocate from the Rust heap rather than from JVM +off-heap memory, but every reservation they make is charged against this same pool, so Comet and +Spark's own off-heap consumers draw down one number. A larger pool lets the operators that can spill, such as sorts, aggregations and the shuffle writer, keep more in memory before they spill to disk. An operator that cannot spill fails the task when it cannot reserve the memory it @@ -75,15 +75,15 @@ The valid pool types are: - `fair_unified` (default when `spark.memory.offHeap.enabled=true` is set) - `greedy_unified` -Both pool types are shared by all the native plans in the same Spark task. A task can run more than -one native plan at a time, for example the native operators on either side of a union or a +Both pool types are shared by all the Rust plans in the same Spark task. A task can run more than +one Rust plan at a time, for example the Rust-implemented operators on either side of a union or a coalesce. The shared pool ensures that their combined memory usage stays within the per-task limit. The `fair_unified` pool keeps memory for the operators that cannot spill, such as the final merge of a sort or the build side of a hash join. Spark limits each task to an even share of `spark.memory.offHeap.size` among the tasks -using it, and the pool divides `spark.memory.offHeap.size` the same way among the tasks running native plans. Within +using it, and the pool divides `spark.memory.offHeap.size` the same way among the tasks running Rust plans. Within a task, it limits each operator that can spill to an even share of what the operators that cannot spill leave of the -task's share, counting the operators of all of the task's native plans. The operators that cannot spill take what +task's share, counting the operators of all of the task's Rust plans. The operators that cannot spill take what Spark grants. So an operator that can spill sometimes spills even when there is memory to spare, in order to leave it for the task's other operators. A limit that small can be less than one batch of wide rows, and a sort cannot spill a batch it has not yet reserved, so it then fails with "Not enough memory to continue external sort". Give the @@ -143,7 +143,7 @@ offer. ## Sizing the Overhead from the Memory Usage Log -While Comet native plans are running, each executor logs its native memory usage at INFO level, +While Comet's Rust plans are running, each executor logs its native memory usage at INFO level, one line every 10 seconds for the whole executor: ``` @@ -152,26 +152,26 @@ Comet native memory usage: allocated 5412.3 MiB, reserved 3890.0 MiB (16 native - `allocated` is the memory allocated through Comet's Rust global allocator and not yet freed, whether or not a pool tracks it. It excludes allocations made by native libraries outside that - allocator, including the embedded Python interpreter and PyArrow when native Arrow UDFs are - enabled. + allocator, including the Python interpreter and PyArrow that Comet embeds to run scalar Arrow + UDFs. - `reserved` is the part that Comet's memory pools have reserved from Spark's off-heap memory. It is charged against `spark.memory.offHeap.size`, so the container already has room for it. A pool sometimes has to track memory that Spark could not grant, such as a spilled batch read back from disk while the off-heap memory is full. `reserved` leaves that memory out, since nothing charges it against `spark.memory.offHeap.size`. - `JVM Arrow allocated` is the Arrow memory Comet holds on the JVM side, such as batches read from - Comet's in-memory cache, broadcast data, and batches exchanged with native code or Python workers. - The part imported from native was allocated by Comet's native code, so `allocated` already counts - it, and the rest the JVM allocated itself. A pool reserves the JVM's part only while a native - operator holds on to one of its batches, such as a sort buffering its input. The split follows - which allocator a buffer is charged to rather than where it was allocated, so treat the JVM's - part as an estimate. + Comet's in-memory cache, broadcast data, and batches exchanged with Rust code or Python workers. + The `imported from native` part was allocated by Comet's Rust code, so `allocated` already counts + it, and the rest the JVM allocated itself. A pool reserves the JVM's part only while a + Rust-implemented operator holds on to one of its batches, such as a sort buffering its input. The + split follows which allocator a buffer is charged to rather than where it was allocated, so treat + the JVM's part as an estimate. When the application writes an event log and runs the Comet plugin, the event log also records a summary of these samples; see [Reading the Memory Usage Log from the Event Log]. Comet's untracked memory is what Comet holds outside the JVM heap that Spark's off-heap pool does -not account for: `allocated`, plus the JVM Arrow figure less the part imported from native, minus +not account for: `allocated`, plus the JVM Arrow figure less the `imported from native` part, minus `reserved`. It is the part of Comet's footprint that has to fit in `spark.executor.memoryOverhead`, alongside the JVM's own non-heap memory. To size the overhead from it: @@ -184,11 +184,11 @@ alongside the JVM's own non-heap memory. To size the overhead from it: non-heap memory, and add the most untracked memory seen on any executor. 3. Add a margin on top. The log can miss the true peak between samples, and none of the figures includes the allocator's fragmentation and retained pages, or memory allocated by native C - libraries such as zstd. With native Arrow UDFs, budget the embedded Python interpreter and - PyArrow allocations in addition to the untracked memory calculated from the log. A PyArrow - result buffer later imported into the JVM can appear in the `imported from native` figure that - the estimate subtracts, even though Rust's `allocated` does not count it. Use the executor's - peak resident memory to size this additional overhead. + libraries such as zstd. When Comet runs scalar Arrow UDFs in embedded Python, budget the + interpreter and PyArrow allocations in addition to the untracked memory calculated from the log. + A PyArrow result buffer later imported into the JVM can appear in the `imported from native` + figure that the estimate subtracts, even though Rust's `allocated` does not count it. Use the + executor's peak resident memory to size this additional overhead. For example, a 16 GiB executor derives an overhead of 1638 MiB. If the line above has the most untracked memory in its log, that is 5412.3 + (310.4 - 96.2) - 3890.0 = 1736.5 MiB. The overhead @@ -211,10 +211,10 @@ cluster, whose workers do not limit an executor's memory, or with any other clus Look more closely before raising the overhead if the untracked memory keeps growing through a run rather than levelling off: memory that is not being released will exhaust any overhead eventually. -The executor logs one more line after its last native plan finishes, and an `allocated` or +The executor logs one more line after its last Rust plan finishes, and an `allocated` or `JVM Arrow allocated` figure there that grows from one query to the next points the same way. -`spark.comet.memory.logInterval` is read when an executor starts its first Comet native plan, so set +`spark.comet.memory.logInterval` is read when an executor starts its first Comet Rust plan, so set it when the application is submitted. Set it to `0` to turn the log off. [Reading the Memory Usage Log from the Event Log]: #reading-the-memory-usage-log-from-the-event-log @@ -231,11 +231,11 @@ over the cluster and often go away with their containers. The driver writes, and flushes, each event of its event log as it goes, so it does not write every sample. For each executor, it takes the samples a minute at a time and writes two of them: the one with the most untracked memory, and the last, which in the minute an executor goes idle is the one -after its last native plan finishes. It writes them at the executor's first heartbeat a minute or +after its last Rust plan finishes. It writes them at the executor's first heartbeat a minute or more after the first of them arrived, which comes every `spark.executor.heartbeatInterval`, 10 seconds by default, busy or not, and when the executor is removed, such as when the cluster manager kills it, or the application stops. So each executor adds about two events a minute while it runs -native plans, however short the interval, and each minute's two reach the event log within about a +Rust plans, however short the interval, and each minute's two reach the event log within about a minute of that minute's first sample arriving. The event with the most untracked memory for an executor is then its sample with the most, apart from samples it takes while the application stops, and samples from the last minute before a driver exits without stopping the application. @@ -284,7 +284,7 @@ already freed but the allocator has not returned to the operating system. Comet' the system allocator, which on Linux is usually glibc's `malloc`. glibc keeps freed memory in arenas so that it can reuse it, and a thread allocates from the arena it is attached to. A lot of what Comet's operators free stays there: each spill drops the batches the operator buffered, and the -next stage's native plans often run on different threads, attached to other arenas, so they reuse +next stage's Rust plans often run on different threads, attached to other arenas, so they reuse only part of what the previous stage left behind. The `allocated` figure in the log drops when the memory is freed, but the executor's resident memory does not, so it grows from one stage and query to the next until the container's limit is reached. @@ -318,7 +318,7 @@ makes it ignore the flag silently. To check whether this is the problem before c configuration, `jcmd System.trim_native_heap` trims a running executor once and prints how much memory it returned. -A trim is not free: while it runs it takes each arena's lock in turn, so native allocations on that +A trim is not free: while it runs it takes each arena's lock in turn, so allocations on that arena wait for it. On one workload, the nested TPC-H benchmark at scale factor 1000 with 64 GiB of off-heap memory per executor, an interval of 5 seconds kept the executors' resident memory at 61 to 70 GiB in 106 GiB containers, where without it a query got 11 executors killed. A single trim @@ -345,9 +345,9 @@ flushes sorted spill files. A value larger than `spark.comet.batchSize` is cappe ## Limiting Spill Disk Usage -Native operators that spill to disk (aggregate, sort, shuffle) are bounded by -`spark.comet.maxTempDirectorySize` (default 100 GB). The operators of one Comet native plan share -the limit. A Spark task can run more than one native plan at a time, for example the native +Rust-implemented operators that spill to disk (aggregate, sort, shuffle) are bounded by +`spark.comet.maxTempDirectorySize` (default 100 GB). The operators of one Comet Rust plan share +the limit. A Spark task can run more than one Rust plan at a time, for example the Rust-implemented operators on either side of a union or a coalesce, so an executor running `N` concurrent tasks may use more than `N` times this value on shared local disks. If the limit is reached, further spills fail and the query errors out. Raise this on workloads with large sort/aggregate/shuffle spills, or @@ -356,7 +356,7 @@ multiple of this value. ## Compressing Spill Files -`spark.comet.exec.spill.compression.codec` sets the codec for the files that native sorts, +`spark.comet.exec.spill.compression.codec` sets the codec for the files that Rust-implemented sorts, aggregations, sort-merge joins, and nested loop joins spill to disk. The default is `lz4`, which Spark also uses for its own spill files by default. `zstd` writes smaller files, which reduces spill I/O and how much of `spark.comet.maxTempDirectorySize` they use, but takes more CPU to diff --git a/docs/source/user-guide/latest/tuning/operators.md b/docs/source/user-guide/latest/tuning/operators.md index 058a6a00cd7..39efea1bd24 100644 --- a/docs/source/user-guide/latest/tuning/operators.md +++ b/docs/source/user-guide/latest/tuning/operators.md @@ -43,10 +43,10 @@ limit, or to a non-positive value to convert every eligible join regardless of s A left-semi join returns each left row if any right row matches. For an unfiltered batch join on a single integer equality key (`=`), set `spark.comet.exec.sortMergeJoin.rightKeyReduction.enabled=true` -to keep one copy of each repeated key per batch before the native sort. The option is off by +to keep one copy of each repeated key per batch before the Comet sort. The option is off by default. The right sort must read the key column directly from a Comet shuffle. With the -`fair_unified` memory pool, the native planner checks at first execution for memory held by another -Spark consumer in the task and skips reduction if it finds any, logging the reason at +`fair_unified` memory pool, Comet's Rust planner checks at first execution for memory held by +another Spark consumer in the task and skips reduction if it finds any, logging the reason at debug level. This best-effort check does not cover consumers that allocate memory later. The reducer skips reduction until it has seen 8,192 rows in a partition and forwards @@ -55,7 +55,7 @@ partition. Batches with too few repeated keys also pass through; after an unprof batch, reduction is only tried again on a larger batch in the same partition. The join reports rows entering, leaving, and bypassing reduction, plus time spent on all attempts. -Left-anti joins and joins with residual conditions keep their ordinary native sort. +Left-anti joins and joins with residual conditions keep their ordinary Comet sort. Extending reduction to those joins or across small batches is separate follow-up work. ### Broadcast Joins @@ -85,10 +85,10 @@ runs in Spark, are sent uncompressed too. ### Join Runtime Filters -Set `spark.comet.exec.join.dynamicFilter.enabled=true` to try experimental native hash join runtime +Set `spark.comet.exec.join.dynamicFilter.enabled=true` to try experimental Comet hash join runtime filtering. It is disabled by default. Eligible joins are inner joins with one direct signed integer -key (`TINYINT`, `SMALLINT`, `INT`, or `BIGINT`) and one native partition per input within each task. -Both broadcast and shuffled hash joins support either Spark build side. Unsupported joins keep +key (`TINYINT`, `SMALLINT`, `INT`, or `BIGINT`) and one DataFusion partition per input within each +task. Both broadcast and shuffled hash joins support either Spark build side. Unsupported joins keep their existing execution path. Once the build completes, its key domain filters probe batches before the hash probe. Eligible @@ -106,7 +106,7 @@ adaptations cannot fail. Nested column pruning still reads only the requested st with supplied file statistics also skip reader attachment. These cases still use runtime filtering on decoded batches. -Filters stay within the task's native plan and do not propagate across Spark exchanges or JVM/Arrow +Filters stay within the task's Rust plan and do not propagate across Spark exchanges or JVM/Arrow boundaries. A shuffled hash join can still filter probe batches after shuffle, but it cannot send its filter back to an earlier scan stage. Compare the [runtime-filter and scan metrics](../metrics.md#hash-joins) with the setting disabled to distinguish reduced hash-probe work from reader I/O savings. @@ -138,7 +138,7 @@ retain their own input/output metrics. #### Join runtime filters through residual predicates -Native join runtime filtering is enabled with +Comet's join runtime filtering is enabled with `spark.comet.exec.join.dynamicFilter.enabled`. It can reject probe rows whose keys are absent from the completed build side and use eligible Parquet readers to prune row groups. By default, an intervening residual filter permits reader propagation @@ -184,7 +184,7 @@ files of the same keys packed into one split, can shuffle many times more rows t skipping disabled. Compare the shuffle write metrics with the setting enabled and disabled before enabling it for a workload. -Eligibility is conservative for the whole fused native plan: any unsupported partial accumulator, +Eligibility is conservative for the whole fused Rust plan: any unsupported partial accumulator, Spark `PartialMerge`, or mixed-mode aggregate disables skipping in that plan. Multi-argument `COUNT` and other accumulators are not admitted. Distribution-required grouping-only stages still fully deduplicate, and non-native-shuffle plans retain ordinary aggregation. @@ -207,8 +207,8 @@ accumulators and modes. ## Local TopK Fusion -Set `spark.comet.exec.topK.fusion.enabled=true` to run an eligible local TopK in the same native -execution as its Parquet scan. This experimental optimization is disabled by default. It currently +Set `spark.comet.exec.topK.fusion.enabled=true` to run an eligible local TopK in the same Rust +plan as its Parquet scan. This experimental optimization is disabled by default. It currently supports a direct native Parquet scan ordered by one signed integer column (`TINYINT`, `SMALLINT`, `INT`, or `BIGINT`). Both sort directions and null orderings are supported. Other inputs use the existing TopK execution path. @@ -217,7 +217,7 @@ Each scan partition keeps enough candidates for both `LIMIT` and `OFFSET`. With Comet shuffles those candidates and performs the final TopK. With one partition, it reuses the local ordering without building a second heap. The final stage applies the offset and output projection. -Fusion reduces the work of passing scan batches between native execution blocks. Fusion alone still +Fusion reduces the work of passing scan batches between separate Rust plans. Fusion alone still reads all input rows. [TopK reader pruning](#topk-reader-pruning) is enabled separately. Fusion can also reduce overlap between scan decoding and TopK processing, so some workloads may run slower. Compare enabled and disabled runs with your @@ -258,19 +258,19 @@ See [TopK metrics](../metrics.md#local-topk). ## Optimizing Sorting on Floating-Point Values Comet normalizes NaN payloads and signed zeros in `FLOAT` and `DOUBLE` ordering keys, including floating-point values -nested in arrays and structs, so `ORDER BY`, window ordering and range partitioning on them match Spark and stay native -even with `spark.comet.exec.strictFloatingPoint=true`. Only the comparison key is normalized; returned values keep their -original NaN representation and zero sign. - -A key that nests values in an array or struct whose type can hold a null element or field stays native under -`ASC NULLS LAST` and `DESC NULLS FIRST` too: Comet separates the key's root null placement from its nested ordering -([#6476](https://github.com/apache/datafusion-comet/issues/6476)). A `RANGE` window frame that has to find a row's -peers over such a key still falls back to Spark +nested in arrays and structs, so `ORDER BY`, window ordering and range partitioning on them match Spark and stay in +the Comet pipeline even with `spark.comet.exec.strictFloatingPoint=true`. Only the comparison key is normalized; +returned values keep their original NaN representation and zero sign. + +A key that nests values in an array or struct whose type can hold a null element or field stays in the Comet pipeline +under `ASC NULLS LAST` and `DESC NULLS FIRST` too: Comet separates the key's root null placement from its nested +ordering ([#6476](https://github.com/apache/datafusion-comet/issues/6476)). A `RANGE` window frame that has to find a +row's peers over such a key still falls back to Spark ([#6477](https://github.com/apache/datafusion-comet/issues/6477)). `sort_array` sorts array elements rather than ordering rows. It follows Spark's floating-point ordering as well, so it -also stays native with `spark.comet.exec.strictFloatingPoint=true`. +also runs in Rust with `spark.comet.exec.strictFloatingPoint=true`. -The default `array_sort` comparator also runs natively. It puts null array elements last while comparing nulls +The default `array_sort` comparator also runs in Rust. It puts null array elements last while comparing nulls inside nested arrays and structs below non-null values. A custom `array_sort` comparator or an array with a non-default string collation continues through the JVM codegen dispatcher. diff --git a/docs/source/user-guide/latest/tuning/shuffle.md b/docs/source/user-guide/latest/tuning/shuffle.md index 2fdaf35682c..e90839a9924 100644 --- a/docs/source/user-guide/latest/tuning/shuffle.md +++ b/docs/source/user-guide/latest/tuning/shuffle.md @@ -45,7 +45,7 @@ back to Spark for shuffle operations. ### Native Shuffle -Comet provides a fully native shuffle implementation, which generally provides the best performance. Native shuffle +Comet provides a native shuffle implementation, which generally provides the best performance. Native shuffle supports `HashPartitioning`, `RangePartitioning`, and `SinglePartition`, plus `RoundRobinPartitioning` when enabled (see [Round-Robin Partitioning](../compatibility/operators.md#round-robin-partitioning)). Range partitioning keys must be scalar types. Hash partitioning keys must be scalar types unless @@ -73,16 +73,16 @@ Comet Columnar shuffle is JVM-based and supports `HashPartitioning`, `RoundRobin `SinglePartition`. This shuffle implementation supports complex data types as partitioning keys. By default, Comet will convert a Spark `ShuffleExchangeExec` to columnar shuffle even when the shuffle's child is a -non-Comet (Spark) plan. The benefit is that the next query stage can start as native Comet execution, since the +non-Comet (Spark) plan. The benefit is that the next query stage can start in the Comet pipeline, since the shuffle output is already in Arrow format. The cost is a row to columnar conversion at the shuffle boundary on the write side. To restrict columnar shuffle to cases where the child is already a Comet plan, set `spark.comet.shuffle.convertFromSparkPlan.enabled=false`. Shuffles whose child is a Spark plan will then be left -as native Spark shuffles, which avoids the row to columnar conversion but means the downstream stage will also start +as Spark's built-in shuffles, which avoids the row to columnar conversion but means the downstream stage will also start on Spark. Setting `spark.comet.convert.shuffleInput.enabled=true` makes such a shuffle use native shuffle instead. Comet converts the child's rows to Arrow, as the `spark.comet.convert.*` settings do for Spark scans, and shuffles the -batches natively. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning +batches in Rust. With hash partitioning that is usually faster than columnar shuffle, while with range partitioning the two come out about even. This applies only where native shuffle supports the partitioning and Comet can convert all of the child's columns, which rules out calendar intervals, and arrays and maps other than `array` and `map`. A shuffle that hashes a string or a value computed from a string stays on columnar shuffle, so @@ -115,7 +115,7 @@ even when both of those aggregates run on Spark. The native shuffle writer buffers a task's output in memory and spills it to disk. By default (`spark.comet.shuffle.native.maxBufferBytes=0`) it spills only when the memory pool denies an allocation, so it holds as much of the task's share of the off-heap pool as it can get. That can leave a sort or an aggregation in the same -task without memory, and the large buffers it frees at the end of the task can stay resident in the native allocator, +task without memory, and the large buffers it frees at the end of the task can stay resident in the memory allocator, adding to the executor's memory footprint. Setting `spark.comet.shuffle.native.maxBufferBytes` makes the writer spill whenever it has buffered that many bytes. @@ -142,5 +142,5 @@ are written as before. `spark.shuffle.compress=false` disables compression for Comet's native shuffle only. It has no effect on the JVM columnar shuffle writer, which always compresses spill files with the codec above. Disabling -compression on the native path may result in faster shuffle times in certain environments, such as -single-node setups with fast NVMe drives, at the expense of increased disk space usage. +compression on the native shuffle path may result in faster shuffle times in certain environments, +such as single-node setups with fast NVMe drives, at the expense of increased disk space usage. diff --git a/docs/source/user-guide/latest/tuning/transitions.md b/docs/source/user-guide/latest/tuning/transitions.md index d0f15e9cda0..ce4ace20bad 100644 --- a/docs/source/user-guide/latest/tuning/transitions.md +++ b/docs/source/user-guide/latest/tuning/transitions.md @@ -23,9 +23,9 @@ When a query stage contains many operators that fall back to Spark row-based exe repeated columnar-to-row and row-to-columnar conversions that dominate stage runtime. Set `spark.comet.exec.transitionRevert.enabled=true` to have Comet revert the entire stage to Spark row execution when the number of columnar-to-row transitions exceeds -`spark.comet.exec.transitionRevert.maxTransitions` (default `2`). This trades native execution of a small -subset of operators for eliminating conversion overhead across the stage. A stage is not reverted when it holds a -native aggregate whose intermediate buffer Spark cannot exchange with Comet across a stage boundary, because +`spark.comet.exec.transitionRevert.maxTransitions` (default `2`). This trades running a small subset of operators +in Comet for eliminating conversion overhead across the stage. A stage is not reverted when it holds a +Comet aggregate whose intermediate buffer Spark cannot exchange with Comet across a stage boundary, because reverting it would split that aggregate between the two engines. ## Wide or Deeply Nested Schemas @@ -35,8 +35,8 @@ columnar-to-row conversion is especially expensive because the conversion work s columns and nested fields. If profiling shows these conversions dominating a query over such a schema, set `spark.comet.exec.transitionRevert.enabled=true` and lower `spark.comet.exec.transitionRevert.maxTransitions` (default `2`) to `1`. Note that this reverts every stage that -exceeds the threshold to Spark row-based execution — Comet removes the stage's native operators rather than running a -mix of native and fallback operators joined by repeated conversions — which can be cheaper than paying the +exceeds the threshold to Spark row-based execution — Comet removes the stage's Comet operators rather than running a +mix of Comet and fallback operators joined by repeated conversions — which can be cheaper than paying the expensive conversions again and again. ## Experimental: Direct Columnar-to-Row Conversion diff --git a/spark/src/main/scala/org/apache/comet/CometConf.scala b/spark/src/main/scala/org/apache/comet/CometConf.scala index 02a0c82425f..78231369f74 100644 --- a/spark/src/main/scala/org/apache/comet/CometConf.scala +++ b/spark/src/main/scala/org/apache/comet/CometConf.scala @@ -88,7 +88,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC) .doc( "Whether to enable Comet extension for Spark. When this is turned on, Spark will use " + - "Comet to read Parquet data source. Note that to enable native vectorized execution, " + + "Comet to read Parquet data source. Note that to run queries in Comet, " + "both this config and `spark.comet.exec.enabled` need to be enabled.") .booleanConf .createWithEnvVarOrDefault("ENABLE_COMET", true) @@ -105,7 +105,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_TESTING) .doc( "Whether to enable native Parquet write through Comet. When enabled, " + - "Comet will intercept Parquet write operations and execute them natively. This " + + "Comet will intercept Parquet write operations and execute them in Rust. This " + "feature is highly experimental and only partially implemented. It should not " + "be used in production.") .booleanConf @@ -116,7 +116,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_SCAN) .doc( "Whether to enable native Iceberg table scan using iceberg-rust. When enabled, " + - "Iceberg tables are read directly through native execution, bypassing Spark's " + + "Iceberg tables are read directly in Rust, bypassing Spark's " + "DataSource V2 API for better performance.") .booleanConf .createWithDefault(true) @@ -145,7 +145,7 @@ object CometConf extends ShimCometConf { val COMET_ICEBERG_NATIVE_WRITE_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.write.iceberg.enabled") .category(CATEGORY_EXEC) - .doc("Whether Comet plans Iceberg writes and writes their data files natively. When " + + .doc("Whether Comet plans Iceberg writes and writes their data files in Rust. When " + "enabled, Comet replaces Spark's combined V2 write operator with a file writer " + "(inside AQE) under a committer (outside AQE), and writes the data files of each " + "eligible write with its native (iceberg-rust) writer. Other writes use Iceberg's own " + @@ -194,7 +194,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.scan.csv.v2.enabled") .category(CATEGORY_TESTING) .doc( - "Whether to use the native Comet V2 CSV reader for improved performance. " + + "Whether to use Comet's native V2 CSV reader for improved performance. " + "Default: false (uses standard Spark CSV reader) " + "Experimental: Performance benefits are workload-dependent.") .booleanConf @@ -232,7 +232,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.convert.parquet.enabled") .category(CATEGORY_EXEC) .doc( - "When enabled, data from Spark (non-native) Parquet v1 and v2 scans will be converted to " + + "When enabled, data from Spark (non-Comet) Parquet v1 and v2 scans will be converted to " + "Arrow format.") .booleanConf .createWithDefault(false) @@ -241,7 +241,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.convert.json.enabled") .category(CATEGORY_EXEC) .doc( - "When enabled, data from Spark (non-native) JSON v1 and v2 scans will be converted to " + + "When enabled, data from Spark (non-Comet) JSON v1 and v2 scans will be converted to " + "Arrow format.") .booleanConf .createWithDefault(false) @@ -249,9 +249,8 @@ object CometConf extends ShimCometConf { val COMET_CONVERT_FROM_CSV_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.convert.csv.enabled") .category(CATEGORY_EXEC) - .doc( - "When enabled, data from Spark (non-native) CSV v1 and v2 scans will be converted to " + - "Arrow format.") + .doc("When enabled, data from Spark (non-Comet) CSV v1 and v2 scans will be converted to " + + "Arrow format.") .booleanConf .createWithDefault(false) @@ -259,7 +258,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.convert.text.enabled") .category(CATEGORY_EXEC) .doc( - "When enabled, data from Spark (non-native) Text v1 and v2 scans will be converted to " + + "When enabled, data from Spark (non-Comet) Text v1 and v2 scans will be converted to " + "Arrow format. Arrow strings must be valid UTF-8, so Comet replaces bytes that are " + "not valid UTF-8, for example in a Latin-1 file, with U+FFFD, and results that " + "depend on the original bytes can differ from Spark. Enable this only for Text " + @@ -272,7 +271,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC) .doc( "When enabled, the rows of `spark.range` and SQL `range()` that Comet does not generate " + - "natively (see `spark.comet.exec.range.enabled`) will be converted to Arrow format.") + "in Rust (see `spark.comet.exec.range.enabled`) will be converted to Arrow format.") .booleanConf .createWithDefault(false) @@ -280,7 +279,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.convert.inMemoryCache.enabled") .category(CATEGORY_EXEC) .doc( - "When enabled, data from in-memory cached tables that Comet's native cache scan does " + + "When enabled, data from in-memory cached tables that Comet's cache scan does " + "not read (see `spark.comet.exec.inMemoryCache.enabled`), such as tables cached in " + "Spark's default format, will be converted to Arrow format.") .booleanConf @@ -319,7 +318,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC) .doc("When enabled, the output of typed Dataset operations, such as `map`, `flatMap`, " + "`mapPartitions` and `groupByKey(...).mapGroups`, will be converted to Arrow format so " + - "that the operators above them can run natively. The user function still runs in " + + "that the operators above them can run in Comet. The user function still runs in " + "Spark. This pays off when the operators above do enough work, such as an " + "aggregation over many groups, and can be slower when they are cheap, such as an " + "aggregation over a few groups after a selective filter.") @@ -342,12 +341,12 @@ object CometConf extends ShimCometConf { val COMET_EXEC_ENABLED: ConfigEntry[Boolean] = conf(s"$COMET_EXEC_CONFIG_PREFIX.enabled") .category(CATEGORY_EXEC) .doc( - "Whether to enable Comet native vectorized execution for Spark. This controls whether " + - "Spark should convert operators into their Comet counterparts and execute them in " + - "native space. Note: each operator is associated with a separate config in the " + + "Whether to enable Comet operators other than shuffle for Spark. This controls whether " + + "Spark should convert operators into their Comet counterparts and run them in " + + "Comet. Note: each operator is associated with a separate config in the " + "format of `spark.comet.exec..enabled` at the moment, and both the " + - "config and this need to be turned on, in order for the operator to be executed in " + - "native.") + "config and this need to be turned on, in order for the operator to run in " + + "Comet.") .booleanConf .createWithDefault(true) @@ -355,10 +354,11 @@ object CometConf extends ShimCometConf { conf(s"$COMET_EXEC_CONFIG_PREFIX.makeDate.operatorFallback.enabled") .category(CATEGORY_EXEC) .doc( - "Keep affected make_date operators in Spark when native batching can evaluate rows " + - "that Spark skips. When false, allow native operators even if make_date can raise " + - "an error on a row Spark would not evaluate. This does not disable expression-level " + - "evaluation safeguards or change make_date's invalid-date handling.") + "Keep affected make_date operators in Spark when Comet, which evaluates whole batches, " + + "can evaluate rows that Spark skips. When false, allow Comet operators even if " + + "make_date can raise an error on a row Spark would not evaluate. This does not " + + "disable expression-level evaluation safeguards or change make_date's invalid-date " + + "handling.") .booleanConf .createWithDefault(true) @@ -388,7 +388,7 @@ object CometConf extends ShimCometConf { defaultValue = false, notes = Some( "Opt-in: DataFusion's CrossJoinExec buffers the left input and cannot spill " + - "(https://github.com/apache/datafusion/issues/26383). Runs natively only without a " + + "(https://github.com/apache/datafusion/issues/26383). Runs in Comet only without a " + "join condition and when each input reports exactly one partition")) val COMET_EXEC_SORT_MERGE_JOIN_RIGHT_KEY_REDUCTION_ENABLED: ConfigEntry[Boolean] = conf(s"$COMET_EXEC_CONFIG_PREFIX.sortMergeJoin.rightKeyReduction.enabled") @@ -402,7 +402,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.exec.existenceJoin.enabled") .category(CATEGORY_ENABLE_EXEC) .doc( - "Whether to enable native ExistenceJoin, produced when EXISTS / NOT EXISTS / IN is " + + "Whether to run ExistenceJoin in Comet, produced when EXISTS / NOT EXISTS / IN is " + "combined with another predicate via OR. Sort-merge joins, residual (non-equi) join " + "conditions, computed (non-column) join keys, and NOT IN combined with OR fall back " + "to Spark.") @@ -435,8 +435,8 @@ object CometConf extends ShimCometConf { "range", defaultValue = false, notes = Some( - "When enabled, Comet generates the rows of `spark.range` and SQL `range()` in native " + - "code, so the operators above them can run natively. It is off by default because " + + "When enabled, Comet generates the rows of `spark.range` and SQL `range()` in Rust, " + + "so the operators above them can run in Comet. It is off by default because " + "it can be slower than Spark when those operators are only cheap expressions, such " + "as a filter, which Spark compiles together with the range into one loop")) val COMET_EXEC_EMPTY_RELATION_ENABLED: ConfigEntry[Boolean] = @@ -449,41 +449,40 @@ object CometConf extends ShimCometConf { defaultValue = false, notes = Some( "Only takes effect on Spark 3.5 and later. Spark 3.4 has no MergeRowsExec. On Spark " + - "4.1 and later, stock V2 writers retain Spark MergeRowsExec for MergeSummary; native " + + "4.1 and later, stock V2 writers retain Spark MergeRowsExec for MergeSummary; Comet's " + "MergeRows is used only where the enclosing Comet write path preserves that contract.")) val COMET_EXEC_IN_MEMORY_CACHE_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.exec.inMemoryCache.enabled") .category(CATEGORY_EXEC) - .doc( - "Whether to enable Comet native scans and fused Spark reads of in-memory cached tables. " + - "Requires spark.comet.enabled=true. On Spark 4.0 and later, it also decides how an " + - "adaptive cached plan is built: when the final stage ends in a Spark row operator, " + - "Comet writes the rows straight into Arrow batches for the cache instead of " + - "having Spark fill its own column vectors first. On Spark 3.4, AQE cannot coalesce the " + - "shuffle partitions of a union that Comet runs with a cached relation in one branch. " + - "At startup, this setting also decides " + - "whether CometDriverPlugin installs Comet's cache serializer, which stores cached " + - "data in Arrow format. The plugin installs it only if spark.comet.enabled and " + - "spark.comet.exec.enabled are also enabled at startup, and only with one of Comet's " + - "shuffle managers while Comet shuffle is enabled. " + - "Because spark.sql.cache.serializer is a " + - "static config, the cached format is fixed for the application, and disabling this " + - "or spark.comet.enabled at runtime sends cached scans back to Spark's execution path " + - "without the fused reader. Relations whose schema Comet's Arrow writer does not " + - "support are always cached in Spark's default " + - "format. Each cached batch is stored as one Arrow IPC record batch with per-buffer " + - "compression, and a scan copies out only the buffers of the columns it " + - "projected, so the unselected ones are never decompressed. Eligible Spark " + - "whole-stage codegen consumers read cached vectors directly when vectorized cache " + - "reading is enabled; other Spark row consumers use a reusable row buffer. Decoding " + - "costs can still make wide numeric reads slower than Spark's default cache. With " + - "spark.kryo.registrationRequired=true, the plugin installs it only if Kryo has " + - "registered Comet's cached batch, as " + - "spark.kryo.registrator=org.apache.comet.CometKryoRegistrator does when set before " + - "creating the SparkContext, because Kryo would otherwise reject a cached block as " + - "soon as it is serialized, including the disk half of the default " + - "MEMORY_AND_DISK storage level.") + .doc("Whether to enable Comet scans and fused Spark reads of in-memory cached tables. " + + "Requires spark.comet.enabled=true. On Spark 4.0 and later, it also decides how an " + + "adaptive cached plan is built: when the final stage ends in a Spark row operator, " + + "Comet writes the rows straight into Arrow batches for the cache instead of " + + "having Spark fill its own column vectors first. On Spark 3.4, AQE cannot coalesce the " + + "shuffle partitions of a union that Comet runs with a cached relation in one branch. " + + "At startup, this setting also decides " + + "whether CometDriverPlugin installs Comet's cache serializer, which stores cached " + + "data in Arrow format. The plugin installs it only if spark.comet.enabled and " + + "spark.comet.exec.enabled are also enabled at startup, and only with one of Comet's " + + "shuffle managers while Comet shuffle is enabled. " + + "Because spark.sql.cache.serializer is a " + + "static config, the cached format is fixed for the application, and disabling this " + + "or spark.comet.enabled at runtime sends cached scans back to Spark's execution path " + + "without the fused reader. Relations whose schema Comet's Arrow writer does not " + + "support are always cached in Spark's default " + + "format. Each cached batch is stored as one Arrow IPC record batch with per-buffer " + + "compression, and a scan copies out only the buffers of the columns it " + + "projected, so the unselected ones are never decompressed. Eligible Spark " + + "whole-stage codegen consumers read cached vectors directly when vectorized cache " + + "reading is enabled; other Spark row consumers use a reusable row buffer. Decoding " + + "costs can still make wide numeric reads slower than Spark's default cache. With " + + "spark.kryo.registrationRequired=true, the plugin installs it only if Kryo has " + + "registered Comet's cached batch, as " + + "spark.kryo.registrator=org.apache.comet.CometKryoRegistrator does when set before " + + "creating the SparkContext, because Kryo would otherwise reject a cached block as " + + "soon as it is serialized, including the disk half of the default " + + "MEMORY_AND_DISK storage level.") .booleanConf .createWithDefault(false) @@ -585,9 +584,9 @@ object CometConf extends ShimCometConf { conf("spark.comet.exec.nativeArrowPythonUDF.enabled") .category(CATEGORY_EXEC) .doc( - "Experimental: execute Spark 4.1+ scalar @arrow_udf functions inside the Comet native " + - "pipeline via PyO3. Requires a native library built with the python-udf Cargo feature " + - "and a compatible Python/PyArrow installation on every executor.") + "Experimental: execute Spark 4.1+ scalar @arrow_udf functions inside a Rust-implemented " + + "Comet plan via PyO3. Requires a native library built with the python-udf Cargo " + + "feature and a compatible Python/PyArrow installation on every executor.") .booleanConf .createWithDefault(false) @@ -600,9 +599,10 @@ object CometConf extends ShimCometConf { val COMET_MEMORY_LOG_INTERVAL: ConfigEntry[Long] = conf("spark.comet.memory.logInterval") .category(CATEGORY_TUNING) .doc( - "How often each executor logs its native memory usage at INFO level while Comet native " + - "plans are running: the bytes the native allocator has handed out, the bytes reserved " + - "in Comet's memory pools, and the Arrow memory Comet holds on the JVM side. The " + + "How often each executor logs its native memory usage at INFO level while Comet's " + + "Rust-implemented plans are running: the bytes the Rust allocator has handed out, " + + "the bytes reserved in Comet's memory pools, and the Arrow memory Comet holds on the " + + "JVM side. The " + "difference between the first two is native memory that the pools are not accounting " + "for. The executor logs one line per interval however many tasks are " + "running, and one more after the last plan finishes. It logs a warning when the " + @@ -611,7 +611,7 @@ object CometConf extends ShimCometConf { "executor also sends its samples to the driver, which writes the one with the most " + "untracked memory and the last of every minute to the event log, as " + "CometExecutorMemoryUsage events. This is an " + - "executor setting, read when an executor starts its first Comet native plan, so it " + + "executor setting, read when an executor starts its first Rust-implemented plan, so it " + "must be set when the application is submitted. An invalid value disables the log " + s"with a warning. Set to 0 to disable. $TUNING_GUIDE.") .timeConf(TimeUnit.MILLISECONDS) @@ -627,7 +627,7 @@ object CometConf extends ShimCometConf { "Note that this requires setting `spark.shuffle.manager` to " + "`org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager` or " + "`org.apache.spark.sql.comet.execution.shuffle.CometCelebornShuffleManager`. " + - "Celeborn requires explicit native mode and compatible application settings. " + + "Celeborn requires explicit `native` mode and compatible application settings. " + "`spark.shuffle.manager` must be set before starting the Spark application and " + "cannot be changed during the application.") .booleanConf @@ -638,8 +638,8 @@ object CometConf extends ShimCometConf { .withAlternative(s"$COMET_EXEC_CONFIG_PREFIX.shuffle.directRead.enabled") .category(CATEGORY_SHUFFLE) .doc( - "When enabled, native operators that consume shuffle output will read " + - "compressed shuffle blocks directly in native code, bypassing Arrow FFI. " + + "When enabled, Rust-implemented operators that consume shuffle output will read " + + "compressed shuffle blocks directly in Rust, bypassing Arrow FFI. " + "Applies to both native shuffle and JVM columnar shuffle. " + "Requires spark.comet.shuffle.enabled to be true.") .booleanConf @@ -661,7 +661,7 @@ object CometConf extends ShimCometConf { conf(s"$COMET_EXEC_CONFIG_PREFIX.broadcast.enabled") .category(CATEGORY_EXEC) .doc( - "Whether to force enabling broadcasting for Comet native operators. " + + "Whether to force enabling broadcasting for Comet operators. " + "Comet broadcast feature will be enabled automatically by " + "Comet extension. But for unit tests, we need this feature to force enabling it " + "for invalid cases. So this config is only used for unit test.") @@ -721,7 +721,7 @@ object CometConf extends ShimCometConf { conf(s"$COMET_EXEC_CONFIG_PREFIX.aggregate.skipPartial.enabled") .category(CATEGORY_EXEC) .doc( - "Experimental opt-in: let a native partial aggregate stop aggregating once its input " + + "Experimental opt-in: let a Comet partial aggregate stop aggregating once its input " + "looks mostly distinct, and send the rest of the task's rows to the shuffle " + "unaggregated. Only applies to partial aggregates that feed Comet native shuffle " + "and whose aggregate functions, if any, are all single-argument COUNT. The check " + @@ -734,13 +734,12 @@ object CometConf extends ShimCometConf { val COMET_EXEC_JOIN_DYNAMIC_FILTER_ENABLED: ConfigEntry[Boolean] = conf(s"$COMET_EXEC_CONFIG_PREFIX.join.dynamicFilter.enabled") .category(CATEGORY_EXEC) - .doc( - "Experimental opt-in: use a completed native hash join build's key domain to prune " + - "eligible native Parquet row groups and filter probe batches before the hash probe. " + - "Supports inner joins with one direct signed integer key and one native partition " + - "per input, including both Spark build sides. The probe filter remains active " + - "when reader pruning is unavailable. Unsupported joins retain their existing " + - "execution path. Filters do not cross Spark exchanges or JVM/Arrow boundaries.") + .doc("Experimental opt-in: use a completed Comet hash join build's key domain to prune " + + "eligible row groups in native Parquet scans and filter probe batches before the hash " + + "probe. Supports inner joins with one direct signed integer key and one DataFusion " + + "partition per input, including both Spark build sides. The probe filter remains " + + "active when reader pruning is unavailable. Unsupported joins retain their existing " + + "execution path. Filters do not cross Spark exchanges or JVM/Arrow boundaries.") .booleanConf .createWithDefault(false) @@ -748,7 +747,7 @@ object CometConf extends ShimCometConf { conf(s"$COMET_EXEC_CONFIG_PREFIX.join.dynamicFilter.allowDeterministicFilterPushdown") .category(CATEGORY_EXEC) .doc( - "Allow native join runtime filters to reach Parquet readers through deterministic " + + "Allow Comet's join runtime filters to reach Parquet readers through deterministic " + "residual predicates. Requires spark.comet.exec.join.dynamicFilter.enabled. " + "This can skip expression errors on rows eliminated before the residual runs, " + "including division-by-zero and ANSI cast errors. Errors on retained rows and " + @@ -763,10 +762,10 @@ object CometConf extends ShimCometConf { conf(s"$COMET_EXEC_CONFIG_PREFIX.topK.fusion.enabled") .category(CATEGORY_EXEC) .doc( - "Experimental opt-in: run an eligible local TopK in the same native execution as its " + - "Parquet scan. Supports one direct signed integer sort key. This changes the local " + - "execution pipeline and can reduce scan/TopK overlap, so it may be slower for some " + - "workloads.") + "Experimental opt-in: run an eligible local TopK in the same Rust-implemented plan as " + + "its Parquet scan. Supports one direct signed integer sort key. This changes the " + + "local execution pipeline and can reduce scan/TopK overlap, so it may be slower for " + + "some workloads.") .booleanConf .createWithDefault(false) @@ -775,9 +774,9 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC) .doc( "Experimental opt-in: use a local TopK heap's improving threshold to prune " + - "eligible native Parquet input. Requires spark.comet.exec.topK.fusion.enabled. " + + "eligible native Parquet scan input. Requires spark.comet.exec.topK.fusion.enabled. " + "Supports one direct signed integer sort key. Thresholds are local to each " + - "native execution and do not cross Spark exchanges or JVM/Arrow boundaries. " + + "Rust-implemented plan and do not cross Spark exchanges or JVM/Arrow boundaries. " + "Unsupported readers retain ordinary TopK execution.") .booleanConf .createWithDefault(false) @@ -787,10 +786,10 @@ object CometConf extends ShimCometConf { conf("spark.comet.nativeUdf.enabled") .category(CATEGORY_EXEC) .doc( - "Whether native UDF libraries may be loaded. A native UDF is a shared library that " + + "Whether UDF shared libraries may be loaded. A UDF library is a shared library that " + "runs with the full privileges of the JVM process, so an operator who does not want " + "any can turn the feature off. When false, `CometNativeUDF.register` fails on the " + - "driver, and a plan that names a native UDF library fails on the executor rather " + + "driver, and a plan that names a UDF library fails on the executor rather " + "than loading it.") .booleanConf .createWithDefault(true) @@ -800,7 +799,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.nativeUdf.allowedPaths") .category(CATEGORY_EXEC) .doc( - "A comma-separated list of directories that native UDF libraries may be loaded from. " + + "A comma-separated list of directories that UDF shared libraries may be loaded from. " + "When set, a library is loaded only if its path is absolute and, after symlinks and " + "`..` are resolved, lies under one of these directories, which are themselves " + "resolved the same way. A path outside them fails on the driver at registration and " + @@ -818,9 +817,9 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC) .doc("Whether to route Spark `ScalaUDF` expressions through Comet's " + "Arrow-direct codegen dispatcher. When enabled, a supported ScalaUDF is compiled into " + - "a per-batch kernel that reads and writes Arrow vectors directly from native " + - "execution. When disabled, plans containing a ScalaUDF fall back to Spark for the " + - "enclosing operator. The same dispatcher backs the regex family (`rlike`, " + + "a per-batch kernel that reads and writes Arrow vectors directly from inside a " + + "Rust-implemented plan. When disabled, plans containing a ScalaUDF fall back to " + + "Spark for the enclosing operator. The same dispatcher backs the regex family (`rlike`, " + "`regexp_replace`, `split`, `regexp_extract`, `regexp_extract_all`, `regexp_instr`) so " + "those route through it by default as well.") .booleanConf @@ -839,13 +838,13 @@ object CometConf extends ShimCometConf { .category(CATEGORY_SHUFFLE) .doc( "Whether to allow nested types (struct, array, map) as hash partitioning keys in " + - "Comet native shuffle. Comet's native Murmur3 kernel hashes nested types " + + "Comet native shuffle. Comet's Rust Murmur3 kernel hashes nested types " + "recursively and shares that kernel, and Spark's seed, with the `hash` expression, " + "so partition assignment matches Spark. A map key additionally requires the " + "`mapsort` normalization that Spark 4.0 and later insert, so maps are rejected on " + "earlier versions. Disabled by default until the performance of the nested hashing " + "paths has been measured: shapes whose leaves are not primitives, such as " + - "`array>`, fall back to a per-element code path in the native hasher " + + "`array>`, fall back to a per-element code path in the Rust hasher " + "rather than a vectorized one.") .booleanConf .createWithDefault(false) @@ -937,7 +936,7 @@ object CometConf extends ShimCometConf { .doc( "When enabled, Comet will convert a Spark `ShuffleExchangeExec` to a Comet columnar " + "shuffle even when its child is a non-Comet (Spark) plan. Disable to leave such " + - "shuffles as native Spark shuffles, restricting Comet shuffle to cases where the " + + "shuffles as Spark's built-in shuffles, restricting Comet shuffle to cases where the " + "child is already a Comet plan.") .booleanConf .createWithDefault(true) @@ -1057,9 +1056,9 @@ object CometConf extends ShimCometConf { .withAlternative("spark.comet.columnar.shuffle.batch.size") .category(CATEGORY_SHUFFLE) .doc( - "Batch size when writing out sorted spill files on the native side. A value larger " + + "Batch size when writing out sorted spill files on the Rust side. A value larger " + "than the batch size (i.e., `spark.comet.batchSize`) is capped at the batch size, so " + - "that the native operators after the shuffle do not receive larger batches than " + + "that the Comet operators after the shuffle do not receive larger batches than " + "expected.") .intConf .checkValue(v => v > 0, "Batch size must be positive") @@ -1154,14 +1153,14 @@ object CometConf extends ShimCometConf { conf("spark.comet.shuffle.rss.maxInFlightBytes") .category(CATEGORY_SHUFFLE) .doc( - "Maximum shuffle bytes admitted concurrently by native Comet map attempts sharing " + - "an executor-side remote shuffle client. Admission includes native encoding " + - "scratch and overlapping native, JNI, and remote shuffle frame copies. " + + "Maximum shuffle bytes admitted concurrently by Comet native shuffle map attempts " + + "sharing an executor-side remote shuffle client. Admission includes Rust-side " + + "encoding scratch and overlapping Rust, JNI, and remote shuffle frame copies. " + "A frame must fit its codec and Arrow workspace as well as its encoded bytes; " + "ordinary uncompressed frames need approximately seven times their size plus " + "schema and transport overhead. Compressed frames also reserve workspace for " + "their uncompressed data. Admission is acquired before encoding. " + - "Encrypted native RSS is not supported; " + + "Encrypted native shuffle to a remote shuffle service is not supported; " + "use ordinary Spark shuffle when spark.io.encryption.enabled is true.") .bytesConf(ByteUnit.BYTE) .checkValue( @@ -1173,12 +1172,11 @@ object CometConf extends ShimCometConf { val COMET_DEBUG_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.debug.enabled") .category(CATEGORY_EXEC) - .doc( - "Whether to enable debug mode for Comet. " + - "When enabled, Comet will do additional checks for debugging purpose. For example, " + - "validating array when importing arrays from JVM at native side. Note that these " + - "checks may be expensive in performance and should only be enabled for debugging " + - "purpose.") + .doc("Whether to enable debug mode for Comet. " + + "When enabled, Comet will do additional checks for debugging purpose. For example, " + + "validating arrays on the Rust side before exporting them to the JVM. Note that these " + + "checks may be expensive in performance and should only be enabled for debugging " + + "purpose.") .booleanConf .createWithDefault(false) @@ -1211,7 +1209,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC_EXPLAIN) .doc( "When this setting is enabled, Comet will provide a tree representation of " + - "the native query plan before execution and again after execution, with " + + "the Rust plan before execution and again after execution, with " + "metrics.") .booleanConf .createWithDefault(false) @@ -1238,7 +1236,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC_EXPLAIN) .doc( "When this setting is enabled, Comet will provide logging explaining the reason(s) " + - "why a query stage cannot be executed natively. Set this to false to " + + "why a query stage cannot run in Comet. Set this to false to " + "reduce the amount of logging.") .booleanConf .createWithDefault(false) @@ -1258,9 +1256,9 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC_EXPLAIN) .doc( "When enabled, Comet logs the plan it would have executed, with a coverage " + - "summary, to the driver log and then lets Spark execute the query unchanged. Native " + - "planning failures are not detected, so the coverage can be optimistic. Requires " + - "`spark.comet.exec.enabled=true`.") + "summary, to the driver log and then lets Spark execute the query unchanged. " + + "Planning failures on the Rust side are not detected, so the coverage can be " + + "optimistic. Requires `spark.comet.exec.enabled=true`.") .booleanConf .createWithDefault(false) @@ -1269,11 +1267,11 @@ object CometConf extends ShimCometConf { .category(CATEGORY_TESTING) .doc( "Test-only. When enabled, Comet throws if it declines to convert an operator that it " + - "could otherwise have converted (all children are already native) without recording a " + - "fallback reason on the operator or on any of its expressions. Without this check, a " + - "serde that returns `None` and forgets to state a reason silently produces a generic " + - "' is not supported' message instead of a visible failure. Enabled for all " + - "Comet test suites via `CometTestBase`.") + "could otherwise have converted (all children are already Comet operators) without " + + "recording a fallback reason on the operator or on any of its expressions. Without " + + "this check, a serde that returns `None` and forgets to state a reason silently " + + "produces a generic ' is not supported' message instead of a visible " + + "failure. Enabled for all Comet test suites via `CometTestBase`.") .internal() .booleanConf .createWithDefault(false) @@ -1282,15 +1280,15 @@ object CometConf extends ShimCometConf { conf("spark.comet.testing.strictTimestampLabels.enabled") .category(CATEGORY_TESTING) .doc( - "Test-only. When enabled, Comet fails a query whose native plan labels a timestamp " + - "with a different timezone than Comet's convention, which is `UTC` for " + - "`TimestampType` and no timezone for `TimestampNTZType`. Every native output batch " + - "is checked against the Spark output types, and the casts that scans and the native " + - "shuffle writer would otherwise apply to repair a label fail instead. A label never " + - "changes the values, so comparing results cannot detect it, but inside a native plan " + - "a wrong label breaks comparisons and type coercion, and a missing one makes kernels " + - "read the values as wall-clock time. Enabled for all Comet test suites via " + - "`CometTestBase`.") + "Test-only. When enabled, Comet fails a query whose Rust-implemented plan labels a " + + "timestamp with a different timezone than Comet's convention, which is `UTC` for " + + "`TimestampType` and no timezone for `TimestampNTZType`. Every output batch of such " + + "a plan is checked against the Spark output types, and the casts that scans and the " + + "native shuffle writer would otherwise apply to repair a label fail instead. A label " + + "never changes the values, so comparing results cannot detect it, but inside a " + + "Rust-implemented plan a wrong label breaks comparisons and type coercion, and a " + + "missing one makes kernels read the values as wall-clock time. Enabled for all " + + "Comet test suites via `CometTestBase`.") .internal() .booleanConf .createWithDefault(false) @@ -1324,7 +1322,7 @@ object CometConf extends ShimCometConf { conf("spark.comet.exec.memoryPool") .category(CATEGORY_TUNING) .doc( - "The type of memory pool to be used for Comet native execution when running Spark in " + + "The type of memory pool to be used by Comet's Rust code when running Spark in " + "off-heap mode. Available pool types are `greedy_unified` and `fair_unified`. " + s"$TUNING_GUIDE.") .stringConf @@ -1379,7 +1377,7 @@ object CometConf extends ShimCometConf { .doc("Test-only. When set, the Comet driver plugin registers a query listener that " + "records every Iceberg write the application runs, the writer that ran it (Comet's " + "native writer, the JVM writer behind Comet's split operator, or Spark's own V2 write) " + - "and the reasons Comet recorded for not writing natively, as JSON lines in this " + + "and the reasons Comet recorded for not using its native writer, as JSON lines in this " + "directory. `dev/ci/summarize-iceberg-writes.py` summarizes them. The Iceberg Spark " + "test jobs set it through the environment variable so the Iceberg diffs need no change.") .stringConf @@ -1416,15 +1414,16 @@ object CometConf extends ShimCometConf { val COMET_CONVERT_BROADCAST_BUILD_SIDE_ENABLED: ConfigEntry[Boolean] = conf("spark.comet.convert.broadcastBuildSide.enabled") .category(CATEGORY_EXEC) - .doc("When enabled, Comet converts a file or V2 scan that it cannot scan natively (for " + + .doc("When enabled, Comet converts a file or V2 scan that it has no native scan for (for " + "example an ORC source) to Arrow when that scan feeds a broadcast join's build side and " + - "every scan on the probe side is already native, so the broadcast join runs natively " + - "over the large probe. The build side is usually small (auto-broadcasts are capped by " + - "the broadcast threshold), so the row to Arrow copy is usually cheap - though an " + - "explicit BROADCAST hint or a selective filter above the scan can make it larger. With " + - "dynamic partition pruning the dimension is also read a second time for the pruning " + - "subquery. Parquet, JSON, CSV and Text scans are converted only by their own " + - "`spark.comet.convert` configs. Set this to false to keep such build sides on Spark.") + "every scan on the probe side is already a native scan, so the broadcast join runs in " + + "Comet over the large probe. The build side is usually small (auto-broadcasts are " + + "capped by the broadcast threshold), so the row to Arrow copy is usually cheap - " + + "though an explicit BROADCAST hint or a selective filter above the scan can make it " + + "larger. With dynamic partition pruning the dimension is also read a second time for " + + "the pruning subquery. Parquet, JSON, CSV and Text scans are converted only by their " + + "own `spark.comet.convert` configs. Set this to false to keep such build sides on " + + "Spark.") .booleanConf .createWithDefault(true) @@ -1443,7 +1442,7 @@ object CometConf extends ShimCometConf { "Parquet files may contain unsigned 8-bit integers (UINT_8) which Spark maps to " + "ShortType. When this config is true (default), Comet falls back to Spark for " + "ShortType columns because we cannot distinguish signed INT16 (safe) from unsigned " + - "UINT_8 (may produce different results). Set to false to allow native execution of " + + "UINT_8 (may produce different results). Set to false to allow Comet to read " + "ShortType columns if you know your data does not contain unsigned UINT_8 columns " + s"from improperly encoded Parquet files. $COMPAT_GUIDE.") .booleanConf @@ -1488,7 +1487,7 @@ object CometConf extends ShimCometConf { .category(CATEGORY_EXEC) .doc( "Whether to enable Comet metrics reporting through Spark's external monitoring system. " + - "When enabled, Comet exposes metrics such as native operators, Spark operators, " + + "When enabled, Comet exposes metrics such as Comet operators, Spark operators, " + "queries planned, transitions, and acceleration ratio. These metrics can be " + "visualized through tools like Grafana when a metrics sink (e.g., Prometheus) is " + "configured. Disabled by default because Spark plan traversal adds overhead and " + @@ -1506,11 +1505,11 @@ object CometConf extends ShimCometConf { conf(s"spark.hadoop.$COMET_LIBHDFS_SCHEMES_KEY") .category(CATEGORY_SCAN) .doc( - "Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side accesses " + + "Defines filesystem schemes (e.g., hdfs, webhdfs) that Comet's Rust code accesses " + "via libhdfs, separated by commas. Valid only when built with hdfs-opendal feature " + "enabled. Read from the Hadoop configuration, so it must be set before the " + - "SparkSession is created. A scheme listed only through a reader option is not read " + - "natively. libhdfs reads the settings of a listed scheme, such as " + + "SparkSession is created. A native scan does not read a scheme listed only through " + + "a reader option. libhdfs reads the settings of a listed scheme, such as " + "`fs..impl`, from the executor's `core-site.xml`.") .stringConf .createOptional @@ -1539,8 +1538,8 @@ object CometConf extends ShimCometConf { .category(CATEGORY_TUNING) .doc( "The maximum amount of data (in bytes) stored inside the temporary directories " + - "used by native operators when spilling. Applied to each Comet native plan " + - "separately, and a Spark task can run more than one native plan at a time, so an " + + "used by Rust-implemented operators when spilling. Applied to each Rust-implemented " + + "plan separately, and a Spark task can run more than one such plan at a time, so an " + "executor running N concurrent tasks may use more than N times this value on shared " + "local disks. Once the limit is reached, further spills will fail and the query will " + "error out.") @@ -1552,9 +1551,9 @@ object CometConf extends ShimCometConf { conf(s"$COMET_EXEC_CONFIG_PREFIX.spill.compression.codec") .category(CATEGORY_TUNING) .doc( - "The codec used to compress the files that native sorts, aggregations, and joins " + - "spill to disk. lz4 and zstd are supported, and none disables compression. zstd " + - "writes smaller files than lz4 but uses more CPU. Comet's shuffle writers use " + + "The codec used to compress the files that Rust-implemented sorts, aggregations, and " + + "joins spill to disk. lz4 and zstd are supported, and none disables compression. " + + "zstd writes smaller files than lz4 but uses more CPU. Comet's shuffle writers use " + s"${COMET_SHUFFLE_COMPRESSION_CODEC.key} instead. $TUNING_GUIDE.") .stringConf .checkValues(Set("lz4", "zstd", "none")) @@ -1566,7 +1565,7 @@ object CometConf extends ShimCometConf { .doc( "Development and testing configuration option to allow DataFusion configs set in " + "Spark configuration settings starting with `spark.comet.datafusion.` to be passed " + - "into native execution.") + "into DataFusion.") .booleanConf .createWithDefault(false)