Offset overflow - Codegen dispatcher writes each batch's output into one Arrow vector #7498

Part of #7267.

Problem

The JVM codegen dispatcher evaluates an expression over a whole Arrow batch and writes a string or binary result into one vector per batch with setSafe (spark/src/main/scala/org/apache/comet/codegen/CometBatchKernelCodegenOutput.scala:258-283). When the batch's output passes Integer.MAX_VALUE bytes, Arrow throws OversizedAllocationException and the task fails. Spark evaluates one row at a time and succeeds.

The dispatcher is on by default for ScalaUDF (spark.comet.exec.scalaUDF.codegen.enabled) and for the regex family (rlike, regexp_replace, split, regexp_extract, regexp_extract_all, regexp_instr). A UDF that returns serialized or JSON-encoded values of about 300 KB per row overflows at 8192 rows, as does a regexp_replace that grows large text.

This is the JVM side of the limit in #7247: one expression's result for one batch has to fit in 2 GiB.

Line references are for main at 4e6b122.

Workaround

Lower spark.comet.batchSize, or set spark.comet.exec.scalaUDF.codegen.enabled=false to run the enclosing operator in Spark.

Proposed fix

Have the kernel stop before the write that would pass the limit and report how many rows it wrote, so the native side can split the input batch at that row and evaluate the rest separately. That evaluates each row once, which matters for nondeterministic UDFs. It needs the projection to emit more than one batch per input batch, the same constraint as #7247.

Tests

A real overflow needs 2 GiB of output. Test the stop-and-resume path with a small budget injected for the test.