Skip to content

Native timestamp casts fail or panic for years beyond chrono's range #7471

Open

Description

@tlfua

Describe the bug

Native casts to and from timestamps fail for values whose year is outside chrono's range (about ±262,143), while Spark supports timestamps up to year 294,247 and returns a result. This happens for TRY casts too, so try_cast raises instead of returning a value.

Query Comet
CAST(ts AS DATE) CometNativeException: Cast error: Cannot convert TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports dates between Jan 1, 262145 BCE and Dec 31, 262143 CE
CAST(ts AS STRING) same error
try_cast(ts AS DATE) same error
CAST(dt AS TIMESTAMP) native panic at spark-expr/src/conversion_funcs/temporal.rs:74:34

Steps to reproduce

As a SQL file test (spark/src/test/resources/sql-tests/...), default Spark 4.1 profile:

statement
CREATE TABLE tsr(ts timestamp, dt date) USING parquet

statement
INSERT INTO tsr VALUES
  (timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
  (timestamp_micros(0), date_from_unix_date(0))

query
SELECT CAST(ts AS DATE) FROM tsr

query
SELECT CAST(ts AS STRING) FROM tsr

query
SELECT try_cast(ts AS DATE) FROM tsr

query
SELECT CAST(dt AS TIMESTAMP) FROM tsr

timestamp_micros(9000000000000000000) is in year 287,000 or so, and date_from_unix_date(100000000) in year 275,000 or so; both are valid Spark values.

Expected behavior

The same results as Spark. Where the native kernel cannot represent the value, the serde should fall back to Spark rather than fail or panic.

Additional context

The error paths:

  • pre_timestamp_cast in native/spark-expr/src/utils.rs (as_datetime(...).ok_or_else(datetime_cast_err)), used by timestamp to string and timestamp to date for every eval mode.
  • timestamp_ntz_to_timestamp and cast_timestamp_to_ntz in the same file, same pattern.
  • Date to timestamp in native/spark-expr/src/conversion_funcs/temporal.rs: epoch + chrono::Duration::days(d) panics. The comment above it says chrono's range covers every i32 day count, but it does not.
  • test_cast_unsupported_timestamp_to_date in conversion_funcs/cast.rs asserts the error for i64::MAX.

Found while auditing which expressions can safely run as TopK later keys in #7457 , which now treats every timestamp cast as able to fail. Suggested triage: priority:high (native panic), area:expressions.

Activity

  1. tlfua commented on Nov 6, 2026

    @tlfua
    Author

    A cast from DATE to STRING has the same problem, raised in review on #7457 . Comet sends it to arrow-cast's value_to_string, whose ValueFormatter::write ignores the safe option, so a date beyond chrono's year range raises in every mode:

    CREATE TABLE big_date(dt date) USING parquet;
    INSERT INTO big_date SELECT date_from_unix_date(100000000);
    SELECT dt, CAST(dt AS STRING) FROM big_date;

    Spark returns the date and its string. Comet fails with Cast error: Failed to convert 100000000 to temporal for Date32 (checked on Spark 4.1 with a CometSqlFileTestSuite fixture). Implicit casts such as concat(s, dt) go the same way. #7457 now treats this cast as able to fail for TopK sort keys.

  2. added
    bugSomething isn't working
    priority:highCrashes, panics, segfaults, major functional breakage
    and removed on Nov 9, 2026
  3. kazuyukitanimura commented on Nov 9, 2026

    @kazuyukitanimura
    Contributor

    we are moving onto jiff FYI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingpriority:highCrashes, panics, segfaults, major functional breakage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions