API: astype to CategoricalDtype matches by lookup instead of casting, so legitimate casts silently become NaN

[jbrockmendel]

astype to a CategoricalDtype resolves values by lookup rather than by cast: Categorical.__init__ ends up in _get_codes_for_values, which does categories.get_indexer_for(values). When the values are perfectly representable in categories.dtype but are currently stored in a different dtype, every lookup misses and the result is silently all-NaN.

Reproducible Example

No pyarrow, no datetimes — just int values against string categories, and vice versa:

import numpy as np
import pandas as pd

pd.Series([1, 2]).astype(pd.CategoricalDtype(pd.Index(["1", "2"], dtype="str")))
# 0    NaN
# 1    NaN
# dtype: category
# Categories (2, str): ['1', '2']

# the two-step cast, which is what was meant:
pd.Categorical(pd.Series([1, 2]).astype("str"),
               dtype=pd.CategoricalDtype(pd.Index(["1", "2"], dtype="str")))
# ['1', '2']
# Categories (2, str): ['1', '2']

pd.Series(np.array(["1", "2"], dtype=object)).astype(pd.CategoricalDtype(pd.Index([1, 2])))
# 0    NaN
# 1    NaN
# dtype: category
# Categories (2, int64): [1, 2]

Both emit Pandas4Warning: Constructing a Categorical with a dtype and values containing non-null entries not in that dtype's categories is deprecated and will raise in a future version. — but the values are in the categories. Only the dtype differs.

Why this needs a decision now

The deprecation added in GH-62142 will turn all of these into a hard error. Its two motivating issues, GH-40996 and GH-59899, are both about values that are genuinely absent from the categories (pd.Categorical(["foo"], dtype=ci.dtype), "c" against ["a", "b"]), and raising there is clearly right.

But the same warning also covers the dtype-mismatch cases above, where nothing is absent. When it is enforced, pd.Series([1, 2]).astype(pd.CategoricalDtype(pd.Index(["1", "2"]))) will raise, with a message pointing at the wrong cause.

Scope

Sweeping 10 value dtypes × 5 category dtypes, comparing one-step astype(CategoricalDtype(cats)) against two-step astype(cats.dtype) + Categorical(...). On 3.1.0.dev0+1520.gebd40365c94 / pyarrow 23.0.1: 50 combos — 25 agree, 11 group A (two-step succeeds, one-step silently all-NaN), 14 group B (two-step raises, one-step silently all-NaN).

Script

import warnings

import numpy as np
import pyarrow as pa

import pandas as pd
from pandas.errors import Pandas4Warning

VALUES = {
    "object[str]":        np.array(["2017-01-01", "2018-01-01"], dtype=object),
    "str":                pd.array(["2017-01-01", "2018-01-01"], dtype="str"),
    "ArrowDtype[string]": pd.array(["2017-01-01", "2018-01-01"], dtype=pd.ArrowDtype(pa.string())),
    "date32[pyarrow]":    pd.array(["2017-01-01", "2018-01-01"], dtype="date32[day][pyarrow]"),
    "object[intstr]":     np.array(["1", "2"], dtype=object),
    "str[intstr]":        pd.array(["1", "2"], dtype="str"),
    "int64":              np.array([1, 2]),
    "Int64":              pd.array([1, 2], dtype="Int64"),
    "int64[pyarrow]":     pd.array([1, 2], dtype="int64[pyarrow]"),
    "float64":            np.array([1.0, 2.0]),
}

CATEGORIES = {
    "M8[s]":   pd.Index(["2017-01-01", "2018-01-01"], dtype="M8[s]"),
    "int64":   pd.Index([1, 2]),
    "Int64":   pd.Index([1, 2], dtype="Int64"),
    "float64": pd.Index([1.0, 2.0]),
    "str":     pd.Index(["1", "2"], dtype="str"),
}


def one_step(values, cdtype):
    """astype straight to the CategoricalDtype."""
    return pd.Series(values).astype(cdtype)


def two_step(values, cats, cdtype):
    """Cast to the categories' dtype first, then build the Categorical."""
    return pd.Categorical(pd.Series(values).astype(cats.dtype), dtype=cdtype)


def run(func):
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        try:
            result = list(func())
        except Exception as err:
            return f"raises {type(err).__name__}", False
    warned = any(issubclass(w.category, Pandas4Warning) for w in caught)
    return "[" + ", ".join(str(x) for x in result) + "]", warned


for vname, values in VALUES.items():
    for cname, cats in CATEGORIES.items():
        cdtype = pd.CategoricalDtype(cats)
        got_one, warned = run(lambda: one_step(values, cdtype))
        got_two, _ = run(lambda: two_step(values, cats, cdtype))
        if got_one == got_two:
            verdict = "agree"
        elif got_two.startswith("raises"):
            verdict = "B: one-step laxer"
        else:
            verdict = "A: SILENTLY WRONG"
        print(f"{vname:20s} {cname:8s} {got_one:46s} {got_two:46s} {verdict}")

Full results (50 combos)

values dtype	categories dtype	one-step	two-step	warns
object[str]	M8[s]	[2017-01-01 00:00:00, 2018-01-01 00:00:00]	[2017-01-01 00:00:00, 2018-01-01 00:00:00]	no	agree
object[str]	int64	[nan, nan]	raises ValueError	yes	B: one-step laxer
object[str]	Int64	[nan, nan]	raises ArrowInvalid	yes	B: one-step laxer
object[str]	float64	[nan, nan]	raises ValueError	yes	B: one-step laxer
object[str]	str	[nan, nan]	[nan, nan]	yes	agree
str	M8[s]	[2017-01-01 00:00:00, 2018-01-01 00:00:00]	[2017-01-01 00:00:00, 2018-01-01 00:00:00]	no	agree
str	int64	[nan, nan]	raises ValueError	yes	B: one-step laxer
str	Int64	[nan, nan]	raises ArrowInvalid	yes	B: one-step laxer
str	float64	[nan, nan]	raises ValueError	yes	B: one-step laxer
str	str	[nan, nan]	[nan, nan]	yes	agree
ArrowDtype[string]	M8[s]	[NaT, NaT]	[2017-01-01 00:00:00, 2018-01-01 00:00:00]	yes	A: silently wrong
ArrowDtype[string]	int64	[nan, nan]	raises ValueError	yes	B: one-step laxer
ArrowDtype[string]	Int64	[nan, nan]	raises ValueError	yes	B: one-step laxer
ArrowDtype[string]	float64	[nan, nan]	raises ValueError	yes	B: one-step laxer
ArrowDtype[string]	str	[nan, nan]	[nan, nan]	yes	agree
date32[pyarrow]	M8[s]	[NaT, NaT]	[2017-01-01 00:00:00, 2018-01-01 00:00:00]	yes	A: silently wrong
date32[pyarrow]	int64	[nan, nan]	raises TypeError	yes	B: one-step laxer
date32[pyarrow]	Int64	[nan, nan]	raises TypeError	yes	B: one-step laxer
date32[pyarrow]	float64	[nan, nan]	raises TypeError	yes	B: one-step laxer
date32[pyarrow]	str	[nan, nan]	[nan, nan]	yes	agree
object[intstr]	M8[s]	[NaT, NaT]	raises ValueError	yes	B: one-step laxer
object[intstr]	int64	[nan, nan]	[1, 2]	yes	A: silently wrong
object[intstr]	Int64	[nan, nan]	[1, 2]	yes	A: silently wrong
object[intstr]	float64	[nan, nan]	[1.0, 2.0]	yes	A: silently wrong
object[intstr]	str	[1, 2]	[1, 2]	no	agree
str[intstr]	M8[s]	[NaT, NaT]	raises ValueError	yes	B: one-step laxer
str[intstr]	int64	[nan, nan]	[1, 2]	yes	A: silently wrong
str[intstr]	Int64	[nan, nan]	[1, 2]	yes	A: silently wrong
str[intstr]	float64	[nan, nan]	[1.0, 2.0]	yes	A: silently wrong
str[intstr]	str	[1, 2]	[1, 2]	no	agree
int64	M8[s]	[NaT, NaT]	[NaT, NaT]	yes	agree
int64	int64	[1, 2]	[1, 2]	no	agree
int64	Int64	[1, 2]	[1, 2]	no	agree
int64	float64	[1.0, 2.0]	[1.0, 2.0]	no	agree
int64	str	[nan, nan]	[1, 2]	yes	A: silently wrong
Int64	M8[s]	[NaT, NaT]	[NaT, NaT]	yes	agree
Int64	int64	[1, 2]	[1, 2]	no	agree
Int64	Int64	[1, 2]	[1, 2]	no	agree
Int64	float64	[1.0, 2.0]	[1.0, 2.0]	no	agree
Int64	str	[nan, nan]	[1, 2]	yes	A: silently wrong
int64[pyarrow]	M8[s]	[NaT, NaT]	[NaT, NaT]	yes	agree
int64[pyarrow]	int64	[1, 2]	[1, 2]	no	agree
int64[pyarrow]	Int64	[1, 2]	[1, 2]	no	agree
int64[pyarrow]	float64	[1.0, 2.0]	[1.0, 2.0]	no	agree
int64[pyarrow]	str	[nan, nan]	[1, 2]	yes	A: silently wrong
float64	M8[s]	[NaT, NaT]	[NaT, NaT]	yes	agree
float64	int64	[1, 2]	[1, 2]	no	agree
float64	Int64	[1, 2]	[1, 2]	no	agree
float64	float64	[1.0, 2.0]	[1.0, 2.0]	no	agree
float64	str	[nan, nan]	[nan, nan]	yes	agree

Note the object[str]/str → M8[s] rows already agree: DatetimeArray._validate_listlike has a string-parsing branch, so get_indexer already applies cast semantics there. That is the behavior the rest of the table is inconsistent with.

Proposal

Cast to dtype.categories.dtype first, then look up. Group A then round-trips correctly, and group B raises a genuine cast error instead of a "not in categories" message that misidentifies the cause. The 25 agreeing combos are unaffected.

The alternative is to keep lookup semantics and accept that legitimate casts raise once GH-62142 is enforced, which seems hard to justify to users.

The cast belongs in _get_codes_for_values, not in _validate_listlike

Worth stating explicitly, because it decides whether the arrow date types are in scope. Equality and castability are separate questions, and pandas already treats them separately: a string is not == a Timestamp, yet get_indexer matches it and astype to CategoricalDtype(M8) works.

Arrow date32/date64 are the same shape — not equal to datetimes (GH-62157, GH-60937), but exactly castable to them. Putting the cast in _validate_listlike would change == and break that, but putting it in _get_codes_for_values fixes the astype without touching either:

date32 -> Categorical[M8[s]]:  [Timestamp('2017-01-01'), Timestamp('2018-01-01')]   # fixed
ser_date32 == ser_datetime64:  [False, False]                                       # GH-62157 unchanged
dti.get_indexer(date32_array):  [-1, -1]                                            # GH-64953 unchanged

astype and the setops already disagree

Worth recording separately, because it is easy to mistake for a consequence of this proposal: the split already exists on current main. Against M8[s] categories:

other dtype	==	get_indexer	isin	intersection	astype(cat)
object[date]	match	match	NO	match	match
str	NO	match	NO	NO	match
ArrowDtype[string]	NO	NO	NO	NO	NO
date32[pyarrow]	NO	NO	NO	NO	NO

Script

import warnings
from datetime import date

import numpy as np
import pyarrow as pa

import pandas as pd

warnings.simplefilter("ignore")

cats = pd.Index(["2017-01-01", "2018-01-01"], dtype="M8[s]")
cdtype = pd.CategoricalDtype(cats)

OTHERS = {
    "object[date]":       np.array([date(2017, 1, 1), date(2018, 1, 1)], dtype=object),
    "str":                pd.array(["2017-01-01", "2018-01-01"], dtype="str"),
    "ArrowDtype[string]": pd.array(["2017-01-01", "2018-01-01"], dtype=pd.ArrowDtype(pa.string())),
    "date32[pyarrow]":    pd.array(["2017-01-01", "2018-01-01"], dtype="date32[day][pyarrow]"),
}


def matches(func):
    try:
        return "match" if func() else "NO"
    except Exception as err:
        return f"raises {type(err).__name__}"


for name, other in OTHERS.items():
    ser = pd.Series(other)
    print(
        f"{name:20s}",
        matches(lambda: (ser == pd.Series(cats)).all()),
        matches(lambda: (cats.get_indexer(other) != -1).all()),
        matches(lambda: cats.isin(other).all()),
        matches(lambda: len(cats.intersection(pd.Index(other))) == 2),
        matches(lambda: ser.astype(cdtype).notna().all()),
    )

str already matches under astype but not under intersection. So cast-first propagates an existing split to more dtypes rather than creating one. Unifying the five operations is a larger API question and is out of scope here — but it is worth knowing that astype matching while intersection does not is the status quo, not a regression introduced by fixing this.

Separately, isin is NO on every row, including object[date], which matches under every other operation. That looks like an independent bug rather than part of this issue.

Related: GH-62051 is one instance of this (ArrowDtype string values, and date32 values, against datetime/timedelta categories), which is what surfaced it.

[Bharath-970]

/take