Add an agent skill for datafusion-python extension authors

[timsaucer]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Writing a datafusion-python extension library means getting a handful of non-obvious lifetime and identity rules right, and there is currently no guidance an agent working in a downstream repo can find. The rules are spread across docs/source/contributor-guide/ffi.md, the docstrings on SessionContext.with_extensions / with_logical_extension_codec, .ai/skills/ffi-capsule-protocol/SKILL.md, and the two example crates. Each of those is written for someone already inside this repository.

The people who need it are not. Ballista and datafusion-distributed each build a separate cdylib plus Python glue against this project's FFI protocol, and they hit the same set of traps that #1679 was opened to fix: components bound to an intermediate context that is later collected, TaskContextProvider went out of scope over FFI boundary, codec ids that do not survive a process boundary, and write_parquet failing across the FFI codec because it has no file-format hooks. Every one of those is knowable in advance and none of them is discoverable from where those authors are standing.

Raised by @ntjohnson1 in review of #1679: "I wonder if it makes sense to have a todo for some datafusion python extension skill/s ... I do suspect it will require some guidance to make sure they are doing it safely."

Describe the solution you'd like

An agent skill for extension authors, plus a contributor-side audit skill that keeps it from rotting.

Location. A new skills/datafusion_python_extension/SKILL.md, sibling to the existing skills/datafusion_python/SKILL.md. Not .ai/skills/ — that tier is for agents working on this repository, and this audience is working on a different one. .ai/skills/ffi-capsule-protocol/SKILL.md stays where it is as the review trigger for changes to the capsule getters themselves, and the two cross-reference rather than duplicate.

Open question this issue needs to settle: how does a downstream agent get the file? skills/ is not currently included in the wheel, so an agent in a repo that depends on datafusion cannot see it. Options, roughly in order of how well they match "keeping it up to date shouldn't be too bad":

1. Ship skills/ as package data in the wheel. Downstream gets the current version with the dependency they already have, and it tracks the release they are actually building against.
2. Publish to agentskills.io, the standard CLAUDE.md already points at, so find-skills discovers it.
3. Document that downstream repos should vendor a copy. Cheapest to do, fastest to go stale.

Content. Drawn from the traps that #1672 and its three split PRs actually surfaced, not invented advice:

• The __datafusion_session_extension__ bundle protocol: create fresh components on every call, never cache bound components, never retain the context that was passed in.
• Take the task-context provider off the supplied context. Never construct a SessionContext inside an extension library. (Link ffi-capsule-protocol; do not restate it.)
• Codec identity: the class-derived id, __datafusion_codec_id__ for pinning one, and anon:<uuid> for a bare PyCapsule. A plan that has to decode in another process requires a declared id — this is the rule a distributed engine hits first.
• Ownership: the context returned from with_extensions is the strong owner. A DataFrame does not keep it alive. TaskContextProvider went out of scope over FFI boundary is the symptom; holding the context is the cure.
• Prefer with_extensions to hand-chaining with_logical_extension_codec / with_physical_extension_codec / set_query_planner, and say why.
• Known gap: FFI_LogicalExtensionCodec has no vtable entries for file formats, so df.write_csv / write_parquet / write_json fail across the boundary. Compose the FFI codec with a DefaultLogicalExtensionCodec fallback on try_encode_file_format / try_decode_file_format.

Sequencing. This should land after the codec-id gap in with_extensions is resolved. SessionExtensionComponents currently has no way to declare an id for a codec handed over as a bare capsule, and with_extensions takes no codec_id=, so the only workaround is a Python shim that declares __datafusion_codec_id__. Writing the skill now would document that workaround as the intended design and require a rewrite when the API gains a way to express it.

Keeping it current. An .ai/skills/audit-extension-skill/ contributor skill following the existing .ai/skills/audit-skill-md/ pattern: diff the skill's claims against the two example crates and the current public API, and report drift. This is the part worth automating, because the failure mode is the skill silently describing a protocol that has since moved.

Describe alternatives you've considered

Extend .ai/skills/ffi-capsule-protocol instead of adding a skill. Wrong audience and wrong trigger. That skill fires when someone touches a __datafusion_*__ getter in this repository. An extension author is not touching those; they are implementing against them, and they need a build-it-from-scratch narrative rather than a review checklist.

Extend docs/source/contributor-guide/ffi.md instead. Already the reference material, and #1679 adds substantially to it. But it is organized around the protocol rather than around the task, and it lives at a URL a downstream agent has no reason to fetch. The skill should point at ffi.md for depth rather than replace it.

Assert in CI that the skill can regenerate the three-library example. This was the original suggestion and it is the right instinct — the example is the best available proof the guidance is sufficient — but it does not survive contact with a workflow. Generated output varies between runs, so a byte-comparison fails on noise and a fuzzy comparison asserts nothing. The deterministic half is the audit skill above. The full generate-from-skill-and-run-the-tests exercise is real value in the wrong shape for CI; it belongs on the release checklist as a manual step.

Additional context

Reference implementations the skill should point at, both exercised in CI via .github/workflows/test.yml:

• examples/datafusion-ffi-example/ — table providers, UDFs, catalog providers, optimizer rules, extension codecs.
• examples/datafusion-ffi-query-planner-example/ — query planner, and MyPlannerExtension in src/extension.rs as a complete Rust implementation of __datafusion_session_extension__.

Follows up on #1679 (part 3 of the #1672 split).