What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning

Rohan Siva^1,* · Neel P. Bhatt^1,2,* · Yunhao Yang^1,2,* · Seoyoung Lee¹ · Nishant Gadde¹
Christian Ellis^1,2 · Alvaro Velasquez^2,3 · Zhangyang Wang¹ · Ufuk Topcu^1,2
1The University of Texas at Austin    ²Neurosymbolic Intelligence    ³University of Colorado Boulder
*Equal Contribution
CoRL 2026

Project Page | arXiv | Dataset | Model Checkpoint

---

TL;DR

A4D reasons about what objects enable, not what they are, mapping visual observations into a functional latent space structured around affordances (e.g., "movable") instead of appearance-based categories, and discovering new affordances when existing ones aren't enough. A4D achieves 94% inference accuracy on existing affordances (+15 points over SOTA VLMs), improves new-affordance inference from 70% to over 90% with fewer than 10% of the original training data, and enables 100x faster inference.

---

Framework Overview

Functional Latent Space for Affordance Reasoning. A4D maps visual observations and affordance descriptions into a shared functional latent space. For each affordance, we construct an affordance axis between an affordance and its antonym (e.g., movable ↔ fixed). Visual observations are projected onto these axes to infer task-relevant object functionalities. Projection proximity is calibrated into uncertainty estimates, enabling the system to identify when additional reasoning is required.

Uncertainty-Aware Affordance Discovery. Given a task and visual observation, A4D first generates candidate affordances relevant to the current planning problem. Affordance inference is performed in the functional latent space, while calibrated uncertainty determines whether existing affordances are sufficient. When uncertainty is high or a new functionality is required, a vision-language model proposes and labels new affordances, which are incorporated through offline learning and added to the affordance memory for future deployment.

Demonstrations

A4D performs affordance-based decision making across diverse scenarios. In the first example, the planner selects the cart as the most movable object for the task. In the second example, existing affordances are insufficient, triggering uncertainty-guided affordance discovery and introducing a new traversable affordance to successfully complete the task.

A4D transfers across robot platforms and domains. The same affordance generation, inference, and discovery framework is deployed on a tabletop robot arm, demonstrating that affordance reasoning is not tied to a specific robot platform. By conditioning generation and labeling on platform-specific capabilities, A4D adapts affordance predictions to new robots and tasks without modifying the underlying framework.

Setup

pip install -r requirements.txt
export OPENAI_API_KEY="your_api_key_here"  # only needed for affordance_generation.ipynb

Demos

1. affordance_generation.ipynb — open-vocabulary affordance discovery with a VLM.
2. classification_uncertainty.ipynb — CLIP-based affordance classification with calibrated uncertainty.

Model Checkpoint

Fine-tuned CLIP checkpoint on Hugging Face: rohansiva/A4D-Checkpoint. Downloaded automatically by classification_uncertainty.ipynb if not present locally.

Dataset

Binary affordance labels for real-world object images, used to train and evaluate A4D's classification, calibration, and discovery pipeline. Available on Hugging Face — rohansiva/A4D-dataset — with a local copy in dataset/.

Images	552
Object classes	29
Affordances	10
(image, affordance) labels	5,096
Label balance	2,146 positive / 2,950 negative

Affordances

containable · graspable · liftable · movable · openable · pushable · rollable · stackable · supportable · traversable

Object classes

Class	Count	Class	Count
cup	30	racket	20
pot	30	sofa	20
banana	28	spatula	20
orange	25	tvm	20
potted_plant	25	bin	19
table	21	bowl	19
bed	20	extinguisher	19
chair	20	stairs	17
drill	20	box	16
hammer	20	bottle	15
knife	20	can	15
lamp	20	door	15
pan	20	strawberry	15
photo_frame	20	mouse	2
		cart	1

Format — one row per (image, affordance) pair in dataset.jsonl:

{"image": "banana_000.jpg", "affordance": "containable", "label": 0}

Citation

If you find this work interesting and use it in your research, please consider citing our paper.

@inproceedings{siva2026objectsenablearefunctional,
            title={What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoning},
            author={Rohan Siva and Neel P. Bhatt and Yunhao Yang and Seoyoung Lee and Nishant Gadde and Christian Ellis and Alvaro Velasquez and Zhangyang Wang and Ufuk Topcu},
            year={2026},
            booktitle={Proceedings of the Tenth Conference on Robot Learning},
            address={Austin, TX, USA},
            publisher={PMLR},
      }