Currently we don't provide API for our framework enough. Ideally we should be able to
import judgearena
from judgarena.datasets import register_dataset
# Use your own dataset, with following some rules to work properly in the pipeline
@register_dataset
class MyClass
...
...
# Load tasks
task = load_custom_task(...)
cfg = {...}
judgearena.run_task(cfg=cfg, task=task...)
This extends the capability of the framework so it can be used as a library as well as framework.
Currently we don't provide API for our framework enough. Ideally we should be able to
This extends the capability of the framework so it can be used as a library as well as framework.