Skip to content

fix(trainer): ContrastiveTrainer crashes with a list of eval datasets (missing batch_size) - #427

Open
Anai-Guo wants to merge 1 commit into
illuin-tech:mainfrom
Anai-Guo:fix/eval-concat-datasets-batch-size
Open

Anai-Guo wants to merge 1 commit into
illuin-tech:mainfrom
Anai-Guo:fix/eval-concat-datasets-batch-size

Conversation

@Anai-Guo

@Anai-Guo Anai-Guo commented Sep 6, 2026

Copy link
Copy Markdown

Problem

ContrastiveTrainer.__init__ handles a list of datasets for both train_dataset and eval_dataset, but only the train branch passes batch_size to concat_datasets:

def concat_datasets(datasets: list[Dataset], batch_size: int) -> Dataset:
    ...

class ContrastiveTrainer(Trainer):
    def __init__(self, loss_func, is_vision_model, compute_symetric_loss=False, *args, **kwargs):
        if isinstance(kwargs["train_dataset"], list):
            train_dataset_list = kwargs["train_dataset"]
            kwargs["train_dataset"] = concat_datasets(train_dataset_list, batch_size=kwargs["args"].train_batch_size)
        ...
        if isinstance(kwargs["eval_dataset"], list):
            eval_dataset_list = kwargs["eval_dataset"]
            kwargs["eval_dataset"] = concat_datasets(eval_dataset_list)     # <- no batch_size

batch_size has no default, so constructing the trainer with a list of eval datasets fails immediately:

TypeError: concat_datasets() missing 1 required positional argument: 'batch_size'

The single-dataset (non-list) path is unaffected, which is why this has gone unnoticed — only multi-dataset evaluation trips it.

Root cause

Vbert (#339, 174055b0) factored the per-dataset round-down that #276 had added for training into the shared concat_datasets(datasets, batch_size) helper, and introduced both call sites in the same diff — the train one with batch_size=, the eval one without:

+def concat_datasets(datasets: list[Dataset], batch_size: int) -> Dataset:
@@
+            kwargs["train_dataset"] = concat_datasets(train_dataset_list, batch_size=kwargs["args"].train_batch_size)
@@
+            kwargs["eval_dataset"] = concat_datasets(eval_dataset_list)

Fix

Pass the eval-side batch size, mirroring the train branch two statements above (that call is the spec):

kwargs["eval_dataset"] = concat_datasets(eval_dataset_list, batch_size=kwargs["args"].eval_batch_size)

TrainingArguments.eval_batch_size is the eval-side counterpart of train_batch_size, so each eval dataset now gets rounded down to a whole number of eval batches — the same invariant concat_datasets was written to enforce for training.

+1 / −1.

Verification

transformers / torch aren't needed for this: concat_datasets and the two isinstance(kwargs[...], list) statements of __init__ were pulled out of the real file with ast.get_source_segment (nothing hand-transcribed) and executed with ConcatDataset stubbed to a list and a minimal Dataset exposing __len__ / take.

check before after
AST: concat_datasets call sites line 38 → 1 pos + batch_size; line 44 → 1 pos, no kwargs both → 1 pos + batch_size
inspect.signature(concat_datasets).bind([datasets]) missing a required argument: 'batch_size' —
__init__ prologue, eval_dataset=[DS(10), DS(9)], eval_batch_size=4 TypeError: concat_datasets() missing 1 required positional argument: 'batch_size' [DS(8), DS(8)] — rounded down to a multiple of 4
same run, train_dataset=[DS(20), DS(17)], train_batch_size=8 (never reached) [DS(16), DS(16)] — unchanged behaviour
eval_dataset=DS(9) (not a list) untouched, no exception untouched, no exception

ruff format --check is clean on the changed file. ruff check reports only the pre-existing I001 that the pristine upstream file also reports when linted outside the repo's own layout, so the change adds no new lint.

🤖 Generated with Claude Code

`concat_datasets(datasets, batch_size)` has no default for `batch_size`, so
`ContrastiveTrainer(..., eval_dataset=[...])` raised
`TypeError: concat_datasets() missing 1 required positional argument`.

Mirror the train branch two statements above and pass
`kwargs["args"].eval_batch_size`, so each eval dataset is rounded down to a
whole number of eval batches.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant