Skip to content

Add basic batching function - #194

Open
sjavis wants to merge 6 commits into
v1.0.0-rcfrom
batching
Open

sjavis wants to merge 6 commits into
v1.0.0-rcfrom
batching

Conversation

@sjavis

@sjavis sjavis commented May 26, 2026

Copy link
Copy Markdown
Collaborator

Adds a basic batching function. It creates a directory for each batch, retrieves / preprocesses data, then runs the tracker.

The function takes these arguments:

  • The tracker object to use.
  • The number of batches to perform.
  • The names of the input files to pass to the tracker during each batch. It accepts * wildcards to match multiple files.
  • A function to retrieve the data / put it in the batch directory.
  • A list of preprocessing steps (the function + a dictionary of arguments to pass when the function gets called). Files in the batch directory can be specified using %BATCH%, e.g. "%BATCH%/input.nc".

It includes options for where to save the outputs, whether to combine the output files (will be implemented in a later PR), and whether to delete the batch directories at the end.

Closes #191

@sjavis sjavis self-assigned this May 26, 2026
@sjavis sjavis mentioned this pull request Jun 1, 2026
5 tasks
@sjavis
sjavis force-pushed the batching branch 2 times, most recently from d18d88e to c1b806e Compare August 26, 2026 22:34
@sjavis
sjavis changed the base branch from main to v1.0.0-rc August 27, 2026 08:28
@sjavis
sjavis force-pushed the batching branch 2 times, most recently from 3990166 to bce272a Compare August 27, 2026 09:15
@sjavis
sjavis marked this pull request as ready for review August 27, 2026 10:55
@sjavis
sjavis changed the base branch from v1.0.0-rc to ruff-rules August 27, 2026 12:44
Comment thread src/tctrack/utils/batching.py
Comment thread src/tctrack/utils/batching.py Outdated

# Optionally delete the batch directory
if delete_batch_dirs:
batch_dir.rmdir()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the directory is non-empty, it will raise error and will not delete it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for spotting. Fixed in efe7b86.

Comment thread src/tctrack/utils/batching.py
Comment thread src/tctrack/utils/batching.py Outdated
# Replace any %BATCH% and %ITER% tags in string arguments
i_iter, batch_dir = batch
for k, v in kwargs.items():
if v == "%ITER%":

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the comparison between a cf-Field and string legit here?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As it turns out, no. In that case the equality seems to return the field for some reason which is truthy. I've moved it inside the check for string types in 4b73781.

@sjavis
sjavis removed this pull request from stack #251 September 10, 2026 15:48
Base automatically changed from ruff-rules to v1.0.0-rc September 10, 2026 15:48
Accept a list preprocessing functions.
Use a registry of fields to keep in memory during preprocessing.
@sjavis
sjavis requested a review from surbhigoel77 September 17, 2026 08:51
@sjavis

sjavis commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator Author

This should be ready. Only the last two commits are new. The rest have just been rebased onto main / v1.0.0-rc branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Basic batching function

2 participants