Skip to content

Add preprocessing functions - #205

Merged
sjavis merged 17 commits into
mainfrom
preprocessing
Jul 7, 2026
Merged

sjavis merged 17 commits into
mainfrom
preprocessing

Conversation

@sjavis

@sjavis sjavis commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Adds preprocessing functions for all of the topics in the 'data preprocessing' page of the docs. Intended for use with the batching (#194). Generally the functions are designed to accept either file paths or cf.Field inputs and return single or lists of cf.Field. Each function can also write an output file by providing the output_file argument.

  • Preprocessing utility functions in src/tctrack/preprocessing.py
  • Tests
  • Documentation & add to the 'data preprocessing' page.
  • Replace preprocessing in tutorial
  • Add checks that esmpy is installed for regridding. Update the docs to reflect that this is now a "proper dependency".

Closes #189

@sjavis sjavis self-assigned this Jun 1, 2026
@sjavis
sjavis force-pushed the preprocessing branch 3 times, most recently from 9ce5cd0 to f15b6b9 Compare June 2, 2026 10:00
@sjavis
sjavis marked this pull request as ready for review June 2, 2026 10:01

@MarionBWeinzierl MarionBWeinzierl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just had a quick look and some comments -- @surbhigoel77 , could you have a more thorough look through this PR?

Comment thread tutorial/preprocess_data.py
Comment thread tests/unit/preprocessing/test_preprocessing.py Outdated
Comment thread docs/data/preprocessing_data.rst Outdated
@sjavis sjavis linked an issue Jun 9, 2026 that may be closed by this pull request
@sjavis sjavis removed a link to an issue Jun 10, 2026
@sjavis
sjavis requested a review from surbhigoel77 June 17, 2026 08:06
@sjavis

sjavis commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

In fc12ab3 I have also added a correction for what I suspect is an error in cf.curl_xy that causes the result to be the negative of the actual value. I will add an issue to cf-python about this when I get time.

This seems to resolve the issue in #156 when combined with updates to TSTORMS. This will be done in another PR.

@sjavis sjavis mentioned this pull request Jun 18, 2026
Comment thread src/tctrack/preprocessing.py
Comment thread src/tctrack/preprocessing.py

@MarionBWeinzierl MarionBWeinzierl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am happy from my part, but leave it to @surbhigoel77 to accept this PR after she has done another review

sjavis added 5 commits July 7, 2026 09:34
Return cf.Field objects instead of size-1 lists.
Also allow size-1 lists to be passed as inputs to functions that expect
single fields.
@sjavis
sjavis merged commit dd49498 into main Jul 7, 2026
8 checks passed
@sjavis
sjavis deleted the preprocessing branch July 7, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Data preprocessing functions

3 participants