Repository navigation
Make fof compare work with radar fof files - #103
Merged
Merged
Conversation
muellch
commented
May 21, 2026
huppd
reviewed
Jun 19, 2026
huppd
left a comment
Collaborator
There was a problem hiding this comment.
Thanks a lot for this new feature. It is minimal invasive, what I like very much.
I only have some minor comments. Final verdict should come from an expert and should be checked if it is compatible with the letkf pipeline.
cghielmini
requested changes
Jun 23, 2026
cghielmini
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for your work, I just have a little modificaiton to suggest to avoid the LETKF check from failing
- Strip the over-allocated padding by the real n_hdr/n_body counts (radar files allocate d_* larger than the real data), replacing the previous zero-append hack with an isel slice - Sort radar observations by the per-observation position dlat/dlon instead of the radar-station lat/lon, so the row order is deterministic - Size the fof_compare consistency gate and tolerance vector by n_body (real count) rather than the padded d_body - Flag veri_data cells that are NaN in one file but real in the other as differences instead of silently passing them - Guard against non-front-packed padding (via the dlat discriminator) and against multiple verification runs (d_veri > 1), failing loudly - Skip empty data frames in fof_compare's detailed log via log_dataframe - Add a realistic radar fixture (2-D veri_data, float fields, NaN padding) and unit/integration tests: padding strip, sort determinism, scattered-padding guard, and fof_compare consistent / not-consistent / different-padding / NaN-vs-real cases
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
- test_fof_compare: pass the fixtures to _run_fof_compare by keyword (too-many-positional-arguments). - test_fof_utils: drop the unused reports variable. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
- fof_compare: size the tolerance by n_body * d_veri. With several verification runs (e.g. LETKF files) veri_data has one row per (d_body, d_veri), so an n_body-long tolerance left most rows unchecked. The size gate compares n_body and d_veri. - check_file_with_tolerances (FOF): fail if the tolerance length does not match the veri_data rows, instead of silently skipping rows. - Fix the row-order comment in split_feedback_dataset and the module docstring of fof_compare. - Tests for both: a change in the last run of late-sorted observations, and a tolerance file one row short. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
cghielmini
requested changes
Nov 5, 2026
TEMP fof files from ICON and LETKF contain dlat/dlon with all values NaN. They were taken for radar files: the front-packing check raised, and the observations would have been sorted by the NaN dlat/dlon. A file is now radar only if every report has obstype 13; all other files are split and sorted as on main. Tests: a fixture shaped like ICON and LETKF SYNOP/TEMP files (LETKF with several verification runs, TEMP with all-NaN dlat/dlon), split and fof-compare tests for each, and a test that the sort keys follow obstype. The radar fixture gets obstype 13, as in real files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
get_report_variables and get_observation_variables picked variables whose first dimension had the size of d_hdr or d_body. In files with one observation per report (n_hdr == n_body) the header variables were also taken as observation variables, and fof-compare raised because the observation table had too many rows; main passed such files. With d_veri == n_hdr (ICON files have one verification run) the veri_* metadata became report variables. Select by dimension name instead. Tests: the classifiers with equal dimension sizes, and fof-compare of files with one observation per report for all ICON/LETKF families. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
muellch
force-pushed
the
radar_fof_fixes
branch
from
November 9, 2026 01:37
c10a634 to
0b2075d
Compare
cghielmini
previously approved these changes
Nov 9, 2026
cghielmini
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for your work :) looks good to me
Same triggers as the pre-commit workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
huppd
approved these changes
Nov 10, 2026
huppd
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for thoroughly testing it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make fof comparison work for radar fof files
(isel slice instead of the zero-append hack), check that the real observations fill [0:n_body], and sort
observations by the per-observation position dlat/dlon instead of the radar-site lat/lon
per report)
and a tolerance whose length does not match veri_data fails
runs, all-NaN dlat); padding strip and guard, sort keys by obstype, sort determinism, variables by dimension;
fof_compare consistent / not consistent / different padding / NaN vs real / several verification runs / one
observation per report; tolerance length mismatch
🤖 Generated with Claude Code
Co-authored-by: Alina Yapparova alina.yapparova@meteoswiss.ch