Skip to content

Make fof compare work with radar fof files - #103

Merged
huppd merged 15 commits into
MeteoSwiss:mainfrom
muellch:radar_fof_fixes
Nov 10, 2026
Merged

huppd merged 15 commits into
MeteoSwiss:mainfrom
muellch:radar_fof_fixes

Conversation

@muellch

@muellch muellch commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Make fof comparison work for radar fof files

  • Radar files (obstype 13 in every report): strip the over-allocated padding by the real n_hdr/n_body counts
    (isel slice instead of the zero-append hack), check that the real observations fill [0:n_body], and sort
    observations by the per-observation position dlat/dlon instead of the radar-site lat/lon
  • Other fof files (ICON, LETKF) are split and sorted as before; TEMP files with all-NaN dlat/dlon are not radar
  • Select report and observation variables by dimension name, not size (sizes can coincide, e.g. one observation
    per report)
  • fof_compare: size check on (n_body, d_veri) instead of the padded d_body, every verification run is compared,
    and a tolerance whose length does not match veri_data fails
  • Log veri_data differences and tolerances via log_dataframe
  • Tests: radar fixture (padding, 2-D veri_data, NaN) and ICON/LETKF SYNOP/TEMP fixtures (several verification
    runs, all-NaN dlat); padding strip and guard, sort keys by obstype, sort determinism, variables by dimension;
    fof_compare consistent / not consistent / different padding / NaN vs real / several verification runs / one
    observation per report; tolerance length mismatch

🤖 Generated with Claude Code

Co-authored-by: Alina Yapparova alina.yapparova@meteoswiss.ch

Comment thread util/fof_utils.py Resolved

@huppd huppd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this new feature. It is minimal invasive, what I like very much.
I only have some minor comments. Final verdict should come from an expert and should be checked if it is compatible with the letkf pipeline.

Comment thread util/dataframe_ops.py Resolved
Comment thread engine/fof_compare.py Resolved
Comment thread engine/fof_compare.py Resolved

@cghielmini cghielmini left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work, I just have a little modificaiton to suggest to avoid the LETKF check from failing

Comment thread util/fof_utils.py Resolved
muellch added 6 commits June 26, 2026 16:13
- Strip the over-allocated padding by the real n_hdr/n_body counts (radar
files allocate d_* larger than the real data), replacing the previous
zero-append hack with an isel slice
- Sort radar observations by the per-observation position dlat/dlon instead
of the radar-station lat/lon, so the row order is deterministic
- Size the fof_compare consistency gate and tolerance vector by n_body (real
count) rather than the padded d_body
- Flag veri_data cells that are NaN in one file but real in the other as
differences instead of silently passing them
- Guard against non-front-packed padding (via the dlat discriminator) and
against multiple verification runs (d_veri > 1), failing loudly
- Skip empty data frames in fof_compare's detailed log via log_dataframe
- Add a realistic radar fixture (2-D veri_data, float fields, NaN padding) and
unit/integration tests: padding strip, sort determinism, scattered-padding
guard, and fof_compare consistent / not-consistent / different-padding /
NaN-vs-real cases
muellch and others added 3 commits October 27, 2026 12:05
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
- test_fof_compare: pass the fixtures to _run_fof_compare by keyword
  (too-many-positional-arguments).
- test_fof_utils: drop the unused reports variable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
- fof_compare: size the tolerance by n_body * d_veri. With several
  verification runs (e.g. LETKF files) veri_data has one row per
  (d_body, d_veri), so an n_body-long tolerance left most rows
  unchecked. The size gate compares n_body and d_veri.
- check_file_with_tolerances (FOF): fail if the tolerance length does
  not match the veri_data rows, instead of silently skipping rows.
- Fix the row-order comment in split_feedback_dataset and the module
  docstring of fof_compare.
- Tests for both: a change in the last run of late-sorted observations,
  and a tolerance file one row short.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
@muellch muellch changed the title Draft: Make fof compare work with radar fof files Make fof compare work with radar fof files Oct 27, 2026
@muellch
muellch requested review from cghielmini and huppd November 3, 2026 10:47
Comment thread util/fof_utils.py Resolved
Comment thread util/fof_utils.py Resolved
Comment thread util/fof_utils.py Resolved
muellch and others added 3 commits November 9, 2026 01:57
TEMP fof files from ICON and LETKF contain dlat/dlon with all values NaN.
They were taken for radar files: the front-packing check raised, and the
observations would have been sorted by the NaN dlat/dlon. A file is now
radar only if every report has obstype 13; all other files are split and
sorted as on main.

Tests: a fixture shaped like ICON and LETKF SYNOP/TEMP files (LETKF with
several verification runs, TEMP with all-NaN dlat/dlon), split and
fof-compare tests for each, and a test that the sort keys follow obstype.
The radar fixture gets obstype 13, as in real files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
get_report_variables and get_observation_variables picked variables whose
first dimension had the size of d_hdr or d_body. In files with one
observation per report (n_hdr == n_body) the header variables were also
taken as observation variables, and fof-compare raised because the
observation table had too many rows; main passed such files. With
d_veri == n_hdr (ICON files have one verification run) the veri_*
metadata became report variables. Select by dimension name instead.

Tests: the classifiers with equal dimension sizes, and fof-compare of
files with one observation per report for all ICON/LETKF families.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL
cghielmini
cghielmini previously approved these changes Nov 9, 2026

@cghielmini cghielmini left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work :) looks good to me

huppd and others added 2 commits November 9, 2026 15:48
Same triggers as the pre-commit workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WFoZEcmmfDRpZR2HBzwmvL

@huppd huppd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for thoroughly testing it.

@huppd
huppd merged commit 82db664 into MeteoSwiss:main Nov 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants