Repository navigation
2113 nimrod regrid - #2269
2113 nimrod regrid#2269BernardClaxton (BernardClaxton) wants to merge 105 commits into
Conversation
Total coverage: 93% (HTML report)Name Stmts Miss Branch BrPart Cover --------------------------------------------------------------------------------------------------- src/CSET/__init__.py 113 4 14 0 97% src/CSET/_common.py 156 0 54 0 100% src/CSET/cset_workflow/app/fetch_fcst/bin/fetch_data.py 117 28 26 0 78% src/CSET/cset_workflow/app/fetch_nimrod/bin/fetch_nimrod.py 85 8 38 16 80% src/CSET/cset_workflow/app/finish_website/bin/finish_website.py 79 2 8 2 95% src/CSET/cset_workflow/app/parbake_recipes/bin/parbake.py 29 0 8 0 100% src/CSET/cset_workflow/app/send_email/bin/send_email.py 25 0 4 0 100% src/CSET/cset_workflow/lib/python/jinja_utils.py 17 0 6 0 100% src/CSET/extract_workflow.py 105 1 26 1 98% src/CSET/graph.py 44 0 14 0 100% src/CSET/operators/__init__.py 89 0 26 0 100% src/CSET/operators/_atmospheric_constants.py 9 0 0 0 100% src/CSET/operators/_colormaps.py 250 3 72 4 98% src/CSET/operators/_stash_to_lfric.py 3 0 0 0 100% src/CSET/operators/_utils.py 200 8 74 6 95% src/CSET/operators/ageofair.py 142 7 64 5 94% src/CSET/operators/aggregate.py 184 8 60 7 94% src/CSET/operators/aviation.py 61 0 18 0 100% src/CSET/operators/collapse.py 162 8 80 3 94% src/CSET/operators/constraints.py 115 7 50 2 93% src/CSET/operators/convection.py 38 4 10 2 88% src/CSET/operators/ensembles.py 27 0 14 0 100% src/CSET/operators/feature.py 44 0 10 0 100% src/CSET/operators/filters.py 67 2 30 0 98% src/CSET/operators/fluxes.py 41 0 10 0 100% src/CSET/operators/humidity.py 135 0 52 0 100% src/CSET/operators/imageprocessing.py 57 0 16 0 100% src/CSET/operators/mesoscale.py 18 0 2 0 100% src/CSET/operators/misc.py 172 1 72 3 98% src/CSET/operators/plot.py 1174 156 440 71 83% src/CSET/operators/power_spectrum.py 156 10 60 8 91% src/CSET/operators/precipitation.py 204 2 92 2 99% src/CSET/operators/pressure.py 41 0 12 0 100% src/CSET/operators/radar_filter.py 111 4 44 4 95% src/CSET/operators/read.py 453 25 200 15 94% src/CSET/operators/regrid.py 147 1 70 3 98% src/CSET/operators/scoreswrappers.py 226 22 74 6 89% src/CSET/operators/temperature.py 121 0 32 0 100% src/CSET/operators/transect.py 63 0 24 0 100% src/CSET/operators/wind.py 46 3 10 2 91% src/CSET/operators/write.py 15 0 6 0 100% src/CSET/recipes/__init__.py 104 0 28 0 100% src/CSET/sample_data/__init__.py 0 0 0 0 100% --------------------------------------------------------------------------------------------------- TOTAL 5445 314 1950 162 93% |
…, np.nanmax, np.nanmean #2113
60bea6d to
4b81b55
Compare
BernardClaxton (BernardClaxton)
linked an issue
|
Note to self:
|
James Ross (james-a-f-ross)
left a comment
James Ross (james-a-f-ross)
left a comment
There was a problem hiding this comment.
initial pass at this.. get in touch if anything isn't clear
| def select_radar_source(input_source_list): | ||
| """Select the preferred radar source from a list of sources.""" | ||
| # Preference list for Nimrod sources, with most preferred heading the list. | ||
| nimrod_preference = ["Nimrod2km", "Nimrodxkm", "Nimrod1km"] | ||
|
|
||
| empty_string = "" | ||
| preferred_nimrod = empty_string | ||
| for prefer in reversed(nimrod_preference): | ||
| if any(prefer in source for source in input_source_list): | ||
| preferred_nimrod = prefer | ||
| return preferred_nimrod |
There was a problem hiding this comment.
needs type annotations. can we handle if preffered_nimrod isn't set?
There was a problem hiding this comment.
also might be able to simplify this a bit depending on how we handle preferred_nimrod = ""
| ``"realization"``. | ||
| strict_title: bool, optional | ||
| Logical switch that if set to True will ensure that the MODEL_NAME | ||
| string is not prepended to the plot title. The default is False. |
There was a problem hiding this comment.
if I understand this, you are removing model name form the start of the plot title, because in the recipe you have it in already. I would rather we didn't do that and we changed the plot title to suit rather than changing the logic here.. we can have more of a discussion about how we set the plot titles.
| @@ -0,0 +1,68 @@ | |||
| category: Surface Spatial Plot | |||
| title: "$MODEL_LABEL $MODEL_VARNAME\n common domain $MODEL_LABEL and $RADAR_LABEL" | |||
There was a problem hiding this comment.
do think more generally the title names could use a bit more thought and to be more descriptive
| nimrod_field: iris.cube.Cube | iris.cube.CubeList, | ||
| nimrod_mask: iris.cube.Cube | iris.cube.CubeList, | ||
| boundary_margin: int = 8, | ||
| outputs: str = "radar", |
There was a problem hiding this comment.
rename variable to required_outputs maybe? as just outputs feels a bit confusing
| title: "$RADAR_NAME $VARNAME $SUBAREA_NAME" | ||
| description: | |
There was a problem hiding this comment.
more descriptive title
|
|
||
| # Loop over the models. | ||
| for model in model_field: | ||
| print("-------> using model ", model) |
There was a problem hiding this comment.
put this in the logger instead please
| nimrod_field: iris.cube.Cube | iris.cube.CubeList, | ||
| nimrod_mask: iris.cube.Cube | iris.cube.CubeList, | ||
| boundary_margin: int = 8, | ||
| outputs: str = "radar", |
There was a problem hiding this comment.
outputs -> required_outputs
| @@ -223,8 +225,8 @@ title=Radar observations | |||
|
|
|||
| [template variables=NIMROD_COMP_XKM] | |||
There was a problem hiding this comment.
What does XKM actually mean? Description suggests 5km??
| os.makedirs(nimrod_dir_wei, exist_ok=True) | ||
| logger.info("Cylc-run Nimrod weights directory: %s", nimrod_dir_wei) | ||
|
|
||
| # Put +1 hour offset for accumulation radar files as |
There was a problem hiding this comment.
Do we really need the shift? Asking as we are loading radar data with a timestamp ending the hourly accumulation period. so we read an hourly accumulation for 11am which is representative for the accumulation from 10-11. Lfric gives us an instantaneous rainfall rate at 11. Apart from teh required conversions and getting accums from lfric isn't the timestamp actually correct?
There was a problem hiding this comment.
I have been comparing the lfric files containing surface_microphysical_rainfall_rate i.e. the timeproc files and the diagnostic files against UK Nimrod
LFRic: When fetching CYCLE_TASK_CYCLE_POINT=00:00:00
- timeproc files give seconds since 00:00:00
point [00:30:00, 01:30:00, 02:30:00]
bounds [00:00:00, 01:00:00] [01:00:00:00, 02:00:00][02:00:00,03:00:00 ] - diagnostics files give seconds since 00:00:00
point [01:00:00, 02:00:00, 03:00:00]
bounds [01:00:00, 01:00:00] [02:00:00, 02:00:00][03:00:00, 03:00:00]
hence timeproc gives a meaned (?) field on the halfway point and diagnostics gives a truly instantaneous point.
UKNimrod radar when fetching CYCLE_TASK_CYCLE_POINT=00:00:00
points [00:00:00]
bounds [23:00:00,00:00:00]
For fetching a filename for lfric and radar for CYCLE_TASk_CYCLE_POINT
- LFRic returns a file with CYCLE_TASk_CYCLE_POINT in the name and data for bounds [CYCLE_TASK_CYCLE_POINT, CYCLE_TASK_CYCLE_POINT+1 ]
- Radar returns a file name with CYCLE_TASK_CYCLE_POINT in the name and data for bounds [CYCLE_TASK_CYCLE_POINT -1, CYCLE_TASK_CYCLE_POINT]
Hence, we need to shift UK nimrod radar data filename by +1 to fetch the correpsonding time bounds between lfric and radar. I suspect that when outputting accumulations in lfric for a filename with CYCLE_TASK_CYCLE_POINT we need to check if the output bounds still start at [CYCLE_TASK_CYCLE_POINT, CYCLE_TASK_CYCLE_POINT+1] with a point at CYCLE_TASK_CYCLE_POINT+1 or if a point exists at CYCLE_TASK_CYCLE_POINT.
There was a problem hiding this comment.
So for now we require the +1 correction to CYCLE_TASK_CYCLE_POINT to fetch the correct radar file.
| title=Produce Nimrod 2D surface plots | ||
| description=Produce 2D surface plots of radar observations. | ||
| help=Switch to produce 2D surface plots utilising Nimrod radar observations. | ||
| Plots are produced of the 2D surface rainfall for each of the selected Nimrod sources. |
There was a problem hiding this comment.
more precision needed: Nimrod provides rainfall amounts
| title=Produce Nimrod radar histograms | ||
| description=Produce histograms incorporating radar observations. | ||
| help=Switch to produce histograms utilising Nimrod radar observations. | ||
| The histograms will combine both model rainfall and radar observations. |
There was a problem hiding this comment.
This will likely change when lfric gives us accumulations as well, but worth being precise her that Nimrod gives as amounts as an accumulation for sum products and a 5min "snapshot" for the other product, whereas the models currently gives us rainfall_rate i.e. amount per time as a snapshot
| def select_radar_source(input_source_list): | ||
| """Select the preferred radar source from a list of sources.""" | ||
| # Preference list for Nimrod sources, with most preferred heading the list. | ||
| nimrod_preference = ["Nimrod2km", "Nimrodxkm", "Nimrod1km"] |
There was a problem hiding this comment.
needs explanation for xkm
| # for this is set in v["weights"]. | ||
| if v["weights"] == "True" and nimrod_weights_exist == "True": | ||
| filename_wei_nc = ( | ||
| f"{nimrod_dir_wei}/{date_use.strftime('%Y%m%d%H%M')}" |
There was a problem hiding this comment.
Now the radar data are written out with a timepoint in the filename that is shifted by 1hour for the radar_offset. So a file with CYCLE_POINT timestamp in the name is now renamed to CYCLE_POINT+1 in the name while the time coordinates inside have not changed. Is this correct given that the original filename has the endtime in the filename?
There was a problem hiding this comment.
Need to check the filenames written to disk: Do we now have CYCLE_POINT+1 as filename, but time bounds in the file with [CYCLE_POINT-1:CYCLE_POINT]?
Carol Halliwell (cehalliwell)
left a comment
Carol Halliwell (cehalliwell)
left a comment
There was a problem hiding this comment.
I have has a look through and left comments/suggestions.
Could you add a description of what this PR is doing?
Have you tested all the different combinations (or as many as possible) of radar type/model?
| ] | ||
|
|
||
| # Specify the boundary margin width to use when trimming model fields. | ||
| boundary_margin = 16 |
There was a problem hiding this comment.
Does this need to be fixed? If not, can it be set by the user?
| description=Produce histograms incorporating radar observations. | ||
| help=Switch to produce histograms utilising Nimrod radar observations. | ||
| The histograms will combine both model rainfall and radar observations. | ||
| The model rainfall variable is selected in the Rose tab Diagnostics/Surface (2D) fields. |
There was a problem hiding this comment.
Is this correct? Other code suggests rainfall variable is fixed, i.e.
"MODEL_VARNAME": "surface_microphysical_rainfall_rate",
|
do the recipes maybe belong in their own directory now? e.g recipes/radar or recipes/surface_fields/radar? |
Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>
Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>
Co-authored-by: Sylvia Bohnenstengel <62748926+Sylviabohnenstengel@users.noreply.github.com>
…plot.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>
…togram_domain_mean.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>
…togram_sequence.yaml Co-authored-by: Carol Halliwell <150140850+cehalliwell@users.noreply.github.com>
|
I have started to make some changes in this branch, some thing I'm not too sure about though. |
This branch has not been deployed
5 participants
Contribution checklist
Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.
Githib copilot was used to help write some of the tests in test_radar_filter.py.