Add CFMIP4 data

[znichollscr]

Description

Checklist

Please confirm that this pull request has done the following:

• Data released on ESGF
• ESGF update pulled in here
• Documentation added (where applicable)
• Changelog item added to changelog/
• Did a new release after merging

[znichollscr]

@daianzhuo sorry I've been so slow on this. Picking up now.

I've had a very quick look at your data. This is going to take a few iterations to get right. To help guide the next step, can you please tell me:

• Why have you registered two source IDs? What is the difference between the two?
• Looking at the data, and what we can handle in the naming convention (DRS), I think we're probably going to have a different source ID for each model that supplies piClim-deltaSST tos information. That's painful, but I don't think we have another mechanism to supply this information unfortunately unless you think you can see one?
• In the HadISST_bcs data, does the suffix 'bcs' indicate whether the variable has been diddled or not? If not, how do you indicate 'diddled' vs. 'not diddled'?

[pceppi]

@znichollscr thanks for looking into this.

1. I think @daianzhuo created one source ID per experiment in the CFMIP protocol: amip-piForcing and piClim-deltaSST.
2. I'm not too sure what that means in practice for the end user; as long as they can find the datasets on ESGF, it's all good.
3. Yes, "_bcs" means diddled. We followed the naming convention used for AMIP boundary conditions.

[znichollscr]

1. I think @daianzhuo created one source ID per experiment in the CFMIP protocol: amip-piForcing and piClim-deltaSST.

Ok. You can do it based on this logic if you wish. You don't have to split this way though (see more below).

2. I'm not too sure what that means in practice for the end user; as long as they can find the datasets on ESGF, it's all good

Very little: it's just a note (but we can't get around it anyway, so not worth worrying about it)

3. Yes, "_bcs" means diddled. We followed the naming convention used for AMIP boundary conditions

Ok this simplifies things.

For the piClim-deltaSST experiments, I think there is only one option: one source ID per supplying model-experiment, so it will have to be something like (I explain what 'middle-bit' is below):

1. ImperialCollege-[middle-bit]-CESM2-abrupt-4xCO2-1-0
2. ImperialCollege-[middle-bit]-CNRM-ESM2-1-abrupt-4xCO2-1-0
3. ImperialCollege-[middle-bit]-CanESM5-abrupt-4xCO2-1-0
4. etc.

For the amip-piForcing data, you can just use a single source ID, so I guess something like ImperialCollege-[middle-bit]-1-0.

Then the question is what to put in this 'middle-bit'. Here there are a few options I could think of quickly:

1. Make the middle bit the experiment name so you would have
   • ImperialCollege-amip-piForcing-1-0
   • ImperialCollege-piClim-deltaSST-CESM2-abrupt-4xCO2-1-0
   • ImperialCollege-piClim-deltaSST-CNRM-ESM2-1-abrupt-4xCO2-1-0
   • ImperialCollege-piClim-deltaSST-CanESM5-abrupt-4xCO2-1-0
   • ...
2. Just don't have a middle bit so you would have
   • ImperialCollege-1-0
   • ImperialCollege-CESM2-abrupt-4xCO2-1-0
   • ImperialCollege-CNRM-ESM2-1-abrupt-4xCO2-1-0
   • ImperialCollege-CanESM5-abrupt-4xCO2-1-0
   • ...
3. A uniform middle bit e.g. if you used "CFMIP4", you would always have
   • ImperialCollege-CFMIP4-1-0
   • ImperialCollege-CFMIP4-CESM2-abrupt-4xCO2-1-0
   • ImperialCollege-CFMIP4-CNRM-ESM2-1-abrupt-4xCO2-1-0
   • ImperialCollege-CFMIP4-CanESM5-abrupt-4xCO2-1-0
   • ...
4. An otherwise varying middle bit (bit more confusing for users and impossible to parse with a machine, but maybe what you want)
   • ImperialCollege-HadISST1-1-0
   • ImperialCollege-piClim-deltaSST-CESM2-abrupt-4xCO2-1-0
   • ImperialCollege-piClim-deltaSST-CNRM-ESM2-1-abrupt-4xCO2-1-0
   • ImperialCollege-piClim-deltaSST-CanESM5-abrupt-4xCO2-1-0
   • ...

[pceppi]

Thanks @znichollscr, all sounds good! For the middle bit, let's follow option 1 with one tweak.

For the amip-piForcing dataset, I would use the following source ID:
ImperialCollege-AMIP-HadISST1p1-1-0
This follows the naming convention for PCMDI AMIP SSTs, and also indicates what SST product we're using. We will add HadISST2 when it becomes available.

[znichollscr]

Ok thank you. I have just pushed those updated source IDs. Next step, we need to rewrite the files. I have put a commented ncdump below which will hopefully give you the guidance you need. You don't need to rename the files yet, I'll explain that in the next step

// global attributes:
		:Conventions = "CF-1.7" ;  # can stay as is
		:activity_id = "input4MIPs" ;  # can stay as is
		:comment = "" ;  # if you want, but is optional
		:contact = "" ;  # must match the CVs entries exactly (maybe already the case)
		:creation_date = "2025-02-28T18:23:33Z" ; # can stay as is
		:dataset_category = "" ; # I don't think this is required so do nothing now, I will confirm when I check in full
		:doi = "" ;  # optional
		:frequency = "mon" ;  # can stay as is
		:further_info_url = "" ; # must match the CVs exactly
		:institution_id = "ImperialCollege" ;  # must match the CVs exactly
		:license = "The input4MIPs data linked to this entry is licensed under a Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6Plus output, including citation requirements and proper acknowledgment. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;  # required
		:license_id = "CC BY 4.0" ;   # required
		:mip_era = "CMIP7" ;   # can stay as is
		:source_id = "" ;  # must match the CVs exactly
		:source_version = "" ; # must match the CVs exactly
		:target_mip = "CFMIP" ;
		:tracking_id = "" ;  # can stay as is
		:variable_id = "" ;  # must match the variable in the file exactly

If you can send me (or upload into the google drive link I shared via email) a single AMIP file and a single piClim-deltaSST, that will be enough for checking, no need to send the full set at the moment

[pceppi]

Thanks. I've now uploaded two sample files to the Google Drive. I used the source IDs we agreed on.

While editing the attributes, I realised that one tweak needs to be made to the CVs: for the AMIP datasets, Jonathan Gregory should be removed from the list of contacts (but he should be included in the piClim-deltaSST files). I'm not GitHub-savvy so I wasn't sure whether I could make this edit myself?

[znichollscr]

Thanks

I'm not GitHub-savvy so I wasn't sure whether I could make this edit myself?

You can if you want (google something like "GitHub edit file in browser and make pull request"), but you don't need to: this change is tiny and I can do it.

I hope to have a more in depth look at the files early next week

[znichollscr]

Alrighty @pceppi sorry I have been so slow replying. Tweaks for the next round

Dtos1x1_CFMIP4_CanESM5_abrupt-4xCO2.nc

2. The creation_date attribute must be of the form YYYY-MM-DDThh:mm:ssZ, i.e. be an ISO 8601 timestamp in the UTC timezone, something like 2025-02-28T18:29:22Z
3. The tracking_id attribute must start with the prefix 'hdl:21.14100/', followed by a version 4 universally unique identifier (UUID4). If it does this, it will match the following regular expression,
   'hdl\:21\.14100\/[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}'. Here is an example of how to generate one in python

siconcbcs_HadISST_187001-202512.nc

1. remove the _FillValue attribute from all of your *_bnds variables
2. add an attribute external_variables and put areacello as the value
3. The creation_date attribute must be of the form YYYY-MM-DDThh:mm:ssZ, i.e. be an ISO 8601 timestamp in the UTC timezone, something like 2025-02-28T18:29:22Z
4. The tracking_id attribute must start with the prefix 'hdl:21.14100/', followed by a version 4 universally unique identifier (UUID4). If it does this, it will match the following regular expression,
   'hdl\:21\.14100\/[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12}'. Here is an example of how to generate one in python

[pceppi]

Thanks @znichollscr. For the tracking_id, I didn't find the Python example you mentioned? Also, do we just pick an ID at random? (Perhaps the example you mentioned addresses this second question.)

Also, for Dtos1x1_CFMIP4_CanESM5_abrupt-4xCO2.nc: does it matter that the coordinate variables are named longitude and latitude (and their _bnds counterparts), as opposed to lon and lat?

[znichollscr]

Apologies I meant to link https://input4mips-validation.readthedocs.io/en/latest/api/input4mips_validation/database/raw/#input4mips_validation.database.raw.Input4MIPsDatabaseEntryFileRaw.tracking_id

import uuid  # part of the standard library

tracking_id = f"hdl:21.14100/{uuid.uuid4()}"

[znichollscr]

Also, for Dtos1x1_CFMIP4_CanESM5_abrupt-4xCO2.nc: does it matter that the coordinate variables are named longitude and latitude (and their _bnds counterparts), as opposed to lon and lat?

Not that I am aware of, but you are free to change them if you wish

[pceppi]

Thanks @znichollscr, have now updated the files accordingly. An additional minor tweak (not requested by you) is that I removed the _FillValue attributes from the lon, lat and time variables in Dtos1x1_CFMIP4_CanESM5_abrupt-4xCO2.nc. I assume that's ok, there are no missing values in those variables.