Updated required data sources to run Examples

[99snowleopards]

Hello - I'm starting a discussion to compile the full list of data sources required to run the examples, since some of the original links have changed. These are the required data:

1. Hydrodynamic model data

This is the NEMO Analysis & Forecast data, that is hosted here:

https://data.marine.copernicus.eu/product/GLOBAL_ANALYSISFORECAST_PHY_001_024/description

It looks like the files nomenclature has changed - the example_Italy_coast_settings.json file shows the filename_style as "psy4v3r1/psy4v3r1-daily_", but the link above has different file names. This document has the new nomenclature.

2. Biogeochemical model data:

This is hosted here:

https://data.marine.copernicus.eu/product/GLOBAL_ANALYSISFORECAST_BGC_001_028/description

3. Wave data

ERA5 wave data is hosted here:

https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels?tab=overview

I don't see the variables u_component_stokes_drift and v_component_stokes_drift listed in the download options.

4. Wind data

This is available from the ERA5 link above

---

The the example_Italy_coast_settings.json file also lists some additional files:

"ocean_mesh": "domain_ORCA0083-N006/PSY4V3R1_mesh_hgr.nc"

"bathymetry_mesh": "domain_ORCA0083-N006/PSY4V3R1_mesh_zgr.nc"

"bgc_mesh": "domain_ORCA025-N006/mesh_hgr_PSY4V3_deg.nc"

"filename": "input_data/NEMO0083/land_current_NEMO0083.nc"

Could you share the download locations of these netcdf files please?

Please let me know if I've missed any other required files.

Thank you!

Zen.

[michaeldenes]

Hi @99snowleopards, originally plasticparcels was built for the native C-grid version of the model you link to above, run by Mercator Ocean international (MOi). Since our last release, they've changed their website, and location of their data, but it should still be available (via a THREDDS data server). There's some additional information here: https://tds.mercator-ocean.fr/userguide/user_guide.html

You can still use the re-interpolated A-grid data from Copernicus though (just make sure that when you load the data, you don't interpret it as C-grid data!).

Regarding the wave data, I use the cdsapi (see https://cds.climate.copernicus.eu/how-to-api) to download the data. For example:

import os
import datetime
import cdsapi #Setup as per https://cds.climate.copernicus.eu/how-to-api
import pandas as pd

dates = pd.date_range(datetime.date(2020,1,1),datetime.date(2023,12,1),freq='MS')

dates = dates[::-1]  # reverse order to start downloading from the most recent data

client = cdsapi.Client()
download_folder = 'path/to/download/to'
name_ = 'ERA5_global_waves_monthly_'

for date_ in dates:

  file_str = os.path.join(download_folder, name_ + str(date_.date()) + '.nc')

  dataset = 'reanalysis-era5-single-levels'
  request = {
    'product_type': ['reanalysis'],
    'variable': ['mean_wave_period','peak_wave_period','u_component_stokes_drift','v_component_stokes_drift'],
    'year': str(date_.year),
    'month': '%2.2i' %date_.month,
    'day': [
                      '01', '02', '03',
                      '04', '05', '06',
                      '07', '08', '09',
                      '10', '11', '12',
                      '13', '14', '15',
                      '16', '17', '18',
                      '19', '20', '21',
                      '22', '23', '24',
                      '25', '26', '27',
                      '28', '29', '30',
                      '31',
                  ],
                  "time": [ # Take 6 hourly data to reduce load on CDS
                        "00:00",
                        "06:00",
                        "12:00",
                        "18:00"
                    ],
    'data_format': 'netcdf',
  }
  target = file_str
  client.retrieve(dataset, request, target)
  print(file_str, "retrieved.")

Again, since the last release of plasticparcels, the ERA5 netcdf output has changed, resulting in some issues in both parcels and plasticparcels... I convert the data using the following code:

# Conversion WAVES

import xarray as xr
import glob
import numpy as np
from glob import glob
import os

def sort_and_zero_data(data_,i_sort_lat,i_sort_lon):

    tmp_1 = data_[:,:,i_sort_lon]
    tmp_2 = tmp_1[:,i_sort_lat,:]

    mask = np.isnan(tmp_2)
    tmp_2[mask] = 0.
    return tmp_2
    
download_folder = 'path/to/download/to'
    
files = glob(os.path.join(download_folder,'ERA5_global_waves_monthly_20*'))
files = files[::-1] # reverse order

for file_ in files:
    
    data_ = xr.open_dataset(file_)
    data_ = data_.rename({"valid_time":"time"})
    data_ = data_.drop_vars('number') # 
    data_ = data_.drop_vars('expver')
    
    lons = data_['longitude'].data
    lons[lons>180] -= 360
    i_sort_lon = np.argsort(lons)
    lats = data_['latitude'].data
    i_sort_lat = np.argsort(lats)
                     
    var_mwp = sort_and_zero_data(data_['mwp'].data,i_sort_lat,i_sort_lon)
    var_pp1d = sort_and_zero_data(data_['pp1d'].data,i_sort_lat,i_sort_lon)
    var_ust = sort_and_zero_data(data_['ust'].data,i_sort_lat,i_sort_lon)
    var_vst = sort_and_zero_data(data_['vst'].data,i_sort_lat,i_sort_lon)
    
    lons_sorted = lons[i_sort_lon]
    lats_sorted = lats[i_sort_lat]
    
    ds = xr.Dataset(
    {"mwp": (("time", "lat", "lon"), var_mwp ),
      "pp1d": (("time", "lat", "lon"), var_pp1d ),
      "ust": (("time", "lat", "lon"), var_ust ),
      "vst": (("time", "lat", "lon"), var_vst )
      #"explanation": 'wave data (mean wave period, peak wave period, Stokes drift) converted from ERA5 data to have lon from -180 to 180, lats sorted from south to north, and zeros instead of NaN'},
    },
    coords={
        "lon": lons_sorted,
        "lat": lats_sorted,
        "time": data_['time'].data,
    },
    )   
   
    encoding_ = {"mwp": {"dtype": "float32"},
                 "pp1d": {"dtype": "float32"},
                "ust": {"dtype": "float32"},
                 "vst": {"dtype": "float32"}}

    file_out = file_[:-13] + 'converted_' + file_[-13:-3] + '.nc'
    ds.to_netcdf(file_out)   
    print(file_out)

I hope this helps!

[michaeldenes]

Also, it's worth mentioning that I'm currently working on integrating plasticparcels to directly ingest Copernicus data using the copernicusmarine toolbox (see https://help.marine.copernicus.eu/en/articles/7949409-copernicus-marine-toolbox-introduction ), but this will require some of the new features of the currently-being-developed parcels v4 (see https://docs.oceanparcels.org/en/v4-dev/v4/) to be performant!

[99snowleopards]

Thank you so much for the detailed response @michaeldenes !

The example_Italy_coast_settings.json file lists the following netcdf files - where can we download those from?

"ocean_mesh": "domain_ORCA0083-N006/PSY4V3R1_mesh_hgr.nc"

"bathymetry_mesh": "domain_ORCA0083-N006/PSY4V3R1_mesh_zgr.nc"

"bgc_mesh": "domain_ORCA025-N006/mesh_hgr_PSY4V3_deg.nc"

"filename": "input_data/NEMO0083/land_current_NEMO0083.nc"

[yinkakosothomas]

Hi @99snowleopards,
I was able to find this website (https://gdex.ucar.edu/datasets/d010049/) and the THREDDS server mentioned (https://thredds.ucar.edu/thredds/catalog/catalog.html) that houses data from Copernicus in netCDF formatted files. I'm working on the Greece coast example and don't know how to change the .json settings file to run without errors?

If you have any other information @michaeldenes about importing the hydrodynamic (and maybe biogeochemical data) that would be much appreciated!