4.2.1. ESA CCI Toolbox Finding Data

This notebook shows the usage of some covenience functions that have been setup to further facilitate the handling of the stores and the use of the ESA CCI Toolbox. A thorough description can be found here: https://esa-climate-toolbox.readthedocs.io/en/latest/api_reference.html#datasets-and-datastores. To see how you can actually open data using these stores, see notebooks 2-5.

To run this Notebook, make sure the ESA CCI Toolbox is setup correctly.

The functions provided here serve the purpose to get an overview about data and from which data store to get it from. They also serve to look for Essential Climate Variables. We may start by listing all that are provided by the ESA CCI Toolbox.

from esa_climate_toolbox.core import list_ecvs

list_ecvs()

['OC',
 'SOILMOISTURE',
 'LC',
 'SEAICE',
 'PERMAFROST',
 'VEGETATION',
 'SEASURFACESALINITY',
 'LST',
 'SST',
 'SEASTATE',
 'CLOUD',
 'AEROSOL',
 'GHG',
 'BIOMASS',
 'SNOW',
 'OZONE',
 'WATERVAPOUR',
 'LAKES',
 'SEALEVEL',
 'ICESHEETS',
 'FIRE']

Now that we see what the toolbox can offer, we can find datasets for one ecv, e.g., WATERVAPOUR.

from esa_climate_toolbox.core import list_ecv_datasets

list_ecv_datasets("WATERVAPOUR")

[('esacci.WATERVAPOUR.day.L3S.TCWV.multi-sensor.multi-platform.TCWV_land_005deg.3-2.r1',
  'esa-cci'),
 ('esacci.WATERVAPOUR.day.L3S.TCWV.multi-sensor.multi-platform.TCWV_land_05deg.3-2.r1',
  'esa-cci'),
 ('esacci.WATERVAPOUR.mon.L3S.TCWV.multi-sensor.multi-platform.TCWV_land_005deg.3-2.r1',
  'esa-cci'),
 ('esacci.WATERVAPOUR.mon.L3S.TCWV.multi-sensor.multi-platform.TCWV_land_05deg.3-2.r1',
  'esa-cci')]

The result consists of a list of tuples, where the first value is a data id and the second value the name of the store that provides access to the data. For information on how to access the stores specifically, using the xcube store framework, see notebooks 2 to 5. Here, we continue with listing all the stores that are registered in the system.

from esa_climate_toolbox.core import list_stores

list_stores()

['esa-cci', 'esa-cci-kc', 'esa-cci-zarr', 'local']

You will see default stores (and maybe some you have added yourself). You can also ask for a store by asking for a specific dataset id.

from esa_climate_toolbox.core import find_data_store

find_data_store(ds_id="esacci.WATERVAPOUR.mon.L3S.TCWV.multi-sensor.multi-platform.TCWV_land_05deg.3-2.r1")

('esa-cci',
 <esa_climate_toolbox.ds.dataaccess.CciCdcDataStore at 0x7f432ff48450>)

The result consists of the name of the store as well as the python object itself. We could also get it by just asking for its name.

from esa_climate_toolbox.core import get_store

get_store("esa-cci")

<esa_climate_toolbox.ds.dataaccess.CciCdcDataStore at 0x7f432ff48450>

We can also ask for the data of a store:

from esa_climate_toolbox.core import list_datasets

list_datasets("esa-cci-zarr")

['ESACCI-BIOMASS-L4-AGB-MERGED-100m-2010-2018-fv2.0.zarr',
 'ESACCI-BIOMASS-L4-AGB-MERGED-100m-2010-2020-fv4.0.zarr',
 'ESACCI-GHG-L2-CH4-SCIAMACHY-WFMD-2002-2011-fv1.zarr',
 'ESACCI-GHG-L2-CO2-OCO-2-FOCAL-2014-2021-v10.zarr',
 'ESACCI-GHG-L2-CO2-SCIAMACHY-WFMD-2002-2012-fv1.zarr',
 'ESACCI-ICESHEETS_Antarctica_GMB-2002-2016-v1.1.zarr',
 'ESACCI-ICESHEETS_Greenland_GMB-2003-2016-v1.1.zarr',
 'ESACCI-L3C_CLOUD-CLD_PRODUCTS-AVHRR_NOAA-1982-2016-fv3.0.zarr',
 'ESACCI-L3C_SNOW-SWE-1979-2018-fv1.0.zarr',
 'ESACCI-L3C_SNOW-SWE-1979-2020-fv2.0.zarr',
 'ESACCI-L4_GHRSST-SST-GMPE-GLOB_CDR2.0-1981-2016-v02.0-fv01.0.zarr',
 'ESACCI-LAKES-L3S-LK_PRODUCTS-MERGED-1992-09-fv2.0.1.zarr',
 'ESACCI-LC-L4-LCCS-Map-300m-P1Y-1992-2015-v2.0.7b.zarr',
 'ESACCI-LST-L3C-LST-MODISA-0.01deg_1DAILY_DAY-2002-2018-fv3.00.zarr',
 'ESACCI-LST-L3C-LST-MODISA-0.01deg_1DAILY_NIGHT-2002-2018-fv3.00.zarr',
 'ESACCI-LST-L3S-LST-IRCDR_-0.01deg_1DAILY_DAY-1995-2020-fv3.00.zarr',
 'ESACCI-LST-L3S-LST-IRCDR_-0.01deg_1DAILY_NIGHT-1995-2020-fv3.00.zarr',
 'ESACCI-LST-L3S-LST-IRCDR_-0.01deg_1MONTHLY_DAY-1995-2020-fv3.00.zarr',
 'ESACCI-LST-L3S-LST-IRCDR_-0.01deg_1MONTHLY_NIGHT-1995-2020-fv3.00.zarr',
 'ESACCI-OC-L3S-IOP-MERGED-1M_MONTHLY_4km_GEO_PML_OCx-QAA-1997-2022-fv6.0.zarr',
 'ESACCI-OC-L3S-IOP-MERGED-1M_MONTHLY_4km_GEO_PML_OCx_QAA-1997-2020-fv5.0.zarr',
 'ESACCI-OC-L3S-IOP-MERGED-1Y_YEARLY_4km_GEO_PML_OCx_QAA-1997-2022-fv6.0.zarr',
 'ESACCI-OC-L3S-IOP-MERGED-1Y_YEARLY_4km_GEO_PML_RRS-1997-2022-fv6.0.zarr',
 'ESACCI-OC-L3S-IOP-MERGED-8D_DAILY_4km_GEO_PML_OCx_QAA-1997-2022-fv6.0.zarr',
 'ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1M_MONTHLY_4km_GEO_PML_OCx_QAA-1997-2022-fv6.0.zarr',
 'ESACCI-OC-L3S-OC_PRODUCTS-MERGED-1Y_YEARLY_4km_GEO_PML_OCx_QAA-1997-2022-fv6.0.zarr',
 'ESACCI-OC-L3S-RRS-MERGED-1M_MONTHLY_4km_GEO_PML_RRS-1997-2022-fv6.0.zarr',
 'ESACCI-PERMAFROST-L4-ALT-MODISLST-AREA4_PP-1997-2018-fv02.0.zarr',
 'ESACCI-SEAICE-L3C-SITHICK-RA2_ENVISAT-NH25KMEASE2-2002-2012-fv2.0.zarr',
 'ESACCI-SEAICE-L3C-SITHICK-SIRAL_CRYOSAT2-NH25KMEASE2-2010-2017-fv2.0.zarr',
 'ESACCI-SEAICE-L4-SICONC-AMSR_50.0kmEASE2-NH-2002-2017-fv2.1.zarr',
 'ESACCI-SEALEVEL-IND-MSLTR-MERGED-1993-2016-fv02.zarr',
 'ESACCI-SEALEVEL-L4-MSLA-MERGED-1993-2015-fv02.zarr',
 'ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED-1978-2020-fv05.3.zarr',
 'ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED-1978-2021-fv07.1.zarr']

4.2.1.1. Adding your own stores

You may also define your own stores. This can come in handy when you want to combine your own data with the data from the toolbox. You may add store to locally hosted data like this:

from esa_climate_toolbox.core import add_local_store

add_local_store(
    root="/path/to/my/data/",
    store_id="my_real_local_store",
    includes="aero*.nc",
    read_only=True,
    persist=False
)

'my_real_local_store'

The root specifies the path to the data that shall be served by the store. The store_id defines by which name the store is registered in the system. With includes you can specify a pattern to determine what data shall be served, read_only causes that the store does not permit write operations, and if you would set persist to true, the store will be saved and available in your next session. Of course, we can always also remove a store. Don’t worry, this only deletes the registry entry. No real data is deleted.

from esa_climate_toolbox.core import remove_store

remove_store("my_real_local_store")

4.2.1.2. Writing data to stores

When you have created a new dataset, you might want to save it somewhere. For that, you can use this command:

from esa_climate_toolbox.core import write_data

# write_data(data, store_id="my_real_local_store")

We leave this open for now, as we don’t want to actually write data now. Note that to write data, you need to define a store and make sure it is not read_only (see above). If you know you want your output in the same place, you can set a default output store:

from esa_climate_toolbox.core import set_output_store

add_local_store(
    root="/path/to/my/data/",
    store_id="my_real_local_store",
    includes="aero*.nc",
    read_only=True,
    persist=False
)
set_output_store("my_real_local_store")

Let’s check this is really the output store:

from esa_climate_toolbox.core import get_output_store_id

get_output_store_id()

'my_real_local_store'

Great! You are set to use these functions for making use of the toolbox. See the other notebooks to find out how to open datasets and use operations. Also, see the API for more detailed information on the functions described here: https://esa-climate-toolbox.readthedocs.io/en/latest/api_reference.html#datasets-and-datastores.