{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "## Gap Filling \n", "\n", "This notebook will show some algorithms to fill gaps of vector data and raster data. The notebook is structured as follows: \n", "\n", "1. Gap Filling of a vector dataset in spatial domain \n", "2. Gap filling of a raster dataset in temporal and spatial domain\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [] }, "outputs": [], "source": [ "from esa_climate_toolbox.core import get_op\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "import numpy as np\n", "import scipy.interpolate as interp\n", "import xarray as xr\n", "from xcube.core.store import new_data_store" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get access to the data, we will initiate the ESA CCI Zarr store in the following cell." ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "store = new_data_store('esa-cci-zarr')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "--- \n", "### 1. Gap Filling of a Vector Dataset in the Spatial Domain\n", "\n", "In this example, we will work with trend data of the Greenland ice sheet, which is provided as a set of spatial points over a given time range. We will apply several interpolation algorithms to fill the gaps between these points, ultimately converting the data into a raster format.\n", "\n", "First we will open the data as usual. " ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
<xarray.Dataset> Size: 43kB\n",
"Dimensions: (meas_ind: 901, time: 10)\n",
"Dimensions without coordinates: meas_ind, time\n",
"Data variables:\n",
" GMB_trend (meas_ind, time) float32 36kB dask.array<chunksize=(901, 10), meta=np.ndarray>\n",
" end_time (time) datetime64[ns] 80B dask.array<chunksize=(10,), meta=np.ndarray>\n",
" latitude (meas_ind) float32 4kB dask.array<chunksize=(901,), meta=np.ndarray>\n",
" longitude (meas_ind) float32 4kB dask.array<chunksize=(901,), meta=np.ndarray>\n",
" start_time (time) datetime64[ns] 80B dask.array<chunksize=(10,), meta=np.ndarray>\n",
" t (time) datetime64[ns] 80B dask.array<chunksize=(10,), meta=np.ndarray>\n",
"Attributes: (12/20)\n",
" Institution: DTU Space - Div. of Geodynamics\n",
" Latitude_max: 83.23728\n",
" Latitude_min: 60.73857\n",
" Longitude_max: -14.706720000000018\n",
" Longitude_min: -71.83566000000002\n",
" Method: GRACE GMB trend following Barletta et al. 2012\n",
" ... ...\n",
" grid_projection: Geographical coordinates relative to WGS84\n",
" netCDF_version: NETCDF3_CLASSIC\n",
" product_version: 1.2\n",
" time_coverage_end: 2016-01-31\n",
" time_coverage_start: 2003-01-01\n",
" time_resolution: 5Y<xarray.Dataset> Size: 29MB\n",
"Dimensions: (lat: 1200, lon: 3050)\n",
"Coordinates:\n",
" * lon (lon) float64 24kB -73.0 -72.98 -72.96 ... -12.04 -12.02\n",
" * lat (lat) float64 10kB 83.0 82.98 82.96 ... 59.06 59.04 59.02\n",
"Data variables:\n",
" GMB_trend_slice (lat, lon) float64 29MB nan nan nan nan ... nan nan nan nan<xarray.Dataset> Size: 29MB\n",
"Dimensions: (lat: 1200, lon: 3050)\n",
"Coordinates:\n",
" * lon (lon) float64 24kB -73.0 -72.98 -72.96 ... -12.04 -12.02\n",
" * lat (lat) float64 10kB 83.0 82.98 82.96 ... 59.06 59.04 59.02\n",
"Data variables:\n",
" GMB_trend_slice (lat, lon) float64 29MB nan nan nan nan ... nan nan nan nan<xarray.Dataset> Size: 654GB\n",
"Dimensions: (time: 15767, lat: 720, lon: 1440)\n",
"Coordinates:\n",
" * lat (lat) float64 6kB 89.88 89.62 89.38 ... -89.38 -89.62 -89.88\n",
" * lon (lon) float64 12kB -179.9 -179.6 -179.4 ... 179.6 179.9\n",
" * time (time) datetime64[ns] 126kB 1978-11-01 ... 2021-12-31\n",
"Data variables:\n",
" dnflag (time, lat, lon) float32 65GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" flag (time, lat, lon) float32 65GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" freqbandID (time, lat, lon) float32 65GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" mode (time, lat, lon) float32 65GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" sensor (time, lat, lon) float64 131GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" sm (time, lat, lon) float32 65GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" sm_uncertainty (time, lat, lon) float32 65GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
" t0 (time, lat, lon) float64 131GB dask.array<chunksize=(16, 720, 720), meta=np.ndarray>\n",
"Attributes: (12/44)\n",
" Conventions: CF-1.9\n",
" cdm_data_type: Grid\n",
" comment: This dataset was produced with funding of t...\n",
" contact: cci_sm_contact@eodc.eu\n",
" creator_email: cci_sm_developer@eodc.eu\n",
" creator_name: Department of Geodesy and Geoinformation, V...\n",
" ... ...\n",
" time_coverage_end_product: 20211231T235959Z\n",
" time_coverage_resolution: P1D\n",
" time_coverage_start: 1978-11-01 00:00:00\n",
" time_coverage_start_product: 19781101T000000Z\n",
" title: ESA CCI Surface Soil Moisture COMBINED acti...\n",
" tracking_id: ad35798e-58e0-488f-b5b9-593874a47700<xarray.Dataset> Size: 18kB\n",
"Dimensions: (time: 367)\n",
"Coordinates:\n",
" lat float64 8B 20.12\n",
" lon float64 8B 20.12\n",
" * time (time) datetime64[ns] 3kB 2000-01-01 ... 2001-01-01\n",
"Data variables:\n",
" dnflag (time) float32 1kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" flag (time) float32 1kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" freqbandID (time) float32 1kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" mode (time) float32 1kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" sensor (time) float64 3kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" sm (time) float32 1kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" sm_uncertainty (time) float32 1kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
" t0 (time) float64 3kB dask.array<chunksize=(13,), meta=np.ndarray>\n",
"Attributes: (12/44)\n",
" Conventions: CF-1.9\n",
" cdm_data_type: Grid\n",
" comment: This dataset was produced with funding of t...\n",
" contact: cci_sm_contact@eodc.eu\n",
" creator_email: cci_sm_developer@eodc.eu\n",
" creator_name: Department of Geodesy and Geoinformation, V...\n",
" ... ...\n",
" time_coverage_end_product: 20211231T235959Z\n",
" time_coverage_resolution: P1D\n",
" time_coverage_start: 1978-11-01 00:00:00\n",
" time_coverage_start_product: 19781101T000000Z\n",
" title: ESA CCI Surface Soil Moisture COMBINED acti...\n",
" tracking_id: ad35798e-58e0-488f-b5b9-593874a47700<xarray.Dataset> Size: 18kB\n",
"Dimensions: (time: 367)\n",
"Coordinates:\n",
" lat float64 8B 20.12\n",
" lon float64 8B 20.12\n",
" * time (time) datetime64[ns] 3kB 2000-01-01 ... 2001-01-01\n",
"Data variables:\n",
" dnflag (time) float32 1kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" flag (time) float32 1kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" freqbandID (time) float32 1kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" mode (time) float32 1kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" sensor (time) float64 3kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" sm (time) float32 1kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" sm_uncertainty (time) float32 1kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
" t0 (time) float64 3kB dask.array<chunksize=(367,), meta=np.ndarray>\n",
"Attributes: (12/44)\n",
" Conventions: CF-1.9\n",
" cdm_data_type: Grid\n",
" comment: This dataset was produced with funding of t...\n",
" contact: cci_sm_contact@eodc.eu\n",
" creator_email: cci_sm_developer@eodc.eu\n",
" creator_name: Department of Geodesy and Geoinformation, V...\n",
" ... ...\n",
" time_coverage_end_product: 20211231T235959Z\n",
" time_coverage_resolution: P1D\n",
" time_coverage_start: 1978-11-01 00:00:00\n",
" time_coverage_start_product: 19781101T000000Z\n",
" title: ESA CCI Surface Soil Moisture COMBINED acti...\n",
" tracking_id: ad35798e-58e0-488f-b5b9-593874a47700<xarray.DataArray 'sm' (lat: 720, lon: 1440)> Size: 4MB\n",
"dask.array<getitem, shape=(720, 1440), dtype=float32, chunksize=(720, 720), chunktype=numpy.ndarray>\n",
"Coordinates:\n",
" * lat (lat) float64 6kB 89.88 89.62 89.38 89.12 ... -89.38 -89.62 -89.88\n",
" * lon (lon) float64 12kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9\n",
" time datetime64[ns] 8B 2000-07-02\n",
"Attributes:\n",
" _CoordinateAxes: time lat lon\n",
" ancillary_variables: sm_uncertainty flag t0\n",
" dtype: float32\n",
" long_name: Volumetric Soil Moisture\n",
" standard_name: soil_moisture_content\n",
" units: m3 m-3\n",
" valid_range: [0.0, 1.0]<xarray.Dataset> Size: 4MB\n",
"Dimensions: (lat: 720, lon: 1440)\n",
"Coordinates:\n",
" * lat (lat) float64 6kB 89.88 89.62 89.38 89.12 ... -89.38 -89.62 -89.88\n",
" * lon (lon) float64 12kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9\n",
" time datetime64[ns] 8B 2000-07-02\n",
"Data variables:\n",
" sm (lat, lon) float32 4MB dask.array<chunksize=(720, 720), meta=np.ndarray>\n",
"Attributes: (12/44)\n",
" Conventions: CF-1.9\n",
" cdm_data_type: Grid\n",
" comment: This dataset was produced with funding of t...\n",
" contact: cci_sm_contact@eodc.eu\n",
" creator_email: cci_sm_developer@eodc.eu\n",
" creator_name: Department of Geodesy and Geoinformation, V...\n",
" ... ...\n",
" time_coverage_end_product: 20211231T235959Z\n",
" time_coverage_resolution: P1D\n",
" time_coverage_start: 1978-11-01 00:00:00\n",
" time_coverage_start_product: 19781101T000000Z\n",
" title: ESA CCI Surface Soil Moisture COMBINED acti...\n",
" tracking_id: ad35798e-58e0-488f-b5b9-593874a47700<xarray.DataArray 'sm' (lat: 720, lon: 1440)> Size: 4MB\n",
"dask.array<fillgaps_lowpass_2d, shape=(720, 1440), dtype=float32, chunksize=(720, 720), chunktype=numpy.ndarray>\n",
"Coordinates:\n",
" * lat (lat) float64 6kB 89.88 89.62 89.38 89.12 ... -89.38 -89.62 -89.88\n",
" * lon (lon) float64 12kB -179.9 -179.6 -179.4 ... 179.4 179.6 179.9\n",
" time datetime64[ns] 8B 2000-07-02\n",
"Attributes:\n",
" _CoordinateAxes: time lat lon\n",
" ancillary_variables: sm_uncertainty flag t0\n",
" dtype: float32\n",
" long_name: Volumetric Soil Moisture\n",
" standard_name: soil_moisture_content\n",
" units: m3 m-3\n",
" valid_range: [0.0, 1.0]