EGU SC 2021 added

50_workshops/202104_egu_short_course/00_index.ipynb

+%% Cell type:markdown id: tags:
+
+<img src='./img/egu21_logo.png' alt='Logo EU Copernicus EUMETSAT' align='left' width='25%'></img><img src='./img/atmos_logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='65%'></img></span>
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+#### EGU Short Course 5.5 - Using Copernicus data for Atmospheric Composition Applications | 28 April + 5 May 2021
+
+%% Cell type:markdown id: tags:
+
+# Discover satellite data for Dust Monitoring
+
+%% Cell type:markdown id: tags:
+
+Jointly organized by [EUMETSAT](https://www.eumetsat.int/website/home/index.html), [CAMS-ECMWF](https://atmosphere.copernicus.eu/) and [ESA](https://www.esa.int), with support from [Copernicus](https://www.copernicus.eu/en).
+
+%% Cell type:markdown id: tags:
+
+This practical session gives you an overview of different satellite data for dust monitoring and provides you hands-on practical example how to retrieve, load, process and visualize the data.
+
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<img src='./img/egu_short_course_logo.jpg' alt='Logo EU Copernicus EUMETSAT' align='center' width='70%'></img>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+## Notebook overview and featured data
+
+%% Cell type:markdown id: tags:
+
+### The following satellite datasets for dust monitoring are featured:
+
+* [CAMS Global Near-Real-Time Forecast](https://apps.ecmwf.int/datasets/data/cams-nrealtime/levtype=sfc/)
+* [CAMS European Air Quality Forecasts and Analyses](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-europe-air-quality-forecasts?tab=overview)
+* [Sentinel-5P TROPOMI Level 2 Ultraviolet Aerosol Index](https://sentinels.copernicus.eu/web/sentinel/data-products/-/asset_publisher/fp37fc19FN8F/content/sentinel-5-precursor-level-2-ultraviolet-aerosol-index)
+* [AC SAF GOME-2 Level 3 Absorbing Aerosol Index](https://acsaf.org/index.html)
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### The following notebooks are available:
+
+#### Data discovery
+
+* [01 - Overview - Atmospheric Composition Data - Overview and Retrieve](./01_overview_atmospheric_composition_data_retrieve.ipynb)
+* [02 - CAMS global near-real-time forecast - Dust Aerosol Optical Depth - Load and browse](./02_CAMS_global_forecast_duaod_load_browse.ipynb)
+* [03 - CAMS European Air Quality Forecasts and Analyses - Dust concentration - Load and browse](./03_CAMS_European_air_quality_forecast_dust_concentration_load_browse.ipynb)
+* [04 - Sentinel-5P TROPOMI Ultraviolet Aerosol Index Level 2 - Load and browse](./04_Sentinel-5P_TROPOMI_AI_L2_load_browse.ipynb)
+* [05 - Metop-ABC GOME-2 Absorbing Aerosol Index Level 3 - Load and browse](./05_Metop-ABC_GOME-2_AAI_L3_load_browse.ipynb)
+
+#### Exercises
+* [12 - CAMS global near-real-time forecast - Dust Aerosol Optical Depth - Exercise](./12_CAMS_global_forecast_duaod_exercise.ipynb)
+* [13 - CAMS European Air Quality Forecasts and Analyses - Dust concentration - Exercise](./13_CAMS_European_air_quality_forecast_dust_concentration_exercise.ipynb)
+* [14 - Sentinel-5P TROPOMI Ultraviolet Aerosol Index Level 2 - Exercise](./14_Sentinel-5P_TROPOMI_AI_L2_exercise.ipynb)
+* [15 - Metop-ABC GOME-2 Absorbing Aerosol Index Level 3 - Exercise](./15_Metop-ABC_GOME-2_AAI_L3_exercise.ipynb)
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+
+**NOTE:** Throughout the course, general functions to `load`, `re-shape`, `process` and `visualize` the datasets are defined. These functions are re-used when applicable. The [functions notebook](./functions.ipynb) gives you an overview of all the functions defined and used for the course.
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+## Learning outcomes
+
+%% Cell type:markdown id: tags:
+
+The course is designed for `beginners and medium-level users`, who have basic Python knowledge.
+
+After the course, you should have:
+* an idea about **satellite and assimilated data products for dust monitoring**,
+* knowledge about the most useful **Python packages** to handle, process and visualise atmospheric composition data
+* an idea about how the **data can help to detect and monitor dust events**
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+## Access to the training platform
+
+%% Cell type:markdown id: tags:
+
+The course material is made available on a JupyterLab training platform, a pre-defined environment that gives learners direct access to the data and Python packages required for following the course.
+
+The `JupyterLab` can be accessed as follows:
+
+%% Cell type:markdown id: tags:
+
+* Web address: [https://training.ltpy.adamplatform.eu](https://training.ltpy.adamplatform.eu)
+* Create an account: [https://login.ltpy.adamplatform.eu](https://login.ltpy.adamplatform.eu)
+* Log into the `JupyterLab` with your account created.
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+<img src='../../img/copernicus_logo.png' alt='Logo EU Copernicus' align='right' width='20%'><br><br><br><br><br>
+
+
+
+<p style="text-align:right;">This project is licensed under the <a href="./LICENSE">MIT License</a> and is developed under a Copernicus contract.

50_workshops/202104_egu_short_course/01_overview_atmospheric_composition_data_retrieve.ipynb

+%% Cell type:markdown id: tags:
+
+<img src='./img/egu21_logo.png' alt='Logo EU Copernicus EUMETSAT' align='left' width='25%'></img><img src='./img/atmos_logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='65%'></img></span>
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<a href="./00_index.ipynb"><< Index</a><span style="float:right;"><a href="./02_CAMS_global_forecast_duaod_load_browse.ipynb">02 - CAMS global forecast - Load and browse >></a></span>
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+# Atmospheric composition data - Overview and data access
+
+%% Cell type:markdown id: tags:
+
+This module gives an overview of the following atmospheric composition data services:
+* [ECMWF Data Archive](#ecmwf_archive)
+* [Copernicus Atmosphere Data Store (ADS)](#ads)
+* [Sentinel-5P Pre-Operations data Hub](#s5p)
+* [TEMIS - Tropospheric Emissions Monitoring Internet Service](#temis)
+
+
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+## <a id='ecmwf_archive'></a>ECMWF data archive
+
+%% Cell type:markdown id: tags:
+
+[CAMS global near-real-time forecasts](https://apps.ecmwf.int/datasets/data/cams-nrealtime/levtype=sfc) can be accessed via the `ECMWF data archive`. ECMWF's data archive is called `Meteorological and Archival Retrieval System (MARS)` and provides access to ECMWF Public Datasets.
+
+The archive can be accessed in two ways:
+* via the [web interface](https://apps.ecmwf.int/datasets/) and
+* via the [ECMWF Web API](https://confluence.ecmwf.int/display/WEBAPI/Access+ECMWF+Public+Datasets).
+
+Subsequently, an example is shown how a you can use the ECMWF Web API to execute a MARS request with Python in order to download data in netCDF.
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### 1. Register for an ECMWF user account
+
+%% Cell type:markdown id: tags:
+
+- Self-register at https://apps.ecmwf.int/registration/
+- Login at https://apps.ecmwf.int/auth/login
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### 2. Install the `ecmwfapi` python library
+
+%% Cell type:markdown id: tags:
+
+`pip install ecmwf-api-client`
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### 3. Retrieve your API key
+
+%% Cell type:markdown id: tags:
+
+You can retrieve your API key at https://api.ecmwf.int/v1/key/. Add the `url`, `key` and `email` information, when you define the `ECMWFDataServer` (see below).
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### 4. Execute a MARS request and download data as a `netCDF` file
+
+%% Cell type:markdown id: tags:
+
+Below, you see the principle of a `data retrieval` request. You can use the web interface to browse through the datasets. At the end, there is the option to generate the `data retrieval` request for the API.
+
+<br>
+
+<div class="alert alert-block alert-success">
+<b>NOTE</b>: <br>
+    Per default, ECMWF data is stored on a grid with longitudes from 0 to 360 degrees. It can be reprojected to a regular geographic latitude-longitude grid, by setting the keyword argument <code>area</code> and <code>grid</code>. Per default, data is retrieved in <code>GRIB</code>. If you wish to retrieve the data in <code>netCDF</code>, you have to specify it by using the keyword argument <code>format</code>.</div>
+
+<br>
+
+The example below requests `Dust Aerosol Optical Depth at 550 nm` forecast data for 19 February 2021 in `netCDF` format and by setting the `area` keyword, the longitude values are reprojected to a [-180,180] grid.
+
+%% Cell type:code id: tags:
+
+``` python
+#!/usr/bin/env python
+from ecmwfapi import ECMWFDataServer
+server = ECMWFDataServer(url="https://api.ecmwf.int/v1", key="#########################", email="########################")
+
+# Retrieve data in NetCDF format
+server.retrieve({
+    "class": "mc",
+    "dataset": "cams_nrealtime",
+    "date": "2021-02-19/to/2021-02-19",
+    "expver": "0001",
+    "levtype": "sfc",
+    "param": "209.210",
+    "step": "0/12/24/36/48/60/72/84/96/108/120",
+    "stream": "oper",
+    "time": "00:00:00",
+    "type": "fc",
+    "format": "netcdf",
+    "area": "90/-180/-90/180",
+    "grid": "0.4/0.4",
+    "target": "../eodata/cams/near_real_time/2021/02/19/20210219_dustAOD.nc",
+})
+```
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+## <a id='ads'></a>Copernicus Atmosphere Data Store (ADS)
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<div style='text-align:left;'>
+<figure><img src='./img/ads_cams.png' width='50%' />
+    <figcaption><i>The web interface of the Copernicus Atmosphere Data Store</i></figcaption>
+</figure>
+</div>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+The [CAMS European air quality forecasts](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-europe-air-quality-forecasts?tab=overview) are available via the `Copernicus Atmosphere Data Store (ADS)`. Data from the `ADS` can be accessed programmtically via the [cdsapi](https://pypi.org/project/cdsapi/) Python library.
+
+To install the [cdsapi](https://pypi.org/project/cdsapi/) follow this [how-to guide](https://cds.climate.copernicus.eu/api-how-to). It includes the following steps:
+
+%% Cell type:markdown id: tags:
+
+### 1. Install the CDS API key
+  * [Self-register](https://ads.atmosphere.copernicus.eu/#!/home) at the CDS registration page
+  * [Login](https://ads.atmosphere.copernicus.eu/user/login) to the ADS portal and go to the [api-how-to page](https://ads.atmosphere.copernicus.eu/api-how-to)
+  * Copy the CDS API key displayed in a file under `$HOME/.cdsapirc` *(Unix / Linux environment)*. If you do not see a URL or key appear in the black terminal box on the `API How To` page, please refresh your browser tab.
+
+
+%% Cell type:code id: tags:
+
+``` python
+%%writefile /home/jovyan/.cdsapirc
+
+url: #########
+key: ####:####################
+```
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### 2. Install the CDS API client
+
+%% Cell type:markdown id: tags:
+
+`pip install cdsapi`
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+### 3. Use the CDS API client for data access
+
+%% Cell type:markdown id: tags:
+
+Below, you see the principle of a `data retrieval` request. You can use the [web interface](https://ads.atmosphere.copernicus.eu/cdsapp#!/search?type=dataset) to browse through the datasets. Under `Download data`, at the end, there is the option to generate the `API request`.
+
+<br>
+
+<div class="alert alert-block alert-success">
+<b>NOTE</b>: <br>
+    per default, ECMWF data is stored on a grid with longitudes from 0 to 360 degrees. It can be reprojected to a regular geographic latitude-longitude grid, by setting the keyword argument <code>area</code> and <code>grid</code>. Per default, data is retrieved in <code>GRIB</code>. If you wish to retrieve the data in <code>netCDF</code>, you have to specify it by using the keyword argument <code>format</code>.</div>
+
+<br>
+
+Below, it is the example request for `dust concentration of the European Air Quality forecasts` data from 19-25 February 2021 in `netCDF` format. By specifying the `area` keyword, the data is reprojected to a [-180,180] grid and geographically subsetted to Europe.
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:code id: tags:
+
+``` python
+import cdsapi
+```
+
+%% Cell type:code id: tags:
+
+``` python
+c = cdsapi.Client()
+
+c.retrieve(
+    'cams-europe-air-quality-forecasts',
+    {
+        'model': 'ensemble',
+        'date': '2021-02-19/2021-02-25',
+        'format': 'netcdf',
+        'variable': 'dust',
+        'level': '1000',
+        'type': 'analysis',
+        'area': [
+            71.56, -24.32, 30,
+            45,
+        ],
+        'time': [
+            '00:00', '06:00', '12:00',
+            '18:00',
+        ],
+        'leadtime_hour': '0',
+    },
+    '../eodata/cams/near_real_time/2021/02/19/20210219-25_regional_nrt_dust_concentation.nc')
+```
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+## <a id='s5p'></a> Sentinel-5P Pre-Operations Data Hub
+
+%% Cell type:markdown id: tags:
+
+Sentinel-5P data is available for browsing and downloading via the [Copernicus Open Access Hub](https://scihub.copernicus.eu/). The Copernicus Open Access Hub provides complete, free and open access to Sentinel-1, Sentinel-2, Sentinel-3 and Sentinel-5P data.
+
+%% Cell type:markdown id: tags:
+
+
+<div style='text-align:center;'>
+<figure><img src='./img/open_access_hub.png' alt='Sentinel-5p data products' align='middle' width='50%'/>
+    <figcaption><i>Interface of the Copernicus Open Access Hub and the Sentinel-5P Pre-Operations Data Hub</i></figcaption>
+</figure>
+</div>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+## <a id='temis'></a> Tropospheric Emission Monitoring Internet Service (TEMIS)
+
+%% Cell type:markdown id: tags:
+
+Gridded (Level 3) Metop-A/B/C GOME-2 Absorbing Aerosol Index (AAI) data are available via the [Tropospheric Emission Monitoring Internet Service (TEMIS)](https://www.temis.nl/airpollution/absaai/). The data of the GOME-2 Absorbing Aerosol Index are provided by KNMI in the framework of the [EUMETSAT Satellite Application Facility on Atmospheric Composition Monitoring (AC SAF)](www.acsaf.org).
+
+TEMIS offers monthly and daily aggregated gridded Metop-A/B/C GOME-2 Absorbing Aerosol Index (AAI) information. The data is available in netCDF for download.
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<div style='text-align:center;'>
+<figure><img src='../../img/temis_aai.png' width='60%' />
+    <figcaption><i>The TEMIS web interface</i></figcaption>
+</figure>
+</div>
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+## Further information
+
+%% Cell type:markdown id: tags:
+
+* [Copernicus Atmosphere Monitoring Service](https://atmosphere.copernicus.eu/)
+
+* [EUMETSAT AC SAF - The EUMETSAT Application Facility on Atmospheric Composition Monitoring](https://acsaf.org/index.html)
+
+* [TEMIS - Tropospheric Emission Monitoring Internet Service](http://temis.nl/index.php)
+
+* [Sentinel-5P Pre-Operations Data Hub](https://s5phub.copernicus.eu/dhus/#/home)
+
+
+
+%% Cell type:markdown id: tags:
+
+<br>
+
+%% Cell type:markdown id: tags:
+
+<a href="./00_index.ipynb"><< Index</a><span style="float:right;"><a href="./02_CAMS_global_forecast_duaod_load_browse.ipynb">02 - CAMS global forecast - Load and browse >></a></span>
+
+%% Cell type:markdown id: tags:
+
+<hr>
+
+%% Cell type:markdown id: tags:
+
+<img src='../../img/copernicus_logo.png' alt='Logo EU Copernicus' align='right' width='20%'><br><br><br><br><br>
+
+<p style="text-align:right;">This project is licensed under the <a href="./LICENSE">MIT License</a> and is developed under a Copernicus contract.

50_workshops/202104_egu_short_course/LICENSE

+MIT License
+
+Copyright (c) 2021 EUMETSAT
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.