Commit 4ce7e653 authored by Julia Wagemann's avatar Julia Wagemann
Browse files

cleaning

parent ae037e43
%% Cell type:markdown id: tags:
<img src='../../../img/EU-Copernicus-EUM_3Logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='50%'></img>
<br>
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
# Module on Dust Aerosol Detection and Monitoring
%% Cell type:markdown id: tags:
This module is a self-contained training on satellite-, model- and groundbased data for dust monitoring and forecasting. The module provides introductory notebooks for different satellite-, model- and ground-based datasets and a practical assignment part, where learners are encouraged to actively complete the tasks and assignments provided.
The module can be used as a self-paced tutorial and as material for online/in-person training activities.
%% Cell type:markdown id: tags:
The course is split in four parts:
* **Part 1-3: Introduction and data discovery**
* Practical data discovery notebooks that provide an overview of key satellite, ground-based and model products for dust forecasting and monitoring
* **Part 4: Practical assignments**
* Practical case studies and exercises
* Perform an analysis with multiple datasets on a real world dust event
*Note: 'Part 4 - Practical Assignments' has `assignment` and `practicals` notebooks. The principle of those notebooks are the same. The only difference is that in a class-room / online training course, the `practical` notebooks are conducted during the course, while the learners shall work on the assignments on their own.*
%% Cell type:markdown id: tags:
## Featured data
%% Cell type:markdown id: tags:
#### Satellite data
* [Sentinel-5P TROPOMI Level 2 Ultraviolet Aerosol Index](https://sentinels.copernicus.eu/web/sentinel/data-products/-/asset_publisher/fp37fc19FN8F/content/sentinel-5-precursor-level-2-ultraviolet-aerosol-index)
* [Polar Multi-Sensor Aerosol Optical Properties (PMAp) Product](https://www.eumetsat.int/media/45738)
* [AC SAF GOME-2 Level 3 Absorbing Aerosol Index](https://acsaf.org/docs/pum/Product_User_Manual_NAR_NAP_ARS_ARP_Apr_2020.pdf)
* [Meteosat Second Generationg L1 true color and dust RGB](https://www.eumetsat.int/0-degree-service)
* [MODIS 10km Level 2 Aerosol product](https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/science-domain/aerosol/)
* [MODIS L1B true color and dust RGB](https://ladsweb.modaps.eosdis.nasa.gov/missions-and-measurements/products/MOD021KM#overview)
%% Cell type:markdown id: tags:
#### Observation data
* [AERONET - AErosol RObotic NETwork](https://aeronet.gsfc.nasa.gov/new_web/index.html)
* [European Aerosol Research Lidar Network (EARLINET) - Backscatter profiles](https://www.earlinet.org/index.php?id=earlinet_homepage)
* [European Environment Agency Air Quality Data](https://www.eea.europa.eu/data-and-maps/data/aqereporting-9)
%% Cell type:markdown id: tags:
#### Model-based data
* [WMO SDS-WAS Dust Forecast - MONARCH](https://sds-was.aemet.es/)
* [CAMS Global atmospheric composition forecasts](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-global-atmospheric-composition-forecasts?tab=overview)
* [CAMS European Air Quality Forecasts](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-europe-air-quality-forecasts?tab=overview)
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
## Module overview
#### 01 - Introduction to satellite data
* [11 - Sentinel-5P TROPOMI Ultraviolet Aerosol Index Level 2](./01_satellite-based_data/11_Sentinel-5P_TROPOMI_AI_L2_load_browse.ipynb)
* [12 - Polar Multi-Sensor Aerosol Optical Properties (PMAp) Product - Aerosol Optical Depth](./01_satellite-based_data/12_PMAp_AOD_L2_load_browse.ipynb)
* [13 - Metop-ABC GOME-2 Absorbing Aerosol Index Level 3](./01_satellite-based_data/13_Metop-ABC_GOME-2_AAI_L3_load_browse.ipynb)
* [14 - Meteosat Second Generation SEVIRI Level 1 true colour and dust RGB](./01_satellite-based_data/14_MSG_dust_product_L1_load_browse.ipynb)
* [15 - MODIS Level 2 10 km aerosol product](./01_satellite-based_data/15_MODIS_10km_aerosol_product_L2_load_browse.ipynb)
* [16 - MODIS Level 1B true color and dust RGB](./01_satellite-based_data/16_MODIS_L1B_RGB_load_browse.ipynb)
#### 02 - Introduction to ground-based observations
* [21 - AERONET - AErosol RObotic NETwork](./02_ground-based_observations/21_AERONET.ipynb)
* [22 - European Aerosol Research Lidar Network (EARLINET) - Backscatter profiles](./02_ground-based_observations/22_EARLINET.ipynb)
* [23 - European Environment Agency Air Quality Data](./02_ground-based_observations/23_EEA_Air_Quality.ipynb)
#### 03 - Introduction to model-based data
* [31 - WMO SDS-WAS - Dust Forecast - MONARCH](./03_model-based_data/31_SDS-WAS_MONARCH_dust_forecast.ipynb)
* [32 - CAMS global atmospheric composition forecast - Dust Aerosol Optical Depth](./03_model-based_data/32_CAMS_global_forecast_duaod_load_browse.ipynb)
* [33 - CAMS European Air Quality Forecasts - Dust concentration](./03_model-based_data/33_CAMS_European_air_quality_forecast_dust_concentration_load_browse.ipynb)
#### 04 - Practical assignments
* [01 - Practicals](./04_assignments/01_practicals.ipynb)
* [02 - Assignment](./04_assignments/02_assignment.ipynb)
* [03 - Practicals](./04_assignments/03_practicals.ipynb)
* [04 - Assignment](./04_assignments/04_assignment.ipynb)
* [05 - Practicals](./04_assignments/05_practicals.ipynb)
%% Cell type:markdown id: tags:
**NOTE:** Throughout the course, general functions to `load`, `re-shape`, `process` and `visualize` the datasets are defined. These functions are re-used when applicable. The [functions notebook](./functions.ipynb) gives you an overview of useful functions defined.
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## Learning outcomes
%% Cell type:markdown id: tags:
The course is designed for `beginners and medium-level users`, who have basic Python knowledge.
After the course, you should have:
* an idea about **satellite-, model-based and ground-based data products for dust monitoring and forecasting**,
* knowledge about the most useful **Python packages** to handle, process and visualise atmospheric composition data
* an idea about how the **data can help to detect and monitor dust events**
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
<img src='../../img/copernicus_logo.png' alt='Logo EU Copernicus' align='right' width='20%'><br><br><br><br>
<p style="text-align:right;">This project is licensed under the <a href="./LICENSE">MIT License</a> and is developed under a Copernicus contract.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Copyright (c) 2012-2019 Satpy developers
#
# This file is part of satpy.
#
# satpy is free software: you can redistribute it and/or modify it under the
# terms of the GNU General Public License as published by the Free Software
# Foundation, either version 3 of the License, or (at your option) any later
# version.
#
# satpy is distributed in the hope that it will be useful, but WITHOUT ANY
# WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR
# A PARTICULAR PURPOSE. See the GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License along with
# satpy. If not, see <http://www.gnu.org/licenses/>.
"""Convert human coordinates (lon and lat) to an area definition.
Here is a usage example.
python coord2area_def.py france stere 42.0 51.5 -5.5 8.0 1.5
The arguments are "name proj min_lat max_lat min_lon max_lon resolution(km)".
The command above yelds the following result.
### +proj=stere +lat_0=46.75 +lon_0=1.25 +ellps=WGS84
france:
description: france
projection:
proj: stere
ellps: WGS84
lat_0: 46.75
lon_0: 1.25
shape:
height: 703
width: 746
area_extent:
lower_left_xy: [-559750.381098, -505020.675776]
upper_right_xy: [559750.381098, 549517.351948]
The first commented line is just a sum-up. The value of "description" can be changed to any descriptive text.
Such a custom yaml configuration can be profitably saved in a local areas.yaml configuration file that won't be
overridden by future updates of SatPy package. For that purpose the local processing script may have suitable
lines as reported below.
# set PPP_CONFIG_DIR for custom composites
import os
os.environ['PPP_CONFIG_DIR'] = '/my_local_path/for_satpy_configuration'
As a further functionality this script may give a quick display of the defined area,
provided the path for the GSHHG library is supplied via the "-s" option
and the modules PyCoast, Pillow and AggDraw have been installed.
python coord2area_def.py france stere 42.0 51.5 -5.5 8.0 1.5 -s /path/for/gshhs/library
The command above would first print the seen area definition and then launch a casual representation
of the area relying on the information about borders involved.
"""
import argparse
import sys
from pyproj import Proj
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("name",
help="The name of the area.")
parser.add_argument("proj",
help="The projection to use. Use proj.4 names, like 'stere', 'merc'...")
parser.add_argument("min_lat",
help="The the latitude of the bottom of the area",
type=float)
parser.add_argument("max_lat",
help="The the latitude of the top of the area",
type=float)
parser.add_argument("min_lon",
help="The the longitude of the left of the area",
type=float)
parser.add_argument("max_lon",
help="The the longitude of the right of the area",
type=float)
parser.add_argument("resolution",
help="The resolution of the area (in km)",
type=float)
parser.add_argument("-s", "--shapes",
help="Show a preview of the area using the coastlines in this directory")
args = parser.parse_args()
name = args.name
proj = args.proj
left = args.min_lon
right = args.max_lon
up = args.min_lat
down = args.max_lat
res = args.resolution * 1000
lat_0 = (up + down) / 2
lon_0 = (right + left) / 2
p = Proj(proj=proj, lat_0=lat_0, lon_0=lon_0, ellps="WGS84")
left_ex1, up_ex1 = p(left, up)
right_ex1, up_ex2 = p(right, up)
left_ex2, down_ex1 = p(left, down)
right_ex2, down_ex2 = p(right, down)
left_ex3, dummy = p(left, lat_0)
right_ex3, dummy = p(right, lat_0)
area_extent = (min(left_ex1, left_ex2, left_ex3),
min(up_ex1, up_ex2),
max(right_ex1, right_ex2, right_ex3),
max(down_ex1, down_ex2))
xsize = int(round((area_extent[2] - area_extent[0]) / res))
ysize = int(round((area_extent[3] - area_extent[1]) / res))
proj4_string = "+" + \
" +".join(("proj=" + proj + ",lat_0=" + str(lat_0) +
",lon_0=" + str(lon_0) + ",ellps=WGS84").split(","))
print('### ' + proj4_string)
print()
print(name + ":")
print(" description: " + name)
print(" projection:")
print(" proj: " + proj)
print(" ellps: WGS84")
print(" lat_0: " + str(lat_0))
print(" lon_0: " + str(lon_0))
print(" shape:")
print(" height: " + str(ysize))
print(" width: " + str(xsize))
print(" area_extent:")
print(" lower_left_xy: [%f, %f]" % (area_extent[0], area_extent[1]))
print(" upper_right_xy: [%f, %f]" % (area_extent[2], area_extent[3]))
if args.shapes is None:
sys.exit(0)
from PIL import Image
from pycoast import ContourWriterAGG
img = Image.new('RGB', (xsize, ysize))
area_def = (proj4_string, area_extent)
cw = ContourWriterAGG(args.shapes)
cw.add_coastlines(img, (proj4_string, area_extent),
resolution='l', width=0.5)
cw.add_grid(img, area_def, (10.0, 10.0), (2.0, 2.0), write_text=False, outline='white', outline_opacity=175,
width=1.0, minor_outline='white', minor_outline_opacity=175, minor_width=0.2, minor_is_tick=False)
img.show()
\ No newline at end of file
%% Cell type:markdown id: tags:
<img src='../../../img/EU-Copernicus-EUM_3Logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='50%'></img>
<br>
%% Cell type:markdown id: tags:
# 01 - Practical exercise
%% Cell type:markdown id: tags:
### About
%% Cell type:markdown id: tags:
> You are working for ENAIRE, the air navigation authority for Spain and western Africa. You know that the Canary Islands are prone to Saharan dust events and for this reason, ENAIRE monitors dust on a daily basis. You are the operations analyst for the week of 21-27 February 2020 and responsible for issuing alerts, and if necessary, to mandate required safety measures.
>
> On 21 February 2020, you are in-charge of analysing the dust forecast and to monitor potential dust events for the coming days. With your new knowledge on aerosol and dust data, you should be able to do this.
%% Cell type:markdown id: tags:
### Tasks
%% Cell type:markdown id: tags:
#### 1. Brainstorm
* What dust forecasts do you know about?
* How do they differ from each other?
* What satellite data do you know about that can be used for dust nowcasting?
* Which variables can be used to monitor and forecast dust?
#### 2. Download and animate the MONARCH dust forecast
* Download the dust forecast from the MONARCH model for 21 February 2020 and animate the forecast
* **Hint**
* [WMO SDS-WAS dust forecast - Example notebook](../../dust_workshop_part1/03_model-based_data/31_SDS-WAS_MONARCH_dust_forecast.ipynb)
* [Data access](https://sds-was.aemet.es/forecast-products/dust-forecasts/files-download) (Username: `sdswas.namee.rc@gmail.com`, Password: `BarcelonaDustRC`)
#### 3. Download the MSG SEVIRI Level 1.5 data and visualize the Dust RGB composite
* Based on the dust forecast for the next days - which day and hour would you choose for getting a near real-time monitoring of dust from the MSG SEVIRI instrument?
* **Hint**
* [MSG SEVIRI Level 1.5 - Example notebook](../../dust_workshop_part1/01_satellite-based_data/14_MSG_dust_product_L1_load_browse.ipynb)
* [Data access](https://data.eumetsat.int/data/map/EO:EUM:DAT:MSG:HRSEVIRI)
#### 4. Interpret the results
* Describe the dust forecast event in comparison with the near real-time monitoring from the satellite.
* What decision as ENAIRE operations analyst do you take? Would you issue an alert? Would you implement some safety measures?
%% Cell type:markdown id: tags:
### Module outline
* [1 - Dust forecasts from the MONARCH model](#sdswas_forecast)
* [2 - Meteosat Second Generation SEVIRI Level 1.5 dust RGB](#msg_seviri)
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
##### Load required libraries
%% Cell type:code id: tags:
``` python
%matplotlib inline
import os
import xarray as xr
import numpy as np
import netCDF4 as nc
import pandas as pd
from IPython.display import HTML
import matplotlib.pyplot as pl
import matplotlib.colors
from matplotlib.cm import get_cmap
from matplotlib import animation
import cartopy.crs as ccrs
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER
import cartopy.feature as cfeature
from matplotlib.axes import Axes
from cartopy.mpl.geoaxes import GeoAxes
GeoAxes._pcolormesh_patched = Axes.pcolormesh
from satpy.scene import Scene
from satpy.composites import GenericCompositor
from satpy.writers import to_image
from satpy.resample import get_area_def
from satpy import available_readers
import glob
import pyresample
import warnings
warnings.simplefilter(action = "ignore", category = RuntimeWarning)
```
%% Cell type:markdown id: tags:
##### Load helper functions
%% Cell type:code id: tags:
``` python
%run ../functions.ipynb
```
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## <a id='sdswas_forecast'>1. Dust forecasts from the MONARCH model
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## <a id='msg_seviri'></a>2. Meteosat Second Generation SEVIRI Level 1.5 dust RGB
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
<img src='../img/copernicus_logo.png' alt='Logo EU Copernicus' align='left' width='20%'><br><br><br><br>
<p style="text-align:right;">This project is licensed under the <a href="./LICENSE">MIT License</a> and is developed under a Copernicus contract.
%% Cell type:markdown id: tags:
<img src='../../../img/EU-Copernicus-EUM_3Logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='50%'></img>
<br>
%% Cell type:markdown id: tags:
# 02 - Assignment
%% Cell type:markdown id: tags:
### About
%% Cell type:markdown id: tags:
> Let us have a closer look at the forecast data from both models for one observation station in Tenerife (Canary Islands). Let us plot the time-series of the CAMS and MONARCH forecasts together in one plot.
%% Cell type:markdown id: tags:
### Tasks
%% Cell type:markdown id: tags:
#### 1. Download and animate the CAMS global atmopsheric composition forecast for 21 February 2020
* Download the CAMS global atmospheric composition forecast for 21 February 2020, with the following specifications:
> Variable on single levels: `Dust aerosol optical depth at 550 nm` <br>
> Date (Start and end): `2020-02-21` <br>
> Time: `12:00` <br>
> Leadtime hour: every three hours from `0 to 90` <br>
> Type: `Forecast` <br>
> Restricted area: `N: 67, W: -30, E: 71, S: -3` <br>
> Format: `Zipped netCDF` <br>
* **Hint**
* [CAMS global atmospheric composition forecasts - Example notebook](../03_model-based_data/32_CAMS_global_forecast_duaod_load_browse.ipynb)
* [Data access](https://ads.atmosphere.copernicus.eu/cdsapp#!/dataset/cams-global-atmospheric-composition-forecasts)
#### 2. Look up the coordinates of the AERONET station *Santa Cruz, Tenerife*
* **Hint**
* You can see an overview of all available AERONET Site Names [here](https://aeronet.gsfc.nasa.gov/cgi-bin/draw_map_display_aod_v3?long1=-180&long2=180&lat1=-90&lat2=90&multiplier=2&what_map=4&nachal=1&formatter=0&level=3&place_code=10&place_limit=0)
#### 3. Select the time-series for CAMS global atmospheric composition forecasts for Santa Cruz, Tenerife
* **Hint**
* With the xarray function `sel()` and keyword argument `method='nearest'` you can select data based on coordinate information
* We also recommend you to transform your xarray.DataArray into a pandas.DataFrame with the function `to_dataframe()` and save it as `csv` with the function `to_csv()`
#### 4. Load the MONARCH dust forecasts and select time-series for Santa Cruz, Tenerife
* **Hint**
* With the xarray function `sel()` and keyword argument `method='nearest'` you can select data based on coordinate information
* We also recommend you to transform your xarray.DataArray into a pandas.DataFrame with the function `to_dataframe()` and save it as `csv` with the function `to_csv()`
#### 5. Visualize both time-series of CAMS and MONARCH forecasts together in one plot
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
### Module outline
* [1 - Load and animate CAMS global atmospheric composition forecasts](#cams_forecast)
* [2 - Select latitude / longitude values for AERONET station *Santa Cruz, Tenerife*](#select_lat_lon_aeronet)
* [3 - Select time-series for CAMS global atmospheric composition forecasts for Santa Cruz, Tenerife](#ts_cams)
* [4 - Load the MONARCH dust forecasts and select time-series for Santa Cruz, Tenerife](#sdswas_forecast_load)
* [5 - Visualize time-series of CAMS and MONARCH forecasts in one plot](#visualize_ts)
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
##### Load required libraries
%% Cell type:code id: tags:
``` python
%matplotlib inline
import os
import xarray as xr
import numpy as np
import netCDF4 as nc
import pandas as pd
from IPython.display import HTML
import matplotlib.pyplot as pl
import matplotlib.colors
from matplotlib.cm import get_cmap
from matplotlib import animation
import cartopy.crs as ccrs
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER
import cartopy.feature as cfeature
from matplotlib.axes import Axes
from cartopy.mpl.geoaxes import GeoAxes
GeoAxes._pcolormesh_patched = Axes.pcolormesh
import warnings
warnings.simplefilter(action = "ignore", category = RuntimeWarning)
```
%% Cell type:markdown id: tags:
##### Load helper functions
%% Cell type:code id: tags:
``` python
%run ../functions.ipynb
```
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
### <a id='cams_forecast'></a>1. Load and animate CAMS global atmospheric composition forecasts
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
### <a id='select_lat_lon_aeronet'></a>2. Select latitude / longitude values for Santa Cruz, Tenerife
%% Cell type:code id: tags:
``` python
```
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
### <a id='ts_cams'></a>3. Select time-series for CAMS global atmospheric composition forecasts