Commit 9c3fdbf6 authored by Julia Wagemann's avatar Julia Wagemann
Browse files

LTPy v0.6 release

parent a05b1af2
%% Cell type:markdown id: tags:
<img src='./img/EU-Copernicus-EUM_3Logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='50%'></img>
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
# LTPy - Learning Tool for Python on Atmospheric Composition Data
%% Cell type:markdown id: tags:
<br>
%% Cell type:markdown id: tags:
**LTPy - Learning tool for Python on Atmospheric Composition Data** is a Python-based training course on Atmospheric Composition Data. The training course covers [10 - DATA ACCESS](#data_access), [20 - DATA DISCOVERY](#data_discovery), [30 - CASE STUDIES](#case_studies) and [40 - EXERCISES](#exercises) of satellite- and model-based data on Atmospheric Composition.
The course is based on [Jupyter notebooks](https://jupyter.org/), which allow for a high-level of interactive learning, as code, text description and visualisation is combined in one place. If you have not worked with `Jupyter Notebooks` before, you can look at the module [01 - Python and Project Jupyter 101](./01_Python_and_Jupyter_101.ipynb) to get a short introduction to Jupyter notebooks and their benefits.
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## Data on Atmospheric Composition
%% Cell type:markdown id: tags:
This course features the following **satellite** data:
* `Metop-A/B GOME-2 Level 2` data
* `Metop-A/B GOME-2 Level 3` reprocessed and regridded data
* `Metop-A/B/C GOME-2 Level 2` data
* `Metop-A/B/C GOME-2 Level 3` reprocessed and regridded data
* `Polar Multi-Sensor Aerosol Optical Properties (PMAp) Level 2` data
* `Metop-A/B IASI Level 2` data
* `Metop-A/B/C IASI Level 2` data
* `Copernicus Sentinel-5P TROPOMI Level 2` data
* `Copernicus Sentinel-3 OLCI Level 1B` data
* `Copernicus Sentinel-3 SLSTR NRT FRP Level 2` data
* `Copernicus Sentinel-3 SLSTR NRT AOD Level 2` data
And the following **model-based** data:
* `Copernicus Atmosphere Monitoring Service (CAMS) Global Reanalysis (EAC4)` data
* `Copernicus Atmosphere Monitoring Service (CAMS) Global Fire Assimilation System (GFAS)` data
* `Coperncus Emergency Management Service (CEMS) Global ECMWF Fire Forecast (GEFF)` data
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## Course material
%% Cell type:markdown id: tags:
The course follows a modular approach and offers modules on:
- [10 - DATA ACCESS](#data_access)
- [20 - DATA DISCOVERY](#data_discovery)
- [30 - CASE STUDIES](#case_studies)
- [40 - EXERCISES](#exercises)
<br>
<div class="alert alert-block alert-info">
<b><a id='data_access'></a>10 - DATA ACCESS</b>
</div>
* [11 - Atmospheric Composition data overview and acccess](./10_data_access/11_ac_data_access_overview.ipynb)
* [12 - WEkEO Harmonized Data Access API](./10_data_access/12_WEkEO_harmonized_data_access_api.ipynb)
<br>
<div class="alert alert-block alert-success">
<b><a id='data_discovery'></a>20 - DATA DISCOVERY</b>
</div>
#### *Metop-A/B/C GOME-2 Level 2 and Level 3 data*
* [211 - Metop-A GOME-2 - Tropospheric NO<sub>2</sub> - Level 2 - Load and browse](./20_data_discovery/211_Metop-A_GOME-2_NO2Tropo_L2_load_browse.ipynb)
* [212 - Metop-A/B GOME-2 - Tropospheric NO<sub>2</sub> - Level 2 - Pre-process](./20_data_discovery/212_Metop-AB_GOME-2_NO2Tropo_L2_preprocess.ipynb)
* [213 - Metop-A/B GOME-2 - Tropospheric NO<sub>2</sub> - Level 3 - Load and browse](./20_data_discovery/213_Metop-AB_GOME-2_NO2Tropo_L3_load_browse.ipynb)
* [214 - Metop-A/B/C GOME-2 - Absorbing Aerosol Index - Level 3 - Load and browse](./20_data_discovery/214_Metop-ABC_GOME-2_AAI_L3_load_browse.ipynb)
#### *Polar Multi-Sensor Aerosol Optical Properties (PMAp) Level 2 data*
* [221 - Polar Multi-Sensor Aerosol Optical Properties (PMAp) - Aerosol Optical Depth - Level 2 - Load and browse](./20_data_discovery/221_PMAp_AOD_L2_load_browse.ipynb)
#### *Metop-A/B IASI Level 2 data*
* [231 - Metop-A/B IASI - Ammonia (NH<sub>3</sub>) - Level 2 - Load and browse](./20_data_discovery/231_Metop-AB_IASI_NH3_L2_load_browse.ipynb)
#### *Sentinel-5P TROPOMI Level 2 data*
* [241 - Sentinel-5P TROPOMI - Carbon Monoxide - Level 2 - Load and browse](./20_data_discovery/241_Sentinel-5P_TROPOMI_CO_L2_load_browse.ipynb)
#### *Sentinel-3 data*
* [251 - Sentinel-3 OLCI - Radiances - Level 1 - Load and browse](./20_data_discovery/251_Sentinel-3_OLCI_radiance_L1_load_browse.ipynb)
* [252 - Sentinel-3 SLSTR NRT - Fire Radiative Power (FRP) - Level 2 - Load and browse](./20_data_discovery/252_Sentinel-3_SLSTR_NRT_FRP_L2_load_browse.ipynb)
* [253 - Sentinel-3 SLSTR NRT - Aerosol Optical Depth (AOD) - Level 2 - Load and browse](./20_data_discovery/253_Sentinel-3_SLSTR_NRT_AOD_L2_load_browse.ipynb)
#### *Copernicus Atmosphere Monitoring Service (CAMS) data*
* [261 - CAMS Global reanalysis (EAC4) - Organic Matter Aerosol Optical Depth - Load and browse](./20_data_discovery/261_CAMS_EAC4_OMAOD_load_browse.ipynb)
* [262 - CAMS Global Fire Assimilation System (GFAS) - Fire Radiative Power - Load and browse](./20_data_discovery/262_CAMS_GFAS_FRPFIRE_load_browse.ipynb)
#### *Copernicus Emergency Management Service (CEMS) data*
* [271 - CEMS Global ECMWF Fire Forecast - Fire Weather Index - Load and browse](./20_data_discovery/271_CEMS_GEFF_FWI_load_browse.ipynb)
* [272 - CEMS Global ECMWF Fire Forecast - Fire Weather Index - Harmonized Danger Classes](./20_data_discovery/272_CEMS_GEFF_FWI_harmonized_danger_classes.ipynb)
* [273 - CEMS Global ECMWF Fire Forecast - Fire Weather Index - Custom Danger Classes](./20_data_discovery/273_CEMS_GEFF_FWI_custom_danger_classes.ipynb)
<br>
<div class="alert alert-block alert-warning">
<b><a id='case_studies'></a>30 - CASE STUDIES</b>
</div>
#### *Fires*
* [311 - Amazon fires 2019](./30_case_studies/311_fire_amazon_2019.ipynb)
* [312 - Siberian fires 2019](./30_case_studies/312_fire_siberia_2019.ipynb)
* [313 - Californian fires 2020](./30_case_studies/313_fire_california_2020.ipynb)
* [314 - Chernobly fires 2020 - Sentinel-3 SLSTR NRT - Fire Radiative Power](./30_case_studies/314_fire_chernobyl_2020_Sentinel-3_SLSTR_NRT_FRP_L2.ipynb)
* [315 - Californian fires 2020 - Sentinel-3 SLSTR NRT - Fire Radiative Power](./30_case_studies/315_fire_california_2020_Sentinel-3_SLSTR_NRT_FRP_L2.ipynb)
* [316 - Californian fires 2020 - Sentinel-3 SLSTR NRT - Aerosol Optical Depth](./30_case_studies/316_fire_california_2020_Sentinel-3_SLSTR_NRT_AOD_L2.ipynb)
#### *Air pollution*
* [321 - Map and time-series analysis - Metop-A/B GOME-2 - Tropospheric NO<sub>2</sub>](./30_case_studies/321_air_pollution_map_time-series_Metop-AB_GOME-2_NO2Tropo_L3.ipynb)
* [322 - Produce gridded dataset - Metop-A/B GOME-2 - Tropospheric NO<sub>2</sub>](./30_case_studies/322_air_pollution_produce_gridded_Metop-AB_GOME-2_NO2Tropo_L2.ipynb)
* [323 - Create an anomaly map - Europe - Metop-A/B GOME-2 - Tropospheric NO<sub>2</sub>](./30_case_studies/323_air_pollution_map_europe_2020_Metop-AB_GOME-2_NO2Tropo_L2.ipynb)
* [324 - Time-series analysis - Europe - Metop-A/B GOME-2 - Tropospheric NO<sub>2</sub>](./30_case_studies/324_air_pollution_time-series_europe_2020_Metop-AB_GOME-2_NO2Tropo_L2.ipynb)
* [325 - Create an anomaly map - Europe - Sentinel-5P TROPOMI - Tropospheric NO<sub>2</sub>](./30_case_studies/325_air_pollution_map_europe_2020_Sentinel-5P_TROPOMI_NO2Tropo_L2.ipynb)
* [326 - Time-series analysis - Europe - Sentinel-5P TROPOMI - Tropospheric NO<sub>2</sub>](./30_case_studies/326_air_pollution_time-series_europe_2020_Sentinel-5P_TROPOMI_NO2Tropo_L2.ipynb)
#### *Stratospheric Ozone*
* [331 - Antarctic ozone hole - Summer 2019](./30_case_studies/331_stratospheric_ozone_2019.ipynb)
* [332 - Antarctic ozone hole - CAMS animation - Summer 2019](./30_case_studies/332_stratospheric_ozone_animation_2019.ipynb)
* [331 - Antarctic ozone hole 2019 - Multi-data](./30_case_studies/331_stratospheric_ozone_Antarctic_2019.ipynb)
* [332 - Antarctic ozone hole 2019 - CAMS animation](./30_case_studies/332_stratospheric_ozone_Antarctic_2019_CAMS_EAC4_animation.ipynb)
* [333 - Antarctic ozone hole 2020 - Metop-A/B/C GOME-2 Level 2](./30_case_studies/333_stratospheric_ozone_Antarctic_2020_Metop-ABC_GOME-2_O3_L2.ipynb)
* [334 - Arctic ozone hole 2020 - Metop-A/B/C IASI Level 2 ](./30_case_studies/334_stratospheric_ozone_Arctic_2020_Metop-ABC_IASI_O3_L2.ipynb)
<br>
<div class="alert alert-block alert-danger">
<b><a id='exercises'></a>40 - EXERCISES</b>
</div>
#### *Sentinel-5P TROPOMI*
* [411 - Sentinel-5P TROPOMI - Carbon Monoxide - Level 2](./40_exercises/411_Sentinel-5P_TROPOMI_CO_L2_exercise.ipynb)
#### *Sentinel-3*
* [421 - Sentinel-3 OLCI - Radiances - Level 1](./40_exercises/421_Sentinel-3_OLCI_radiance_L1_exercise.ipynb)
* [422 - Sentinel-3 SLSTR NRT - Fire Radiative Power](./40_exercises/422_Sentinel-3_SLSTR_NRT_FRP_L2_exercise.ipynb)
* [423 - Sentinel-3 SLSTR NRT - Aerosol Optical Depth](./40_exercises/423_Sentinel-3_SLSTR_NRT_AOD_L2_exercise.ipynb)
#### *Copernicus Atmosphere Monitoring Service*
* [431 - CAMS Global Reanalysis (EAC4) - Total Column Carbon Monoxide](./40_exercises/431_CAMS_EAC4_tcco_exercise.ipynb)
#### *Metop-A/B/C GOME-2 and IASI*
* [441 - Metop-A/B/C GOME-2 - Ozone](./40_exercises/441_Metop-ABC_GOME-2_O3_L2_exercise.ipynb)
* [442 - Metop-A/B/C IASI - Ozone](./40_exercises/442_Metop-ABC_IASI_O3_L2_exercise.ipynb)
<br>
**NOTE:** Throughout the course, general functions to `load`, `re-shape`, `process` and `visualize` the datasets are defined. These functions are re-used when applicable. The [functions notebook](./functions.ipynb) gives you an overview of all the functions defined and used for the course.
If a notebook makes use of these functions, they are loaded as **helper functions** at the beginning of the notebook. With `?function_name`, you can load the function's docstring to see what it does and which keyword arguments the function requires.
See the example to load the docstring of the function [visualize_pcolormesh](./functions.ipynb#visualize_pcolormesh):
%% Cell type:code id: tags:
``` python
%run ./functions.ipynb
```
%% Cell type:code id: tags:
``` python
?visualize_pcolormesh
```
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## Learning outcomes
%% Cell type:markdown id: tags:
The course is designed for `medium-level users`, who have basic Python knowledge and understanding of Atmospheric composition data.
After the course, you should have:
* an idea about the **different datasets on Atmospheric Composition data**,
* knowledge about the most useful **Python packages** to handle, process and visualise large volumes of Earth Observation data
* an idea about different **data application areas**
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## Access to the `LTPy JupyterHub`
%% Cell type:markdown id: tags:
The course material is made available on a JupyterHub instance, a pre-defined environment that give learners direct access to the data and Python packages required for following the course.
The `JupyterHub` can be accessed as follows:
%% Cell type:markdown id: tags:
* Web address: [https://ltpy.adamplatform.eu](https://ltpy.adamplatform.eu)
* Create an account: [https://login.ltpy.adamplatform.eu/](https://login.ltpy.adamplatform.eu/)
* Log into the `JupyterHub` with your account created.
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
## Reproduce LTPy on Atmospheric Compostion data locally
%% Cell type:markdown id: tags:
In case you wish to reproduce the course modules on your local setup, the following Python version and Python packages will be required:
* Python version: **Python3.8**
* Python packages: see [requirements.txt](./requirements.txt)
Python packages can be installed as follows: `pip install -r requirements.txt`.
%% Cell type:markdown id: tags:
The `eodata` folder with all the data required for the training course can be accessed and downloaded from [https://sftp.eumetsat.int](https://sftp.eumetsat.int/login). Find the user name and password in order to be able to login [here](https://gitlab.eumetsat.int/eumetlab/atmosphere/atmosphere/-/blob/master/sftp_login.txt).
%% Cell type:markdown id: tags:
<hr>
%% Cell type:markdown id: tags:
<p><img src='./img/copernicus_logo.png' align='left' alt='Logo EU Copernicus' width='25%'></img></p>
<br clear=left>
<p style="text-align:left;">This project is licensed under the <a href="./LICENSE">MIT License</a> <span style="float:right;"><a href="https://gitlab.eumetsat.int/eumetlab/atmosphere/atmosphere">View on GitLab</a> | <a href="https://training.eumetsat.int/">EUMETSAT Training</a>
......
%% Cell type:markdown id: tags:
 
<img src='../img/EU-Copernicus-EUM_3Logos.png' alt='Logo EU Copernicus EUMETSAT' align='right' width='50%'></img>
 
%% Cell type:markdown id: tags:
 
<br>
 
%% Cell type:markdown id: tags:
 
<a href="../00_index.ipynb"><< Index</a><br>
<a href="./325_air_pollution_map_europe_2020_Sentinel-5P_TROPOMI_NO2Tropo_L2.ipynb"><< 325 - Sentinel-5P TROPOMI anomaly map</a><span style="float:right;"><a href="./331_stratospheric_ozone_2019.ipynb
">331 - Stratospheric Ozone 2019 >></a></span>
<a href="./325_air_pollution_map_europe_2020_Sentinel-5P_TROPOMI_NO2Tropo_L2.ipynb"><< 325 - Sentinel-5P TROPOMI anomaly map</a><span style="float:right;"><a href="./331_stratospheric_ozone_Antarctic_2019.ipynb
">331 - Stratospheric Ozone 2019 - Antarctic >></a></span>
 
%% Cell type:markdown id: tags:
 
<div class="alert alert-block alert-warning">
<b>30 - CASE STUDIES - AIR POLLUTION</b></div>
 
%% Cell type:markdown id: tags:
 
<div class="alert alert-block alert-warning">
 
<b>PREREQUISITES </b>
 
The following **20 - DATA DISCOVERY** modules are prerequisites:
 
- [241 - Sentinel-5P TROPOMI - Carbon Monoxide - Level 2 - Load and browse](../20_data_discovery/241_Sentinel-5P_TROPOMI_CO_L2_load_browse.ipynb)
- [325 - Sentinel-5P TROPOMI anomaly map](../30_case_studies/325_air_pollution_map_europe_2020_Sentinel-5P_TROPOMI_NO2Tropo_L2.ipynb)
 
 
It is recommended to go through these modules before you start with this module.
</div>
 
%% Cell type:markdown id: tags:
 
<hr>
 
%% Cell type:markdown id: tags:
 
# 3.2.6 Sentinel-5P TROPOMI Tropospheric NO<sub>2</sub> time-series analysis
 
%% Cell type:markdown id: tags:
 
This workflows will guide you through the different steps of how you can generate daily average time-series data of tropospheric NO<sub>2</sub> based on Copernius Sentinel-5P TROPOMI Level 2 data.
 
The aim is to have two time-series of `tropospheric NO2` for two periods and three different regions:
- **Time periods**:
- December 2018 to 15 May 2019, and
- December 2019 to 15 May 2020
- **Regions**:
- Europe
- Povalley (Italy)
- Hubei (China)
 
### Data used
The workflow and analyses are based on the following data:
- Daily Sentinel-5P Level 2 files retrieved from the [Open Data Registry on AWS](https://registry.opendata.aws/sentinel5p/)
 
### Workflow outline
The workflow has the following outline:
- [Define a list with dictionaries holding bounding boxes for different regions](#region_list)
- [*Optional: Loop through the files of daily tropopsheric NO<sub>2</sub> and store the average values for each region as dictionary*](#optional_loop)
- [Open JSON file with daily average tropospheric NO<sub>2</sub> values for specific regions](#open_json)
- [Subset the time-series data and create 7-day and 15-day moving averages](#subset)
- [Visualize the Sentinel-5P TROPOMI NO<sub>2</sub> time-series data](#visualize)
 
%% Cell type:markdown id: tags:
 
#### Load required libraries
 
%% Cell type:code id: tags:
 
``` python
import glob
import os
import xarray as xr
import numpy as np
import pandas as pd
from datetime import datetime
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import json
```
 
%% Cell type:markdown id: tags:
 
#### Load helper functions
 
%% Cell type:code id: tags:
 
``` python
%run ../functions.ipynb
```
 
%% Cell type:markdown id: tags:
 
#### Function to help read and load .json files
 
%% Cell type:code id: tags:
 
``` python
def myconverter(o):
if isinstance(o, datetime):
return o.__str__()
```
 
%% Cell type:markdown id: tags:
 
<hr>
 
%% Cell type:markdown id: tags:
 
## <a id='region_list'></a>Define a list with dictionaries holding bounding boxes for different regions
 
%% Cell type:markdown id: tags:
 
You can define several regions as a dictionary, e.g. the bounding box information for Europe, the Po valley in Italy or the Hubei region in China. The dictionaries are stored in a list.
 
%% Cell type:code id: tags:
 
``` python
eur_extent={'name': 'eur_extent',
'lonmin':-10.0,
'lonmax':35,
'latmin':35.0,
'latmax':60.0}
 
# Italy (Po Valley, Milano)
povalley_extent={'name': 'povalley_extent',
'lonmin':7.0,
'lonmax':13.0,
'latmin':44.0,
'latmax':47.0}
 
 
# China (Hubei, Wuhan)
hubei_extent={'name': 'hubei_extent',
'lonmin':108.3,
'lonmax':116.1,
'latmin':29.1,
'latmax':33.3}
 
region_list =[eur_extent,
povalley_extent,
hubei_extent]
```
 
%% Cell type:markdown id: tags:
 
## <a id='optional_loop'></a>*Optional: Loop through the files of daily tropopsheric NO<sub>2</sub> and store the average values for each region as dictionary*
 
%% Cell type:markdown id: tags:
 
This part of the workflow is optional. It opens daily Sentinel-5P Level 2 `Cloud-Optimised GeoTiff` files, creates the average NO<sub>2</sub> value of a geographical subset for a specified region and stores the daily mean value and datetime stamp in two lists. The two lists combined as a dictionary are returned by the function.
 
The downloaded `COG` files are in the folder `/eodata/sentinel5p/no2/`. The time-series data have already been computed and stored as a `JSON` file in the directory `/eodata/sentinel5p/no2/json/`. Thus, you can skip this step and continue directly with the next [workflow step](#open_json).
 
%% Cell type:markdown id: tags:
 
#### List all files ending with `.tif` and store them in a list of files
 
%% Cell type:code id: tags:
 
``` python
fileList = glob.glob('../eodata/sentinel5p/no2/*.tif')
fileList_sort = sorted(fileList)
```
 
%% Cell type:markdown id: tags:
 
#### Function that loops over each region and each file and saves the daily average tropospheric NO<sub>2</sub> values as dictionary
 
%% Cell type:markdown id: tags:
 
For each region in the region list, the loop executes the following steps:
- Loop over each file in the file list
- Open the file as `xarray.DataArray()`
- Rename `x` and `y` to `longitude` and `latitude`
- Create a geographical subset with the function [generate_geographical_subset](../functions.ipynb#generate_geographical_subset) based on the region's bounding box
- Flag negative values
- Compute the spatial mean of the geographical subset
- Append the mean value to a list of mean values
- Do the same for the date time instance
 
For each region, the dictionary holds two lists, with datetime information and the average tropospheric NO<sub>2</sub> values for the respective day and region.
 
%% Cell type:code id: tags:
 
``` python
region_dict={}
# Loop through each region in the region list
for region in region_list:
print(region)
# Initiate an empty list for date and mean values
date_list=[]
mean_list=[]
# Loop through each file in the file list
for i in range(len(fileList)):
print(i)
# Open the tif file as xarray.DataArray with the function 'open_rasterio()'
tmp = xr.open_rasterio(fileList[i])
# Rename x and y to longitude and latitude in order to use the function 'generate_geographical_subset'
tmp= tmp.rename({'x': 'longitude', 'y':'latitude'})
 
# Create a geographical subset for the specific region
tmp_sub = generate_geographical_subset(xarray=tmp,
latmin=region['latmin'],
latmax=region['latmax'],
lonmin=region['lonmin'],
lonmax=region['lonmax'])
 
# Flag negative values
tmp_sub_flag = tmp_sub.where(tmp_sub>0,np.nan)
 
# Create the daily average tropospheric NO2 value for the specific region
tmp_sub_mean = tmp_sub_flag.mean()
 
# Add the mean value to the list
mean_list.append((tmp_sub_mean.values).item())
 
# Retrieve the date from the file name
date = (fileList[i].split('_'))[8]
# Create a datetime object and append it to the list of dates
date_dt = datetime.strptime(date, '%Y%m%d')
date_list.append(date_dt)
 
# Combine both lists in a dictionary
region_dict[region['name']] = {'date':date_list,
'mean':mean_list}
```
 
%% Output
 
{'name': 'eur_extent', 'lonmin': -10.0, 'lonmax': 35, 'latmin': 35.0, 'latmax': 60.0}
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
{'name': 'povalley_extent', 'lonmin': 7.0, 'lonmax': 13.0, 'latmin': 44.0, 'latmax': 47.0}
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
{'name': 'hubei_extent', 'lonmin': 108.3, 'lonmax': 116.1, 'latmin': 29.1, 'latmax': 33.3}
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
 
%% Cell type:markdown id: tags:
 
#### Save time-series dictionary as JSON file
 
%% Cell type:markdown id: tags:
 
Save the dictionary with daily average tropospheric NO<sub>2</sub> values for each region as `JSON` file. You can use the function `json.dump()` to store a dictionary as `JSON`.
 
%% Cell type:code id: tags:
 
``` python
with open('../region_dict.json', 'w') as f:
json.dump(region_dict, f, default=myconverter)
```
 
%% Cell type:markdown id: tags:
 
<br>
 
%% Cell type:markdown id: tags:
 
## <a id='open_json'></a>Open `JSON` file with daily average tropospheric NO<sub>2</sub> values for specific regions
 
%% Cell type:markdown id: tags:
 
Load the `JSON` file that contains a dictionary with three regions as keys and each key has two lists, one with daily `datetime` information and one with average tropospheric NO<sub>2</sub> values for each day for the specific region. You can load a `JSON` file with the function `json.load()`.
 
The `JSON` file is in the directory `/eodata/sentinel5p/no2/json/`.
 
%% Cell type:code id: tags:
 
``` python
with open("../eodata/sentinel5p/no2/json/region_dict.json",encoding='utf-8', errors='ignore') as json_data:
data = json.load(json_data, strict=False)
 
```
 
%% Cell type:markdown id: tags:
 
## <a id='subset'></a>Subset the time-series data and create 7-day and 15-day moving averages
 
%% Cell type:markdown id: tags:
 
For each region in the region list, you do the following:
- Load the dictionry with the region key as `pandas.DataFrame`
- Sort the `pandas.DataFrame` by `date`
- Create 7-day and 15-day moving averages
- Set the `date` column as index
- Subset the time-series for the years 2019 and 2020 with the pandas function `df_subset`
- Reformat the `datetime` index
- Merge the two time-series
- Store the values of the three aggregation values (`mean`, `7day_mean` and `15day_mean`) as keys in the dictionary
 
%% Cell type:code id: tags:
 
``` python
no2_1920_dict = {}
for region in region_list:
print(region)
# Open time-series information as pandas.DataFrame and sort the values based on date
tmp_df = pd.DataFrame(data[region['name']])
tmp_df = tmp_df.sort_values(by='date')
 
# Create 7-day and 15-day rolling windows average values
tmp_df['7day_mean'] = tmp_df.iloc[:,1].rolling(window=7, min_periods=3, center=False).mean()
tmp_df['15day_mean'] = tmp_df.iloc[:,1].rolling(window=15, min_periods=3, center=False).mean()
 
# Take the data columns and set it as index of the pandas.DataFrame
tmp_df = tmp_df.set_index('date')
 
#Create two subsets for 2019 and 2020
tmp_df_2019, tmp_df_2020 = df_subset(tmp_df, '2018-11-30', '2019-05-15', '2019-11-30', '2020-05-15')
 
# Reformat the timestamp of the datatime index
tmp_df_2019.index = pd.DatetimeIndex(tmp_df_2019.index).strftime('%m-%d')
tmp_df_2020.index = pd.DatetimeIndex(tmp_df_2020.index).strftime('%m-%d')
 
#Merge time-series subsets together
tmp_df_merged = tmp_df_2019.merge(tmp_df_2020, how='left', left_index=True, right_index=True)
print(region)
# Create a dictionary of the pandas.DataFrame
no2_1920_dict[region['name']] = {
'mean': {
'date': tmp_df_merged.index,
'2019': tmp_df_merged['mean_x'].to_list(),
'2020': tmp_df_merged['mean_y'].to_list()},
'7day_mean': {
'date': tmp_df_merged.index,
'2019': tmp_df_merged['7day_mean_x'].to_list(),
'2020': tmp_df_merged['7day_mean_y'].to_list()},
'15day_mean': {
'date': tmp_df_merged.index,
'2019': tmp_df_merged['15day_mean_x'].to_list(),
'2020': tmp_df_merged['15day_mean_y'].to_list()}
}
```
 
%% Output
 
{'name': 'eur_extent', 'lonmin': -10.0, 'lonmax': 35, 'latmin': 35.0, 'latmax': 60.0}
{'name': 'eur_extent', 'lonmin': -10.0, 'lonmax': 35, 'latmin': 35.0, 'latmax': 60.0}
{'name': 'povalley_extent', 'lonmin': 7.0, 'lonmax': 13.0, 'latmin': 44.0, 'latmax': 47.0}
{'name': 'povalley_extent', 'lonmin': 7.0, 'lonmax': 13.0, 'latmin': 44.0, 'latmax': 47.0}
{'name': 'hubei_extent', 'lonmin': 108.3, 'lonmax': 116.1, 'latmin': 29.1, 'latmax': 33.3}
{'name': 'hubei_extent', 'lonmin': 108.3, 'lonmax': 116.1, 'latmin': 29.1, 'latmax': 33.3}
 
%% Cell type:markdown id: tags:
 
#### Optional: store the dictionary as `JSON` file
 
%% Cell type:markdown id: tags:
 
Save the dictionary of the temporal subset for values as `JSON`. You can use again the function `json.dump()` to store the content of a dictionary as `JSON`. This step is optional, as the `JSON` has already been stored and is available in the directory `/eodata/sentinel5p/no2/json/`.
 
%% Cell type:code id: tags:
 
``` python
with open('./s5p_ts_2019_2020_till_may.json', 'w') as f:
json.dump(no2_1920_dict, f, default=myconverter)
```
 
%% Cell type:markdown id: tags:
 
<br>
 
%% Cell type:markdown id: tags:
 
## <a id='visualize'></a>Visualize the Sentinel-5p TROPOMI NO<sub>2</sub> time-series data
 
%% Cell type:markdown id: tags:
 
The last step is to visualize the time-series for 2019 and 2020 together in one plot. This helps to compare the difference of tropopsheric NO<sub>2</sub> between both years.
 
%% Cell type:markdown id: tags:
 
For each region, plot the two time-series data (2019 and 2020) of daily tropospheric NO<sub>2</sub> data for six months (December to mid May).
 
You can loop over the different regions in the dictionary, select the list of interest and load it as `pandas.DataFrame`. The function `plot.line()` allows to plot two different columns based on the same datetime index.
 
The grey line corresponds to daily tropospheric NO<sub>2</sub> data in 2019 and the darkred line to values of 2020 accordingly.
 
%% Cell type:markdown id: tags:
 
#### Define conversion factor for Sentinel-5P data
 
%% Cell type:markdown id: tags:
 
Let us define again the conversion factor for Sentinel-5P data to convert the NO<sub>2</sub> from `mol per m`<sup>`2`</sup> to `molecules per cm`<sup>`2`</sup> and apply it to the data to be plotted.
 
%% Cell type:code id: tags:
 
``` python
conversion_factor=6.02214*1e+19
```
 
%% Cell type:markdown id: tags:
 
### Visualize the daily `mean` values of tropospheric NO<sub>2</sub>
 
%% Cell type:markdown id: tags:
 
As dictionary key, you select `['mean']` and you apply the `plot.line()` function of the pandas library, which easily allow to do a line plot of a pandas.DataFrame.
 
%% Cell type:code id: tags:
 
``` python
for region in no2_1920_dict.keys():
print(region)
djfm_mean = pd.DataFrame(no2_1920_dict[region]['mean'])
djfm_mean = djfm_mean.set_index('date')
 
ax=(djfm_mean*conversion_factor*1e-15).plot.line(figsize=(20,10), linestyle='dashed',color=['darkgrey','firebrick'],label=['DJFM 2019','DJFM 2020'])
 
plt.xticks(fontsize=16)
plt.yticks(fontsize=16)
plt.title("Sentinel 5p NO2 Daily averages for " + region + " - DJFMAM 2019 vs DJFMAM 2020", fontsize=20, pad=20)
plt.ylabel('*1e-15 mol/cm2', fontsize=16)
plt.xlabel('Month', fontsize=16)
plt.legend(fontsize=16,loc=1)
plt.show()
```
 
%% Output
 
eur_extent