Merge branch 'pre-2.3.0-rc5' into 'master'

2.3.0-rc6

Closes EPCTO-84, EPCTO-91, EPCTO-89, #256, #198, #212, #155, #255, #244, #249, #252, #211, #190, #182, #186, #183, #181, #59, #154, #164, #144, #149, #150, #86, #85, #129, #102, #107, #123, #117, #95, and #74

See merge request epct/epct!131

.gitattributes

 test-data/** filter=lfs diff=lfs merge=lfs -text
-project-docs/** filter=lfs diff=lfs merge=lfs -text
+project-docs/* filter=lfs diff=lfs merge=lfs -text
+package/epct-installer-2.1.0-py3.6-full.tar.gz filter=lfs diff=lfs merge=lfs -text
+add filter=lfs diff=lfs merge=lfs -text
+conda/msg-gdal-driver/gdal_MSG.so filter=lfs diff=lfs merge=lfs -text
+conda/msg-gdal-driver/gdal_MSG.dll filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -3,3 +3,8 @@ compresson/compressed/*.h5
 .DS_store
 _build
 ~$*
+
+# Jupyter Notebook
+.ipynb_checkpoints/
+nbs/
+**/.idea/

CHANGES.md

@@ -5,6 +5,52 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+
+### Changed
+
+### Fixed
+
+
+## [2.3.0]
+### Added
+ - Customisation framework allows to use customisation plugins without changes to the code
+ - New REST-based Web Service Interface
+ - GUI is now a web application with improved design
+ - Flexible deployment approach allows to select plugins be install
+ - Products: added support for EPS ASCATL2SMR
+ - Products: added support for EPS ASCATL2SMO 
+ - API: added support for asynchronous chain execution
+ - API/CLI: added interface to query EPCT configuration and settings
+ - Plugins: "epct_plugin_umarf" allows to use the msg15toISCCP plugin from UMARF
+   (enables format conversion of MSG HRSEVIRI L1.5 products to ISCCP formats)
+ - Plugins: new "epct_plugin_umarf" allows to use the msg15toXrit executable from UMARF
+ - Plugins: "epct_plugin_fist_iasil1c" allows to use the IASIL1c_v9A plugin from FIST
+   (enables conversion of EPS IASI L1C products to  BUFR).
+ - EPCS prototype, with support for authenticated retrieval of information through the
+   EUMETSAT API gateway.
+ - Automated validation tests for format conversion.
+
+### Changed
+ - Restored support for HRSEVIRI_HRIT, HRSEVIRI_HRIT_HRV
+ - API, CLI: Entry points removed (choices, load)
+ - API, CLI: "chain_run" entry point deprecated in favour of "run_chain".
+ - CLI: specification of the product for `run_chain` now only possible in the chain configuration.
+ - Architecture: new EPS importer (in the "epct_plugin_gis" plugin)
+ - Architecture: updated MSG importer.
+ - GDAL dependency completely removed from EPCT core components
+ - GDAL-based customisation engine is now a plugin
+ - Improved configurability (working directory, base path).
+ - Simplified deployment; now all packages have their own conda-based building system.
+ 
+### Removed
+ - Conditional layer filtering for GLBSST no more supported
+ - ASCII format no more supported.
+ - JAVA GUI application no more available
+ - Support for deployment on Docker containers no longer available.
+
+### Fixed
+
 
 ## [2.2.0] - 2019-07-10
 ### Added
@@ -44,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
  - Installation: minimum version for 'cfgrib' imposed (fixes crash on BUFR conversion)
- - Installation: added dependencies for CentOS installatio
+ - Installation: added dependencies for CentOS installation
  - Fixed crash when naming convention referring to non-existing field
  - Fix erroneously duplicated bands for GOME_PMA_AOP
  - Use standard BUFR missing values instead of dummy ones

CONTRIBUTING.md

@@ -15,9 +15,10 @@ The following main branches always exist:
  
 When a developer wants to work on a new feature or a major bugfix, he needs to:
 
-* [ ] create an issue on the project GitLab repository with a short descriptive title
+* [ ] create an issue on the project GitLab repository with a short descriptive title and giving it a "weight" (complexity): 0: straightforward, and assinging a milestone
 * [ ] from the issue GitLab page, create a new branch with "development" as a parent. The branch name will be automatically generated by GitLab from the issue title
-* [ ] work on the new feature on the new branch, and commit changes to it
+* [ ] when working on a issue, put it in a "Doing" state (issue tag)
+* [ ] work on the new feature on the new branch, and commit changes to it, quoting the issue
 * [ ] once the work is complete, create a merge request to "development", quoting the issue number and resolution
 * [ ] update the "[unpublished]" section in CHANGES.md to add a one-line description of the closed issues (for humans) as described [here](https://keepachangelog.com/en/0.3.0/)
 * [ ] update/close the issue

INSTALL-from-source.linux.rst

-
-Install on Linux from source
-----------------------------
-
-This document describes how to install the EPCT on a Linux Operating System (64-bit) machine from the source code.
-
-The installation process requires conda.
-
-If it is not available as a command:
- * download it from https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
- * run the script to install conda, by executing::
-
-   bash Miniconda3-latest-Linux-x86_64.sh
-
-Take note of the instructions at the end of the installation if conda has not been activated by default.
-Also note that (as the instructions say) if conda is activated by default, you need to exit 
-and reopen a shell for the installation of conda to be available.
-
-
-Building the individual packages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Install conda-build::
-
-   conda install -y conda-build
-
-Download or clone the epct source code, and enter the directory.
-
-
-To build the epct package, execute::
-
-   conda build -c local -c anaconda -c conda-forge ./conda/epct
-
-
-The package will be found in:
-   <path_to_conda_install>/conda-bld/linux-64/
-  
-To build the epct-desktop package (requires the epct package), execute::
-
-    conda build -c local -c anaconda -c conda-forge ./conda/epct-desktop
-
-
-In a default conda installation, the packages will be build in the 
-$HOME/miniconda3/conda-bld directory
-
-Now you can install either the Server package or the Desktop package as follows.
-
-Installing the EPCT Server package
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To install the Server package, execute the following::
-
-   conda env update -f ./conda/epct/environment.yaml 
-
-This create the epct "environment" with the epct installed.
-It needs to be activated in the terminal to use epct.
-This can be done each time a new terminal is opened, with::
-
-   conda activate epct
-
-Or it can be activated by default by adding it to .bashrc::
-
-   echo conda activate epct >> $HOME/.bashrc
-
-Once the epct environment is active, one can use epct; e.g. to retrieve the version::
-
-   ecpt --version
-
-
-Installing the EPCT Desktop package
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To install the Desktop package, execute the following::
-
-   conda env update -f ./conda/epct-desktop/environment.yaml 
-
-This create the epct-desktop "environment" with the epct installed.
-It needs to be activated in the terminal to use epct.
-This can be done each time a new terminal is opened, with::
-
-   conda activate epct-desktop
-
-Or it can be activated by default by adding it to .bashrc::
-
-   echo conda activate epct-desktop >> $HOME/.bashrc
-
-Once the environment is active, one can use the EPCT GUI and the CLI.
-E.g. to launch the GUI::
-
-   ecpt_gui &
-

INSTALL-from-source.win10pro.rst

-
-Install on Windows 10 Professional from source
----------------------------------------------
-
-This document describes how to install the EPCT on a Windows 10 Professional (64-bit) machine from the source code.
-
-System setup
-------------
-* Install conda as described in https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html, with condabin set on user's PATH environment variable
- (needed: about 3GB disk space required, about 1GB RAM)
-* Install Visual Studio 2015 with C++ 
- (needed: about 7GB disk space, about 1GB RAM)
-* Obtain the epct source code
-
-
-Building the individual packages
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Access the 'anaconda prompt' from the Start menu.
-
-From the terminal, execute the following to install conda-build::
-
-   conda install -y conda-build
-
-Download or clone the epct source code, and enter the directory.
-
-The package will be found in:
-   <path_to_conda_install>\conda-bld\win-64\
-
-To build the epct-desktop package (requires the epct package), execute::
-
-   conda build -c local -c anaconda -c conda-forge conda\epct-desktop
-
-The package will be found in:
-   <path_to_conda_install>\conda-bld\win-64\
-
-In a default conda installation, the packages will be build in the
-%HOMEPATH%\Miniconda3\conda-bld directory
-
-
-Now you can install either the Server package or the Desktop package as follows.
-
-Installing the EPCT Server package
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To install the Server package, execute the following from the 'anaconda prompt'::
-
-   conda env update -f conda\epct\environment.yaml 
-
-This create the epct "environment" with the epct installed.
-It needs to be activated in the terminal to use epct.
-This can be done each time a new terminal is opened, with::
-
-   conda activate epct
-
-Or it can be activated by default by adding it to .bashrc::
-
-   echo conda activate epct >> $HOME/.bashrc
-
-Once the epct environment is active, one can use epct; e.g. to retrieve the version::
-
-   ecpt --version
-
-
-Installing the EPCT Desktop package
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To install the Desktop package, execute the following from the anaconda prompt::
-
-   conda env update -f conda\epct-desktop\environment.yaml 
-
-This create the epct-desktop "environment" with the epct installed.
-It needs to be activated in the anconda prompt to use epct.
-This can be done each time a new anaconda prompt is opened, with::
-
-   conda activate epct-desktop
-
-Once the environment is active, one can use the EPCT GUI and the CLI.
-E.g. to launch the GUI::
-
-   ecpt_gui
-

INSTALL.linux.rst

-EPCT Installation on Linux
---------------------------
-
-Prerequisites
-~~~~~~~~~~~~~
-
-To install the EPCT, you need:
-   * either the EPCT Desktop installation package, which includes the EPCT GUI, CLI and API,
-   * or the EPCT Server installation package, which includes the EPCT CLI and API.
-
-The EPCT installation does not require administration credentials.
-
-
-Install (EPCT Server Installation Package)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The EPCT server installation package is a single bash script, whose name starts with epct-
-followed by the version number, e.g. epct-2.2.0_rc1-Linux-x86_64.sh
-
-To install it, execute::
-
-       bash epct-2.2.0_rc1-Linux-x86_64.sh
-
-and follow the instructions on-screen. You will have to accept the EPCT license.
-At the end of the installation, you will have the choice to activate the epct environment
-each time you open the shell, or to activate it when it's needed.
-Note that if you choose to activate it every time, you will still need to exit the current shell and open a new one.
-
-After that, you can start the ecpt CLI, e.g.::
-
-       epct --version
-
-
-Install (EPCT Desktop Installation Package)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The EPCT server installation package is a single bash script, whose name starts with epct-desktop-
-followed by the version number, e.g. epct-desktop-2.2.0_rc1-Linux-x86_64.sh
-
-It is installed in the same way as the Server Installation package.
-
-Once the installation completes, exit the current shell and open a new one.
-
-To launch the EPCT GUI, execute at the prompt::
-
-       epct_gui &

INSTALL.rst

+Install the EPCT
+----------------
+
+This document describes how to build and install the EPCT core components and
+the customisation plugins from the source code, for the following Operating Systems:
+
+- Ubuntu Linux 18.04 64 bit
+- CentOS v.7 64 bit
+- Windows 10 Professional 64bit
+
+Hardware pre-requisites
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Installation of the EPCT needs at least:
+
+- 64bit platform
+- 3 GB of disk space
+- 4 GB of free memory
+
+Software pre-requisites for the core components
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Building and installation require:
+
+- the `epct` source code.
+
+- `conda`, installed as described in
+  `https://conda.io/projects/conda/en/latest/user-guide/install/index.html`.
+
+
+Additional software pre-requisites for the core components (Windows 10 Pro)
+---------------------------------------------------------------------------
+
+- Visual Studio 2015 with C++ installed
+
+
+
+Building the individual packages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Install conda-build::
+
+   conda install -y conda-build
+
+
+Enter the `epct` source directory.
+
+
+Core components
+'''''''''''''''
+
+Build the `epct` package::
+
+    conda build -c local -c anaconda -c conda-forge epct
+
+
+Build the `epct-restapi` package::
+
+    conda build -c local -c anaconda -c conda-forge epct-restapi
+
+Build the `epct-webui` package::
+
+    conda build -c local -c anaconda -c conda-forge epct-webui
+
+
+Building the customisation plugins
+'''''''''''''''''''''''''''''''''''
+
+Customisation plugins are optional packages which
+add customisation functionalities to the EPCT.
+
+At least one must be present to use the EPCT for customisation.
+
+epct_plugin_gis
+...............
+
+Supported platforms: all.
+
+To build the `epct_plugin_gis`, execute::
+
+    conda build -c local -c anaconda -c conda-forge epct_plugin_gis
+
+
+msg-gdal-driver
++++++++++++++++
+
+The `msg-gdal-driver` driver package which adds the support
+for MSG HRV products to the `epct_plugin_gis`.
+
+Download or clone the epct source code, and enter the
+`conda/msg-gdal-driver` (`conda\msg-gda-driver` on Windows) directory.
+
+Download the Public WaveLet Transform Decompression Library
+from https://www.eumetsat.int/website/home/Data/DataDelivery/SupportSoftwareandTools/index.html.
+
+Put the `PublicDecompWT.zip` file from the Public WaveLet Transform Decompression Library in the `src` dir.
+
+Go back to the `epct` source code root directory and execute the following
+to build the `msg-gdal-driver` package::
+
+   conda build -c local -c anaconda -c conda-forge conda
+
+
+
+epct-plugin-umarf
+.................
+
+Supported platforms: Linux.
+
+Build the `epct_plugin_umarf` package as follows::
+
+    conda build -c local -c anaconda -c conda-forge epct_plugin_umarf
+
+Note: the package does not contain the executables `msg15toISCCP` and `msg15toXrit` required to use it.
+They must be present on the target machine in the user's path. If not, the user should
+contact ops@eumetsat.int.
+
+Note: the plugin is optional. If the above executables are not present on the machine,
+installation of the plugin does not fail, but the plugin will not be used by EPCT.
+
+
+epct-plugin-fist-iasil1c
+.........................
+
+Supported platforms: Linux.
+
+Build the `epct_plugin_fist_iasil1c` package as follows:
+
+    conda build -c local -c anaconda -c conda-forge epct_plugin_fist-iasil1c
+
+Note: the package does not contain the `IASIL1c_v9A` executable from FIST required to use it.
+It must be present on the target machine in the user's path. If not, the user should
+contact ops@eumetsat.int.
+
+Note: the plugin is optional. If the above executable are not present on the machine,
+installation of the plugin does not fail, but the plugin will not be used by EPCT.
+
+
+Installing EPCT
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To install the EPCT on:
+
+- a Linux platform, execute the following::
+
+   conda env create -f conda/epct-desktop-linux-environment.yaml
+
+
+- on Windows 10::
+
+   conda env create -f conda\epct-desktop-win10-environment.yaml
+
+
+
+This create the `epct-desktop` "environment" with all the supported components installed.
+It needs to be activated in the terminal to use epct.
+This can be done each time a new terminal is opened, with::
+
+   conda activate epct-desktop
+
+On Linux, it can be activated by default by adding it to `.bashrc`::
+
+   echo conda activate epct-desktop >> $HOME/.bashrc
+
+Once the epct environment is active, one can use epct; e.g. to retrieve the version::
+
+   epct --version
+
+
+Important notes:
+
+- (Linux only) to have the `epct_plugin_umarf` customisation plugin working, the `msg15toISCCP`
+  `msg15toXrit` executables  must be in the user's path; they are not installed by conda
+
+- (Linux only) to have the `epct_plugin_fist_iasil1c` customisation backend working, the `IASIL1c_v9A`
+  executable must be in the user's path; it is not installed by conda
+
+Starting the EPCT Web Service and the GUI
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Once the environment is active, one can use the EPCT GUI and the CLI.
+
+To use the GUI, the EPCT Web Service must be active. If not:
+
+1.  set the DEVEL variable to 1.
+
+    on Windows::
+
+        set DEVEL=1
+
+
+    on Linux::
+
+        export DEVEL=1
+
+2. launch the service with (modify the port as needed)::
+
+    waitress-serve --call epct_restapi:app
+
+Test it by accessing with a browser the following URL: `http://localhost:8080/api/v1/products`.
+The browser should return the list of supported products
+
+Then to launch the GUI, execute::
+
+    epct_webui
+

INSTALL.win10pro.rst

-EPCT Installation on Windows 10 Pro
------------------------------------
-
-Prerequisites
-~~~~~~~~~~~~~
-
-The EPCT installation does not require administration credentials.
-
-The EPCT installation requires a version of the conda package manager installed.
-If it is not available as a command, install  it as described in https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html, with condabin set on user's PATH environment variable
-
-
-Install (EPCT Server Installation Package)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The EPCT server installation package is a single executable file, whose name starts with epct-
-followed by the version number, e.g. epct-2.2.0_rc1-Windows-x86_64.exe
-
-To install it, run the executable file, and follow the instructions on-screen.
-You will have to accept the EPCT license and choose an installation directory.
-
-After that, you will find "Anaconda3" in your windows Start menu, with the item "Anaconda Prompt".
-From the anaconda prompt, go to the installation directory chosen during the installation
-process, and from there to the 'Scripts' folder::
-
-    cd <installation folder>\Scripts
-
-Now you can start the ecpt CLI, e.g.::
-
-       epct --version
-
-
-Install (EPCT Desktop Installation Package)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The EPCT server installation package is a single bash script, whose name starts with epct-desktop-
-followed by the version number, e.g. epct-desktop-2.2.0_rc1-Windows-x86_64.exe
-
-It is installed in the same way as the Server Installation package.
-
-Once the installation completes, you will find "Anaconda3" in your windows Start menu,
-with the item "Anaconda Prompt".
-From the anaconda prompt, go to the installation directory chosen during the installation
-process, and from there to the 'Scripts' folder::
-
-    cd <installation folder>\Scripts
-
-To launch the EPCT GUI, execute at the prompt::
-
-       epct_gui.exe
\ No newline at end of file

README.rst

@@ -16,12 +16,11 @@ The EPCT can be installed on:
 The minimum hardware requirements are:
 
 - 64bit platform
-- TBC of disk space
+- 3 GB of disk space
 - 4 GB of free memory 
 
-Installation procedures for Linux-based platforms are in described in the INSTALL.linux.rst file.
+Installation procedures are in described in the INSTALL.rst file.
 
-Installation procedures for Window 10 Pro are in described in the INSTALL.win10pro file.
 
 How to get User Support
 -----------------------

compression/Dockerfile

@@ -60,7 +60,6 @@ RUN cd /tmp \
     && sed -ie 's|# Check the Java Runtime Environment version|JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/|' cviirs.sh
 
 #RUN cd /tmp \
-#    && curl -LO ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_L1c_Compressed-Datasets_and_Decompression-Plugin_April2017/Decompression_Plugin/EUMETSAT-FCIDECOMP_V1.0.1.tar.gz \
 #    && tar -xzvf EUMETSAT-FCIDECOMP_V1.0.1.tar.gz \
 #    && tar -xzvf FCIDECOMP_V1.0.1/Software/fcidecomp_sources-V1.0.1.tar.gz \
 #    && cd FCIDECOMP_SOURCES \

compression/benchmark.py

+# Copyright 2017-2019, European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 import logging
 import os.path

compression/cviirs-benchmark.py

+# Copyright 2017-2019, European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
 
 import logging
 import pathlib

conda/README.linux

+Install the EPCT on a Linux platform
+-------------------------------------
 
-Build on Linux
---------------
+To install EPCT on a Linux platform, follow the instructions in the
+`INSTALL.rst` file in the EPCT root directory.
 
-The build process requires conda.
-
-If it is not available as a command:
- * download it from https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
- * run the script to install conda, by executing::
-
-   bash Miniconda3-latest-Linux-x86_64.sh
-
-Take note of the instructions at the end of the installation if conda has not been activated by default.
-Also note that (as the instructions say) if conda is activated by default, you need to exit