Commit 229b0ec3 authored by Daniel Lee's avatar Daniel Lee 🐐
Browse files

Merge branch 'pre-2.3.0-rc5' into 'master'

2.3.0-rc6

Closes EPCTO-84, EPCTO-91, EPCTO-89, #256, #198, #212, #155, #255, #244, #249, #252, #211, #190, #182, #186, #183, #181, #59, #154, #164, #144, #149, #150, #86, #85, #129, #102, #107, #123, #117, #95, and #74

See merge request epct/epct!131
parents ce4ca341 69ae4726
test-data/** filter=lfs diff=lfs merge=lfs -text
project-docs/** filter=lfs diff=lfs merge=lfs -text
project-docs/* filter=lfs diff=lfs merge=lfs -text
package/epct-installer-2.1.0-py3.6-full.tar.gz filter=lfs diff=lfs merge=lfs -text
add filter=lfs diff=lfs merge=lfs -text
conda/msg-gdal-driver/gdal_MSG.so filter=lfs diff=lfs merge=lfs -text
conda/msg-gdal-driver/gdal_MSG.dll filter=lfs diff=lfs merge=lfs -text
......@@ -3,3 +3,8 @@ compresson/compressed/*.h5
.DS_store
_build
~$*
# Jupyter Notebook
.ipynb_checkpoints/
nbs/
**/.idea/
......@@ -5,6 +5,52 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
### Changed
### Fixed
## [2.3.0]
### Added
- Customisation framework allows to use customisation plugins without changes to the code
- New REST-based Web Service Interface
- GUI is now a web application with improved design
- Flexible deployment approach allows to select plugins be install
- Products: added support for EPS ASCATL2SMR
- Products: added support for EPS ASCATL2SMO
- API: added support for asynchronous chain execution
- API/CLI: added interface to query EPCT configuration and settings
- Plugins: "epct_plugin_umarf" allows to use the msg15toISCCP plugin from UMARF
(enables format conversion of MSG HRSEVIRI L1.5 products to ISCCP formats)
- Plugins: new "epct_plugin_umarf" allows to use the msg15toXrit executable from UMARF
- Plugins: "epct_plugin_fist_iasil1c" allows to use the IASIL1c_v9A plugin from FIST
(enables conversion of EPS IASI L1C products to BUFR).
- EPCS prototype, with support for authenticated retrieval of information through the
EUMETSAT API gateway.
- Automated validation tests for format conversion.
### Changed
- Restored support for HRSEVIRI_HRIT, HRSEVIRI_HRIT_HRV
- API, CLI: Entry points removed (choices, load)
- API, CLI: "chain_run" entry point deprecated in favour of "run_chain".
- CLI: specification of the product for `run_chain` now only possible in the chain configuration.
- Architecture: new EPS importer (in the "epct_plugin_gis" plugin)
- Architecture: updated MSG importer.
- GDAL dependency completely removed from EPCT core components
- GDAL-based customisation engine is now a plugin
- Improved configurability (working directory, base path).
- Simplified deployment; now all packages have their own conda-based building system.
### Removed
- Conditional layer filtering for GLBSST no more supported
- ASCII format no more supported.
- JAVA GUI application no more available
- Support for deployment on Docker containers no longer available.
### Fixed
## [2.2.0] - 2019-07-10
### Added
......@@ -44,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- Installation: minimum version for 'cfgrib' imposed (fixes crash on BUFR conversion)
- Installation: added dependencies for CentOS installatio
- Installation: added dependencies for CentOS installation
- Fixed crash when naming convention referring to non-existing field
- Fix erroneously duplicated bands for GOME_PMA_AOP
- Use standard BUFR missing values instead of dummy ones
......
......@@ -15,9 +15,10 @@ The following main branches always exist:
When a developer wants to work on a new feature or a major bugfix, he needs to:
* [ ] create an issue on the project GitLab repository with a short descriptive title
* [ ] create an issue on the project GitLab repository with a short descriptive title and giving it a "weight" (complexity): 0: straightforward, and assinging a milestone
* [ ] from the issue GitLab page, create a new branch with "development" as a parent. The branch name will be automatically generated by GitLab from the issue title
* [ ] work on the new feature on the new branch, and commit changes to it
* [ ] when working on a issue, put it in a "Doing" state (issue tag)
* [ ] work on the new feature on the new branch, and commit changes to it, quoting the issue
* [ ] once the work is complete, create a merge request to "development", quoting the issue number and resolution
* [ ] update the "[unpublished]" section in CHANGES.md to add a one-line description of the closed issues (for humans) as described [here](https://keepachangelog.com/en/0.3.0/)
* [ ] update/close the issue
......
Install on Linux from source
----------------------------
This document describes how to install the EPCT on a Linux Operating System (64-bit) machine from the source code.
The installation process requires conda.
If it is not available as a command:
* download it from https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
* run the script to install conda, by executing::
bash Miniconda3-latest-Linux-x86_64.sh
Take note of the instructions at the end of the installation if conda has not been activated by default.
Also note that (as the instructions say) if conda is activated by default, you need to exit
and reopen a shell for the installation of conda to be available.
Building the individual packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Install conda-build::
conda install -y conda-build
Download or clone the epct source code, and enter the directory.
To build the epct package, execute::
conda build -c local -c anaconda -c conda-forge ./conda/epct
The package will be found in:
<path_to_conda_install>/conda-bld/linux-64/
To build the epct-desktop package (requires the epct package), execute::
conda build -c local -c anaconda -c conda-forge ./conda/epct-desktop
In a default conda installation, the packages will be build in the
$HOME/miniconda3/conda-bld directory
Now you can install either the Server package or the Desktop package as follows.
Installing the EPCT Server package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the Server package, execute the following::
conda env update -f ./conda/epct/environment.yaml
This create the epct "environment" with the epct installed.
It needs to be activated in the terminal to use epct.
This can be done each time a new terminal is opened, with::
conda activate epct
Or it can be activated by default by adding it to .bashrc::
echo conda activate epct >> $HOME/.bashrc
Once the epct environment is active, one can use epct; e.g. to retrieve the version::
ecpt --version
Installing the EPCT Desktop package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the Desktop package, execute the following::
conda env update -f ./conda/epct-desktop/environment.yaml
This create the epct-desktop "environment" with the epct installed.
It needs to be activated in the terminal to use epct.
This can be done each time a new terminal is opened, with::
conda activate epct-desktop
Or it can be activated by default by adding it to .bashrc::
echo conda activate epct-desktop >> $HOME/.bashrc
Once the environment is active, one can use the EPCT GUI and the CLI.
E.g. to launch the GUI::
ecpt_gui &
Install on Windows 10 Professional from source
---------------------------------------------
This document describes how to install the EPCT on a Windows 10 Professional (64-bit) machine from the source code.
System setup
------------
* Install conda as described in https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html, with condabin set on user's PATH environment variable
(needed: about 3GB disk space required, about 1GB RAM)
* Install Visual Studio 2015 with C++
(needed: about 7GB disk space, about 1GB RAM)
* Obtain the epct source code
Building the individual packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Access the 'anaconda prompt' from the Start menu.
From the terminal, execute the following to install conda-build::
conda install -y conda-build
Download or clone the epct source code, and enter the directory.
The package will be found in:
<path_to_conda_install>\conda-bld\win-64\
To build the epct-desktop package (requires the epct package), execute::
conda build -c local -c anaconda -c conda-forge conda\epct-desktop
The package will be found in:
<path_to_conda_install>\conda-bld\win-64\
In a default conda installation, the packages will be build in the
%HOMEPATH%\Miniconda3\conda-bld directory
Now you can install either the Server package or the Desktop package as follows.
Installing the EPCT Server package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the Server package, execute the following from the 'anaconda prompt'::
conda env update -f conda\epct\environment.yaml
This create the epct "environment" with the epct installed.
It needs to be activated in the terminal to use epct.
This can be done each time a new terminal is opened, with::
conda activate epct
Or it can be activated by default by adding it to .bashrc::
echo conda activate epct >> $HOME/.bashrc
Once the epct environment is active, one can use epct; e.g. to retrieve the version::
ecpt --version
Installing the EPCT Desktop package
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the Desktop package, execute the following from the anaconda prompt::
conda env update -f conda\epct-desktop\environment.yaml
This create the epct-desktop "environment" with the epct installed.
It needs to be activated in the anconda prompt to use epct.
This can be done each time a new anaconda prompt is opened, with::
conda activate epct-desktop
Once the environment is active, one can use the EPCT GUI and the CLI.
E.g. to launch the GUI::
ecpt_gui
EPCT Installation on Linux
--------------------------
Prerequisites
~~~~~~~~~~~~~
To install the EPCT, you need:
* either the EPCT Desktop installation package, which includes the EPCT GUI, CLI and API,
* or the EPCT Server installation package, which includes the EPCT CLI and API.
The EPCT installation does not require administration credentials.
Install (EPCT Server Installation Package)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The EPCT server installation package is a single bash script, whose name starts with epct-
followed by the version number, e.g. epct-2.2.0_rc1-Linux-x86_64.sh
To install it, execute::
bash epct-2.2.0_rc1-Linux-x86_64.sh
and follow the instructions on-screen. You will have to accept the EPCT license.
At the end of the installation, you will have the choice to activate the epct environment
each time you open the shell, or to activate it when it's needed.
Note that if you choose to activate it every time, you will still need to exit the current shell and open a new one.
After that, you can start the ecpt CLI, e.g.::
epct --version
Install (EPCT Desktop Installation Package)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The EPCT server installation package is a single bash script, whose name starts with epct-desktop-
followed by the version number, e.g. epct-desktop-2.2.0_rc1-Linux-x86_64.sh
It is installed in the same way as the Server Installation package.
Once the installation completes, exit the current shell and open a new one.
To launch the EPCT GUI, execute at the prompt::
epct_gui &
Install the EPCT
----------------
This document describes how to build and install the EPCT core components and
the customisation plugins from the source code, for the following Operating Systems:
- Ubuntu Linux 18.04 64 bit
- CentOS v.7 64 bit
- Windows 10 Professional 64bit
Hardware pre-requisites
~~~~~~~~~~~~~~~~~~~~~~~~
The Installation of the EPCT needs at least:
- 64bit platform
- 3 GB of disk space
- 4 GB of free memory
Software pre-requisites for the core components
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Building and installation require:
- the `epct` source code.
- `conda`, installed as described in
`https://conda.io/projects/conda/en/latest/user-guide/install/index.html`.
Additional software pre-requisites for the core components (Windows 10 Pro)
---------------------------------------------------------------------------
- Visual Studio 2015 with C++ installed
Building the individual packages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Install conda-build::
conda install -y conda-build
Enter the `epct` source directory.
Core components
'''''''''''''''
Build the `epct` package::
conda build -c local -c anaconda -c conda-forge epct
Build the `epct-restapi` package::
conda build -c local -c anaconda -c conda-forge epct-restapi
Build the `epct-webui` package::
conda build -c local -c anaconda -c conda-forge epct-webui
Building the customisation plugins
'''''''''''''''''''''''''''''''''''
Customisation plugins are optional packages which
add customisation functionalities to the EPCT.
At least one must be present to use the EPCT for customisation.
epct_plugin_gis
...............
Supported platforms: all.
To build the `epct_plugin_gis`, execute::
conda build -c local -c anaconda -c conda-forge epct_plugin_gis
msg-gdal-driver
+++++++++++++++
The `msg-gdal-driver` driver package which adds the support
for MSG HRV products to the `epct_plugin_gis`.
Download or clone the epct source code, and enter the
`conda/msg-gdal-driver` (`conda\msg-gda-driver` on Windows) directory.
Download the Public WaveLet Transform Decompression Library
from https://www.eumetsat.int/website/home/Data/DataDelivery/SupportSoftwareandTools/index.html.
Put the `PublicDecompWT.zip` file from the Public WaveLet Transform Decompression Library in the `src` dir.
Go back to the `epct` source code root directory and execute the following
to build the `msg-gdal-driver` package::
conda build -c local -c anaconda -c conda-forge conda
epct-plugin-umarf
.................
Supported platforms: Linux.
Build the `epct_plugin_umarf` package as follows::
conda build -c local -c anaconda -c conda-forge epct_plugin_umarf
Note: the package does not contain the executables `msg15toISCCP` and `msg15toXrit` required to use it.
They must be present on the target machine in the user's path. If not, the user should
contact ops@eumetsat.int.
Note: the plugin is optional. If the above executables are not present on the machine,
installation of the plugin does not fail, but the plugin will not be used by EPCT.
epct-plugin-fist-iasil1c
.........................
Supported platforms: Linux.
Build the `epct_plugin_fist_iasil1c` package as follows:
conda build -c local -c anaconda -c conda-forge epct_plugin_fist-iasil1c
Note: the package does not contain the `IASIL1c_v9A` executable from FIST required to use it.
It must be present on the target machine in the user's path. If not, the user should
contact ops@eumetsat.int.
Note: the plugin is optional. If the above executable are not present on the machine,
installation of the plugin does not fail, but the plugin will not be used by EPCT.
Installing EPCT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the EPCT on:
- a Linux platform, execute the following::
conda env create -f conda/epct-desktop-linux-environment.yaml
- on Windows 10::
conda env create -f conda\epct-desktop-win10-environment.yaml
This create the `epct-desktop` "environment" with all the supported components installed.
It needs to be activated in the terminal to use epct.
This can be done each time a new terminal is opened, with::
conda activate epct-desktop
On Linux, it can be activated by default by adding it to `.bashrc`::
echo conda activate epct-desktop >> $HOME/.bashrc
Once the epct environment is active, one can use epct; e.g. to retrieve the version::
epct --version
Important notes:
- (Linux only) to have the `epct_plugin_umarf` customisation plugin working, the `msg15toISCCP`
`msg15toXrit` executables must be in the user's path; they are not installed by conda
- (Linux only) to have the `epct_plugin_fist_iasil1c` customisation backend working, the `IASIL1c_v9A`
executable must be in the user's path; it is not installed by conda
Starting the EPCT Web Service and the GUI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once the environment is active, one can use the EPCT GUI and the CLI.
To use the GUI, the EPCT Web Service must be active. If not:
1. set the DEVEL variable to 1.
on Windows::
set DEVEL=1
on Linux::
export DEVEL=1
2. launch the service with (modify the port as needed)::
waitress-serve --call epct_restapi:app
Test it by accessing with a browser the following URL: `http://localhost:8080/api/v1/products`.
The browser should return the list of supported products
Then to launch the GUI, execute::
epct_webui
EPCT Installation on Windows 10 Pro
-----------------------------------
Prerequisites
~~~~~~~~~~~~~
The EPCT installation does not require administration credentials.
The EPCT installation requires a version of the conda package manager installed.
If it is not available as a command, install it as described in https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html, with condabin set on user's PATH environment variable
Install (EPCT Server Installation Package)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The EPCT server installation package is a single executable file, whose name starts with epct-
followed by the version number, e.g. epct-2.2.0_rc1-Windows-x86_64.exe
To install it, run the executable file, and follow the instructions on-screen.
You will have to accept the EPCT license and choose an installation directory.
After that, you will find "Anaconda3" in your windows Start menu, with the item "Anaconda Prompt".
From the anaconda prompt, go to the installation directory chosen during the installation
process, and from there to the 'Scripts' folder::
cd <installation folder>\Scripts
Now you can start the ecpt CLI, e.g.::
epct --version
Install (EPCT Desktop Installation Package)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The EPCT server installation package is a single bash script, whose name starts with epct-desktop-
followed by the version number, e.g. epct-desktop-2.2.0_rc1-Windows-x86_64.exe
It is installed in the same way as the Server Installation package.
Once the installation completes, you will find "Anaconda3" in your windows Start menu,
with the item "Anaconda Prompt".
From the anaconda prompt, go to the installation directory chosen during the installation
process, and from there to the 'Scripts' folder::
cd <installation folder>\Scripts
To launch the EPCT GUI, execute at the prompt::
epct_gui.exe
\ No newline at end of file
......@@ -16,12 +16,11 @@ The EPCT can be installed on:
The minimum hardware requirements are:
- 64bit platform
- TBC of disk space
- 3 GB of disk space
- 4 GB of free memory
Installation procedures for Linux-based platforms are in described in the INSTALL.linux.rst file.
Installation procedures are in described in the INSTALL.rst file.
Installation procedures for Window 10 Pro are in described in the INSTALL.win10pro file.
How to get User Support
-----------------------
......
......@@ -60,7 +60,6 @@ RUN cd /tmp \
&& sed -ie 's|# Check the Java Runtime Environment version|JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/|' cviirs.sh
#RUN cd /tmp \
# && curl -LO ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_L1c_Compressed-Datasets_and_Decompression-Plugin_April2017/Decompression_Plugin/EUMETSAT-FCIDECOMP_V1.0.1.tar.gz \
# && tar -xzvf EUMETSAT-FCIDECOMP_V1.0.1.tar.gz \
# && tar -xzvf FCIDECOMP_V1.0.1/Software/fcidecomp_sources-V1.0.1.tar.gz \
# && cd FCIDECOMP_SOURCES \
......
# Copyright 2017-2019, European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import os.path
......
# Copyright 2017-2019, European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT)
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import logging
import pathlib
......
Install the EPCT on a Linux platform
-------------------------------------
Build on Linux
--------------
To install EPCT on a Linux platform, follow the instructions in the
`INSTALL.rst` file in the EPCT root directory.
The build process requires conda.
If it is not available as a command:
* download it from https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
* run the script to install conda, by executing::
bash Miniconda3-latest-Linux-x86_64.sh
Take note of the instructions at the end of the installation if conda has not been activated by default.
Also note that (as the instructions say) if conda is activated by default, you need to exit