How to create and share your own Services in F-TEP

One of the key collaborative aspects of F-TEP is the ability for users to create and share services with others. This will give F-TEP users the ability to shape the services that F-TEP offers and give other users the chance to use bespoke and novel algorithms and processors on a satellite data of relevance to them.

The Developer interface is where Expert Users will be able to see and access services, both their own services as well as the range of F-TEP core services. If you cannot see the Developer Interface you are not an Expert User and hence you need to request an upgrade via the Helpdesk interface.

The Services Panel on the left of the screen lists all the services that that user has access to, which will be made up of:

  • Core services, provide by F-TEP and available to all 
  • My services, created by the user themselves, which may be not be visible to others
  • Shared services, provided by other F-TEP users

Users can filter within the services panels by All, Mine or Shared criteria or they can type in a keyword.

Creating a Service

Key components of any F-TEP service include:

  • Defined input files - the type of input files are defined by the service developer, with the actual files chosen by the service user 
  • Defined input parameters - the parameters defined by the service developer, with the actul values chosen by the service user 
  • Defined output files - the type of output files are defined by the service developer, with the actual file(s) chosen by the service user
  • The actual service engine, which is normally created within a Docker container

It is expected that the service creator has some basic knowledge of Docker and shell scripting, although template files and guidance are provided, and users can contact the F-TEP Helpdesk for further guidance.

The basic steps for creating a service includes:

  1. Naming the service: an Expert user logs into the F-TEP portal and navigates to the Developer Interface. They then click on the large plus button towards to bottom of the Services Panel to ‘Create New Service’. They then fill in the pop-up box with the name and the description of the service and after they have accepted the F-TEP Terms and Conditions, click ‘create’ to create the Service.
  2. Building the Service Framework: The service framework is then automatically created, which consists of a series of files which are needed for the service. These include:
    1. A DockerFile: this default file provides SNAP and Orfeo toolbox on top of the Ubuntu 16.04 Operating System. This is expected to be sufficient in many cases for creating bespoke EO data processing services that utilise SNAP algorithms.
    2. A workflow.sh File: this required executable bash script file allows the user to arbitrarily define the service procedure. The default file demonstrates the expected directories for input and output data, and how to read input parameter values. If the service developer requires an alternative scripting language, workflow.sh can be updated to call another script, or the ENTRYPOINT parameter in the Dockerfile may be changed.
    3. Defining the service: within the main developer panel the user can enter various different pieces of information including:
      1. Docker Tag: automatically generated by F-TEP
      2. Version: the developer can assign a version number for their service 
      3. Service Type: this can either be a Processor, Application or Bulk Processor
      4. Summary: Free text for the user to describe their service
      5. The Input Definitions tab allows users to define the service’s Input parameters. Required fields include Field ID, Field Title, Field Description, parameter type. A minimum of 1 input parameter is needed.
      6. The Output Definitions tab allows users to define the service’s Output parameters. Required fields include Field ID, Field Title, Field Description, parameter type. A minimum of 1 output parameter is needed.
    4. Testing the service: Once a user is happy with the service they can then test it via the Explorer interface. Within the Explorer interface the service should be visible. During the testing the service may be shared with other groups for testing.
    5. Sharing & Publishing the service: Once ready a user can make their service accessible to other via the Manage / Share page. Publishing is where the service is made avaialble to all and hence is not instantaneous as any new services need to be reviewed by F-TEP to ensure they are safe to be used by others.

An example service creation

Below is a simple example service that can be created to demonstrate the key steps:

  1. User creates a new service and keeps the default Dockerfile template.
  2. Input parameter definitions are created, for example string parameters "foo" and "bar".
  3. Output parameter definitions are created, for example "baz".
  4. The default workflow.sh file is updated with these parameters in mind, for example see below.
  5. The service is saved and tested via the Explorer view. It may be shared with other groups for testing.
    1. When the service is run, the user will see in the job log the standard output/error as well as step-by-step command output from the workflow.sh execution. The result of the below example script will produce a single output file containing the input parameters concatenated.
  6. When ready, the service is requested to be published.
  7. An F-TEP Content Authority will review the service and if approved, take the necessary publishing actions. As a result, the new service will be visible to all users in the Explorer view, and exposed via WPS for external usage.

Example workflow.sh

#!/usr/bin/env bash

set -x -e

# F-TEP service environment
WORKFLOW=$(dirname $(readlink -f "$0"))
WORKER_DIR="/home/worker"
IN_DIR="${WORKER_DIR}/workDir/inDir"
OUT_DIR="${WORKER_DIR}/workDir/outDir"
WPS_PROPS="${WORKER_DIR}/workDir/FTEP-WPS-INPUT.properties"
PROC_DIR="${WORKER_DIR}/procDir"
TIMESTAMP=$(date --utc +%Y%m%d_%H%M%SZ)

mkdir -p ${PROC_DIR}
mkdir -p ${OUT_DIR}/baz

# Input params
source ${WPS_PROPS}

# Processing
echo "${foo} ${bar}" >${OUT_DIR}/baz/MY_SERVICE_OUTPUT_${TIMESTAMP}.txt