Package `ftep_util`

Generic F-TEP related utility classes and methods

Requires Python modules requests, boto3, feedparser and GDAL. Note that GDAL module requires that GDAL binaries and development headers have been installed on the host. GDAL python module installation requires that Python development headers have been installed. On e.g. Ubuntu 20.04 install them with command

apt-get install python3-dev gdal-bin libgdal-dev

Automatic GDAL installation by pip may try to install a version of GDAL that differs from the version of the installed binaries. GDAL version can be checked with e.g.

gdalinfo --version

If needed install a specific version of GDAL before installing ftep_util module, e.g.

pip install GDAL==3.0.4

If installing an older GDAL version fails one reason can be too new setuptools module, e.g. installing GDAL 3.0.4 fails with setuptools >58. In this case install an older version, e.g.

pip install setuptools==57.5.0

Version 0.8 (19.12.2023)

Functions

def convert_to_python_re(userRegExp)

Converts the given expression to Python regular expression.

Makes it a global pattern by adding ^ and $ in the ends, escapes all dots to literal dots and converts all asterisks to .*

def detectFileParam(param_name, supported_extensions=None, pattern=None, verbose=False, input_dir='/home/worker/workDir/inDir/')

Detects files from the input directory of the given parameter, with additional extensions and pattern filtering.

Returns a tuple (files, filtered_files). The returned files have path. Returns empty lists if files are not found, e.g. user has not specified the input or it is of an unsupported file type.

If extensions list is given collects only those files that end with one of the extensions.

If a pattern is given only file names matching the pattern are returned in the filtered list. The pattern is applied to the full path of the files.

Raises ValueError if dir is not valid.

def detectFiles(dir, supported_extensions=None, pattern=None, verbose=False)

Detects files from a directory tree starting from the given dir

Returns a tuple (files, filtered_files). The returned files have path.

If extensions list is given collects only those files that end with one of the extensions.

If a pattern is given only file names matching the pattern are returned in the filtered list. The pattern is applied to the full path of the files.

Raises ValueError if dir is not valid.

def detectImageFiles(dir, supported_extensions=None, pattern=None, verbose=False)

Detects files from a directory tree starting from the given dir. Accepts all GDAL supported image files, with additional extensions and pattern filtering.

Returns a tuple (files, filtered_files). The returned files have path.

If extensions list is given collects only those files that end with one of the extensions.

If a pattern is given only file names matching the pattern are returned in the filtered list. The pattern is applied to the full path of the files.

Raises ValueError if dir is not valid.

def detectImageParam(param_name, supported_extensions=None, pattern=None, verbose=False, input_dir='/home/worker/workDir/inDir/')

Detects all GDAL supported image files from the input directory of the given parameter, with additional extensions and pattern filtering.

Returns a tuple (files, filtered_files). The returned files have path. Returns empty lists if files are not found, e.g. user has not specified the input or it is of an unsupported file type.

If extensions list is given collects only those files that end with one of the extensions.

If a pattern is given only file names matching the pattern are returned in the filtered list. The pattern is applied to the full path of the files.

Raises ValueError if dir is not valid.

def download_eodata(item_key, local_dir='.')

Downloads the requested item from CREODIAS EODATA S3 bucket to the local disk.

Args

item_key: The item key in the s3 bucket, e.g. Sentinel-2/MSI/L1C/2016/05/06/S2A_MSIL1C_20160506T195902_N0202_R128_T18XWR_20160506T195900.SAFE/. Can point to a folder or a file. The S3Path value from search_eodata results can be used as such (the unnecessary /eodata/ start is ignored).
local_dir: The local path where to download the data. The data will be stored in subdirectories under the given directory using the same directory names as in EODATA.

Returns

True if ok, False if the item key was not found.

def download_eodata_from_atom_url(atom_url, local_dir='.')

Downloads the requested items from CREODIAS EODATA S3 bucket to the local disk.

Args

atom_url: The resource catalogue url to the search result atom feed, e.g. https://resource-catalogue.new.f-tep.com/csw/?mode=opensearch&service=CSW&version=3.0.0&request=GetRecords&elementsetname=full&resulttype=results&typenames=csw:Record&recordids=S2A_MSIL1C_20160506T195902_N0202_R128_T18XWR_20160506T195900.SAFE.
local_dir: The local path where to download the data. The data will be stored in subdirectories under the given directory using the same directory name as in EODATA.

def download_s3_folder(bucket, s3_folder, local_dir='.')

Download the contents of a S3 folder to a local directory

Args

bucket: the boto3 s3 bucket object
s3_folder: the folder path in the s3 bucket, e.g. Sentinel-2/MSI/L1C/2016/05/06/S2A_MSIL1C_20160506T195902_N0202_R128_T18XWR_20160506T195900.SAFE/
local_dir: the local directory where to download the data

def search_eodata(collection, start_date, end_date, aoi_wkt, count=100, skip=0, cloud_cover=None)

Query Creodias EODATA content Arguments:

collection – One of SENTINEL-1, SENTINEL-2, SENTINEL-3, SENTINEL-5P, SENTINEL-6, SENTINEL-1-RTC, LANDSAT-5, LANDSAT-7, LANDSAT-8, SMOS, TERRAAQUA, COP-DEM, ENVISAT, S2GLC.
start_date – Content start date in UTC in format yyyy-mm-ddThh:mm:ssZ (e.g. 2023-06-01T00:00:00Z).
end_date – Content end date in UTC in format yyyy-mm-ddThh:mm:ssZ (e.g. 2023-06-01T00:00:00Z).
aoi_wkt – Area of interest that must intersect with the images as a EPSG:4326 WKT POLYGON definition (e.g. POLYGON ((20 60, 21 60, 21 61, 20 61, 20 60))).
count – The number of items to return (1-1000).
skip – The number of items to skip from the beginning of the results. Use this to browse results that contain over 1000 items.
cloud_cover – The maximum cloud cover percentage for the images. Can be used with collections that support it.

Returns a list of dict objects with the following keys: Name, S3Path, ContentLength, ContentDateStart, ContentDateEnd, Footprint, CloudCover (if applicable). The S3Path can be used to locate the data in the /eodata folder or to download the data from Creodias EODATA S3 interface using download_eodata method.

Raises ValueError if the input is incorrect. Raises RuntimeError if Creodias search does not return HTTP 200.

Classes

class Params (list_variables=[])

A convenience class for handling service parameters in F-TEP.

Example:

# Default parameter values
params = ftep.Params()
params.setValue('band', 1)
params.setValue('window', 5)
params.setValue('set_nodata', True)

# Add parameters from file
params.readFile(parameterFile)

Methods

def appendValue(self, key, value)
def getBoolean(self, key): Returns True if the parameter value is case-insensitive true or 1, False otherwise. Return False if the parameter value does not exist.
def getFloat(self, key, minvalue=None, maxvalue=None): Return the value of the parameter as a float, or None if the parameter does not exist.

Raises ValueError if the value cannot be converted to a float.

Raises ValueError if the value is not within the optional bounds.
def getFloatList(self, key, minvalue=None, maxvalue=None, delimiter=None): Return the value of the parameter as a list of floats, or None if the parameter does not exist.

Raises ValueError if the value cannot be converted to a float.

Raises ValueError if the value is not within the optional bounds.
def getInt(self, key, minvalue=None, maxvalue=None): Return the value of the parameter as an integer, or None if the parameter does not exist.

Raises ValueError if the value cannot be converted to an integer.

Raises ValueError if the value is not within the optional bounds.
def getIntList(self, key, minvalue=None, maxvalue=None, delimiter=None): Return the value of the parameter as a list of integers, or None if the parameter does not exist.

Raises ValueError if the value cannot be converted to an integer.

Raises ValueError if the value is not within the optional bounds.
def getListOfIntLists(self, key, minvalue=None, maxvalue=None, delimiter=None, listdelimiter=';'): Return the value of the parameter as a list of lists of integers, or None if the parameter does not exist. The lists should be separated with listdelimiter string.

Raises ValueError if the value cannot be converted to an integer.

Raises ValueError if the value is not within the optional bounds.
def getString(self, key): Return the value of the parameter as a string, or None if the parameter does not exist or if the value is None.
def getValue(self, key): Gets the value of a parameter as an object.
def readFile(self, file): Reads a properties file with key=value pairs

If value starts with def (case insensitive) does not modify the default value. If value equals none (case insensitive) sets the value explicitly to None. Lines starting with # are comments. Leading and trailing whitespace is removed.
def setValue(self, key, value): Sets the value of a parameter.