eogrow.pipelines.download

Implements different customizeable pipelines for downloading data.

class eogrow.pipelines.download.RaySessionActor(session)

Bases: object

This object can share a session object with multiple workers.

Parameters:

session (SentinelHubSession) –

get_valid_session()
pydantic model eogrow.pipelines.download.RescaleSchema[source]

Bases: Schema

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:
field dtype: dtype | None = None

The output dtype of data

Validated by:

  • optional_parse_dtype

field features_to_rescale: List[Tuple[FeatureType, str]] [Required]
field rescale_factor: float = 1

Amount by which the selected features are multiplied

pydantic model eogrow.pipelines.download.PostprocessingRescale[source]

Bases: Schema

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:
field rescale_schemas: List[RescaleSchema] [Optional]

Specify different ways to rescale features

class eogrow.pipelines.download.BaseDownloadPipeline(*args, **kwargs)[source]

Bases: Pipeline

Base pipeline for downloading satellite data

Parameters:

  • config – A dictionary with configuration parameters

  • raw_config – The configuration parameters pre-validation, for logging purposes only

  • args (Any) –

  • kwargs (Any) –

pydantic model Schema[source]

Bases: Schema

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:

  • output_folder_key (str)

  • postprocessing (eogrow.pipelines.download.PostprocessingRescale | None)

  • threads_per_worker (int | None)

field output_folder_key: str [Required]

Storage manager key pointing to the path where downloaded EOPatches will be saved.

Validated by:

  • validate_storage_key

field postprocessing: PostprocessingRescale | None = None

Parameters used in post-processing tasks

field threads_per_worker: int | None = None

Maximum number of parallel threads used during download by each worker. If set to None it will use 5 * N threads, where N is the number of CPUs on the machine

config: Schema
filter_patch_list(patch_list)[source]

EOPatches are filtered according to existence of specified output features

Parameters:

patch_list (List[Tuple[str, BBox]]) –

Return type:

List[Tuple[str, BBox]]

static get_postprocessing_node(postprocessing_config, previous_node)[source]

Provides node that applies postprocessing to data after download is complete

Parameters:
Return type:

EONode

build_workflow(session_loader)[source]

Method that builds a workflow

Parameters:

session_loader (Callable[[], SentinelHubSession] | None) –

Return type:

EOWorkflow

get_execution_arguments(workflow, patch_list)[source]

Adds required bbox and time_interval parameters for input task to the base execution arguments

Parameters:

  • workflow (EOWorkflow) – EOWorkflow used to download images

  • patch_list (List[Tuple[str, BBox]]) –

Return type:

Dict[str, Dict[EONode, Dict[str, object]]]

run_procedure()[source]

Execution procedure of pipeline. Can be overridden if needed.

By default, builds the workflow by using a build_workflow method, which must be additionally implemented.

Returns:

A pair of lists representing successful and unsuccessful executions.

Return type:

tuple[list[str], list[str]]

pydantic model eogrow.pipelines.download.CommonDownloadFields[source]

Bases: Schema

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:
field data_collection: DataCollection [Required]

Data collection from which data will be downloaded. See utils.validators.parse_data_collection for more info on input options.

Validated by:
field maxcc: float | None = None

Maximal cloud coverage filter.

Constraints:

  • minimum = 0

  • maximum = 1

field resampling_type: ResamplingType | None = None

A type of downsampling and upsampling used by Sentinel Hub service. Default is NEAREST

field resolution: float | None = None

Resolution of downloaded data in meters. Exactly one of the parameters resolution and size has to be specified.

field size: Tuple[int, int] | None = None

A pair (width, height) of downloaded data in pixels. . Exactly one of the parameters resolution and size has to be specified.

Validated by:

  • cannot_be_used_with_resolution

pydantic model eogrow.pipelines.download.TimeDependantFields[source]

Bases: Schema

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:
field mosaicking_order: MosaickingOrder | None = None

The mosaicking order used by Sentinel Hub service. Default is mostRecent

field time_difference: float | None = None

Time difference in minutes between consecutive time frames

field time_period: Tuple[date, date] [Required]
Validated by:
class eogrow.pipelines.download.DownloadPipeline(*args, **kwargs)[source]

Bases: BaseDownloadPipeline

Pipeline to download data via SentinelHubInputTask

Parameters:

  • config – A dictionary with configuration parameters

  • raw_config – The configuration parameters pre-validation, for logging purposes only

  • args (Any) –

  • kwargs (Any) –

pydantic model Schema[source]

Bases: Schema, CommonDownloadFields, TimeDependantFields

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:

  • additional_data (List[Feature])

  • bands (Optional[List[str]])

  • bands_feature_name (str)

  • use_dn (bool)

field additional_data: List[Feature] [Optional]

Additional data to download

field bands: List[str] | None = None

Names of bands to download

field bands_feature_name: str [Required]

Name of a feature in which bands will be saved

field use_dn: bool = False

Save bands as float32 reflectance (default), or int16 digital numbers.

config: Schema
class eogrow.pipelines.download.DownloadEvalscriptPipeline(*args, **kwargs)[source]

Bases: BaseDownloadPipeline

Pipeline to download through an evalscript

Parameters:

  • config – A dictionary with configuration parameters

  • raw_config – The configuration parameters pre-validation, for logging purposes only

  • args (Any) –

  • kwargs (Any) –

pydantic model Schema[source]

Bases: Schema, CommonDownloadFields, TimeDependantFields

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:

  • evalscript_folder_key (str)

  • evalscript_path (str)

  • features (List[Feature])

field evalscript_folder_key: str = 'input_data'

Storage manager key pointing to the path where the evalscript is loaded from.

Validated by:

  • validate_storage_key

field evalscript_path: str [Required]
field features: List[Feature] [Required]

Features to construct from the evalscript

config: Schema
class eogrow.pipelines.download.DownloadTimelessPipeline(*args, **kwargs)[source]

Bases: BaseDownloadPipeline

Pipeline to download timeless data

Parameters:

  • config – A dictionary with configuration parameters

  • raw_config – The configuration parameters pre-validation, for logging purposes only

  • args (Any) –

  • kwargs (Any) –

pydantic model Schema[source]

Bases: Schema, CommonDownloadFields

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:

  • feature_name (str)

field feature_name: str [Required]

Name of the resulting feature

config: Schema