eogrow.core.logging

Implementation of LoggingManager and different handlers used for logging.

class eogrow.core.logging.LoggingManager(config, storage)[source]

Bases: EOGrowObject

A class that manages logging specifics

Parameters:

  • config (Schema) – A configuration file

  • storage (StorageManager) – An instance of StorageManager class

pydantic model Schema[source]

Bases: ManagerSchema

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Fields:

  • capture_warnings (bool)

  • eoexecution_ignore_packages (Tuple[str, ...])

  • include_logs_to_report (bool)

  • pipeline_ignore_packages (Tuple[str, ...])

  • pipeline_logs_backup_interval (float)

  • save_logs (bool)

  • show_logs (bool)

  • stdout_log_packages (Tuple[str, ...])

field capture_warnings: bool = True

If warnings should be treated as logs and with save_logs=True written into log files instead of being printed in stderr.

field eoexecution_ignore_packages: Tuple[str, ...] = ('...',)

Names of packages for which the logs will not be written to EOExecution log files. You can reference the defaults with “…”, for example, adding another package can be done with [”…”, “some_package”]

field include_logs_to_report: bool = False

If log files should be parsed into an EOExecution report file or just linked. When working with larger number of EOPatches the recommended option is False.

field pipeline_ignore_packages: Tuple[str, ...] = ('...',)

Names of packages for which the logs will not be written to the main pipeline log file. You can reference the defaults with “…”, for example, adding another package can be done with [”…”, “some_package”]

field pipeline_logs_backup_interval: float = 60

When working with a remote storage this parameter defines a minimal number of seconds between two consecutive times when pipeline log file will be copied into the remote storage.

field save_logs: bool = False

A flag to determine if pipeline logs and reports will be saved to logs folder. This includes potential EOExecution reports and logs.

field show_logs: bool = False

Shows basic pipeline execution logs at stdout.

field stdout_log_packages: Tuple[str, ...] = ('...',)

Names of packages for which the logs will be written to stdout. You can reference the defaults with “…”, for example, adding another package can be done with [”…”, “package_to_display”]

config: Schema
get_pipeline_logs_folder(pipeline_execution_name, full_path=False)[source]

Provides path to the folder where logs of this pipeline execution will be saved

Parameters:

  • pipeline_execution_name (str) – Name of current pipeline execution

  • full_path (bool) – If it should provide a full absolute path or a path relative to the filesystem object

Return type:

str

start_logging(logger, pipeline_execution_name, filename)[source]

Creates a folder for logs and sets up (and returns) logging handlers

Supported handlers: - Writing to a file in pipeline logs folder - Printing logs to a standard output

Parameters:

  • logger (Logger) –

  • pipeline_execution_name (str) –

  • filename (str) –

Return type:

list[logging.Handler]

stop_logging(logger, handlers)[source]

Updates logs, removes pipeline handlers from the global logger and puts global logging level back to default

Parameters:

  • logger (Logger) –

  • handlers (list[logging.Handler]) –

Return type:

None

update_pipeline_report(pipeline_execution_name, pipeline_config, pipeline_raw_config, pipeline_id, pipeline_timestamp, elapsed_time=None)[source]

A method in charge of preparing a report about pipeline run.

Content of a report:

  • pipeline configuration parameters,

  • pipeline execution stats,

  • versions of Python and Python packages,

  • information about a compute instance on which the pipeline is running.

Parameters:

  • pipeline_execution_name (str) –

  • pipeline_config (Schema) –

  • pipeline_raw_config (RawConfig | None) –

  • pipeline_id (str) –

  • pipeline_timestamp (str) –

  • elapsed_time (float | None) –

Return type:

None

save_eopatch_execution_status(pipeline_execution_name, finished, failed)[source]

Saves lists of EOPatch names for which execution either finished successfully or failed

Parameters:

  • pipeline_execution_name (str) –

  • finished (list[str]) –

  • failed (list[str]) –

Return type:

None

class eogrow.core.logging.FilesystemHandler(path, filesystem, encoding='utf-8', **kwargs)[source]

Bases: FileHandler

A filesystem abstraction of FileHandler

In case the handler gets a local path it behaves the same as FileHandler. In case it gets a remote path it writes logs first to a local path and then copies them to the remote location.

IMPORTANT: This handler will by default have an extra FilesystemFilter which will ignore logs from packages that produce logs during LocalFile.copy_to_remote call. Otherwise, a log that would be created within an emit call would be recursively sent back to the handler. That would either trigger an infinite recursion or make the process stuck waiting for a thread lock release.

Parameters:

  • path (str) – A path to a log file that is relative to the given filesystem object.

  • filesystem (FS | bytes) – A filesystem to where logs will be written. It can either be an instance of a filesystem object or its pickled copy.

  • encoding (str | None) – Encoding used to write log files.

  • kwargs (Any) – Keyword arguments that will be propagated to FileHandler.

close()[source]

Closes logging and closes the local file

Return type:

None

class eogrow.core.logging.RegularBackupHandler(path, filesystem, backup_interval, **kwargs)[source]

Bases: FilesystemHandler

A customized FilesystemHandler that makes a copy to a remote location regularly after given amount of time.

Parameters:

  • path (str) – A path to a log file that is relative to the given filesystem object.

  • filesystem (FS | bytes) – A filesystem to where logs will be written. It can either be an instance of a filesystem object or its pickled copy.

  • backup_interval (float | int) – A minimal number of seconds before handler will back up the log file to the remote location. The backup will only happen when the next log record will be emitted.

  • kwargs (Any) –

emit(record)[source]

Save a new record and backup to remote if the backup hasn’t been done in the given amount of time.

Parameters:

record (LogRecord) –

Return type:

None

class eogrow.core.logging.EOExecutionHandler(path, filesystem, encoding='utf-8', **kwargs)[source]

Bases: FilesystemHandler

A customized FilesystemHandler that makes a copy to a remote location every time a new node in a workflow is started.

Parameters:

  • path (str) – A path to a log file that is relative to the given filesystem object.

  • filesystem (FS | bytes) – A filesystem to where logs will be written. It can either be an instance of a filesystem object or its pickled copy.

  • encoding (str | None) – Encoding used to write log files.

  • kwargs (Any) – Keyword arguments that will be propagated to FileHandler.

emit(record)[source]

Save a new record. In case a new node in EOWorkflow is started it will copy the log file to remote.

Parameters:

record (LogRecord) –

Return type:

None

class eogrow.core.logging.FilesystemFilter(name='')[source]

Bases: Filter

The sole purpose of this filter is to capture any log that happens during LocalFile.copy_to_remote call. Any log that would not be captured would break the entire runtime.

Initialize a filter.

Initialize with the name of the logger which, together with its children, will have its events allowed through the filter. If no name is specified, allow every event.

IGNORE_HARMFUL_LOGS = ('botocore', 'boto3.resources', 's3transfer')
filter(record)[source]

Ignores logs from certain low-level packages

Parameters:

record (LogRecord) –

Return type:

bool

class eogrow.core.logging.StdoutFilter(*args, log_packages, **kwargs)[source]

Bases: Filter

Filters log messages passed to standard output

Parameters:

  • log_packages (Sequence[str]) – Names of packages which logs to include.

  • args (Any) –

  • kwargs (Any) –

DEFAULT_LOG_PACKAGES = ('eogrow', '__main__', 'root', 'sentinelhub.api.batch')
filter(record)[source]

Shows only logs from eo-grow type packages and high-importance logs

Parameters:

record (LogRecord) –

Return type:

bool

class eogrow.core.logging.LogFileFilter(*args, ignore_packages, **kwargs)[source]

Bases: Filter

Filters log messages passed to log file

Parameters:

  • ignore_packages (Sequence[str]) – Names of packages which logs will be ignored.

  • args (Any) –

  • kwargs (Any) –

DEFAULT_IGNORE_PACKAGES = ('eolearn.core', 'botocore', 's3transfer', 'matplotlib', 'fiona', 'rasterio', 'graphviz', 'urllib3', 'boto3')
filter(record)[source]

Shows everything from the main thread and process except logs from packages that are on the ignore list. Those packages send a lot of useless logs.

Parameters:

record (LogRecord) –

Return type:

bool

class eogrow.core.logging.EOExecutionFilter(*args, ignore_packages, **kwargs)[source]

Bases: Filter

Filters logs that will be saved by EOExecutor

Parameters:

  • ignore_packages (Sequence[str]) – Names of packages which logs will be ignored.

  • args (Any) –

  • kwargs (Any) –

DEFAULT_IGNORE_PACKAGES = ('botocore', 's3transfer', 'urllib3', 'rasterio', 'numba', 'fiona.ogrext', 'fiona.env', 'fiona._env')
filter(record)[source]

Ignores logs from certain low-level packages

Parameters:

record (LogRecord) –

Return type:

bool