eogrow.utils.fs

Module containing utilities for working with filesystems

class eogrow.utils.fs.BaseLocalObject(path, *, mode='r', filesystem=None, config=None, always_copy=False, **temp_fs_kwargs)[source]

Bases: object

An abstraction for working with a local version of remote objects.

If object’s original location is remote (e.g. on S3) then this will ensure working with a local copy. If object’s original location is already on local filesystem then it will work with that, unless always_copy=True is used.

Parameters:

  • path (str) – Either a full path to a remote file or a path to remote file which is relative to given filesystem object.

  • mode (str) – One of the option r’, ‘w’, and ‘rw, which specify if a file should be read or written to remote. The default is ‘r’.

  • filesystem (FS | None) – A filesystem of the remote. If not given, it will be determined from the path.

  • config (SHConfig | None) – A config object with which AWS credentials could be used to initialize a remote filesystem object.

  • always_copy (bool) – If True it will always make a local copy to a temporary folder, even if a file is already in the local filesystem.

  • temp_fs_kwargs (Any) – Parameters that will be propagated to fs.tempfs.TempFS

property path: str

Provides an absolute path to the copy in the local filesystem

close()[source]

Close the local copy

Return type:

None

copy_to_local()[source]

A public method for copying from remote to local location. It can be called anytime.

Return type:

None

copy_to_remote(raise_missing=True)[source]

Copy from local to remote location

Parameters:

raise_missing (bool) –

Return type:

None

class eogrow.utils.fs.LocalFile(path, *, mode='r', filesystem=None, config=None, always_copy=False, **temp_fs_kwargs)[source]

Bases: BaseLocalObject

An abstraction for working with a local version of a remote file.

Check BaseLocalObject for more info.

Parameters:

  • path (str) – Either a full path to a remote file or a path to remote file which is relative to given filesystem object.

  • mode (str) – One of the option r’, ‘w’, and ‘rw, which specify if a file should be read or written to remote. The default is ‘r’.

  • filesystem (FS | None) – A filesystem of the remote. If not given, it will be determined from the path.

  • config (SHConfig | None) – A config object with which AWS credentials could be used to initialize a remote filesystem object.

  • always_copy (bool) – If True it will always make a local copy to a temporary folder, even if a file is already in the local filesystem.

  • temp_fs_kwargs (Any) – Parameters that will be propagated to fs.tempfs.TempFS

class eogrow.utils.fs.LocalFolder(path, *args, filesystem=None, config=None, walker=None, workers=None, **kwargs)[source]

Bases: BaseLocalObject

An abstraction for working with a local version of a remote folder and its content.

Check BaseLocalObject for more info.

Parameters:

  • path (str) – Either a full path to a remote folder or a path to remote folder which is relative to given filesystem object.

  • args (Any) – Positional arguments propagated to the base class.

  • filesystem (FS | None) – A filesystem of the remote. If not given, it will be determined from the path.

  • config (SHConfig | None) – A config object with which AWS credentials could be used to initialize a remote filesystem object.

  • walker (Walker | None) – An instance of fs.walk.Walker object used to configure advanced copying parameters. It is possible to set max folder depth of copy (default is entire tree), how to handle errors (by default are ignored), and what to include or exclude.

  • workers (int | None) – A maximal number of threads used for parallel copies between local and remote locations. The default is 5 times the number of CPUs.

  • kwargs (Any) – Keyword arguments propagated to the base class.