pygobbler package

Submodules

pygobbler.allocate_upload_directory module

pygobbler.allocate_upload_directory.allocate_upload_directory(staging)[source]

Allocate a subdirectory in the staging directory to prepare files for upload via upload_directory().

Parameters:

staging (str) – Path to the staging directory.

Return type:

str

Returns:

Path to a new subdirectory for staging uploads.

pygobbler.approve_probation module

pygobbler.approve_probation.approve_probation(project, asset, version, staging, url)[source]

Approve a probational upload of a version of a project’s asset.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset of the project.

  • version (str) – The name of the version of the asset to approve.

  • staging (str) – Path to the staging directory.

  • url (str) – URL to the Gobbler REST API.

pygobbler.clone_version module

pygobbler.clone_version.clone_version(project, asset, version, destination, registry)[source]

Clone the directory structure for a versioned asset into a separate location. This is typically used to prepare a new version for a lightweight upload.

More specifically, cloning involves creating a directory at destination that has the same structure as that of the specified project-asset-version. All files in the version are represented as symlinks from destination to the corresponding file in the registry.The idea is that, when destination is used in upload_directory(), the symlinks are converted into upload links. This allows users to create new versions very cheaply as duplicate files are not stored in the backend.

Users can more-or-less do whatever they want inside the cloned destination, but the symlink targets should be read-only as they refer to immutable files in the registry. If a file in destination needs to be modified, the symlink should be deleted and replaced with a new file.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset of the project.

  • version (str) – The name of the version of the asset to clone.

  • destination (str) – Path to a destination directory at which to create the clone.

  • registry (str) – Path to the registry.

pygobbler.create_project module

pygobbler.create_project.create_project(project, staging, url, owners=None, uploaders=None)[source]

Create a new project in the registry.

Parameters:

  • project (str) – Name of the project to create.

  • staging (str) – Path to the staging directory.

  • url (str) – URL for the Gobbler REST API.

  • owners (Optional[List]) – List of user IDs of the owners of this project. If not provided, the current user will be set as the sole owner.

  • uploaders (Optional[List]) – List specifying the authorized uploaders for this project. See the uploaders field in fetch_permissions() return value for the expected format.

Returns:

On success, the requested project is created in the registry.

pygobbler.fetch_directory module

pygobbler.fetch_directory.fetch_directory(path, registry, url, cache=None, force_remote=False, overwrite=False, concurrent=1)[source]

Obtain the path to a directory in the registry. This may create a local copy of the subdirectory’s contents if the caller is not on the same filesystem as the registry.

Parameters:

  • path (str) – Relative path to a subdirectory within the registry. This usually takes the form of PROJECT/ASSET/VERSION/*.

  • registry (str) – Path to the registry.

  • url (str) – URL to the Gobbler REST API. Only used for remote queries.

  • cache (Optional[str]) – Path to a cache directory. If None, an appropriate location is automatically chosen. Only used for remote access.

  • force_remote (bool) – Whether to force remote access. This will download all files in the path via the REST API and cache them locally, even if registry is present on the same filesystem.

  • overwrite (bool) – Whether to overwrite existing files in the cache.

  • concurrent (int) – Number of concurrent downloads.

Return type:

str

Returns:

Path to the subdirectory on the caller’s filesystem. This is either a path to the registry if it is accessible, or a path to a local cache of the registry’s contents otherwise.

pygobbler.fetch_file module

pygobbler.fetch_file.fetch_file(path, registry, url, cache=None, force_remote=False, overwrite=False)[source]

Obtain the path to a file in the registry. This may create a local copy if the caller is not on the same filesystem as the registry.

Parameters:

  • path (str) – Relative path to a file within the registry. This usually takes the form of PROJECT/ASSET/VERSION/*.

  • registry (str) – Path to the registry.

  • url (str) – URL to the Gobbler REST API. Only used for remote queries.

  • cache (Optional[str]) – Path to a cache directory. If None, an appropriate location is automatically chosen. Only used for remote access.

  • force_remote (bool) – Whether to force remote access. This will download all files in the path via the REST API and cache them locally, even if registry is present on the same filesystem.

  • overwrite (bool) – Whether to overwrite existing files in the cache.

  • concurrent – Number of concurrent downloads.

Return type:

str

Returns:

Path to the file on the caller’s filesystem. This is either a path to the file in the registry if it is accessible, or a path to a local cache of the registry’s contents otherwise.

pygobbler.fetch_latest module

pygobbler.fetch_latest.fetch_latest(project, asset, registry, url, force_remote=False)[source]

Fetch the latest version of an asset of a project.

Parameters:

  • project (str) – Name of a project.

  • asset (str) – Name of an asset in the project.

  • registry (str) – Path to the Gobbler registry.

  • url (str) – URL of the REST API. Only used for remote queries.

  • force_remote (bool) – Whether to force a remote query via url, even if the registry is present on the current filesystem.

Return type:

Optional[str]

Returns:

The name of the latest version of the asset, or None if no latest version exists.

pygobbler.fetch_manifest module

pygobbler.fetch_manifest.fetch_manifest(project, asset, version, registry, url, cache=None, force_remote=False, overwrite=False)[source]

Fetch the manifest for a version of a project asset.

Parameters:

  • project (str) – Name of a project.

  • asset (str) – Name of an asset in the project.

  • version (str) – Name of a version of the asset.

  • registry (str) – Path to the Gobbler registry.

  • url (str) – URL of the REST API. Only used for remote queries.

  • cache (Optional[str]) – Path to a cache directory. If None, a default cache location is selected. Only used for remote queries.

  • force_remote (bool) – Whether to force a remote query via url, even if the registry is present on the current filesystem.

  • overwrite (bool) – Whether to overwrite existing entries in the cache. Only used for remote queries.

Return type:

Dict[str, Any]

Returns:

Dictionary containing the manifest. Each key is a relative path to a file in this version of the project asset, and each value is a dictionary with the following fields:

  • size, integer specifying the size of the file in bytes.

  • md5sum, string containing the file’s hex-encoded MD5 checksum.

  • link (optional): a list specifying the link destination for a file. This contains the strings project, asset, version and path; if the link destination is also a link, an ancestor dictionary will be present containing the final location of the file after resolving all intermediate links.

pygobbler.fetch_permissions module

pygobbler.fetch_permissions.fetch_permissions(project, registry, url, asset=None, force_remote=False)[source]

Fetch permissions for a project.

Parameters:

  • project (str) – Name of a project.

  • registry (str) – Path to the Gobbler registry.

  • url (str) – URL of the REST API. Only used for remote queries.

  • asset (Optional[str]) – Name of the asset inside the project. If supplied, permissions are returned for this asset rather than the entire project.

  • force_remote (bool) – Whether to force a remote query via url, even if the registry is present on the current filesystem.

Return type:

Dict[str, Any]

Returns:

A dictionary containing the permissions for the project. This contains owners, a list of strings with the user IDs of the owners of this project; and uploaders, a list of dictionaries where each dictionary has the following fields:

  • id, string containing a user ID that is authorized to upload.

  • asset (optional), string containing the name of the asset that the uploader is allowed to upload to. If not present, there is no restriction on the uploaded asset name.

  • version (optional), string containing the name of the version that the uploader is allowed to upload to. If not present, there is no restriction on the uploaded version name.

  • until (optional), string containing the expiry date of this authorization in Internet Date/Time format. If not provided, the authorization does not expire.

  • trusted (optional), whether the uploader is trusted. If not provided, defaults to false.

The top-level dictionary may also contain global_write, a boolean indicating whether global writes are supported. If true, any user may create a new asset in this project, and each user can upload new versions to any asset they created under this mode.

If asset is provided, the returned dictionary contains owners and uploaders to describe the owners and uploaders, respectively, for the specified asset.

pygobbler.fetch_summary module

pygobbler.fetch_summary.fetch_summary(project, asset, version, registry, url, cache=None, force_remote=False, overwrite=False)[source]

Fetch the summary for a particular version of a project asset.

Parameters:

  • project (str) – Name of a project.

  • asset (str) – Name of an asset in the project.

  • version (str) – Name of a version of the asset.

  • registry (str) – Path to the Gobbler registry.

  • url (str) – URL of the REST API. Only used for remote queries.

  • cache (Optional[str]) – Path to a cache directory. If None, a default cache location is selected. Only used for remote queries.

  • force_remote (bool) – Whether to force a remote query via url, even if the registry is present on the current filesystem.

  • overwrite (bool) – Whether to overwrite existing entries in the cache. Only used for remote queries.

Returns:

  • uploader_user_id, string containing the UID of the uploading user.

  • upload_start_time, string containing the upload start time in Internet Date/Time format.

  • upload_finish_time, string containing the upload finish time in Internet Date/Time format.

  • on_probation (optional), boolean indicating whether this version of the asset is a probational upload.

Return type:

Dictionary containing summary information including

pygobbler.fetch_usage module

pygobbler.fetch_usage.fetch_usage(project, registry, url, force_remote=False)[source]

Fetch the disk usage of a project.

Parameters:

  • project (str) – Name of a project.

  • registry (str) – Path to the Gobbler registry.

  • url (str) – URL of the REST API.

  • force_remote (bool) – Whether to force a remote query via url, even if the registry is present on the current filesystem.

Return type:

Optional[str]

Returns:

The current usage (in bytes) for this project.

pygobbler.list_assets module

pygobbler.list_assets.list_assets(project, registry, url, force_remote=False)[source]

List all assets in a project.

Parameters:

  • project (str) – The name of the project.

  • registry (str) – Path to the registry.

  • url (str) – URL to the Gobbler REST API. Only used for remote access.

  • force_remote (bool) – Whether to force remote access via the API, even if registry is on the same filesystem as the caller.

Return type:

List[str]

Returns:

List of strings containing the project names.

pygobbler.list_files module

pygobbler.list_files.list_files(project, asset, version, registry, url, prefix=None, include_dotdot=True, force_remote=False)[source]

List the contents of a version of a project asset.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset in project.

  • version (str) – The name of the version of the asset.

  • registry (str) – Path to the registry.

  • url (str) – URL to the Gobbler REST API. Only used for remote access.

  • prefix (Optional[str]) – Prefix for the path within this version’s subdirectory. If provided, files are only listed if they have a relative path (i.e., inside the version subdirectory) that starts with this prefix. If None, all files associated with this version are listed.

  • include_dotdot (bool) – Whether to list files with path components that start with ...

  • force_remote (bool) – Whether to force remote access via the API, even if registry is on the same filesystem as the caller.

Return type:

List[str]

Returns:

List of strings containing the relative paths of files associated with the versioned asset. All paths will start with prefix if provided.

pygobbler.list_projects module

pygobbler.list_projects.list_projects(registry, url, force_remote=False)[source]

List all projects in the registry.

Parameters:

  • registry (str) – Path to the registry.

  • url (str) – URL to the Gobbler REST API. Only used for remote access.

  • force_remote (bool) – Whether to force remote access via the API, even if registry is on the same filesystem as the caller.

Return type:

List[str]

Returns:

List of strings containing the project names.

pygobbler.list_projects.list_registry_directories(path, registry, url, force_remote)[source]
Return type:

List[str]

pygobbler.list_versions module

pygobbler.list_versions.list_versions(project, asset, registry, url, force_remote=False)[source]

List all versions of a project asset.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset in project.

  • registry (str) – Path to the registry.

  • url (str) – URL to the Gobbler REST API. Only used for remote access.

  • force_remote (bool) – Whether to force remote access via the API, even if registry is on the same filesystem as the caller.

Return type:

List[str]

Returns:

List of strings containing the project names.

pygobbler.refresh_latest module

pygobbler.refresh_latest.refresh_latest(project, asset, staging, url)[source]

Recompute the latest version of a project’s asset. This is useful on rare occasions where multiple simultaneous uploads cause the latest version to be slightly out of sync.

Parameters:

  • project (str) – Name of the project.

  • asset (str) – Name of the asset.

  • staging (str) – Path to the staging directory.

  • url (str) – URL of the gobbler REST API.

Return type:

Optional[str]

Returns:

Latest version of the project, or None if there is no non-probational version.

pygobbler.refresh_usage module

pygobbler.refresh_usage.refresh_usage(project, staging, url)[source]

Recompute the quota usage of a project. This is useful on rare occasions where multiple simultaneous uploads cause the usage calculations to be out of sync.

Parameters:

  • project (str) – Name of the project.

  • staging (str) – Path to the staging directory.

  • url (str) – URL of the gobbler REST API.

Return type:

int

Returns:

Total quota usage of this project, in bytes.

pygobbler.reindex_version module

pygobbler.reindex_version.reindex_version(project, asset, version, staging, url)[source]

Reindex a version of a project asset in the registry. This regenerates all of the internal ..manifest and ..links files.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset of the project.

  • version (str) – The name of the version of the asset to remove.

  • staging (str) – Path to the staging directory.

  • url (str) – URL to the Gobbler REST API.

Returns:

The requested version is reindexed.

pygobbler.reject_probation module

pygobbler.reject_probation.reject_probation(project, asset, version, staging, url, force=False)[source]

Reject a probational upload of a version of a project’s asset.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset of the project.

  • version (str) – The name of the version of the asset to reject.

  • staging (str) – Path to the staging directory.

  • url (str) – URL to the Gobbler REST API.

  • force (bool) – Whether the version should be forcibly rejected and removed if it contains invalid files. If this needs to be set to True, users may need to call refresh_usage`() afterwards to correct project-level usage statistics.

pygobbler.remove_asset module

pygobbler.remove_asset.remove_asset(project, asset, staging, url, force=False)[source]

Remove an asset of a project from the registry. This should only be performed by Gobbler instance administrators. Consider running reroute_links() beforehand to avoid dangling references to files in this asset.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset to remove.

  • staging (str) – Path to the staging directory.

  • url (str) – URL to the Gobbler REST API.

  • force (bool) – Whether the asset should be forcibly removed if it contains invalid files. If this needs to be set to True, users may need to call refresh_usage`() afterwards to correct project-level usage statistics.

pygobbler.remove_project module

pygobbler.remove_project.remove_project(project, staging, url)[source]

Remove a project from the registry. This should only be performed by Gobbler instance administrators. Consider running reroute_links() beforehand to avoid dangling references to files in this project.

Parameters:

  • project (str) – String containing the project to remove.

  • staging (str) – Path to the staging directory.

  • url (str) – URL for the Gobbler REST API.

pygobbler.remove_version module

pygobbler.remove_version.remove_version(project, asset, version, staging, url, force=False)[source]

Remove a version of a project asset from the registry. This should only be performed by Gobbler instance administrators. Consider running reroute_links() beforehand to avoid dangling references to files in this version.

Parameters:

  • project (str) – The name of the project.

  • asset (str) – The name of the asset of the project.

  • version (str) – The name of the version of the asset to remove.

  • staging (str) – Path to the staging directory.

  • force (bool) – Whether the version should be forcibly removed if it contains invalid files. If this needs to be set to True, users may need to call refresh_usage`() afterwards to correct project-level usage statistics. Similarly, refresh_latest() may also need to be called.

  • url (str) – URL to the Gobbler REST API.

pygobbler.service_info module

pygobbler.service_info.service_info(url)[source]

Get information about the Gobbler service, namely the locations of the staging directory and registry.

Parameters:

url (str) – URL of the gobbler REST API.

Return type:

Dict[str, Any]

Returns:

Dictionary containing the location of the staging and registry directories.

pygobbler.set_permissions module

pygobbler.set_permissions.set_permissions(project, registry, staging, url, asset=None, owners=None, uploaders=None, global_write=None, append=True)[source]

Set the owner and uploader permissions for a project.

Parameters:

  • project (str) – Name of the project.

  • registry (str) – Path to the Gobbler registry.

  • staging (str) – Path to the staging directory.

  • url (str) – URL of the REST API.

  • asset (Optional[str]) – Name of the asset inside the project. If supplied, permissions are set on this asset rather than the entire project.

  • owners (Optional[List]) – List of user IDs for owners of this project. If None, no change is made to the existing owners in the project permissions.

  • uploaders (Optional[Dict]) – List of dictionaries specifying the authorized uploaders for this project. See the uploaders field in the return value of fetch_permissions() for the expected format. If None, no change is made to the existing uploaders.

  • global_write (Optional[bool]) – Whether to enable global writes for this project, see the global_write field in the return value of fetch_permissions() for more details. If None, no change is made to the global write status. Ignored if asset is provided.

  • append (bool) – Whether owners and uploaders should be appended to the existing owners and uploaders, respectively. If False, the owners and uploaders are used to replace the existing values in the project permissions.

pygobbler.start_gobbler module

pygobbler.start_gobbler.start_gobbler(staging=None, registry=None, port=None, wait=1, version='0.4.0', overwrite=False)[source]

Start a test Gobbler service.

Parameters:

  • registry (Optional[str]) – Path to a registry directory. If None, a temporary directory is automatically created.

  • staging (Optional[str]) – Path to a registry directory. If None, a temporary directory is automatically created.

  • port (Optional[int]) – Port number for the Gobbler API to receive requests. If None, an open port is automatically chosen.

  • wait (float) – Number of seconds to wait for the service to initialize before use.

  • version (str) – Version of the service to run.

  • overwrite (bool) – Whether to overwrite the existing Gobbler binary.

Return type:

Tuple[bool, str, str, str]

Returns:

A tuple indicating whether a new test service was created (or an existing instance was re-used), the path to the staging directory, the path to the registry, and the chosen URL. If a service is already running, this function is a no-op and the configuration details of the existing service will be returned.

pygobbler.start_gobbler.stop_gobbler()[source]

Stop any gobbler test service started by start_gobbler(). If no test service was running, this function is a no-op.

pygobbler.unpack_path module

pygobbler.unpack_path.unpack_path(path)[source]

Unpack a path to its combination of project, asset, version, and (optionally) path, for easier use in the various pygobbler functions.

Parameters:

path (str) – Relative path within the Gobbler registry.

Return type:

Dict[str, str]

Returns:

Dictionary with the project, asset, version and path keys. All values are strings except for path, which may be None.

pygobbler.upload_directory module

pygobbler.upload_directory.upload_directory(project, asset, version, directory, staging, url, probation=False, consume=None, ignore_dot=True)[source]

Upload a directory as a new versioned asset of a project in the registry.

Parameters:

  • project (str) – The name of an existing project.

  • asset (str) – The name of a new or existing asset in project.

  • version (str) – The name of a new version of asset.

  • directory (str) – Path to a directory to be uploaded. For best performace, this should be a subdirectory of staging, e.g., as created by allocate_upload_directory().

  • staging (str) – Path to the staging directory.

  • url (str) – URL for the Gobbler REST API.

  • probation (bool) – Whether to upload a probational version.

  • consume (Optional[bool]) – Whether the contents of directory can be consumed by the upload process. If true, the Gobbler will attempt to move files from directory into the registry. Otherwise, the contents of directory will not be modified by the upload. Defaults to true if the contents of directory need to be copied to staging.

  • ignore_dot (bool) – Whether to skip dotfiles in directory during upload.

pygobbler.version_path module

pygobbler.version_path.version_path(project, asset, version, **kwargs)[source]

Obtain the path to a versioned asset in the registry.

Parameters:

  • project – The name of the project.

  • asset – The name of the asset of the project.

  • version – The name of the version of the asset to obtain.

  • kwargs – Further arguments to fetch_directory().

Return type:

str

Returns:

The path to the directory containing the desired version.

Module contents