Resources

class sane.resources.Resource

A quantifiable positive integer resource

Resource is a wrapper class on quatifiable values to facilitate common operations such as basic arithmetic, reduction to human-readable scaled units (in binary metric prefix, e.g. kibi, mebi, etc.), and type-checking operations between matching resource types or scalars.

Resource.amount can be set to anything that follows the following regular expression:

(\d+)(k|m|g|t)?(b|w)?

The first capture group (1st set of parenthesis) expects any number of numeric literals.

The second capture group (2nd set of parenthesis) optionally can be a binary scale

The third capture group (3rd set of parenthesis) optionally can be a unit (limited to ‘b’ or ‘w’ currently)

Examples of valid and invalid resource amounts:

# valid
1
8k
512mb

# invalid
1.0
-2
7.6gw

All supported binary operations on a Resource return a new Resource with the resultant amount except the division of two resources which results in an int value. The following operations are supported:

res_a = sane.resource.Resource( "mem", "4gb" )
res_b = sane.resource.Resource( "mem", "2gb" )
# addition and subtraction
res_a + res_b or res_a - res_b
res_a + 12345 or res_a - 12345
# multiplication and division
res_a * 2 or res_a / 2
             res_a / res_b # Note: This results in an int and mult is not supported

res_c = sane.resource.Resource( "cpus", 12 )
# Unsupported operations
res_a + res_c         # incompatible resouce type
res_a + 1.2345        # float add/sub
res_a - 1.2345        # float add/sub
res_a * res_b         # undefined behavior
res_a - ( res_a * 2 ) # negative amount not allowed

Note

Resource.amount must be an integer value, even if scaled. The total or current values will always be integer values, even if unscaled or multiplied/divided by a float value.

Scaling amounts:

prefix	multiplier
k	1024
m	1024 2
g	1024 3
t	1024 4

__init__(resource, amount=0, unit='')

Create a Resource with type resource

Parameters:

• resource (str) – Set the type of the resource. Resources of different types cannot interoperate.

• amount – Set the amount that this resource is. See class summary for valid syntax.

• unit – If set, overrides the detected unit (if any) passed in via amount

static is_resource(potential_resource)

Check if the input value (str or int) follows valid Resource.amount syntax

property resource

The type of this resource

property unit

The unit of this resource

property amount

The original amount of this resource

Parameters:

amount – If set, changes this resources amount entirely, but not the resource type.

property total: int

The total unscaled (expanded) numeric value

property current: int

The total

property total_str

The total scaled (reduced) value, including units

property current_str

The current scaled (reduced) value, including units

class sane.resources.AcquirableResource

Bases: Resource

A Resource that also tracks the amount acquired internally

This class internally contains another Resource that tracks the acquirable amount of the resouce. The original acquirable amount always matches the amount set at instantiation, and setting amount afterwards results in undefined behavior.

All supported binary operations on an AcquirableResource return a new AcquirableResource and operate on the underlying acquirable amount. The AcquirableResource.amount is unaffected.

For example:

res_a = sane.resource.Resource( "mem", "4gb" )
res_b = sane.resource.Resource( "mem", "1gb" )

res_c = res_a - res_b
res_c.current_str # outputs "3gb"
res_c.total_str   # outputs the original "4gb"

__init__(resource, amount)

Create a Resource with type resource

Parameters:

• resource – Set the type of the resource. Resources of different types cannot interoperate.

• amount – Set the amount that this resource is. See class summary for valid syntax.

• unit – If set, overrides the detected unit (if any) passed in via amount

acquirable

The underlying Resource tracking the amount of resource currently acquirable

property current

The current acquirable amount

Parameters:

amount – If set, changes the acquirable.amount to this value.

property used

The total used amount of resources, total - acquirable.total

property used_str

The used scaled (reduced) value, including units

class sane.resources.ResourceRequestor

Bases: OptionLoader

Aggregates any arbitrary resource requests to be made to a ResourceProvider

Note

Resources listed here are stored verbatim and thus can be anything. The onus of allocation of resources is left to the ResourceProvider implementation. The default classes only support resources that would match the Resource syntax.

User Interface

User Methods

add_resource_requirements(resource_dict)

Add resource requirements to this requestor

Hint

See Resource for syntax on default supported values

Add an arbitrary dict of key-value pairs to this requestor. The key-value pairs in this dict will eventually be requested from a ResourceRequestor for:

• acquire_resources()

• release_resources()

• resources_available()

If the value in a key-value pair is of type dict, it will be considered an override resource request specific to the name of the key. This override dict will be kept separately in an internal location to be used later. Multiple nested dict overrides are not allowed.

See resources() for more info.

As an example:

{
  "cpus"  : 12,
  "mem"   : "1gb",
  "slots" : 1
  "specific_provider" :
  {
    "cpus" : 36,
    "mem"  : "3gb"
  }
}

The values of "specific_provider" will only be used (in addition to any unmodified values in the top-level dict) if the current_host matches this key.

Note

While the resource_dict can be arbitrary values, it is on the ResourceProvider (i.e. sane.Host) to be able to provide these resources.

Notably, any resources that do not follow the Resource syntax are left strictly to the provider class implementation. The default classes do not support values outside of Resource unless specified.

Parameters:

resource_dict (dict)

User Attributes & Properties

resources(override=None)

Return a copy of the current resources requested

During workflow execution, the current_host will be provided as the override value when requesting resources from the sane.Host via resources.ResourceProvider.acquire_resources()

Parameters:

override (str) – If an override dict exists in the resources, return a copy of the resources recursively updated to prioritize the overriden values.

Return type:

dict

local

Control if the resources requested should be provided from a local pool if a NonLocalProvider is used. If set to None then resource delegation is left to the provider, True to force local, and False to force non-local. If the provider is a normal ResourceProvider this option has no effect.

Internal API

load_core_options(options, origin)

From OptionLoader.load_core_options():

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From ResourceRequestor.load_core_options():

Load ResourceRequestor resource requirements

The following key is loaded verbatim into add_resource_requirements():

• "resources"

The following key is loaded if possible, defaulting to None:

• "local" => local

Parameters:

• options (dict)

• origin (str)

class sane.resources.ResourceProvider

Bases: OptionLoader

Manages and provides use of AcquirableResources

During workflow execution the ResourceProvider will be given the resources of each runnable ResourceRequestor (Action) to check for availability, acquire, and finally release the resources at completion.

User Interface

User Methods

add_resources(resource_dict, override=False)

Add resources to this provider that can be acquired

Hint

See Resource for syntax on default supported values

Add a dict of key-value pairs to this provider. The key-value pairs in this dict will be used to create AcquirableResource instances that track resource requests from a ResourceRequestor.

An example options dict

{
  "cpus"  : 12,
  "mem"   : "1gb",
  "slots" : 2
}

The above resource_dict would provide 12 counts of "cpus", 1024 ³ b units of "mem", and 2 counts of "slots".

Important

The ResourceProvider has no inherent understanding of the units, amounts, or names of resources. Internally, resources will be tracked but at acquisition these resources will not correspond to any form of real hardware / software allocations or locks unless logic within a custom implementation of a derived ResourceProvider specifies.

Parameters:

resource_dict (dict)

User Attributes & Properties

property resources: Dict[str, AcquirableResource]

A copy of the available resources

Returns:

a deep copy of the internal resources in their current state

Return type:

dict[str, AcquirableResource]

Internal API

load_core_options(options, origin)

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From ResourceProvider.load_core_options():

Load the available resources for this ResourceProvider

The following key is loaded verbatim into add_resources() and thus should use the same amount syntax:

• "resources"

The following key is loaded into the internal _mapper as a dict[str,list[str]] via add_mapping(), where each key in the dict is a resource name/type and the value is a list of strings to use as aliases:

• "mapping"

An example options dict

{
  "resources" :
  {
    "cpus" : 123,
    "mem"  : "64mb",
  }
  "mapping" : 
  {
    "ncpus" : [ "cpus", "cpu", "procs", "proc", "processors" ]
  }
}

The above options would provide map "cpus" to "ncpus" and add an AcquirableResource of type "ncpus" with amount 123 and an AcquirableResource of type "mem" with amount 64mb to the resources.

Note

When using a mapping, the ResourceRequestor and ResourceProvider can use any of the alias names or map key itself in its resources. Within the ResourceProvider, all resources will always be internally mapped to the corresponding map key if available.

This allows the creation of ResourceRequestor and ResourceProvider that for one reason or another wish to refer to the same resources by different names.

resources_available(resource_dict, requestor, log=True)

Check if the all resources in the requested resource_dict are currently available

Parameters:

• resource_dict (dict)

• requestor (ResourceRequestor)

Return type:

bool

acquire_resources(resource_dict, requestor)

Acquire all the resources in the resource_dict

Parameters:

• resource_dict (dict)

• requestor (ResourceRequestor)

release_resources(resource_dict, requestor)

Release all the resources in the resources_dict

Parameters:

• resource_dict (dict)

• requestor (ResourceRequestor)

class sane.resources.NonLocalProvider

Bases: ResourceProvider

An abstract base class specialization of ResourceProvider

This class introduces the concept of a “local” pool of resources separate from the rest of the resources this provider offers. This nomenclature suggests that all resources directly in this class (not in the local_resources) are thus nonlocal.

The internal local pool is itself a ResourceProvider. The local pool and this instance share a common ResourceMapper.

User Interface

User Attributes & Properties

local_resources

An internal ResourceProvider to manage local resource requests

default_local

Set whether ResourceRequestor with local set to None should default to using local_resources

force_local

Set to force all ResourceRequestor to acquire from local_resources, regardless of local setting

Internal API

load_core_options(options, origin)

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From NonLocalProvider.load_core_options():

Load local resources into local_resources and control flags

The following key is loaded verbatim into local_resources via add_resources():

• "local_resources"

The following keys are loaded, if available, into their respective attribute. If not present, the attributes are unmodified:

• "default_local" => default_local

• "force_local" => force_local

An example options dict

{
  "local_resources" :
  {
    "cpus" : 4,
    "mem"  : "2gb"
  },
  "default_local" : True
}

From ResourceProvider.load_core_options():

Load the available resources for this ResourceProvider

The following key is loaded verbatim into add_resources() and thus should use the same amount syntax:

• "resources"

The following key is loaded into the internal _mapper as a dict[str,list[str]] via add_mapping(), where each key in the dict is a resource name/type and the value is a list of strings to use as aliases:

• "mapping"

An example options dict

{
  "resources" :
  {
    "cpus" : 123,
    "mem"  : "64mb",
  }
  "mapping" : 
  {
    "ncpus" : [ "cpus", "cpu", "procs", "proc", "processors" ]
  }
}

The above options would provide map "cpus" to "ncpus" and add an AcquirableResource of type "ncpus" with amount 123 and an AcquirableResource of type "mem" with amount 64mb to the resources.

Note

When using a mapping, the ResourceRequestor and ResourceProvider can use any of the alias names or map key itself in its resources. Within the ResourceProvider, all resources will always be internally mapped to the corresponding map key if available.

This allows the creation of ResourceRequestor and ResourceProvider that for one reason or another wish to refer to the same resources by different names.

Customizable Functions

The NonLocalProvider class is by design an abstract base class to force a specialized implementation to be defined. This class has no inherent knowledge abount how nonlocal resources would be requested or managed.

abstractmethod nonlocal_resources_available(resource_dict, requestor, log=True)

Tell us how to determine if nonlocal resources are available

Parameters:

requestor (ResourceRequestor)

abstractmethod nonlocal_acquire_resources(resource_dict, requestor)

Tell us how to acquire nonlocal resources

Parameters:

requestor (ResourceRequestor)

abstractmethod nonlocal_release_resources(resource_dict, requestor)

Tell us how to release nonlocal resources

Parameters:

requestor (ResourceRequestor)

class sane.resources.ResourceMapper

add_mapping(resource, aliases)

Add a mapping of resource name to a list of aliases

Parameters:

• aliases (list[str]) – A set of aliases to associate this resource name with.

• resource (str)