Host

class sane.Host[source]

Bases: NameMatch, SaveState, ResourceProvider

Primary ResourceProvider and container for Environment available within a workflow.

User Interface

User Methods

__init__(name, aliases=[])[source]

Create a host with name and optional aliases

add_environment(env)[source]

Add an Environment to the environments using Environment.name as the key

Parameters:

env (Environment)

add_resources(resource_dict, override=False)[source]

Add resources to this provider that can be acquired

Hint

See Resource for syntax on default supported values

Add a dict of key-value pairs to this provider. The key-value pairs in this dict will be used to create AcquirableResource instances that track resource requests from a ResourceRequestor.

An example options dict

{
  "cpus"  : 12,
  "mem"   : "1gb",
  "slots" : 2
}

The above resource_dict would provide 12 counts of "cpus", 1024 3 b units of "mem", and 2 counts of "slots".

Important

The ResourceProvider has no inherent understanding of the units, amounts, or names of resources. Internally, resources will be tracked but at acquisition these resources will not correspond to any form of real hardware / software allocations or locks unless logic within a custom implementation of a derived ResourceProvider specifies.

Parameters:

resource_dict (dict)

User Attributes & Properties

property name

The name of this object set at instantiation

property aliases

Aliases that this object may be referenced as

environments: utdict.UniqueTypedDict[Environment]

The unique set of environments within this host.

The environments are stored in a unique-key type-enforced dictionary. Only Environment instance objects may be stored (derived types valid)

config: dict

A user-defined dict that can hold anything picklable

This is meant as a general information container without the need for defining a custom Host with custom attributes.

property default_env: Environment

Return the default Environment or None if no default is set

Param:

set the default by using the Environment.name

property base_env

The base Environment to always setup when running this Host before any specific Environment requested by Action.environment

Hint

If a base env is provided, all Environment in environments copy specific attributes over using Environment._copy_from_base

Parameters:

env – A Environment to use as the base

property resources: Dict[str, AcquirableResource]

A copy of the available resources

Returns:

a deep copy of the internal resources in their current state

Return type:

dict[str, AcquirableResource]

Customizable Functions

Action and Host classes are designed to be modified by users.

While any function could generally be overwritten with some care, certain functions within each class are designed specifically for user modification without the need to worry about internal logic.

Defining these functions does not require calling super or other intrinsic Python knowledge beyond the interface of the function and any user logic.

load_extra_options(options, origin=None)[source]

Load any extra options after load_core_options().

Any processed field should be removed from the options dict, with everything else ignored.

See load_options() for parameters.

Parameters:
pre_run_actions(actions)[source]

Called just before entering the main workflow loop of Orchestrator.run_actions()

Parameters:

actions (dict[str,Action]) – The current set of Action queued in this workflow, stored by id

pre_launch(action)[source]

Called within the main thread just before calling Action.launch()

Parameters:

action (Action)

launch_wrapper(action, dependencies)[source]
Parameters:
post_launch(action, retval, content)[source]

Called within the main thread afetr completing Action.launch() with its return values

Parameters:

action (Action)

post_run_actions(actions)[source]

Called just after exiting the main workflow loop of Orchestrator.run_actions()

Parameters:

actions (dict[str,Action]) – The current set of Action queued in this workflow, stored by id

property watchdog_func

Return a callable function that takes one positional argument corresponding to the current set of Action queued in this workflow, stored by id

The callable returned by this function will be started on a separate thread just before Host.pre_run_actions() is called. It is on the user to ensure that this thread can exit when kill_watchdog is set to True.

Default is None

Returns:

If None is returned, no watchdog thread is created.

Internal API

The following documentation is provided for advanced use in the creation of custom Hosts.

property info

Host info provided as a dict to the Action at runtime via Action.host_info

The default implementation provides the following:

{
  "file"   : save_file,
  "name"   : name,
  "config" : config,
}
valid_host(override_host=None)[source]

Check if this host should be used for this workflow.

The default check uses the FQDN of current machine as the full string to find a partial substring match to this Host.name.

Parameters:

override_host (str) – If provided, use this string instead of the FQDN to check for substring matches.

Returns:

True if this host is valid for running this workflow on this machine

Return type:

bool

load_options(options, origin=None)[source]

Base class implementation for loading of dict-based attributes into instance

Take a options dict of relevant attributes and load them via load_core_options() then load_extra_options(). The options dict should be modified in each call to remove processed fields so that at the very end of this method, any unused keys in the options dict may be logged.

The load_extra_options() is meant as a user-overwritable method so that load_core_options() may retain core underlying base class implementation details without the risk of base class loading not being called.

To keep track of every time this function is called and potentially modifying this instance an origin may be provided, noting where the change is coming from.

Parameters:

  • options (dict) –

    A dict of class-specific attributes.

    Important

    The options dict is modified such that only unused values are left in it at the end of this method

  • origin (str) – A string identifier of where this load is coming from

load_core_options(options, origin)[source]

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From Host.load_core_options():

Load the Host options into this instance.

Below is the expected layout, where all fields are optional and "<>" fields are user-specified:

{
  "aliases" : [ ...str.. ],
  "default_env" : "<env-name>",
  "config" : { ...anything... }
  "base_env" : { "type" : "<some_env_type>", ...env options... },
  "environments" :
  {
    "<env-name>" : { "type" : "<some_env_type>", ...env options... },
    ...other env declarations...
  }
}

The following keys are loaded to their respective attribute. If not present, the attributes are unmodified.

The following key is loaded and calls recursive_update() preserve any unmodified existing values:

The "base_env" key, if present, is processsed to create an Environment that is used to set the base_env. Inside of the dict of this key, the "type" field informs which type of Environment to create. If no "type" is specified, the default is Environment.

The "environments" key is processed by iterating over each "<env-name>" and its dict. Inside of this respective "<env-name>" dict, the "type" field informs which type of Environment to create. If no "type" is specified, the default is Environment.

For both "environments" and "base_env", once the environment instance is created, its respective dict is loaded via its own Environment.load_options(). Then the created Environment is added with add_environment()

Hint

See search_type() for more info on how the "type" field should be specified.

An example options dict:

{
  "aliases" : [ "basic", "simple" ]
  # Recall that config is a generic dict
  "config"      : { "foo" : [ 1, 2, 3 ], "bar" : "file" },
  "base_env" :
  {
    "type" : "sane.Environment",
    ...env options...
  },
  "environments" :
  {
    "gnu" : { "type" : "sane.Environment", ...env options... }
  }
}

From ResourceProvider.load_core_options:

Load the available resources for this ResourceProvider

The following key is loaded verbatim into add_resources() and thus should use the same amount syntax:

  • "resources"

The following key is loaded into the internal _mapper as a dict[str,list[str]] via add_mapping(), where each key in the dict is a resource name/type and the value is a list of strings to use as aliases:

  • "mapping"

An example options dict

{
  "resources" :
  {
    "cpus" : 123,
    "mem"  : "64mb",
  }
  "mapping" : 
  {
    "ncpus" : [ "cpus", "cpu", "procs", "proc", "processors" ]
  }
}

The above options would provide map "cpus" to "ncpus" and add an AcquirableResource of type "ncpus" with amount 123 and an AcquirableResource of type "mem" with amount 64mb to the resources.

Note

When using a mapping, the ResourceRequestor and ResourceProvider can use any of the alias names or map key itself in its resources. Within the ResourceProvider, all resources will always be internally mapped to the corresponding map key if available.

This allows the creation of ResourceRequestor and ResourceProvider that for one reason or another wish to refer to the same resources by different names.

search_type(type_str, noexcept=False)[source]

Match a type (as an input string) to an actualy python type

If at any point a search is successful, the function immediately returns the found type.

Search priority:

  1. type_str using pydoc locate() (effectively search current context for type of that fully qualified name )

  2. Split type_str on last . in name and search any user-loaded module that contains the prefix for an attribute matching the suffix. If no split occurs all user modules are searched.

Valid type examples:

import sane
import user_mod.nested.foo # module foo has CustomType

# ... in the context of this class ...
self.search_type( "sane.Action" )
self.search_type( "sane.host.Host" )
self.search_type( "user_mod.nested.foo.CustomType" )
# Using search method (2) if foo was loaded into the user modules by the workflow
# since "foo" is a substring of "user_mod.nested.foo"
self.search_type( "foo.CustomType" )
Returns:

type corresponding to the type_str

Parameters:

type_str (str)

resources_available(resource_dict, requestor, log=True)[source]

Check if the all resources in the requested resource_dict are currently available

Parameters:
Return type:

bool

acquire_resources(resource_dict, requestor)[source]

Acquire all the resources in the resource_dict

Parameters:
release_resources(resource_dict, requestor)[source]

Release all the resources in the resources_dict

Parameters:
__orch_wake__()[source]

Wake up the Orchestrator from another thread.

This should be used as an event trigger to induce re-evaluation of completed Actions in the current workflow run. See Orchestrator.__wake__ for more info.

kill_watchdog

Control when to kill the watchdog_func