Environment

class sane.Environment

Bases: NameMatch, OptionLoader

Control the setup of an environment on a particular Host

User Interface

User Methods

__init__(name, aliases=[], lmod_path=None)

Create a host with name and optional aliases and lmod_path

If a Host has a Host.base_env, by default the lmod_path of that base env will be copied over during setup()

setup_env_vars(cmd, var, val=None, category='unassigned')

Store environment variable commands to execute later during setup()

These commands will eventually be executed within the isolated subprocess of Action.run(). The cmd should be one of { "set", "unset", "prepend", "append" }. The val is used for all eventual calls except "unset". The cmd corresponds to a respective call used in setup()

Example usage for recreating export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/new/path/:

self.setup_env_vars( "append", "LD_LIBRARY_PATH", "/new/path/" )

Warning

os.environ uses values verbatim and does not expand values. setup_env_vars( "append", "LD_LIBRARY_PATH", "$NEW_PATH" ) with NEW_PATH=/new/path would not produce the intended effect. Currently, the env_var_* methods do not expand values automatically.

If a category is passed in, commands are grouped based on the category and during evaluation are executed in order of category creation then command insertion order (first category and first command input go first).

Parameters:

• cmd (str) – one of { "set", "unset", "prepend", "append" }

• var (str) – environment variable to modify

• val – value to use during modification, if applicable

• category – category to group this command under

setup_lmod_cmds(cmd, *args, category='unassigned', **kwargs)

Store lmod commands to execute later during setup()

These commands will eventually be executed within the isolated subprocess of Action.run(). The cmd should be the first argument one would use for module command. All arguments that follow are appended to execution. **kwargs is reseved for the python module() function implementation.

Example usage for recreating module purge && module load gcc/12.4.0:

self.setup_lmod_cmds( "purge" )
self.setup_lmod_cmds( "load", "gcc/12.4.0" )

If a category is passed in, commands are grouped based on the category and during evaluation are executed in order of category creation then command insertion order (first category and first command input go first).

Parameters:

• cmd (str) – first argument to command line module command

• args (str) – all other arguments to command line module command

• category – category to group this command under

setup_scripts(*scripts)

Store scripts execute later during setup()

These scripts should modify the execution environment silently. Internally, to capture the effects from a subprocess, the difference in env output after running the script and output python commands to emulate this effect.

Danger

The script MUST be silent in output.

Parameters:

scripts (str) – Any number of script paths to execute during setup()

User Attributes & Properties

property name

The name of this object set at instantiation

property aliases

Aliases that this object may be referenced as

lmod_path

Path to lmod python module containing the module() function call

Customizable Functions

The Environment is also designed to be modified by users.

While any function could generally be overwritten with some care, certain functions within each class are designed specifically for user modification without the need to worry about internal logic.

Defining these functions does not require calling super or other intrinsic Python knowledge beyond the interface of the function and any user logic.

load_extra_options(options, origin=None)

Load any extra options after load_core_options().

Any processed field should be removed from the options dict, with everything else ignored.

See load_options() for parameters.

Parameters:

• options (dict)

• origin (str)

pre_setup()

Called just before setup()

post_setup()

Called just after setup()

Internal API

The following documentation is provided for advanced use in the creation of a custom Environment.

setup()

Setup the environment

If a base env is present, call the setup() for that Environment first. Then call _copy_from_base() to ensure this instance has the relevant information.

Setup of this Environment follows this order:

1. Scripts stored from setup_scripts() are executed via env_script()

2. lmod commands stored from setup_lmod_cmds() are executed via module()

3. env commands stored from setup_env_vars() are executed using the respective command call:

• "set" => env_var_set()

• "unset" => env_var_unset()

• "prepend" => env_var_prepend()

• "append" => env_var_append()

Setup commands with category are executed in order of category creation then command insertion order (first category and first command input go first).

load_options(options, origin=None)

Base class implementation for loading of dict-based attributes into instance

Take a options dict of relevant attributes and load them via load_core_options() then load_extra_options(). The options dict should be modified in each call to remove processed fields so that at the very end of this method, any unused keys in the options dict may be logged.

The load_extra_options() is meant as a user-overwritable method so that load_core_options() may retain core underlying base class implementation details without the risk of base class loading not being called.

To keep track of every time this function is called and potentially modifying this instance an origin may be provided, noting where the change is coming from.

Parameters:

• options (dict) –

  A dict of class-specific attributes.

  Important

  The options dict is modified such that only unused values are left in it at the end of this method

• origin (str) – A string identifier of where this load is coming from

load_core_options(options, origin)

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From Environment.load_core_options():

Load the options into this Environment

The following keys are loaded to their respective attribute. If not present, the attributes are unmodified.

• "aliases" => aliases

• "lmod_path" => lmod_path

The following key is iterated over, where each dict is then expanded to keyword arguments directly calling setup_env_vars()

• "env_vars"

The following key is iterated over, where for each dict "cmd" and "args" are extracted as positional arguments and the remainder is expanded to keyword arguments directly calling setup_lmod_cmds()

• "lmod_cmds"

The following key is loaded verbatim using setup_scripts():

• "env_scripts"

An example options dict:

{
  "aliases" : [ "generic", "maybe something else" ],
  "lmod_path" : "<path to lmod>",
  "env_vars"  :
  [
    { "cmd" : "prepend", "var" : "foo", "val" : 0 },
    { "cmd" : "append", "var" : "bar", "val" : "/path/" }
  ],
  "lmod_cmds" :
  [
    { "cmd" : "load", "args" : [ "gcc", "netcdf" ] }
  ]
  "env_scripts" :
  [
    "/etc/profile.d/z00_modules.sh",
    "/some/other/profile/script.sh"
  ]
}

env_var_prepend(var, val)

Prepend val to environment variable var

env_var_append(var, val)

Append val to environment variable var

env_var_set(var, val)

Set environment variable var to val

env_var_unset(var)

Unset environment variable var

module(cmd, *args, **kwargs)

Execute lmod commands in a subprocess.Popen and return the python commands to emulate the environment changes.

env_script(script)

Execute a script in a subprocess.Popen and return the python commands to emulate the environment changes.