JSON Interface

In this tutorial we will work through creating a workflow for a project called mango. The files will be added gradually as we build up our understanding of SANE workflows.

We will be recreating the workflow created in the Python Interfacing tutorial, however you do not need to go through that tutorial first.

Hint

SANE workflows tries to keep the python usage and JSON representation as as similar as possible. What you learn in one tutorial will translate smoothly to the other.

Additionally, the JSON and python interfaces can be used in tandem within workflows.

Preface

The JSON interface of SANE workflows is minimal, but provides rich control over the default classes provided, and allows using custom classes that add support for the JSON interface.

Confused about *how* are JSON files loaded? Better yet, what is a JSON file?

Tip

JSON loading of files relies on the json python module. Reading in JSON files, the files are read one-to-one as simple python objects ( {} as dict, [] as list, "" as str, and # as int)

A JSON file itself is nothing more than a “file format … to store and transmit data objects”, the contents of which are decided by us. See https://en.wikipedia.org/wiki/JSON for more detail.

Attention

Much like the python interface, it will be important to understand at a very high level our entry point.

ALL files (JSON or python) are only ever loaded by the Orchestrator. JSON files specifically are read in via the Orchestrator.load_config_files() function. All subsequent loading functions (load_core_options) operates only on the specific subsections of the file they are provided.

As we go through this section of the tutorial, we will go over various load_core_options functions that digest information from our config files. To prepare us, we will briefly look at an example JSON file. For now, do not worry about understanding the content, and instead focus on layout:

Example config file

// Section A: Everything between these {} will be loaded as options
// by Orchestrator.load_core_options()
{
  "hosts" :
  {
    // Section B: Everything between {} for each <key> : { <value> } will be loaded by <type>.load_core_options()
 // <key>            : { <value> }
 // vvvvvvvvv            vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
    "my_host"        : { "type" : "sane.Host", "resources" : { "cpus" : 12 } },
    "my_larger_host" : { "type" : "sane.Host", "resources" : { "cpus" : 24 } }
  },
  "actions" :
  {
    // Section B again
    "my_action" :
    {
      "type" : "sane.Action",
      "resources" : { "cpus" : 8 },
      "config" : { "command" : "script.sh", "arguments" : [ 1, 2, 3 ] }
    }
  }
  "patches" :
  {
    "priority" : 0,
    "hosts"   :
    {
      // Section C: Everything between {} for each value will be loaded by respective load_core_options()
      "my_larger_host" : { "resources" : { "gpus" : 1 } }
    }
    "actions" :
    {
      // Section C again
      "my_action"  : { "config" : { "arguments" : [ 3, 2, 1 ] } }
    }
  }
}

Reviewing the example above:

Section A is the entire config file top-level {} that is further digested into smaller subsections of options
Section B breaks out the { <value> } in "<key>" : { <value> } pairs that will be loaded as a subsection by <type>.load_core_options()
Section C* is similar to Section B but type is not used

If this doesn’t make sense, don’t worry. This is just an overview to gain context before the walkthrough.

config disambiguation

We will often refer to a config being used within the Action. Do not confuse this with the config file itself.

The Action.config is a generalized container to hold data in the object instance, whereas the config file is just the set of options read in.

The Action.config will always be referred to with code or reference link styling.

Orchestrator

The primary loading function is Orchestrator.load_core_options(). You do not need to call this function, however it is noted here as the values this method loads are the primary way we interact with the JSON interface.

This function will take the entire options dict of the JSON file and attempt to load valid keys. Thus, this is effectively the functions that determines the top-level structure of these JSON config files.

Below is a quick look at the function:

Quick Reference (click to open/close)

Orchestrator.load_core_options(options, origin)[source]

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From Orchestrator.load_core_options():

Load the provided options dict, creating any Host or Action as necessary and recording patches.

Below is the expected layout, where all fields are optional and "<>" fields are user-specified:
{
  "hosts" :
  {
    "<host-name>" : { "type" : "<some_host_type>", ...host options... },
    ...other host declarations...
  },
  "actions" :
  {
    "<action-id>" : { "type" : "<some_action_type>", ...action options... },
    ...other action declarations...
  }
  "patches" :
  {
    "priority" : int,
    "hosts"   : ...same as above *except* "type"...
    "actions" : ...same as above *except* "type"...
  }
}
The "hosts" key is processed first, iterating over each "<host-name>" and its dict. Inside of this respective "<host-name>" dict, the "type" field informs which type of Host to create. If no "type" is specified, the default is Host. The "<host-name>" is used as the Host.name during instantiation.

Once the host instance is created, its respective dict is loaded via its own Host.load_options(). Then the created host is added with add_host()

Next, the "actions" key is processed in a similar fashion, except the default "type" is Action and added via add_action()

Hint

See search_type() for more info on how the "type" field should be specified.

Finally, the "patches" key is processed. A default priority of 0 is used if no priority is specified. Everything in the "patches" dict (except the "priority") is saved for later use in process_patches() in an internal patch priority queue. The content of this can generally be the same as when declaring "hosts" or "actions", with limitations left the type’s implementation of loading the options for which the patch would be applied to (e.g. a derived Action may allow more or less fields in its load_options/load_core_options/load_extra_options). Each entry should correspond to an existing object in the workflow found in hosts or actions - objects to be patched do not need to be created via JSON config file.

Hint

See process_patches() or process_patch_dict() for advanced usage of patching objects, including using patch filters.

Note

"type" is not a valid field in any of the "patches" sub-dicts as the options will be applied to existing object instances and "type" is only used for initial creation of objects in this method.

Parameters:

options (Dict[str, object])
origin (str)

Using the above Quick Reference or Example config file as guidance:

JSON files for SANE workflows are always composed of three optional keys within the root dictionary:

"hosts"
"actions"
"patches"

The "patches" dictionary mirrors the layout of the parent JSON dictionary, except that "type" is no longer valid in any referenced host or action (and you can’t have more nested "patches").

Within the "hosts" and "actions" dictionaries, the unique keys are used as the name or id for the underlying objects they will create. The valid fields inside the corresponding options { <value> } of the unique key name/id are left without much specification because the "type" key may change how that options subsection is loaded, as we will see later in the Custom Classes section.

For now, we will assume we are using the default classes. We then can simply use their respective load_core_options. Let’s start making some file content.

Separate config files?

Since all options at the top-level are optional, we can categorically organize our config files to contain different subsections. The separate files will each be processed into the Orchestrator cummulatively, building up the workflow definition piecemeal.

Hosts

To begin our workflow, let’s create the .sane/mango/hosts/forest.jsonc file and create a host in this file.

.sane/
└── mango
    └── hosts
        └── forest.jsonc

We will start by creating a Host options under the "hosts" key. Recall that the unique key under "hosts" acts as the Host.name.

.sane/mango/hosts/forest.jsonc

{
  "hosts" :
  {
    "forest" :
    {
      //...what to put here...
    }
  }
}

As far as creating a host we are basically done… Well, not really. It is a valid host, but it does not provided much. Additionally, when running a workflow, the expectation is that any Action will always run in an Environment, even if one is not needed. Thus, hosts must provide at least one Environment to even be somewhat useful.

Let’s continue to flesh out this Host.

To find what fields/keys to put in our "forest" options { <value> } area, let’s take a quick look at the Host.load_core_options():

Quick Reference (click to open/close):

Host.load_core_options(options, origin)[source]

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From Host.load_core_options():

Load the Host options into this instance.

Below is the expected layout, where all fields are optional and "<>" fields are user-specified:
{
  "aliases" : [ ...str.. ],
  "default_env" : "<env-name>",
  "config" : { ...anything... }
  "base_env" : { "type" : "<some_env_type>", ...env options... },
  "environments" :
  {
    "<env-name>" : { "type" : "<some_env_type>", ...env options... },
    ...other env declarations...
  }
}
The following keys are loaded to their respective attribute. If not present, the attributes are unmodified.

"aliases" => aliases

"default_env" => default_env

The following key is loaded and calls recursive_update() preserve any unmodified existing values:

"config" => config

The "base_env" key, if present, is processsed to create an Environment that is used to set the base_env. Inside of the dict of this key, the "type" field informs which type of Environment to create. If no "type" is specified, the default is Environment.

The "environments" key is processed by iterating over each "<env-name>" and its dict. Inside of this respective "<env-name>" dict, the "type" field informs which type of Environment to create. If no "type" is specified, the default is Environment.

For both "environments" and "base_env", once the environment instance is created, its respective dict is loaded via its own Environment.load_options(). Then the created Environment is added with add_environment()

Hint

See search_type() for more info on how the "type" field should be specified.

An example options dict:
{
  "aliases" : [ "basic", "simple" ]
  # Recall that config is a generic dict
  "config"      : { "foo" : [ 1, 2, 3 ], "bar" : "file" },
  "base_env" :
  {
    "type" : "sane.Environment",
    ...env options...
  },
  "environments" :
  {
    "gnu" : { "type" : "sane.Environment", ...env options... }
  }
}

From ResourceProvider.load_core_options:

Load the available resources for this ResourceProvider

The following key is loaded verbatim into add_resources() and thus should use the same amount syntax:

"resources"

The following key is loaded into the internal _mapper as a dict[str,list[str]] via add_mapping(), where each key in the dict is a resource name/type and the value is a list of strings to use as aliases:

"mapping"

An example options dict
{
  "resources" :
  {
    "cpus" : 123,
    "mem"  : "64mb",
  }
  "mapping" : 
  {
    "ncpus" : [ "cpus", "cpu", "procs", "proc", "processors" ]
  }
}
The above options would provide map "cpus" to "ncpus" and add an AcquirableResource of type "ncpus" with amount 123 and an AcquirableResource of type "mem" with amount 64mb to the resources.

Note

When using a mapping, the ResourceRequestor and ResourceProvider can use any of the alias names or map key itself in its resources. Within the ResourceProvider, all resources will always be internally mapped to the corresponding map key if available.

This allows the creation of ResourceRequestor and ResourceProvider that for one reason or another wish to refer to the same resources by different names.

Some interesting things to note about the keys available to us:

All keys are optional (this is the case for all default classes)
Support for different keys is provided via subclass calls implementing their own load_core_options:
- Keys supported by other subclasses are aggregates (not replacements) to the supported key list

Hint

All keys across all default classes are always optional.

Thus, taking into account all suported key aggregated from the different classes we can see that we have quite a few options options:

From Host.load_core_options():

"aliases"

"default_env"

"config" (see Host.config or config disambiguation for disambiguation)

"base_env"

"environments"

From ResourceProvider.load_core_options:

"resources"

"mapping"

That is a lot of options, so make this introduction simple we will focus on the key options that will make this host useful: "environments" and "resources".

Resources & Environment

Maps:

"resources" to Host.add_resources() / Host.resources
"environments" to Host.add_environment() / Host.environments

Let’s take a second to look at the options documentation for "resources".

We see that it takes a dict that uses sane.resources.Resource syntax. The simple explanation is that the resource must be a non-negative integer followed by optional binary scaling ( ‘k’ : 1024, ‘m’ : 1024 ², and so on), and an optional unit designator.

The simple explanation is that the resource must be a non-negative integer followed by optional binary scaling ( ‘k’ : 1024, ‘m’ : 1024 ², and so on), and an optional unit designator.

So we have a forest Host and want to add resources for our mango project workflow… Let’s add "trees" (these could nominally be "cpus" or whatever resource your Action would take to run):

.sane/mango/hosts/forest.jsonc

{
  "hosts" :
  {
    "forest" :
    {
      "resources" :
      {
        // this coud be cpus but we'll use trees because
        // that is where mangos grow
        "trees" : 12
      }
    }
  }
}

Next, we need an Environment options.

Important

Currently, actions ALWAYS need to run with an Environment set up. Therefore, hosts must have at least one Environment declared that isn’t the base_env

We see once again that the details of the environment options are sparse due to a "type" specification, similar to the one used under "hosts" and "actions" configs. Likewise, if no "type" is specified, the internal default class is used.

Therefore, let’s assume we are using the default Environment.load_core_options:

Quick Reference (click to open/close):

Environment.load_core_options(options, origin)[source]

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From Environment.load_core_options():

Load the options into this Environment

The following keys are loaded to their respective attribute. If not present, the attributes are unmodified.

"aliases" => aliases

"lmod_path" => lmod_path

The following key is iterated over, where each dict is then expanded to keyword arguments directly calling setup_env_vars()

"env_vars"

The following key is iterated over, where for each dict "cmd" and "args" are extracted as positional arguments and the remainder is expanded to keyword arguments directly calling setup_lmod_cmds()

"lmod_cmds"

The following key is loaded verbatim using setup_scripts():

"env_scripts"

An example options dict:
{
  "aliases" : [ "generic", "maybe something else" ],
  "lmod_path" : "<path to lmod>",
  "env_vars"  :
  [
    { "cmd" : "prepend", "var" : "foo", "val" : 0 },
    { "cmd" : "append", "var" : "bar", "val" : "/path/" }
  ],
  "lmod_cmds" :
  [
    { "cmd" : "load", "args" : [ "gcc", "netcdf" ] }
  ]
  "env_scripts" :
  [
    "/etc/profile.d/z00_modules.sh",
    "/some/other/profile/script.sh"
  ]
}

A lot of options option again:

"aliases"
"lmod_path"
"env_vars"
"lmod_cmds"
"env_scripts"

For demonstrative purposes, we will say that our environments have different GROWTH_RATE values can be used within the workflow.

To do this and maintain the simple approach for now, we will only use "env_vars" under our unique keys that create our Environment.

Our .sane/mango/hosts/forest.jsonc should now look like:

.sane/mango/hosts/forest.jsonc

{
  "hosts" :
  {
    "forest" :
    {
      "resources" :
      {
        // this coud be cpus but we'll use trees because
        // that is where mangos grow
        "trees" : 12
      },
      "environments" :
      {
        "valley" : { "env_vars" : [ { "cmd" : "set", "var" : "GROWTH_RATE", "val" : 85 } ] },
        "river"  : { "env_vars" : [ { "cmd" : "set", "var" : "GROWTH_RATE", "val" : 65 } ] }
      }
    }
  }
}

We now have a functional host. However, it can’t do a whole lot without an Action to exercise the host and its environments. If we were to run this host, we would get some lackluster output:

sane_runner -p .sane/ -n -sh forest -v --run

2025-12-12 17:33:31 INFO     [sane_runner]            Logging output to /home/aislas/mango/log/runner.log
2025-12-12 17:33:31 INFO     [orchestrator]           Searching for workflow files...
2025-12-12 17:33:31 INFO     [orchestrator]             Searching .sane/ for *.json
2025-12-12 17:33:31 INFO     [orchestrator]             Searching .sane/ for *.jsonc
2025-12-12 17:33:31 INFO     [orchestrator]               Found .sane/mango/hosts/forest.jsonc
2025-12-12 17:33:31 INFO     [orchestrator]             Searching .sane/ for *.py
2025-12-12 17:33:31 INFO     [orchestrator]           Loading config file .sane/mango/hosts/forest.jsonc
2025-12-12 17:33:31 INFO     [sane_runner]            No actions selected
...help info...

Tip

This output can be reproduced by using the source repo example found at /home/aislas/frameflow/docs/examples/mango/json_basic_host/.sane

The default search patterns found our file and loaded it, but nothing was done since no actions were found.

Actions

Let’s now try to create our first JSON-based Action!

We will create the .sane/mango/actions/grow.jsonc file, as well as create a helper script at .sane/scripts/grow.sh:

.sane/
├── mango
│   ├── actions
│   │   └── grow.jsonc
│   └── hosts
│       └── forest.jsonc
└── scripts
    └── grow.sh

Confused as to how we are able to use separate files? See the separate config files tip for clarification.

We will start by creating our .sane/mango/actions/grow.jsonc file:

.sane/mango/actions/grow.jsonc

{
  "actions" :
  {
    "grow_action" :
    {
      //...
    }
  }
}

Similar to what we’ve done previously, let us look at the Action.load_core_options():

Quick Reference (click to open/close):

Action.load_core_options(options, origin)[source]

From OptionLoader.load_core_options:

Any processed field should be removed from the options dict, with everything else ignored. All listed options are cummulative and optional unless specified otherwise.

See load_options() for parameters.

From Action.load_core_options():

Load Action settings from the provided options dict, all keys are optional.

The following keys are loaded verbatim into their respective attribute:

"environment" => environment

"working_directory" => working_directory

The following key is loaded and calls recursive_update() preserve any unmodified existing values:

"config" => config

The following key is loaded directly to add_dependencies() as key-value tuple pairs via dict.items()

"dependencies"

An example options dict
{
  "environment" : "gnu",
  # Recall that config is a generic dict
  "config"      : { "foo" : [ 1, 2, 3 ], "bar" : "file" },
  # if loading via plain-text (e.g. JSON), use the text value
  # of DependencyType as noted in add_dependencies()
  "dependencies" : { "action_b" : "afterok", "action_c" : "afternotok" }
}

From ResourceRequestor.load_core_options:

Load ResourceRequestor resource requirements

The following key is loaded verbatim into add_resource_requirements():

"resources"

The following key is loaded if possible, defaulting to None:

"local" => local

To quickly summarize, the Action.load_core_options() supports:

From Action.load_core_options():

"environment"

"working_directory"

"config"

"dependencies"

From ResourceRequestor.load_core_options:

"resources"

"local"

Most important to us will be:

"config"
"environment"
"resources"
"dependencies" (covered later)

We shall go over relevance of each of these.

`"config"`

Maps to Action.config (see config disambiguation)

Arguably the most versatile field in the default Action, this generic dict is meant to hold anything picklable that your action may need in running. Later (during Attribute Dereferencing) we will learn to more effectively use this general data container.

For now, the main use is in the config["command"] and config["arguments"] keys which are used by the default Action.run(). As their names imply, during Action execution, the config["command"] and config["arguments"] will be used as the command and arguments to execute for that action, respectively.

We are going to use the helper script located at .sane/mango/scripts.grow.sh to perform our main logic. This thereby allows us to have an Action with customizable execution without needing to write Custom Actions. The path we use to the script should be a relative path from the working directory, in this case ./. Only use absolute paths for scripts that will always be in the same place (e.g. host scripts)!

Let’s quickly modify our .sane/mango/actions/grow.jsonc:

.sane/mango/actions/grow.jsonc

{
  "actions" :
  {
    "grow_action" :
    {
      "config" :
      {
        "command" : ".sane/scripts/grow.sh",
        // this must be a list
        "arguments" : [ 4 ]
      }
    }
  }
}

As for the contents of .sane/mango/scripts/grow.sh let’s use:

.sane/mango/scripts/grow.sh

#!/usr/bin/bash
NTREES=$1
echo "Growing with $NTREES trees with $GROWTH_RATE% growth rate..."

for i in $(seq 1 $NTREES ); do
  NMANGOS=$(( ($RANDOM % 10) * $GROWTH_RATE / 100 + 1 ))
  echo "  Tree $i grew $NMANGOS mangos!"
  echo $NMANGOS > mango_tree_$i.txt
done

Note

The string provided to config["command"] must be an executable. Commands from your PATH will work, but if you use a script it must have the executable property. Try running:

chmod +x <command script>

if you are having issues getting your Action to run your config["command"]

`"environment"`

Maps to Action.environment

Each Action must run within an Environment, however the actual object instance is owned by the Host, as each host is left to implement that specific environment.

The communication from Action to Host about which environment to use is facilitated by a str match between Action.environment and Environment.name (or Environment.aliases) within the Host.environments.

Let’s update our .sane/mango/actions/grow.jsonc again, saying we need a "valley" environment (recall forest.jsonc):

.sane/mango/actions/grow.jsonc

{
  "actions" :
  {
    "grow_action" :
    {
      "config" :
      {
        "command" : ".sane/scripts/grow.sh",
        // this must be a list
        "arguments" : [ 4 ]
      },
      "environment": "valley"
    }
  }
}

`"resources"`

Maps to Action.add_resource_requirements() / Action.resources

Any Action that makes use of quantifiable resources tracked by Host.resources should ideally note them by calling this function.

These resource requests are how the Orchestrator coordinates tasking with a given Host. This is to ensure the resources that the host does provide are not oversubscribed (keeping usage at or below limit), and that any Action that requires resources that are not provided are caught.

Danger

If an Action does not request resources, regardless of the internal logic and actual resources usage, it will ALWAYS be run. If you use untracked or more resources than requested, you may inadvertently use more resources than the Host provides.

It is generally a good idea to track critical resources needed in both the Action and Host

To grow our mangos we will need "trees" (recall forest.jsonc):

.sane/actions/grow.jsonc

{
  "actions" :
  {
    "grow_action" :
    {
      "config" :
      {
        "command" : ".sane/mango/scripts/grow.sh",
        // this must be a list
        "arguments" : [ 4 ]
      },
      "environment": "valley",
      // We need trees to grow on
      "resources" : { "trees" : 4 }
    }
  }
}

`"dependencies"`

Maps to Action.add_dependencies() / Action.dependencies

This is the first Action in our workflow, so it will not have any dependencies. See the Adding "dependencies" for details on adding dependencies.

Final `Action` Result

Our final .sane/mango/actions/grow.jsonc should now look like:

.sane/actions/grow.jsonc

{
  "actions" :
  {
    "grow_action" :
    {
      "config" :
      {
        "command" : ".sane/mango/scripts/grow.sh",
        // this must be a list
        "arguments" : [ 4 ]
      },
      "environment": "valley",
      // We need trees to grow on
      "resources" : { "trees" : 4 }
    }
  }
}

Running

Now that we have everything set up, we should be able to run. We will be using the following optional flags:

-sh Specific Host option to ensure our host is selected
-n New Run option to always rerun our workflow entirely
-v Verbose option to get full output in one location rather than split amongst multiple files

sane_runner -p .sane/ -sh forest -n -v -r

2025-12-12 19:58:08 INFO     [sane_runner]            Logging output to /home/aislas/mango/log/runner.log
2025-12-12 19:58:08 INFO     [orchestrator]           Searching for workflow files...
2025-12-12 19:58:08 INFO     [orchestrator]             Searching .sane/ for *.json
2025-12-12 19:58:08 INFO     [orchestrator]             Searching .sane/ for *.jsonc
2025-12-12 19:58:08 INFO     [orchestrator]               Found .sane/mango/hosts/forest.jsonc
2025-12-12 19:58:08 INFO     [orchestrator]               Found .sane/mango/actions/grow.jsonc
2025-12-12 19:58:08 INFO     [orchestrator]             Searching .sane/ for *.py
2025-12-12 19:58:08 INFO     [orchestrator]           Loading config file .sane/mango/hosts/forest.jsonc
2025-12-12 19:58:08 INFO     [orchestrator]           Loading config file .sane/mango/actions/grow.jsonc
2025-12-12 19:58:08 INFO     [orchestrator]           No previous save file to load
2025-12-12 19:58:08 INFO     [orchestrator]           Requested actions:
2025-12-12 19:58:08 INFO     [orchestrator]             grow_action
2025-12-12 19:58:08 INFO     [orchestrator]           and any necessary dependencies
2025-12-12 19:58:08 INFO     [orchestrator]           Full action set:
2025-12-12 19:58:08 INFO     [orchestrator]           Full action set:
2025-12-12 19:58:08 INFO     [orchestrator]             grow_action
2025-12-12 19:58:08 INFO     [orchestrator]           Checking host "forest"
2025-12-12 19:58:08 INFO     [orchestrator]           Running as 'forest'
2025-12-12 19:58:08 INFO     [orchestrator]           Checking ability to run all actions on 'forest'...
2025-12-12 19:58:08 INFO     [orchestrator]             Checking environments...
2025-12-12 19:58:08 INFO     [orchestrator]             Checking resource availability...
2025-12-12 19:58:08 INFO     [orchestrator]           * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
2025-12-12 19:58:08 INFO     [orchestrator]           * * * * * * * * * *            All prerun checks for 'forest' passed            * * * * * * * * * *
2025-12-12 19:58:08 INFO     [orchestrator]           * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
2025-12-12 19:58:08 INFO     [orchestrator]           Saving host information...
2025-12-12 19:58:08 INFO     [orchestrator]           Setting state of all inactive actions to pending
2025-12-12 19:58:08 INFO     [orchestrator]           No previous save file to load
2025-12-12 19:58:08 INFO     [orchestrator]           Using working directory : '/home/aislas/mango'
2025-12-12 19:58:08 INFO     [orchestrator]           Running actions...
2025-12-12 19:58:08 INFO     [orchestrator]           Running 'grow_action' on 'forest'
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]      Action logfile captured at /home/aislas/mango/log/grow_action.log
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]      Saving action information for launch...
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]      Using working directory : '/home/aislas/mango'
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]      Running command:
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]        /home/aislas/frameflow/sane/action_launcher.py /home/aislas/mango /home/aislas/mango/tmp/action_grow_action.json
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]      Command output will be captured to logfile /home/aislas/mango/log/grow_action.runlog
2025-12-12 19:58:08 INFO     [thread_0]  [grow_action::launch]      Command output will be printed to this terminal
2025-12-12 19:58:08 INFO     [grow_action::launch]    ***************Inside action_launcher.py***************
2025-12-12 19:58:08 INFO     [grow_action::launch]    Current directory: /home/aislas/mango
2025-12-12 19:58:08 INFO     [grow_action::launch]    Loaded Action "grow_action"
2025-12-12 19:58:08 INFO     [grow_action::launch]    Loaded Host "forest"
2025-12-12 19:58:08 INFO     [grow_action::launch]    Using Environment "valley"
2025-12-12 19:58:08 INFO     [valley]                 Running env cmd: 'set' with var: 'GROWTH_RATE' and val: '85'
2025-12-12 19:58:08 INFO     [valley]                   Environment variable GROWTH_RATE=85
2025-12-12 19:58:08 INFO     [grow_action::run]       Running command:
2025-12-12 19:58:08 INFO     [grow_action::run]         .sane/mango/scripts/grow.sh 4
2025-12-12 19:58:08 INFO     [grow_action::run]       Command output will be printed to this terminal
2025-12-12 19:58:08 STDOUT   [grow_action::run]       Growing with 4 trees with 85% growth rate...
2025-12-12 19:58:08 STDOUT   [grow_action::run]         Tree 1 grew 7 mangos!
2025-12-12 19:58:08 STDOUT   [grow_action::run]         Tree 2 grew 5 mangos!
2025-12-12 19:58:08 STDOUT   [grow_action::run]         Tree 3 grew 6 mangos!
2025-12-12 19:58:08 STDOUT   [grow_action::run]         Tree 4 grew 2 mangos!
2025-12-12 19:58:08 INFO     [grow_action::launch]    ***************Finished action_launcher.py***************
2025-12-12 19:58:08 INFO     [orchestrator]           [FINISHED] ** Action 'grow_action'            completed with 'success'
2025-12-12 19:58:08 INFO     [orchestrator]           Finished running queued actions
2025-12-12 19:58:08 INFO     [orchestrator]             grow_action: success
2025-12-12 19:58:08 INFO     [orchestrator]           All actions finished with success
2025-12-12 19:58:08 INFO     [orchestrator]           Finished in 0:00:00.199341
2025-12-12 19:58:08 INFO     [orchestrator]           Logfiles at /home/aislas/mango/log
2025-12-12 19:58:08 INFO     [orchestrator]           Save file at /home/aislas/mango/tmp/orchestrator.json
2025-12-12 19:58:08 INFO     [orchestrator]           JUnit file at /home/aislas/mango/log/results.xml
2025-12-12 19:58:08 INFO     [sane_runner]            Finished

Tip

This output can be reproduced by using the source repo example found at docs/examples/mango/json_grow_action/.sane/

A quick walkthrough of the above output, focusing on the highlighted regions:

The Orchestrator finds and loads our files
The Orchestrator, after verifying the selected Host, runs our Action on the host
During execution inside our Action (action_launcher.py):
1. The Action loading itself and the Host
2. Sets up the Environment
3. Calls our config["command"] with config["arguments"]
4. Outputs the command output with log tag STDOUT (stderr also goes here)
Final logs and results information is left at the bottom for our convenience

Special notes:

Everything betwwen ***...Inside action_launcher.py...*** and ***...Finished action_launcher.py...*** for a respective Action is actually the direct output of the Action.run()
(3.a) occurs because the action_launcher.py (Action.run()) occurs in a totally separate subprocess
(3.d) captures all command output (stdout and stderr)

Extending the workflow

Now that we have a basis for creating our workflow, let’s harvest our mangos. We will be adding a .sane/mango/actions/harvest.jsonc file and supporting .sane/mango/scripts/harvest.sh script:

.sane/
└── mango
    ├── actions
    │   ├── grow.jsonc
    │   └── harvest.jsonc
    ├── hosts
    │   └── forest.jsonc
    └── scripts
        ├── grow.sh
        └── harvest.sh

Adding `"dependencies"`

Before we create a new Action for our workflow, we should first discuss how it will tie in and depend on our "grow_action".

When running multiple Action in a workflow, it may be beneficial to have them run in a very particular order. This is where the directed acyclic graph (DAG) nature of SANE workflows comes into play.

Dependencies between Action are mapped out based on \(child \rightarrow N parents\) relationships, where each Action lists the parents it is dependent on.

Action can have any number of dependencies, so long as the dependencies never form a closed loop. In other words, an Action CAN NEVER be the ancestor (parent) to any of its own ancestors. If thought of as a family tree, any single Action CAN NEVER reappear earlier in its lineage.

BAD: Dependency Graph with cycles

digraph foo {
"bar" -> "baz";
"baz" -> "boo";
"baz" -> "foo";
"foo" -> "bar";
}

OKAY: Dependency Graph with no cycles

digraph foo {
"bar" -> "baz";
"baz" -> "boo";
"baz" -> "foo";
"foo" -> "zoo";
}

This is the first Action in our workflow, so it will not have any dependencies.

If we were to add depedencies to an Action, we would list the Action.id and DependencyType string value for each dependency:

Quick Reference (click to open/close)

class sane.DependencyType[source]

Types of dependencies between actions from child to parent

AFTEROK = 'afterok': after successful run (this is the default)

AFTERNOTOK = 'afternotok': after failure

AFTERANY = 'afterany': after either failure or success

AFTER = 'after': after the step starts

Example JSON dependency

// ...JSON file excerpt...
"actions" :
{
  "a" : {},
  "b" : { "dependencies" : { "a" : "afterok" },
  "c" : {},
  "d" : { "dependencies" : { "b" : "afterok", "c" : "afternotok" },
}

New files

We can model the new "harvest_action" after the grow.jsonc example, but change a few things such as the script we will run and adding a dependency to our initial "grow_action":

.sane/mango/actions/harvest.jsonc

{
  "actions" :
  {
    "harvest_action" :
    {
      "config" :
      {
        "command" : ".sane/mango/scripts/harvest.sh"
      },
      "environment": "valley",
      // no resource requirements listed for this action
      // set dependency to "grow_action"
      "dependencies" : { "grow_action" : "afterok" }
    }
  }
}

And our helper script as:

.sane/mango/scripts/harvest.sh

#!/usr/bin/bash
echo "Harvesting mangos..."
echo -ne "Collected : "
awk 'BEGIN { sum=0 } { sum+=$1 } END { print sum }' mango_tree_*.txt

Note that we listed the dependencies using the Action.id string value.

Since the dependency graph is constructed at runtime, there are NO checks on dependency graph validity during Action instantiation. Checks are done ony after all Action are loaded and attempted to be run.

This slight caveat gives us the flexibilty to declare our Action objects across different files and functions in virtually any order we want. We can create workflows without worrying about when an Action is created, and instead focus on proper dependency mapping as the final result.

Let’s run with new action:

sane_runner -p .sane/ -sh forest -n -v -r

2025-12-12 20:14:31 INFO     [sane_runner]            Logging output to /home/aislas/mango/log/runner.log
2025-12-12 20:14:31 INFO     [orchestrator]           Searching for workflow files...
2025-12-12 20:14:31 INFO     [orchestrator]             Searching .sane/ for *.json
2025-12-12 20:14:31 INFO     [orchestrator]             Searching .sane/ for *.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]               Found .sane/mango/actions/grow.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]               Found .sane/mango/actions/harvest.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]               Found .sane/mango/hosts/forest.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]             Searching .sane/ for *.py
2025-12-12 20:14:31 INFO     [orchestrator]           Loading config file .sane/mango/actions/grow.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]           Loading config file .sane/mango/actions/harvest.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]           Loading config file .sane/mango/hosts/forest.jsonc
2025-12-12 20:14:31 INFO     [orchestrator]           No previous save file to load
2025-12-12 20:14:31 INFO     [orchestrator]           Requested actions:
2025-12-12 20:14:31 INFO     [orchestrator]             grow_action     harvest_action
2025-12-12 20:14:31 INFO     [orchestrator]           and any necessary dependencies
2025-12-12 20:14:31 INFO     [orchestrator]           Full action set:
2025-12-12 20:14:31 INFO     [orchestrator]           Full action set:
2025-12-12 20:14:31 INFO     [orchestrator]             grow_action     harvest_action
2025-12-12 20:14:31 INFO     [orchestrator]           Checking host "forest"
2025-12-12 20:14:31 INFO     [orchestrator]           Running as 'forest'
2025-12-12 20:14:31 INFO     [orchestrator]           Checking ability to run all actions on 'forest'...
2025-12-12 20:14:31 INFO     [orchestrator]             Checking environments...
2025-12-12 20:14:31 INFO     [orchestrator]             Checking resource availability...
2025-12-12 20:14:31 INFO     [orchestrator]           * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
2025-12-12 20:14:31 INFO     [orchestrator]           * * * * * * * * * *            All prerun checks for 'forest' passed            * * * * * * * * * *
2025-12-12 20:14:31 INFO     [orchestrator]           * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
2025-12-12 20:14:31 INFO     [orchestrator]           Saving host information...
2025-12-12 20:14:31 INFO     [orchestrator]           Setting state of all inactive actions to pending
2025-12-12 20:14:31 INFO     [orchestrator]           No previous save file to load
2025-12-12 20:14:31 INFO     [orchestrator]           Using working directory : '/home/aislas/mango'
2025-12-12 20:14:31 INFO     [orchestrator]           Running actions...
2025-12-12 20:14:31 INFO     [orchestrator]           Running 'grow_action' on 'forest'
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]         Action logfile captured at /home/aislas/mango/log/grow_action.log
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]         Saving action information for launch...
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]         Using working directory : '/home/aislas/mango'
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]         Running command:
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]           /home/aislas/frameflow/sane/action_launcher.py /home/aislas/mango /home/aislas/mango/tmp/action_grow_action.json
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]         Command output will be captured to logfile /home/aislas/mango/log/grow_action.runlog
2025-12-12 20:14:31 INFO     [thread_0]  [grow_action::launch]         Command output will be printed to this terminal
2025-12-12 20:14:31 INFO     [grow_action::launch]    ***************Inside action_launcher.py***************
2025-12-12 20:14:31 INFO     [grow_action::launch]    Current directory: /home/aislas/mango
2025-12-12 20:14:31 INFO     [grow_action::launch]    Loaded Action "grow_action"
2025-12-12 20:14:31 INFO     [grow_action::launch]    Loaded Host "forest"
2025-12-12 20:14:31 INFO     [grow_action::launch]    Using Environment "valley"
2025-12-12 20:14:31 INFO     [valley]                 Running env cmd: 'set' with var: 'GROWTH_RATE' and val: '85'
2025-12-12 20:14:31 INFO     [valley]                   Environment variable GROWTH_RATE=85
2025-12-12 20:14:31 INFO     [grow_action::run]       Running command:
2025-12-12 20:14:31 INFO     [grow_action::run]         .sane/mango/scripts/grow.sh 4
2025-12-12 20:14:31 INFO     [grow_action::run]       Command output will be printed to this terminal
2025-12-12 20:14:31 STDOUT   [grow_action::run]       Growing with 4 trees with 85% growth rate...
2025-12-12 20:14:31 STDOUT   [grow_action::run]         Tree 1 grew 7 mangos!
2025-12-12 20:14:31 STDOUT   [grow_action::run]         Tree 2 grew 4 mangos!
2025-12-12 20:14:31 STDOUT   [grow_action::run]         Tree 3 grew 6 mangos!
2025-12-12 20:14:31 STDOUT   [grow_action::run]         Tree 4 grew 3 mangos!
2025-12-12 20:14:31 INFO     [grow_action::launch]    ***************Finished action_launcher.py***************
2025-12-12 20:14:31 INFO     [orchestrator]           [FINISHED] ** Action 'grow_action'            completed with 'success'
2025-12-12 20:14:31 INFO     [orchestrator]           Running 'harvest_action' on 'forest'
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]      Action logfile captured at /home/aislas/mango/log/harvest_action.log
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]      Saving action information for launch...
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]      Using working directory : '/home/aislas/mango'
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]      Running command:
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]        /home/aislas/frameflow/sane/action_launcher.py /home/aislas/mango /home/aislas/mango/tmp/action_harvest_action.json
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]      Command output will be captured to logfile /home/aislas/mango/log/harvest_action.runlog
2025-12-12 20:14:31 INFO     [thread_0]  [harvest_action::launch]      Command output will be printed to this terminal
2025-12-12 20:14:31 INFO     [harvest_action::launch] ***************Inside action_launcher.py***************
2025-12-12 20:14:31 INFO     [harvest_action::launch] Current directory: /home/aislas/mango
2025-12-12 20:14:31 INFO     [harvest_action::launch] Loaded Action "harvest_action"
2025-12-12 20:14:31 INFO     [harvest_action::launch] Loaded Host "forest"
2025-12-12 20:14:31 INFO     [harvest_action::launch] Using Environment "valley"
2025-12-12 20:14:31 INFO     [valley]                 Running env cmd: 'set' with var: 'GROWTH_RATE' and val: '85'
2025-12-12 20:14:31 INFO     [valley]                   Environment variable GROWTH_RATE=85
2025-12-12 20:14:31 INFO     [harvest_action::run]    Running command:
2025-12-12 20:14:31 INFO     [harvest_action::run]      .sane/mango/scripts/harvest.sh
2025-12-12 20:14:31 INFO     [harvest_action::run]    Command output will be printed to this terminal
2025-12-12 20:14:31 STDOUT   [harvest_action::run]    Harvesting mangos...
2025-12-12 20:14:31 STDOUT   [harvest_action::run]    Collected : 20
2025-12-12 20:14:31 INFO     [harvest_action::launch] ***************Finished action_launcher.py***************
2025-12-12 20:14:31 INFO     [orchestrator]           [FINISHED] ** Action 'harvest_action'         completed with 'success'
2025-12-12 20:14:31 INFO     [orchestrator]           Finished running queued actions
2025-12-12 20:14:31 INFO     [orchestrator]             grow_action   : success  harvest_action: success
2025-12-12 20:14:31 INFO     [orchestrator]           All actions finished with success
2025-12-12 20:14:31 INFO     [orchestrator]           Finished in 0:00:00.360383
2025-12-12 20:14:31 INFO     [orchestrator]           Logfiles at /home/aislas/mango/log
2025-12-12 20:14:31 INFO     [orchestrator]           Save file at /home/aislas/mango/tmp/orchestrator.json
2025-12-12 20:14:31 INFO     [orchestrator]           JUnit file at /home/aislas/mango/log/results.xml
2025-12-12 20:14:31 INFO     [sane_runner]            Finished

Tip

This output can be reproduced by using the source repo example found at docs/examples/mango/json_harvest_action/.sane/

Again, reviewing the highlighted regions:

Our "harvest_action" is only executed after the "grow_action" has completed
The config["command"] is executed (this time with no config["arguments"])
The STDOUT shows that we harvested 20 mangos. Quite the haul!

✨ Congratulations! ✨

You’ve gone through the basic JSON interface tutorial and are ready to make some workflows!

If you’re looking to add more control to your workflows or for an extra challenge, check out the Advanced Topics.

JSON Interface

Preface

Orchestrator

Hosts

Resources & Environment

Actions

"config"

"environment"

"resources"

"dependencies"

Final Action Result

Running

Extending the workflow

Adding "dependencies"

New files

`"config"`

`"environment"`

`"resources"`

`"dependencies"`

Final `Action` Result

Adding `"dependencies"`