Attribute Dereferencing

SANE supports GitHub-Actions-style attribute dereferencing of strings within any instance of Action (including derived), usable within the JSON and Python interface.

Attribute dereferencing lets you reference values defined within an Action object dynamically at runtime. While the actual dereference operation (Action.dereference_str()) only acts on str, the main function Action.dereference() can fully operate on strings, lists, and nested dictionaries.

Dereferencing uses the syntax:

"${{ <attribute> }}""

A more detailed explanation can be viewed from this API excerpt:

Quick Reference (click to open/close):
Action.dereference_str(input_str, log=True, noexcept=False)[source]

Dereference an input string using GitHub Actions style syntax scoped to the current object

Continuously dereferences strings within the current object until no more substitutions can be made. This means that dereference strings can be nested. All attributes and properties can be referenced, but dereferencing will work best with attributes that are dict, list, str, or int values.

Dict referencing can be achieved with . operator (key as next field)
Index referencing can be achieved with [] operator (positive integer)

Valid syntax examples:

action_a = sane.Action( "action_a" )
action_b = sane.Action( "action_b" )
action_a.add_dependencies( "action_b" )

action_a.config["foo"] = "1"
action_a.config["bar"] = "2"
action_a.config["zoo"] = [ 3, 4 ]
action_a.config["boo"] = 7
action_a.config["moo"] = { "loo" : [ { "goo" : "5", "hoo" : [ 6, "${{ config.boo" }} ] }, 0, 0, 0 ] }
action_a.outputs["outfile"] = "something"
action_a.add_resource_requirements( { "cpus" : 12, "mem" : "1gb" } )

action_b.outputs["some_file"] = "fill_this_in"

# Within the context of action_a these are valid
"${{ config.foo }}"       => "1"
"${{ config.bar }}"       => "2"
"${{ config.zoo[1] }}"    => "4"
"${{ resources.cpus }}"   => "12"
"${{ outputs.outfile }}"  => "something"
# At runtime this would be valid, with this value if the value in action_b has not changed
"${{ dependencies.action_b.outputs.some_file }}" => "fill_this_in"

# A complex dereference
"${{ config.moo.loo[0].hoo[1] }}" => "7"

# A nested dereference
"${{ config.moo.loo[0].hoo[ ${{ config.moo.loo[ ${{ config.foo }} ] }} ] }} => "6"
#                                               ^^^^^^^^^^^^^^^^^ => "1"
#                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ => "0"

Attention

During the substitution, if indexing to the next attribute yields None an Exception will be thrown. Thus, at the time of dereferencing, the string input MUST be valid.

Parameters:

  • log – enable logging

  • noexcept – disable exceptions and instead allow failed dereference

  • input_str (str)

Returns:

string fully dereferenced

Return type:

str

Action.dereference(obj, log=True, noexcept=False)[source]

Fully dereference all strings within the obj passed in

For dict and list objects, each will be iterated over and this function will be recursively call for each iterated value (not key), and then assigned back to itself, presumably modified. For str objects dereference_str() will be called.

For all other object types, the obj will be unmodified.

The obj is then returned, modified if necessary (and possible) to contain only fully dereferenced strings.

Returns:

obj with all strings dereferenced


The Action.config is resolved at an action’s default run()1, so the values must be present and valid at dereference time.

Important

1 Not all Action attributes are dereferenced by default. The base class will always have fully dereferenced config and the info of dependencies (that parent Action's outputs and config by default) at the start of run().

Dependency info is always dereferenced.

Users are free to call dereference() at any time within any method of a derived Action class. If users override the Action.info() or Action.run() method they should take care to dereference() any attributes needed.

Basic JSON Example

The JSON interface commonly uses attribute dereferencing inside the "config" dictionary for an Action. For example:

{
  "actions" :
  {
    "hello" :
    {
      "config" :
      {
        "command" : "echo",
        "arguments" : [ "Running action ${{ id }}" ]
      }
    }
  }
}

When Action "hello" is run the string ${{ id }} will be replaced with the Action.id (hello in this example).

Basic Python Example

You can also use dereferencing from within the Python interface when you setup config dictionaries within sane.Action objects. For example:

import sane

@sane.register
def create_actions( orch ):
  a = sane.Action( "a" )
  b = sane.Action( "b" )

  b.add_dependencies( a.id )
  # Setup so Action 'b' can reference values from itself and dependencies
  a.config["greeting"] = "Hello from ${{ id }}"
  b.config["message"] = "Depends on '${{ dependencies.a.config.greeting }}'"
  orch.add_action( a )
  orch.add_action( b )

  a.config["command"] = "echo"
  a.config["arguments"] = [ "${{ config.greeting }}" ]

  b.config["command"] = "echo"
  b.config["arguments"] = [ "${{ config.message }}" ]

When Action "b" is run it will recursively replace the reference strings in "arguments" until the final output is "Depends on 'Hello from a'".

Hint

Note that the dependency info passed to "b" from "a" is dereferenced beforehand within the context of "a"

Callables

If an attribute resolves to a callable, it will be called WITH NO ARGUMENTS first before continuing dereferencing. An extra special case of this is the sane.Action.resources() method, where it will be invoked with the current host name (via Action.host_info) if available to ensure the resource dict returned matches the context of the current Host.

How SANE Implements Dereferencing

The bulk of implementation is within the Action.dereference_str() method. Feel free to click on the [source] hyperlink to follow along with the code walkthrough.

The method operates by building up a history of the string to dereference. On each pass:

  1. Check if the output string is not in the history

  2. The previous output string is appended to the history

  3. Matches are generated using the internal regular expression for dereference syntax

  4. For every match this iteration

    • Step through attributes in reference string until complete

    • Take final value and perform in-place substitution on output string

  5. Continue until output string is present in history, meaning there are no more substitutions

This method of dereferencing allows us to break down complex references or even detect cycles to prevent accidental infinite loops:

Dereferenced [0] '${{ config.two[ ${{ config.arr[ ${{ config.three.${{ config.foobar }} }} ] }} ] }}'
             [1] '${{ config.two[ ${{ config.arr[ ${{ config.three.foo }} ] }} ] }}'
             [2] '${{ config.two[ ${{ config.arr[ 3 ] }} ] }}'
             [3] '${{ config.two[ 0 ] }}'
     into => [4] '2'

Direct API Usage in Python

When you implement custom actions or need explicit control of dereferencing, call the Action helpers directly. Remember that dereferencing operates at the scope of the current Action and will throw an error on bad values unless the noexcept=True argument is provided.

import sane

class MyAction( sane.Action ):
  def load_data( self, file ):
    ...do work...

  def run( self ):
    # Our action will need a file from some previous action run
    datafile = self.dereference( "${{ dependencies.${{ config.data_from }}.output.data }}" )
    self.load_data( datafile )

This is a very simplified example where you could easily replace the datafile variable creation with datafile = self.dependencies[config["data_from"]]["output"]["data"]. However, it allows us to define this operation in a manner that goes beyond single resolution. Consider:

import sane

class MyAction( sane.Action ):
  def __init__( self, id ):
    # Data normally comes from this location
    self.datafile = "${{ dependencies.${{ config.data_from }}.output.data }}"

  def load_extra_options( self, options, origin ):
    # Allow instances of this action type to change where the data comes from
    self.datafile = options.pop( "datafile", self.datafile )
    super().load_extra_options( self, options, origin )

  def load_data( self, file ):
    ...do work...

  def run( self ):
    # Our action will need a file from some previous action run
    datafile = self.dereference( "${{ datafile }}" )
    self.load_data( datafile )

We’ve now set the datafile to something that cannot be used upon creation as it depends on runtime information. Likewise, we’ve given any user of this custom Action the option to change the file that gets loaded. At run() the logic for default setup stays exactly the same, but it is now more flexible to configurability. Replicating this with logic contained to just run() may look something like:

import sane

class MyAction( sane.Action ):
  def __init__( self, id ):
    # Data normally comes from this location
    self.datafile = None

  def load_extra_options( self, options, origin ):
    # Allow instances of this action type to change where the data comes from
    self.datafile = options.pop( "datafile", self.datafile )
    super().load_extra_options( self, options, origin )

  def load_data( self, file ):
    ...do work...

  def run( self ):
    if datafile is None:
      datafile = self.dependencies[config["data_from"]]["output"]["data"]
    # Our action will need a file from some previous action run
    self.load_data( datafile )

While subtle, notice that this “equivalent logic” still loses some configurability: if we override the datafile value at option load we cannot access any non-default runtime information. For example, we would not be able to set it to use output data from a build dependency instead (e.g. ${{ dependencies.build.output.data }}).

Fundamentally, while much of the logic for dereferencing could in theory be replaced with conditional or use-specific code the main use is allowing flexible runtime variable referencing for simpler configuration logic.

Runtime Notes and Safety

Dereferencing is a substitution-only mechanism and does not evaluate arbitrary code. It only accesses attributes and items available within the referenced objects, starting at the Action that called it.

Because resolution happens only at function execution, the referenced attributes must be present at least by then otherwise an exception is raised unless noexcept=True is passed to the API. Keep this in mind when making use of deferred evaluation.

Indexing into non-list or out-of-range indices raises an exception.

More Examples

The test suite contains focused examples of dereferencing behavior. See tests/test_action.py (the test_action_dereference* tests) for concrete cases demonstrating nested, indexed, and multi-stage dereferencing.

Summary

Attribute dereferencing is a compact, expressive mechanism to inject dynamic workflow values into configuration. In the default Action.run() the config attribute is dereferenced, but users are free to use this capability in any custom Action classes they write!