Skip to main content
Version: 3.0.0

Working with Workflows

Workflows are a flexible framework for achieving an outcome.

There are some best practices that can be followed to make understanding and developing easier.

This document outlines best practices for working with workflows, it is not an exhaustive description of workflows - please see the Getting Started guide for an overview.

Configuration recap​

Recapping opscotch configurations:

Fully understand what you are trying to achieve​

It is essential to understand what you are trying to achieve and how you'd go about doing that, as if you were not using opscotch.

A common approach is to use command line tools such as curl, or other tools like Postman or the browser network tools.

Here are some tips:

  • when writing a workflow, write out the discrete steps that you would perform
  • understand how you need to authenticate to hosts for each call
  • understand how you'd split and iterate data
  • use the comment property to describe what each workflow is trying to achieve 
{
    "comment" : [
        "Retrieve leave requests from HR system",
        "Flow:",
        "1. Pull the last check date from persistence and query the api for leaves modified since the last check date",
        "2. For each leave response: filter for leave with the same modified date and date submitted",
        "3. Pull user for the leave and generate the leave message",
        "4. Send to notifying system"
    ]
}
{
    "comment" : ["1. Pull the last check date from persistence and query the api for leaves modified since the last check date"]
}

How to set a header on all requests to a host​

Opscotch allows headers be defined in the bootstrap file host section and these headers will be applied to each request to that host.

For example to set the Content-Type : application/json header on all requests to myHost

{
    "myHost" : {
        "host" : "https://example.com",
        "headers" : {
            "Content-Type" : "application/json"
        }
    }
}

How to create and use processor libraries​

Processor code can be kept in files, referred to as "resources".

When packaging, set the resourceDirs property to the directory roots that contain the resource files.

How to reuse processors​

Processors can take a single resource or script property (the script will override the resource)

  • script allows you to write javascript directly into the processor
{
    "urlGenerator" : {
        "script" : "console.log('hello')"
    }
}
  • resource allows you to write into a file which is loaded into the script property a package time
{
    "urlGenerator" : {
        "resource" : "myProcessor.js"
    }
}

Because resource is a file, you can reuse it on multiple processors.

There is one more construct that can be used: processors - this allows you to mix processors into a composite that is executed in order:

{
    "urlGenerator" : {
        "processors" : [
            {
                "comment" : "this one runs first",
                "script" : "console.log('hello')"
            },
            {
                "resource" : "myProcessor.js"
            },
            {
                "comment" : "this one runs last",
                "script" : "console.log('done')"
            }
        ]
    }
}

How do processors work​

There is only a few things to know about processors.

  1. The are executed in order, in the context of the workflow that is running
  2. They are written in Javascript
  3. You can write code directly with the script property
  4. You can reuse resource files with the resource property
  5. You can pass data in via the data property
  6. They run in isolation and can only interact with the agent via the context object

Lets take an example: say you want to make a request to the URL: https://www.example.com/this/is/a/url

Because we're working on the URL, we'd do this in the urlGenerator processor.

The to interact with the agent, in this case setting URL, we can only use the context object.

Looking through the methods available on the context we can see setUrl(String hostref, String path). This takes a bootstrap host reference, and the url path.

If we had a bootstrap host like the following, then the host reference would be example, and the path would be /this/is/a/url

{
    "hosts" : {
        "example" : {
            "host" : "https://www.example.com"
        }
    }
}

Now we can use the the context.setUrl(...) function on the processor:

{
    "stepId" : "call-example",
    "urlGenerator" : {
        "script" : "context.setUrl('example', '/this/is/a/url')"
    }
}

Thats it, the url is now set to call https://www.example.com/this/is/a/url

This will default to a GET HTTP method. If you need to specify another HTTP method you need to use the context.setHttpMethod(String method) function in the processor. There are a few ways to do this.

You could append it to the script that you just wrote, but this gets a bit ugly:

{
    "stepId" : "call-example",
    "urlGenerator" : {
        "script" : "context.setUrl('example', '/this/is/a/url'); context.setHttpMethod('POST')"
    }
}

You could split the code lines into separate processors using the processors property:

{
    "stepId" : "call-example",
    "urlGenerator" : {
        "processors" [
            {
                "script" : "context.setUrl('example', '/this/is/a/url')"
            },
            {
                "script" : "context.setHttpMethod('POST')"
            }
        ]
    }
}

Or if this is something that you're going to use a lot, perhaps in other workflows you can make it reusable by putting it in a resource file in your resource directory (as set-url.js):

context.setUrl('example', '/this/is/a/url');
context.setHttpMethod('POST');

Then you can reference set-url.js in the processor:

{
    "stepId" : "call-example",
    "urlGenerator" : {
        "resource" : "set-url.js"
    }
}

However, you hopefully realized that while in theory its reusable, we've hard coded the host reference, URL path and the HTTP method! Read onwards for the solution.

How does the data property work​

The data property is an object on the following configurations:

The data property can be used to pass data that can be accessed by processors using the context.getData() or context.getRestrictedDataFromHost(String host) functions. In practice this can effectively be thought of as "parameter passing" to processors.

An important characteristic of the data properties is that they are merged along the configuration hierarchy tree, such that a property declared in a bootstrap is available in a host. Likewise a property that is declared in a bootstrap, or workflow or step is available in the processor

Data merging flow:

Data propertyMerged objects
bootstrap.databootstrap.data
host.databootstrap.data + host.data
workflow.databootstrap.data + workflow.data
step.databootstrap.data + workflow.data + step.data
processor.databootstrap.data + workflow.data + step.data + processor.data

The context.getData() functions return a string - if you are expecting a JSON element you will need to parse it:

data = JSON.parse(context.getData())

To use this in practice, consider the URL example above:

We defined the set-url.js processor file:

context.setUrl('example', '/this/is/a/url');
context.setHttpMethod('POST');

Then referenced set-url.js in the processor:

{
    "stepId" : "call-example",
    "urlGenerator" : {
        "resource" : "set-url.js"
    }
}

However, you hopefully realized that we hard coded the host reference, URL path and the HTTP method.

Lets fix this using the data property. First lets declare the data property, along with the host reference, path and HTTP method that we want to use:

{
    "stepId" : "call-example",
    "urlGenerator" : {
        "resource" : "set-url.js"
    },
    "data" : {
        "host-ref" : "example",
        "path" : "/this/is/a/url",
        "httpMethod" : "POST"
    }
}

Then we can change the set-url.js file to use the properties from the data property:

hostref = context.getData("host-ref");
path = context.getData("path");
method = context.getData("httpMethod");


context.setUrl(hostref, path);
context.setHttpMethod(method);

Now we can reuse this set-url.js processor file with little fuss:

{
    "steps" : [
        {
            "stepId" : "call-example-1",
            "urlGenerator" : {
                "resource" : "set-url.js"
            },
            "data" : {
                "host-ref" : "example",
                "path" : "/this/is/a/url",
                "httpMethod" : "POST"
            }
        },
        {
            "stepId" : "call-example-2",
            "urlGenerator" : {
                "resource" : "set-url.js"
            },
            "data" : {
                "host-ref" : "example",
                "path" : "/yet/another/url",
                "httpMethod" : "PUT"
            }
        }
    ]
}

Using pre-made processor resources​

Processor code can be reused by using files called "resources". When using resource files, store them in a resource directory - you need to configure this for packaging so that the resource file is imported into the workflow.

Resources often take parameters/arguments/input in the form of data properties. The expected input should be clear in the comments of the resource file.

Resource directories and files have no required structure, however they are generally grouped by the solution or problem they are solving. Resources that are for no specific solution, or for general use are available under the general directory - these resources perform common tasks like:

  • standard-url-generator.js : constructs a basic url
  • standard-forwarder.js : forward to another step
  • standard-data-as-body.js : use a data property as the body of a request
  • standard-json-resultprocessor.js : encapsulates complex json result aggregation
  • standard-restricted-data-as-auth.js : use host data for authentication
  • standard-restricted-set-cookie-as-auth.js : use a cookie for authentication

It is recommended that you skim over the files when you use them to understand what they are trying to achieve - this will help with understanding what is available and to learn how to implement new functionality.

How to call another step​

From any processor a step can be called just like a function.

Use the context method context.sendToStep("nextStepId", "message to send to step")

{
    "steps" [
        {
            "stepId" : "stepOne",
            "resultsProcessor" : {
                "processors" : [
                    {
                        "script" : "console.log('this is stepOne')"
                    },
                    {
                        "script" : "context.sendToStep('stepTwo', 'message')"
                    }
                ]
            }
        },
        {
            "stepId" : "stepTwo",
            "resultsProcessor" : {
                "script" : "console.log('this is stepTwo')"
            }
        }
    ]
}

When this executes it will

  • print out this is stepOne
  • call to stepTwo
  • print out this is stepTwo

Authentication Processing​

tip

You need to set an authentication processor on EACH step you need to perform authentication for the HTTP call

The authentication processor's job is to securely make changes to the HTTP state so that the HTTP call will pass authentication when the HTTP request is executed. For example setting a username and password or API key.

The authentication processor is unique among the processors in that operates in a restricted context: it can access sensitive information (such as credentials), it CANNOT log, data it produces is automatically redacted from the workflow logs, it can only access host data in the bootstrap that is set as "authenticationHost": true, it can not call out of the authentication context - ie you can not call to a standard step.

To work with the Authentication processor you can imagine that you are calling a function that will make changes to the HTTP state. You can call to any "scripted-auth" type step (imagine that as a function) but you can not call to any non "scripted-auth" steps. After the processor has done its job it will "return" just like a function and continue with the normal step processing.

DO NOT APPLY AUTHENTICATION REQUIREMENTS OUTSIDE OF AN AUTHENTICATION PROCESSOR

There are many ways to manipulate the HTTP request to add authentication requirements, however it is ESSENTIAL to running securely that you use an authentication processor to perform all authentication operations.

Header Authorization / Basic Authorization / Bearer Authorization​

You should already have an understanding of Header based Authorization when you read this section

A simple example of the authentication processor is adding a HTTP Authorization header to the request.

Common examples of the HTTP Authorization Header schemes (in the scope of this example, it does not matter which authentication scheme is used):

  • Basic <credentials>
  • Bearer <token>

Here is an example of the header in use: Authorizaton: Bearer nT7S9CsE7HWnM6LSQ

Opscotch allows headers be defined in the bootstrap file host section and these headers will be applied to each request to that host. You COULD add the Authorization header, however they won't be treated securely ie redacted from logs etc - setting headers in the bootstrap should only be used for non-sensitive headers.

DONT DO THIS

Do not use authorisation in the host headers

{
    "host" : "https://this-is-insecure.com",
    "headers" : {
        "Authorizaton" : "Bearer nT7S9CsE7HWnM6LSQ"
    }
}

We use the authentication processor to set sensitive headers with all the security precautions taken.

First up, the bootstrap host needs to have the following property set: "authenticationHost": true. Second, we can use the data object to create a property to hold the Basic Authorization header value:

{
    "hosts" : {
        "myAuthHost" : {
            "authenticationHost": true,
            "host" : "https://www.more-secure.com",
            "data" : {
                "authorizationHeader" : "Bearer nT7S9CsE7HWnM6LSQ"
            }
        }
    }   
}

This alone will do nothing. On the step(s) that we are applying authentication to, configure the authentication processor.

We will use a pre-defined processor that will do just what we need, the standard-restricted-data-as-header.js - this will do what it says in the name: Take some data from a restricted hosted (where the "authenticationHost": true property has been set) and set it as a HTTP header. We need to provide the value of the header name to set (Authorization as per the HTTP Authorization spec) and the host data property name that has the header value (we called the property authorizationHeader in the example above):

From the bootstrap host example above, if we want to securely set a header equivalent to this on the next HTTP request: Authorizaton: Bearer nT7S9CsE7HWnM6LSQ

  • we can use the pre-defined standard-restricted-data-as-header.js authentication processor
  • and set the processor data property "fromHost" : "myAuthHost" to reference the myAuthHost in the bootstrap
  • and set the processor data property "keyOfValue" : "authorizationHeader" to reference the fromHost data property authorizationHeader value Bearer nT7S9CsE7HWnM6LSQ
  • and set the processor data property "headerName" : "Authorization" to set the header name to Authorization:
{
    "type": "scripted",
    "stepId": "exampleStep",
    "authenticationProcessor" : {
        "resource" : "/general/authentication/standard-restricted-data-as-header.js",
        "data" : {
            "fromHost" : "myAuthHost",
            "keyOfValue" : "authorizationHeader",
            "headerName" : "Authorization"
        }
    }
    ...
}

When this executes, it will access the authentication restricted host myAuthHost data object, extract the property authorizationHeader and set that as the value for the header named Authorization.

The comment property​

The comment property is somewhat undocumented feature that allows you to add comments throughout the workflow configuration.

A comment can be a string:

{
    "comment" : "This is a valid comment"
}

or an array of strings:

{
    "comment" : [
        "These are also",
        "valid comments"
    ]
}

Comments can be used on:

  • workflows
  • steps
  • processors