Skip to main content
Version: 3.0.x

API Reference

This documentation describes the functionality that is available for configuration authors.

Documentation describing the functionality and schema of configuration (workflow, bootstrap etc.) is referred to as the model and is available below.

Documentation describing the functionality and schema of the testing harness is available below.

opscotch configuration model​

The opscotch model is relatively straight forward, and uses repeated patterns - once you grasp these you should be able to understand the entire structure.

Here is the outline of the entire structure of the two configurations: the bootstrap and the workflow. Note that not every field is required, so be sure to check the detailed descriptions.

The bootstrap schema overview​

[
    {
        "enabled" : true,
        "agentPrivateKey": "",
        "deploymentId": "",
        "remoteConfiguration": "",
        "remoteConfigurationAuth": "",
        "remoteConfigurationTimeout" : 0,
        "frequency": 0,
        "reloadTimeout" : 0,
        "errorHandling": {
            "enableLocalLogging": true,
            "metrics": {
                "enabled": true,
                "routingToken": "",
                "outputUrl": "",
                "outputAuthorization": ""
            },
            "logs": {
                "enabled": true,
                "routingToken": "",
                "outputUrl": "",
                "outputAuthorization": ""
            },
            "redactionPatterns": [
                ""
            ]
        },
        "workflow" : {
            "metricOutput": {
                "enabled": true,
                "routingToken": "",
                "outputUrl": "",
                "outputAuthorization": ""
            },
            "errorHandling": {
                "enableLocalLogging": true,
                "metrics": {
                    "enabled": true,
                    "routingToken": "",
                    "outputUrl": "",
                    "outputAuthorization": ""
                },
                "logs": {
                    "enabled": true,
                    "routingToken": "",
                    "outputUrl": "",
                    "outputAuthorization": ""
                }
            }
        },
        "allowExternalHostAccess": [
            {
                "authenticationHost" : true,
                "id" : "",
                "host": "",
                "headers": {
                    "": ""
                },
                "allowList" : [
                    {
                        "method" : "",
                        "uriPattern" : ""
                    }
                ],
                "data" : {}
            }
        ],
        "allowFileAccess": [
            {
                "id" : "",
                "directoryOrFile" : "",
                "patterns" : [""],
                "LIST" : true,
                "READ" : true,
                "WRITE" : true,
                "DELETE" : true
            }
        ],
        "allowHttpServerAccess": [
            {
                "id" : "",
                "port" : 0,
                "serveFromPackagedAsset" : {
                    "packagedAssetFileId": ""
                }
            }
        ],
        "persistenceRoot" : "",
        "data": {}
    }
]

The workflow schema overview​

{
    "workflows": [
        {
            "enabled": true,
            "name": "",
            "steps": [
                {
                    "enabled": true,
                    "debug": true,
                    "type": "",
                    "stepId": "",
                    "trigger" : {
                        "fileWatcher": {
                            "bootstrapFileId": "",
                            "eventSplitter": "",
                            "patterns": [ "" ],
                            "splitAtEnd": false
                        },
                        "http": {
                            "method": "",
                            "path": "",
                            "server" : ""
                        },
                        "timer" : {
                            "delay" : 0,
                            "period" : 0
                        },
                        "runOnce" : true
                    },
                    "authenticationProcessor": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "splitGenerator": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "urlGenerator": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "payloadGenerator": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "itemResultProcessor": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "resultsProcessor": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "httpStatusHandling": {
                        "" : {
                            "script": "",
                            "resource": "",
                            "processors" : [
                                {
                                "script": "",
                                "resource": ""
                                "data": {}
                                }
                            ],
                            "data": {}
                        }
                    },
                    "httpConnectFailedProcessor": {
                        "script": "",
                        "resource": "",
                        "processors" : [
                            {
                                "script": "",
                                "resource": ""
                                "data": {}
                            }
                        ],
                        "data": {}
                    },
                    "httpTimeout" : 0,
                    "singleThreaded" : "",
                    "persistenceFile" : "",
                    "data": {}
                }
            ],
            "data" : {},
        }
    ]
}

Bootstrap JSON Schemas​

JSON Schema in the Bootstrap Package:

Bootstrap​

A configuration that is loaded into the agent at startup and determines how the agent loads the main configuration is loaded.

Please note that the bootstap file that is authored is an ARRAY of these objects ie:

[
      {
          "agentPrivateKey": ...
          "deploymentId": ...
      }
]

It will be provided to the agent as an argument, either via a base64 encoded json text or a path to a json file

JSON Properties​

The following properties are available to set on the JSON schema.

agentPrivateKey​

Type: String

Required The private key for the agent to decrypt the remote configuration.

This should be a base64 encoded private key in pkcs8 format

This is how to create an appropriate key

1. Create key pair
openssl genrsa -out keypair.pem 2048

2. Extract public part
openssl rsa -in keypair.pem -pubout -out public.key

3. Extract private part
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in keypair.pem -out private.key

4. Base64 the contents of this file
cat private.key | base64
{
  "agentPrivateKey": "LS0tLS1CRUdJTiBQUklWQV...."
}

This property is required

allowExternalHostAccess​

Type: List of Host

A list of Host objects that describes access to external http(s) services that can be called from the workflow configuration.

{
  "allowExternalHostAccess": [
      {
          "id" : "myHost",
          "host": "https://www.example.com",
          "headers": {
              "Content-Type": "application/json;charset=UTF-8",
              "Accept": "application/json, text/plain, *\/*"
          },
          "allowList" : [
              { "method" : "GET", "uriPattern" : "/this/is/allowed.*"},
              { "method" : "POST", "uriPattern" : "/this/.*\/allowed"}
          ],
          "data" : {

          }
      }
  }
]

This property is optional

allowFileAccess​

Type: List of FilePermitter

A list of File Permitters that describe access to files and directories.

When a directory is specified the permissions apply to all files in that directory.

Available permissions are:

  • LIST
  • READ
  • WRITE
{
  "allowFileAccess": [
      {
          "id" : "myTxtFiles",
          "directoryOrFile" : "/path/to/a/directory",
          "pattern" : ".*\\.txt",
          "READ" : true
      }
  ]
}

This property is optional

allowHttpServerAccess​

Type: List of HttpServerPermitter

A list of Http Server Permitters that describe access https servers running in the agent.

This is a configuration for starting HTTP servers in the agent.

{
  "allowHttpServerAccess": [
      {
          "id" : "myHttpServer",
          "port" : 1234
      }
  ]
}

This property is optional

data​

Type: Object

A JSON Object for adding properties.

These properties will be merged with the WorkflowConfiguration.data configuration field and be available to all child elements in the main configuration tree.

{
  "data": {
      "somedata" : {
          "abc: 123
      },
      "someotherdata" : [ 1, 2, 3]
  }
}

This property is optional

deploymentId​

Type: String

Required A Service Provider defined value that must be unique per deployed agent configuration.

This property is used by Service Provider downstream processor to enrich the transmitted data.

{
  "deploymentId": "a7d2f8"
}

This property is required

enabled​

Type: Boolean

Disable is bootstrap by setting enabled : false

Defaults to true

{
  "enabled": false
}

This property is optional

errorHandling​

Type: ErrorHandling

Determines the behavior of error handling while loading the main configuration

{
  "errorHandling": {
      ...
  }
}

This property is optional

frequency​

Type: Integer

When using a URL Bootstrap.remoteConfiguration, setting this property will set the polling period in milliseconds

Defaults to 60000 (60 seconds)

{
  "remoteConfiguration": "http://www.example.com",
  "remoteConfigurationAuth" : "xodsjfujrhdjfk",
  "frequency" : 60000
}

This property is optional

hosts​

DEPRICATED: use allowExternalHostAccess Type: Object of Host

A set of named Host objects. This allows the bootstrap to define http(s) host that can be called from the main configuration.

{
  "hosts": {
      "mynamedhost": {
          "host": "https://www.example.com",
          "stepId": "example.com",
          "headers": {
              "Content-Type": "application/json;charset=UTF-8",
              "Accept": "application/json, text/plain, *\/*"
          },
          "allowlist" : [
              ["GET", "/this/is/allowed.*"],
              ["POST", "/this/.*\/allowed"]
          ],
          "data" : {

          }
      }
  }
}

This property is optional

persistenceRoot​

Type: String

Defines the filesystem storage root for persistence

{
  "persistenceRoot": "/storage"
}

This property is optional

reloadTimeout​

Type: Integer

Defines the maximum time in milliseconds to wait for a workflow to terminate running flows before loading a new version

Defaults to 10000 (10 seconds)

{
  "reloadTimeout": 1000
}

This property is optional

remoteConfiguration​

Type: String

Required A URL or file path to the main configuration.

The main configuration does not exist, the agent will wait until it becomes available. When referencing a file, its important to remember that the path is relative to the opscotch working directory, NOT the bootstrap file

{
  "remoteConfiguration": "http://www.example.com",
  "remoteConfigurationAuth" : "xodsjfujrhdjfk",
  "frequency" : 60000
}

This property is required

remoteConfigurationAuth​

Type: String

When using a URL remoteConfiguration, setting this property will set the "Authorization" header

{
  "remoteConfiguration": "http://www.example.com",
  "remoteConfigurationAuth" : "xodsjfujrhdjfk",
  "frequency" : 60000
}

This property is optional

remoteConfigurationTimeout​

Type: Integer

When using a URL remoteConfiguration, setting this property will set the http timeout period in milliseconds

The remoteConfigurationTimeout must be less than the frequency

Defaults to 1000 (1 second)

{
  "remoteConfiguration": "http://www.example.com",
  "remoteConfigurationAuth" : "xodsjfujrhdjfk",
  "frequency" : 60000,
  "remoteConfigurationTimeout" : 1000
}

This property is optional

workflow​

Type: WorkflowOutputs

Determines metric and log outputs for the workflows

{
  "workflow": {
      ...
  }
}

This property is optional

ErrorHandling​

A configuration defining if and how to send error metrics or log to a remote system

JSON Properties​

The following properties are available to set on the JSON schema.

enableLocalLogging​

Type: Boolean

Enable or disables logging of errors and information to the agent standard out

Defaults to true

{
  "enableLocalLogging": true
}

This property is optional

logs​

Type: Output

Defines how error logs are sent to a remote system.

Error logs are brief messages from a fault in the agent and are sent to a remote system for attention.

Error log configuration only apply to the process configuration they are defined for. For example the Bootstrap.errorHandling defines error handling while loading configurations, and the WorkflowConfiguration.errorHandling defines error handling during the execution of configuration steps.

{
  "logs": {
      ...
  }
}

This property is optional

metrics​

Type: Output

Defines how error metrics are sent to a remote system.

Error metrics represent a fault in the agent and are sent to a remote system for attention.

Error metric configuration only apply to the process configuration they are defined for. For example the Bootstrap.errorHandling defines error handling while loading configurations, and the WorkflowConfiguration.errorHandling defines error handling during the execution of configuration steps.

{
  "metrics": {
      ...
  }
}

This property is optional

redactionPatterns​

Type: Array of String

Defines regex patterns to apply redaction to log statements

Log statements from will have these redaction patterns applied before logging.

{
  "redactionPatterns": [
      "[0-9]+"
  ]
}

This property is optional

FilePermitter​

Describes controlled access to files and directories.

Regular Expressions can be used to further restrict access to specific files.

JSON Properties​

The following properties are available to set on the JSON schema.

DELETE​

Type: Boolean

When true files will be deletable.

Defaults to false

{
  "DELETE": true
}

This property is optional

LIST​

Type: Boolean

When true files will be listable.

Defaults to false

{
  "LIST": true
}

This property is optional

READ​

Type: Boolean

When true files will be readable.

Defaults to false

{
  "READ": true
}

This property is optional

WRITE​

Type: Boolean

When true files will be writable.

Defaults to false

{
  "WRITE": true
}

This property is optional

directoryOrFile​

Type: String

A path to a directory or file.

The directory or file does not need to exist. If a directory is specified it must end in a / (or the OS file separator)

Directory example:

{
  "directoryOrFile": "/path/to/directory/"
}

File example:

{
  "directoryOrFile": "/path/to/directory/or/file"
}

This property is required

id​

Type: String

An id that must be unique to this bootstrap.

Use this id in the workflow step file trigger.

{
  "id": "myTextFiles"
}

This property is required

patterns​

Type: List of Strings

A list of (Java) Regular Expressions that can be used to further restrict access to files in the directory

{
  "patterns": [
      ".*\\.txt$",
      ".*\\.log$"
  ]
}

This property is optional

Host​

A HTTP(S) host that the agent is expected to communicate with.

This only applies to customer specific hosts, not to Service Provider hosts.

Hosts are defined in Bootstrap.data and WorkflowConfiguration.data

JSON Properties​

The following properties are available to set on the JSON schema.

allowList​

Type: List of HostAllow

A list of allowed paths that may be called on this host.

A list of regex that will be matched to URL path. Matching results will be allowed, non-matching results will be blocked and logged.

{
  "allowList": [
      { "method" : "GET", "uriPattern" : "/this/is/allowed"},
      { "method" : "POST", "uriPattern" : "/this/.*\/allowed"}
  ]
}

This property is optional

authenticationHost​

Type: Boolean

When set to true the host can only be used by an authentication context, when set to false (default) the host can only be used by a non-authentication context.

{
  "authenticationHost": true
}

Defaults to false

This property is optional

data​

Type: Object

A JSON Object with Additional properties that will accessible during authentication steps.

You can use environment variables in the form of ${NAME} anywhere in the data structure: the environment variable will be substituted prior to parsing.

Strings in the data structure (after environment variable substitution) can be encrypted using the opscotch packager. Values will be decrypted at access time.

Encryption is done via the packaging tool and requires the agent public key.

{
  "data": {
      "somedata" : {
          "abc: 123
      },
      "someotherdata" : [ 1, 2, 3],
      "someencryptedstring" : {
          "abc" : "some encrypted value"
      }
  }
}

This property is optional

headers​

Type: Object of Strings

A object of HTTP headers to be sent with every call.

{
  "headers": {
      "Content-Type": "application/json;charset=UTF-8",
      "Accept": "application/json, text/plain, *\/*"
  }
}

This property is optional

host​

Type: String

Required A URL that is used as the base for constructing URLs to call.

{
  "host": "https://www.example.com:8080"
}

This property is required

id​

Type: String

Required A unique string that identifies this host in other configurations.

{
  "id": "myHost"
}

This property is required

HostAllow​

Defines allowed http access patterns

JSON Properties​

The following properties are available to set on the JSON schema.

method​

Type: String

HTTP Method to match on. One of:

  • GET
  • HEAD
  • POST
  • PUT
  • DELETE
  • OPTIONS
  • PATCH
{
  "method": "GET"
}

This property is required

uriPattern​

Type: String

Regex pattern to match on the URI.

{
  "uriPattern": ".*"
}

This property is required

HttpServerPermitter​

Creates an HTTP server bound to the specified port.

Steps for this bootstrap using a http trigger can bind to this http server.

A special feature is being able to serve static content from a zip file - see serveFromPackagedAsset

JSON Properties​

The following properties are available to set on the JSON schema.

id​

Type: String

Required A unique string that identifies this http server in other configurations.

{
  "id": "myAPI"
}

This property is required

port​

Type: String

Required A port on the host that this http server will bind to. Normal port binding rules apply.

{
  "port": 1234
}

This property is required

serveFromPackagedAsset​

Type: ServePackagedAsset

Describes how to serve content from a zip file.

{
  "serveFromPackagedAsset": {
      ...
  }
}

This property is optional

OldHost​

DEPRICATED This is the old host structure and has been replaced with the new host structure. See allowExternalHostAccess

A HTTP(S) host that the agent is expected to communicate with.

This only applies to customer specific hosts, not to Service Provider hosts.

Hosts are defined in Bootstrap.data and WorkflowConfiguration.data

JSON Properties​

The following properties are available to set on the JSON schema.

allowlist​

Type: List of HostAllow

A list of allowed paths that may be called on this host.

A list of regex that will be matched to URL path. Matching results will be allowed, non-matching results will be blocked and logged.

{
  "allowlist": [
      ["GET","/this/is/allowed"],
      ["POST","/this/.*\/allowed"]
  ]
}

This property is optional

authenticationHost​

Type: Boolean

When set to true the host can only be used by an authentication context, when set to false (default) the host can only be used by a non-authentication context.

{
  "authenticationHost": true
}

Defaults to false

This property is optional

data​

Type: Object

A JSON Object with Additional properties that will accessible during authentication steps.

You can use environment variables in the form of ${NAME} anywhere in the data structure: the environment variable will be substituted prior to parsing.

Strings in the data structure (after environment variable substitution) can be encrypted using the opscotch packager. Values will be decrypted at access time.

Encryption is done via the packaging tool and requires the agent public key.

{
  "data": {
      "somedata" : {
          "abc: 123
      },
      "someotherdata" : [ 1, 2, 3],
      "someencryptedstring" : {
          "abc" : "some encrypted value"
      }
  }
}

This property is optional

headers​

Type: Object of Strings

A object of HTTP headers to be sent with every call.

{
  "headers": {
      "Content-Type": "application/json;charset=UTF-8",
      "Accept": "application/json, text/plain, *\/*"
  }
}

This property is optional

host​

Type: String

Required A URL that is used as the base for constructing URLs to call.

{
  "host": "https://www.example.com:8080"
}

This property is required

Output​

Configuration for how to send data to a remote system.

JSON Properties​

The following properties are available to set on the JSON schema.

enabled​

Type: Boolean

Enables or disables the sending of data. Defaults to false

{
  "enabled": true
}

This property is optional

outputAuthorization​

Type: String

Set the "Authorization" header.

{
  "outputAuthorization" : "xodsjfujrhdjfk"
}

This property is optional

outputUrl​

Type: String

Required The URL that data is transmitted to.

{
  "outputUrl": "http://www.example.com"
}

This property is required

persistenceRoot​

Type: String

If set, defines the location to store output queue files

{
     "persistenceRoot" : "/tmp/output-queues"
}
routingToken​

Type: String

Required A Service Provider defined value that must be unique per deployed agent.

This property is used by Service Provider downstream processor to direct and enrich the transmitted data.

{
  "routingToken": "dogurj"
}

This property is required

ServePackagedAsset​

Defines how to serve static files from a packaged asset (zip file)

JSON Properties​

The following properties are available to set on the JSON schema.

packagedAssetFileId​

Type: String

Identifies a FilePermitter in the bootstrap to serve from.

THe FilePermitter should be a path to a zip file.

{
  "packagedAssetFileId": "myZipFile"
}

This property is required

WorkflowConfiguration​

The configuration that describes the main activities of the agent

JSON Properties​

The following properties are available to set on the JSON schema.

data​

Type: Object

A JSON Object for adding properties.

These properties will be merged with the Bootstrap.data configuration field and be available to all child elements.

{
  "data": {
      "somedata" : {
          "abc: 123
      },
      "someotherdata" : [ 1, 2, 3]
  }
}

This property is optional

workflows​

Type: Array of Workflow

Required A list of Workflow describing the agent tasks

{
  "workflows" [
      ...
  ]
}

This property is required

WorkflowOutputs​

Determines the behaviour of error handling and metrics for the workflow

JSON Properties​

The following properties are available to set on the JSON schema.

errorHandling​

Type: ErrorHandling

Determines the behavior of error handling while running the main configuration

{
  "errorHandling": {
      ...
  }
}

This property is optional

metricOutput​

Type: Output

Metrics represent a data point that is interesting, generated by the agent and are sent to a remote system for analysis.

{
  "metricOutput": {
      ...
  }
}

This property is optional

Workflow JSON Schemas​

JSON Schema in the Workflow Package:

Scripting context objects in the Workflow Package:

FileWatchingTrigger​

Defines a trigger that fires when new lines are added to a file.

Each event (split by the lineSplit property) will be sent to the step processor. A sample event:

{
   "log": {
     "file": {
       "path": "/path/to/file.txt"
     },
     "offset": 4130
   },
   "message": "this is a line from a file",
   "input": {
     "type": "log"
   },
   "host": {
     "name": "hostname",
     "ip": "[fe80:0:0:0:a6d7:feea:f601:902%wlp1s0, 192.168.0.27]"
   },
   "agent": {
     "type": "opscotch",
     "version": "3.0.0"
   },
   "ecs": {
     "version": "1.12"
   }
}
JSON Properties​

The following properties are available to set on the JSON schema.

bootstrapFileId​

Type: String

Defines the allowed file access from the bootstrap

This must match an id in a bootstrap.allowFileAccess.id

{
   "bootstrapFileId": "myTestFiles"
}

This property is required

eventSplitter​

Type: String

Defines the (Java) regular expressions used to split events.

The delimiter will be present on the event message. The events that are collected from this trigger will be passed into the step processor as a list. The event has select field from the ECS for base, log, host, agent.

{
   "eventSplitter": "\\n"
}

This property is required

patterns​

Type: List of String

A list of (Java) regular expressions used for file selection

{
   "patterns": [
      ".+\\.txt$",
      ".+\\.log$"
   ]
}

This property is optional

splitAtEnd​

Type: Boolean

When true the delimiter is at the end of the line, when false the delimiter will be at the start of the line.

Defaults to true

{
   "splitAtEnd": false
}

This property is optional

HttpRequestTrigger​

Describes how to bind to a http request

JSON Properties​

The following properties are available to set on the JSON schema.

method​

Type: String

The HTTP method to listen for.

Defaults to GET

{
   "method": "POST"
}

This property is optional

path​

Type: String

A regular expression that matches against the URI path

{
   "path": "/api/.*"
}

This property is required

server​

Type: String

Identifies the bootstrap http server to listen to

{
   "server": "myAPI"
}

This property is required

JavaScriptSource​

A JavaScript that is executed during the processing of a Step

JSON Properties​

The following properties are available to set on the JSON schema.

data​

Type: Object

A JSON Object for adding properties.

Used to pass parameters and data to the script.

These properties will be merged with all other data fields in the json path, available to the step script elements.

{
      "data": {
          "somedata" : {
              "abc: 123
          },
          "someotherdata" : [ 1, 2, 3],
      }
}

This property is optional

resource​

Type: String

A file path to the script to execute.

Must be supplied if JavaScriptSource.script is not supplied

{
  "resource": "scripts/myscript.js"
}

This property is optional

script​

Type: String

The script to execute.

A json-escaped script. Must be supplied if JavaScriptSource.resource is not supplied

{
  "script": "console.log(\\"hello world\\");"
}

This property is optional

JavascriptProcessor​

JSON Properties​

The following properties are available to set on the JSON schema.

processors​

Type: List of JavaScriptSource

A list of JavaScriptSource that will be executed in order

{
  "processors" : [
      {
          "resource" : "..."
      },
      {
          "resource" : "..."
      }
  ]
}

This property is optional

PersistentQueueParams​

NOT IMPLEMENTED

JSON Properties​

The following properties are available to set on the JSON schema.

maxFileBytes​

HARD CODED TO 5MB

maxMemoryBytes​

HARD CODED TO 1MB (1024 * 1024)

queuefilePrefix​

HARD CODED TO persistenceFile + "-queue-" + deploymentId + "-" + stepId

Step​

The Step is the primary configuration item for the agent configuration.

This configuration describes how to perform a single step or action in a Workflow.

There are different types of steps, however they all have the same execution structure:

  • Prepare
  • Execute
  • Process

Steps take three forms of input:

  • The response from a call to a Host (context.getMessageBodyAsString())
  • The message passed from another Step (context.getPassedMessageAsString())
  • Data from the configuration (context.getData())

The different types of Steps allow for specialised behaviour:

  • "scripted" type: this is the default type with the following flow:

    1. Optionally generate a URL (urlGenerator)

    2. Optionally (requires a url to have been set) generate a payload (payloadGenerator)

    3. Optionally perform authentication steps (authenticationProcessor, call other steps to perform authentication)

    4. Optionally call out to a named Bootstrap Host with the generated URL and payload

    5. Optionally handle specific HTTP status codes with httpStatusHandling

      OR

    6. Process the response or perform after-call processing (resultsProcessor)

      Perhaps:

      • Send a message to another Step
      • Produce some metrics

  • "scripted-split-aggregate" type: this is a modified "scripted" type that has the following workflow of:

    1. transforming the input into a list (splitGenerator)

    2. calling a URL with each item in the list:

      1. Optionally generate a URL (urlGenerator)

      2. Optionally (requires a URL to have been set) generate a payload (payloadGenerator)

      3. Optionally perform authentication steps (authenticationProcessor, call other steps to perform authentication)

      4. Optionally call out to a named Bootstrap Host with the generated URL and payload

      5. Optionally handle specific HTTP status codes with httpStatusHandling

        OR

      6. Optionally Process each item's response (itemResultProcessor)

      7. Collect the result into a response list

    3. The list of collected responses can then be processed and per normal (resultsProcessor)

      Perhaps:

      • Send a message to another Step
      • Produce some metrics

      Please note that in the case that one of the HTTP requests fail, it is your responsibility to handle this case by using the httpStatusHandling, itemResultProcessor, and resultsProcessor

  • "scripted-auth" type: this is identical to "scripted" except that it has an alternative javascript context - it is not allowed to send metrics or call to non-authentication steps, but has access to the bootstrap data which may contain secrets.

  • "httpFile" type: this is a specialized step that serve static content from a zip file by mapping a url path into a zip file path. This type REQUIRES a http trigger to be defined, an only allows GET requests

At least one step in a Workflow will define a Step.trigger that will trigger the start of Workflow processing.

These forms of input and different type allows for the construction of flexible pipelines that should be sufficient to perform most tasks.

JSON Properties​

The following properties are available to set on the JSON schema.

authenticationProcessor​

Type: JavascriptProcessor

Performs authentication tasks. Generally this step might call to authentication steps and/or apply headers.

{
    "authenticationProcessor": {
        ...
    }
}

This property is optional

data​

Type: Object

A JSON Object for adding properties.

Used to pass parameters and data to the JavaScriptSource elements.

These properties will be merged with all other data fields in the json path, available to the step JavaScriptSource elements.

{
    "data": {
        "somedata" : {
            "abc: 123
        },
        "someotherdata" : [ 1, 2, 3],
    }
}

This property is optional

debug​

Type: Boolean

Enables or disables the debug logs for this step. Defaults to false.

{
    "debug": true
}

This property is optional

enabled​

Type: Boolean

Enables or disables the execution of this step. Defaults to true.

{
   "enabled": true
}

This property is optional

httpConnectFailedProcessor​

Type: JavascriptProcessor

Defines how to handle HTTP connection failures using a standard processor

{
    "httpConnectFailedProcessor": {
        ...
    }
}

This property is optional

httpStatusHandling​

Type: Map of String to JavascriptProcessor

A set of JavascriptProcessor to process depending on the http status.

The key is the HTTP status either as a discrete number ie "401" or a range "400-499" and the value is a (standard processor)(#JavascriptProcessor).

When a status code matches a status handler and the handler is executed the resultProcessor or itemProcessor will not be executed.

Note: When supplying ANY httpStatusHandling you will also need explicitly declare success codes ie 200 as the resultsProcessor will not be called

{
    "httpStatusHandling": {
      "301-302" : {
          ...
      },
      "401" : {
          ...
      }
    }
}

This property is optional

httpTimeout​

Type: Integer

Set the http timeout period in milliseconds

Defaults to 10000 (10 seconds)

{
  "httpTimeout" : 10000
}

This property is optional

itemResultProcessor​

Type: JavascriptProcessor

When using Step.type "scripted-split-aggregate", defines if and how to process items before they're added to the list.

{
    "itemResultProcessor": {
        ...
    }
}

This property is optional

payloadGenerator​

Type: JavascriptProcessor

Defines how to generate the call payload, including setting headers.

This requires the urlGenerator to be present and will throw a validation exception. If this processor is not used, then the HTTP body will be removed.

{
    "payloadGenerator": {
        ...
    }
}

This property is optional

persistenceFile​

Type: String

Defines a cache/memory/key-value store that the Step has access to.

Required if using various functions. If required and not defined an error will be logged

{
    "persistenceFile": "persistenceFile.data"
}

This property is optional

resultsProcessor​

Type: JavascriptProcessor

Required Defines how to process the result of the call to the host.

Note: You might setup result processing via the httpStatusHandling property - you will still need to define a resultsProcessor though, even if it is an empty string.

{
    "resultsProcessor": {
        ...
    }
}

This property is required

singleThreaded​

Type: String

Determines the behaviour when the step is called while it's still executing.

Options are:

  • "none" - step is not single threaded and can be run simultaneously (default)
  • "return" - step is single threaded: if the step is already running, any calls to start the step will return immediately without running

Future options are likely : queue - execution will wait for the running execution to complete

This property is optional

splitGenerator​

Type: JavascriptProcessor

When using Step.type "scripted-split-aggregate", defines how to generate the list to be operated on.

{
    "splitGenerator": {
        ...
    }
}

This property is optional

stepId​

Type: String

Required Sets the id of the step.

Must be unique.

{
    "stepId": "the-first-step-in-my-route"
}

This property is required

timer​

Depricated: use Trigger

Type: String

Defines a timer to trigger the execution of a step.

There will likely only be one timer in a Workflow.

Accepts Apache Camel Timer query string parameters

{
    "timer": "fixedRate=true&period=3600000"
}

This property is optional

trigger​

Type: Trigger

Defines how a workflow is started.

{
     "trigger": {
         ...
     }
}

This property is optional

type​

Type: String

Sets the type of the step.

Either "scripted", "scripted-split-aggregate", "scripted-auth" as "httpFile" as defined above.

Defaults to "scripted"

{
    "type": "scripted-split-aggregate"
}

This property is optional

urlGenerator​

Type: JavascriptProcessor

Required Defines how to generate the URL

This processor should not attempt to set the payload (the current internal logic may result in unexpected results). To set the payload, use the payloadGenerator

{
    "urlGenerator": {
        ...
    }
}

This property is optional

Timer​

Defines a repeating, fixed-rate period trigger

JSON Properties​

The following properties are available to set on the JSON schema.

delay​

Type: Long

Defines a delay in milliseconds before the first invocation.

Defaults to 0.

{
   "delay": 60000
}

This property is optional

period​

Type: Long

Defines a delay in milliseconds between invocations.

Defaults to 1000.

{
   "period": 60000
}

This property is optional

Trigger​

Defines the starting condition for a workflow

JSON Properties​

The following properties are available to set on the JSON schema.

fileWatcher​

Type: FileWatchingTrigger

Defines a trigger that fires when new lines are added to a file.

{
   "fileWatcher": {
       ...
   }
}

This property is optional

http​

Type: HttpRequestTrigger

Defines a trigger that fires when a matching http request is received.

{
   "http": {
       ...
   }
}

This property is optional

runOnce​

Type: Boolean

Sets trigger to run only once.

Defaults to false

{
   "runOnce": true
}

This property is optional

timer​

Type: Timer

Defines a repeating period trigger

{
   "timer": {
       ...
   }
}

This property is optional

Workflow​

Defines a set of Steps that the agent will perform in order to generate metrics

JSON Properties​

The following properties are available to set on the JSON schema.

data​

Type: Object

A JSON Object for adding properties.

Used to pass parameters and data to the JavaScriptSource elements.

These properties will be merged with all other data fields in the json path, available to the step JavaScriptSource elements.

{
  "data": {
      "somedata" : {
          "abc: 123
      },
      "someotherdata" : [ 1, 2, 3],
  }
}

This property is optional

enabled​

Type: Boolean

Enable or disables the loading of the route.

Defaults to true

{
  "enabled": false
}

This property is optional

name​

Type: String

Required The user friendly name of the route.

{
  "name": "my route"
}

This property is required

steps​

Type: Array of Step

Required The list of Steps or actions to take to perform the intention of the Workflow.

Please note that steps are not executed in order, rather they are a group of actions that can send messages to one and other. The order of execution is defined by the actions taken by the Step

{
  "steps": [
      ...
  ]
}

This property is required

Workflow Javascript Context​

AuthenticationJavascriptContext​

This is a specialised Javascript Context for authentication steps.

All the methods on Javascript Context are still available, the methods listed below are specialisations.

Methods​

The following methods are available to call on the context object.

String getRestrictedDataFromHost(String host)​

ACCESSES RESTRICTED DATA

Gets the json string of restricted data for a host name

Returns (String): json object string of host data

hostData = JSON.parse(context.getRestrictedDataFromHost("myhost"));
String getAuthenticationPropertiesFromStep(String stepId, String key)​

ACCESSES RESTRICTED DATA

Gets the authentication properties from another step. These properties can expire.

Returns (String): json string of authentication properties value or null if expired

void setAuthenticationPropertiesOnStep(String stepName, Long expiresInMs, String key, String authenticationProperties)​

ACCESSES RESTRICTED DATA

Sets the authentication properties from another step. These properties can expire.

void sendMetric(String routingToken, long timestamp, String key, double value, Map metadata)​

This method is not allowed in the authentication context

void sendToStep(String stepId, String body, Map headers)​

This method is not allowed in the authentication context

void sendMetric(String key, double value)​

This method is not allowed in the authentication context

void sendMetric(long timestamp, String key, double value)​

This method is not allowed in the authentication context

void sendMetric(long timestamp, String key, double value, Map metadata)​

This method is not allowed in the authentication context

FilesContext​

The FileContext provides workflow access to files. The FileContext is got via context.files(fileId) method. Only files defined in the bootstrap allowFileAccess property can be accessed.

Methods​

The following methods are available to call on the context object.

String list(String path)​

Lists the directory

Returns (String): A JSON array as a string of { "name" : "...", "type" : ("FILE"|"DIRECTORY"), "bytes" : 0, "modified" : 0 }

String read(String file)​

Read the contents of the file as a string

Returns (String): string contents of file

void write(String file, String body)​

Write the string contents to a file

void copy(String srcPath, String dstFileId, String dstPath, boolean mkdir, boolean overwrite)​

Copies a file from one location to another

void move(String srcPath, String dstFileId, String dstPath, boolean mkdir, boolean overwrite)​

Moves a file from one location to another

JavascriptContext​

This is a really important class. It is the only class that javascript files can (should) interact with. Anything that a javascript needs to do must be referenced here.

This class is referenced from javascript as context

Methods​

The following methods are available to call on the context object.

String getPassedMessageAsString()​

Returns (String): the message body that was passed to this step.

passedMessage = context.getPassedMessageAsString();

Note that is will likely be in JSON format and need to be parsed to be useful

String getMessageBodyAsString()​

Returns (String): the current message body, this could be the response from an http call for example

body = context.getMessageBodyAsString();
String getBody()​

A shorthand for: check getMessageBodyAsString()then check getPassedMessageAsString()

Returns (String): the contents eiter passed into the step or set by the step.

body = context.getBody();
void setBody(Object body)​

An alias for setMessage(...)

context.setBody("hello");
void sendToStep(String stepName, String body, Map headers)​

Send a message to another step

context.sendToStep("myStep", "theBody", { "header1" : "abc"});
void sendToStep(String stepName, String body)​

Send a message to another step

context.sendToStep("myStep", "theBody");
void sendMetric(String key, double value)​

Send a metric

context.sendMetric("theKey", 123);
void sendMetric(String routingToken, long timestamp, String key, double value, Map metadata)​

Send a metric

context.sendMetric("theRoutingToken", theTimestamp, "theKey", 123, { "someMeta" : "Data"});
void sendMetric(long timestamp, String key, double value)​

Send a metric

context.sendMetric(theTimestamp, "theKey", 123);
void sendMetric(long timestamp, String key, double value, Map metadata)​

Send a metric

context.sendMetric(theTimestamp, "theKey", 123, { "someMeta" : "Data"});
void diagnosticLog(String message)​

Write a diagnostic log. This will only be emitted in development mode, or in Production mode if the step is enabled via a "enableDiagnostics" message

context.diagnosticLog("Hello World");
long getTimestamp()​

Returns (long): the current timestamp (note that the timestamp might be set by a test or other mechanism)

ts = context.getTimestamp();

date = new Date(context.getTimestamp())
TimestampManager getTimestampManager()​

The timestamp manager allows for persistence of timestamps

Returns (TimestampManager): the TimestampManager for the step

tm = context.getTimestampManager();
void setProperty(String key, Object value)​

Sets a property onto this running context to be available in downstream step execution

This differs from setPersistedItem in that setPersistedItem sets onto the STEP to be available on THIS STEP in future runs, while setProperty makes an item available to OTHER steps

context.setProperty("aKey", "aValue");
Object getProperty(String key)​

Gets a property from this running context

Returns (Object): the propery for the key

property = context.getProperty("aKey");
void setPersistedItem(String key, String value)​

Sets an item to the step persistence

context.setPersistedItem("myItem", "theValue");
String getPersistedItem(String key)​

Gets an item from the step persistence

Returns (String): the string value

item = context.getPersistedItem("myItem");
Map getStepProperties()​

Returns the set properties for the Step. These properties will persist after requests but not over config reloads/restarts

Returns (Map): the propery object

property = context.getStepProperties().put("myProperty", "ok");
void setMessage(Object message)​

Sets the message body for this running exchange

context.setMessage("hello world");
void addSplitReturnItem(String item)​

When the current step is a "scripted-split-aggregate" type (see Step the processed item must be added to the aggregate using this method.

context.addSplitReturnItem("hello world");
String getData()​

Returns (String): the JSON string for the data for this step

data = JSON.parse(context.getData());
String getData(String key)​

Returns (String): the string for the key in the data for this step

data = context.getData(key);
void setData(String data)​

Set the JSON data for this step

context.setData(JSON.stringify(data));
void setHeader(String name, Object value)​

Add a header to the next http request

context.setHeader("aHeader", "aValue");
String getHeader(String name)​

Get a header from the previous request/response

Returns (String): the header value as a json array, or an empty array

headerArray = context.getHeader("aHeader");
header = headerArray[0];
void removeHeader(String name)​

Remove a header from the current state

context.removeHeader("aHeader");
void setUrl(String hostref, String path)​

Set the url for the next http action with reference to a named host

context.setUrl("myHost", "/a/path");
void setHttpMethod(String method)​

Set the HTTP method for the next http action

This is generally used when a method other than the default is required.

The HTTP method will default to GET unless there is a payload body then it will default to POST

context.setHttpMethod("POST");
String hash(String toHash)​

When hashing a value, the original and the hash are emitted to the log to be referred to at a later date. The hash can be used in metrics etc so that the original value is not transmitted

Returns (String): the hashed value

hashedValue = context.hash("hello world");
double delta(String key, Double currentValue)​

Returns the differences between the current value and the last value stored for given key

Returns (double): currentValue - lastValue

delta = context.delta("myTs", 12);
List regexMatch(String regex, String input)​

Returns the regex matches

Returns (List): array of matches

String mergeJsonStrings(String one, String two)​

Returns a merged json string

Returns (String): the merged json object

String xmlToJson(String xml)​

Returns a json string from an xml string

Returns (String): json string

String jsonPath(String json, String expression)​

Returns the result from a JSONPath expression

Returns (String): json string

void sleep(long ms)​

Force the currently running script to sleep for the given milliseconds

context.sleep(100);
void end()​

Termaintes the running flow

context.end();
FilesContext files(String id)​

Request access to the bootstrap file permission

Returns (FilesContext): FilesContext

PersistentQueueContext queue()​

Returns the high performance persistent queue

double counter(String name, double add)​

Atomically add to or update a thread safe atomic counter and return the updated value

To just get the value pass 0 for add

Returns (double): the updated value

void setCounter(String name, double value)​

Atomically set the counter to an absolute value

PersistentQueueContext​

A high performance implementation of persistent FIFO queue.

The persistence context is got via context.queue().

Once the queue is obtained, items can be added and removed over multiple runs.

The queue will store a small amount of items in memory and overflow to disk.

Queue items will persist over agent restarts.

Methods​

The following methods are available to call on the context object.

void push(String item)​

Add an item to the end of the persistent queue.

context.queue().push("adding item");
void pushArray(Collection array)​

Add the contents of an array of item to the end of the persistent queue.

context.queue().pushArray(["adding", "multiple", "items"]);
void returnItem(String item)​

Return the item to the head of the queue

void returnArray(Collection array)​

Return the array items to the head of the queue

List take(int count)​

Request the given amount of items from the queue or an empty array when no items available.

NOTE: the returned amount may not be the entire available items. The method should be called repeatedly until the desired amount is collected. This function waits for a short period for data to be loaded from file and may not load all the data.

var items = context.queue().take(1000);

TimestampManager​

Looks after timestamps for metrics, for example recording the last known timestamp for a metric

TimestampManager is local to the step.

Timestamps can have a named key, or if not provided with a key, will just store with a default key.

Methods​

The following methods are available to call on the context object.

Long get()​

Returns (Long): the last timestamp set without a key

Long get(String key)​

Returns (Long): the last timestamp set for the given key

void set(String timestamp)​

Sets the timestamp without a key see: set(String key, String timestamp)

void set(String key, String timestamp)​

Sets the timestamp for the given key

void setIfLastest(String key, String timestamp)​

Only performs the set operation if the passed timestamp is later than the previously stored.

Long firstTimestamp()​

Returns (Long): the earliest timestamp stored

Integer minutesAgo(Long timestamp)​

Returns (Integer): the number of minutes since the given timestamp

opscotch testing model​

The opscotch testing model describes how to define tests that are executed against a running opscotch agent.

The test runner configuration is passed to the test runner and describes the locations of tests and resource files. The test configuration describes each individual test.

Here is the outline of the entire structure of the opscotch testing models: the runner configuration and the test configuration.

Note that not every field is required, so be sure to check the detailed descriptions.

The test runner schema overview​

{
    "resourceDir": "",
    "resourceDirs": [
        ""
    ],
    "testDir": "",
    "testDirs": [
        ""
    ],
    "license": "",
    "concurrency" : 0,
    "iterations" : 0,
    "debug": true,
    "stopOnFailure" : true

}

The test schema​

{
    "ignore": true,
    "fromDirectory": "",
    "testThisStep": "",
    "useThisTestConfigFile": "",
    "useThisTestBootstrapFile": "",
    "mockEndpointResponses" : [
        {
            "whenThisIsCalled": "",
            "expectedPayload": "",
            "expectedHeaders": {
                "": [""]
            },
            "returnThisFile": "",
            "returnThisStatusCode" : 0,
            "mockedHeaders" : {
                "": [""]
            },
            "thisManyTimes" : 0
        }
    ],
    "theseMetricsShouldHaveBeenSent" : [
        {
            "timestamp": 0,
            "key": "",
            "value": 0.0,
            "dimensionMap" : {
                "": ""
            }
        }
    ],
    "theseLogsShouldHaveBeenSent" : [
        ""
    ],
    "logsShouldNotContain" : [
        ""
    ],
    "makeTheseCalls" : [
        {
            "url": "",
            "expectedResponse": "",
            "expectedSize": 0,
            "method": "",
            "headers": {
                "": [""]
            },
            "body": "",
            "timeoutSeconds" : 0
        }
    ],
    "timestampOverride": 0,
    "secondsToWaitForCompletion" : 0
}

JSON Schema in the Test Package:

HttpCall​

Make an http call and assert results

JSON Properties​

The following properties are available to set on the JSON schema.

body​

The http body to send

This property is optional

expectedResponse​

The expected response

This property is optional

expectedSize​

The expected size of the response

This property is optional

headers​

The http headers to use

This property is optional

method​

The http method to use

Defaults to 'get'

This property is optional

timeoutSeconds​

The allowed timeout of the http call in seconds

Defaults to 10

url​

The url to call

This property is required

MockHttpResponse​

Instructs the mock http server what urls to expect and how to respond.

These requests and responses are required to have been called by the test, otherwise the test will fail.

JSON Properties​

The following properties are available to set on the JSON schema.

delay​

Type: Integer

The number of milliseconds to delay the response This will default to 0

{
  "delay" : 0
}

This property is optional

expectedHeaders​

Type: Object with string key with array of string property

If set, will assert that expectedHeader's key/value pair is found within the HTTP header. As headers can have multiple values the value is represented as an array.

{
  "expectedHeaders" : {"some": ["thing"] }
}

This property is optional

expectedPayload​

Type: String

If set will assert that the exact payload was received.

{
  "expectedPayload": "{\"some\":\"thing\"}"
}

This property is optional

mockedHeaders​

Type: Object with string key with array of string property

If set will return these headers with the response. As headers can have multiple values the value is represented as an array.

{
  "mockedHeaders" : {
      "Set-Cookie": ["JSESSIONID=A","X-CSRF-TOKEN=B"]
  }
}

This property is optional

returnThisFile​

Type: String

The file path of the response to return.

{
  "returnThisFile" : "test.response.summary.json"
}

This property is optional

returnThisStatusCode​

Type: Integer

The status code to return

{
  "returnThisStatusCode" : 302
}

This property is optional

thisManyTimes​

Type: Integer

The number of times this endpoint should be invoked This will default to 1

{
  "thisManyTimes" : 2
}

This property is optional

whenThisIsCalled​

Type: String

Required The complete url that the mock server will listen for.

The "mockserver" host is replaced out to the actual host and port for the mock server.

{
  "whenThisIsCalled" : "http://mockserver/controller/restui/licenseRule/getAllRulesSummary"
}

This property is required

withMethod​

Type: String

The method expected ie GET, POST etc This will default to GET and if an expectedPayload is provided it will default to POST

{
  "withMethod" : "GET"
}

This property is optional

SendMetricVerification​

Defines expected metrics to have been published from the test. If these metrics are not received the test will fail.

Pay special note to the timestamp field.

{
  "theseMetricsShouldHaveBeenSent": [
      {
          "key": "mymetric",
          "value": 67.0
      }
  ]
}
JSON Properties​

The following properties are available to set on the JSON schema.

dimensionMap​

Type: Object

Assert that received and expected dimensions are exact.

{
  "dimensionMap" : {
      "ad" : "ad_mh"
  }
}

This property is optional

key​

Type: String

Required Assert the exact key on the metric.

{
  "key" : "abcde"
}

This property is required

timestamp​

Type: Long

If set will assert the exact timestamp on the metric, otherwise the timestamp will be ignored during assertion.

{
  "timestamp" : 1593658546131
}

This property is optional

toThisRoute​
value​

Type: Double

Required Assert the exact value on the metric.

{
  "value" : "1234.56"
}

This property is required

TestConfig​

Describes how to execute a test and verify results

JSON Properties​

The following properties are available to set on the JSON schema.

fromDirectory​

Type: String

Required The relative path from the test directory to the test resources (test workflow configurations, mock response files etc).

{
  "fromDirectory" : "/mytest/"
}

This property is required

ignore​

Type: Boolean

If true the test will not be run.

{
  "ignore" : true
}

This property is optional

logsShouldNotContain​

Type: Array of String

These logs are not expected and the test will fail if they are generated by the workflow configuration.

This is useful to ensure that secrets are not logged.

{
  "logsShouldNotContain" : [
      ...
  ]
}

This property is optional

makeTheseCalls​

Type: Array of HttpCall?

Make http calls to the agent and assert the response

{
  "makeTheseCalls" : [
      ...
  ]
}
mockEndpointResponses​

Type: Array of MockHttpResponse

Required Sets up a http server with mocked responses.

{
  "mockEndpointResponses" : [
      ...
  ]
}

This property is required

secondsToWaitForCompletion​

Type: Integer

Determines how long to wait for logs and metric assertions before the test is stopped

Defaults to 10

{
  "secondsToWaitForCompletion" : 20
}

This property is optional

testThisStep​

Type: String

Required The name of the step in the workflow configuration to trigger to start the test.

{
  "testThisStep" : "start-application"
}

This property is required

theseLogsShouldHaveBeenSent​

Type: Array of String

These logs are expected and the test will fail if they are not generated by the workflow configuration

{
  "theseLogsShouldHaveBeenSent" : [
      ...
  ]
}

This property is optional

theseMetricsShouldHaveBeenSent​

Type: Array of SendMetricVerification

Required These metrics are expected and the test will fail if they are not generated by the workflow configuration

{
  "theseMetricsShouldHaveBeenSent" : [
      ...
  ]
}

This property is required

timestampOverride​

Type: Long

If set (to a millisecond timestamp) the system clock will be set to this time.

{
  "timestampOverride" : 1593658546131
}

This property is optional

useThisTestBootstrapFile​

Type: String

The bootstrap configuration to include for the test.

{
  "useThisBootstrapFile" : "test.bootstrap.json"
}

This property is optional

useThisTestConfigFile​

Type: String

Required The workflow configuration file to load for the test.

{
  "useThisTestConfigFile" : "test.config.json"
}

This property is required

TestRunnerConfig​

JSON configuration required for defining test configurations against agents.

This configuration file is passed to the test runner as a command line argument and tells it where to find the tests and resource files.

JSON Properties​

The following properties are available to set on the JSON schema.

concurrency​

Type: Integer

Number of tests to run at the same time This will default to half the number of processors available or 1 if there is only one. The maximum value is the number of processors available -2 or 1 if there are less than 3 processors available

This property is optional

debug​

Type: Boolean

If true, will enable debug for the test

iterations​

Type: Integer

Speficies the number of times to run the tests Defaults to 1.

license​

Type: String

Required The license key

This property is required

resourceDir​

Deprecated: use resourceDirs

Type: String

Required The path to the configuration resource directory

This property is optional

resourceDirs​

Type: List of String

Required The paths to the configuration resource directories

This property is required

stopOnFailure​

Type: Boolean

Indicates whether the test runner should stop executing tests after the first failure is encountered.

Defaults to false, meaning the test runner will continue executing all configured tests, regardless of whether a failure occurs.

If set to true, the test runner will immediately terminate testing upon the first failed test. Currently running tests will continue till completion.

This property is optional.

testDir​

Deprecated: use testDirs

Type: String

Required The path to the configuration test directory

This property is required

testDirs​

Type: List of String

Required The paths to the configuration test directories

This property is required

Opscotch Error Codes​

Many errors will have an Opscotch Error Code that may be referenced in logs, metrics or messages. Please see the list below for descriptions of error codes:

AJC1: Can not decrypt encrypted data​

This error occurs when the host data property accessed in an authentication context contains encrypted data but failed to decrypt.

Check that the encrypted data is complete. Check that the data was encrypted with a compatible packaging tool version. Check that the matching bootstrap agentPrivateKey public key was used for encryption.

BL1: Failed to load bootstrap​

This is a generic error code that indicates that the specific individual bootstrap failed to load.

Check the error message for additional Opscotch Error Codes

BL2: Duplicate deploymentId​

This error occurs when a bootstrap is loaded with a deploymentId that is already loaded.

Check the bootstrap file for duplicated deploymentId properties.

BL3: Failed to load bootstrap - Decryption​

This is a generic error code that indicates that the specific individual bootstrap failed to decrypting using the internal opscotch keys.

Check that the version of the packaging tool matches the opscotch agent.

BS1: Malformed url​

This error occurs when the url specified in a Output.outputUrl is malformed.

Check the outputUrl property is correct.

BS2: Failed send for metrics or logs​

This error occurs during an unexpected operation. The data you were trying to send has likely been lost.

Please contact opscotch to report the fault.

CD1: The agent private key is invalid​

The agentPrivateKey was read and Base64 decoded from the bootstrap record, but could not read as an RSA PKCS8 private key.

Check that the bootstrap agentPrivateKey is a Base64 encoded private key in pkcs8 format

CD2: Encoding versions do not match​

This error occurs when the Workflow configuration is packaged with an old packaging tool version.

Check that the version of the packaging tool matches the opscotch agent.

CD3: Could not decrypt config​

This error code occurs when the Workflow configuration can not be decrypted.

Check that the matching public and private keys are used. Check that the version of the packaging tool matches the opscotch agent. Check that you're deploying the intended Workflow configuration to the correct agent.

CD4: Workflow JSON file format is not supported​

This error occurs when attempting to load a raw workflow json file.

Ensure that the bootstrap remoteConfiguration property refers to a packaged workflow configuration.

CDE1: Workflow Configuration not loaded​

This error occurs when the Workflow configuration signature can not be validated, even after decryption.

This implies that the content of the encrypted Workflow configuration was manipulated before encryption. This is a very strange state and you should contact opscotch immediately.

CP1:​

This is a generic error that occurs when the Workflow configuration fails to load.

Please see additional messages

CP2:​

This is a generic error that occurs when the Workflow configuration fails to load.

Please see additional messages

CP3: Invalid agentPrivateKey in bootstrap​

This error occurs when the Base64 decode of the bootstrap agentPrivateKey property fails.

Ensure that the bootstrap agentPrivateKey property is a Base64 encoded string

CP4: agentPrivateKey is required in bootstrap in Prod Mode​

This error occurs when running in Prod Mode and the bootstrap agentPrivateKey property is missing.

Ensure that the bootstrap agentPrivateKey property is a Base64 encoded string

CR1: Failed loading configuration​

This is a generic error that occurs after json validation when trying to load the steps.

Please see additional messages.

CR2: Step requested does not exist​

This occurs when the requested step has not been loaded.

Step requested does not exist

CR3: Workflow did not finish active runs in time. Forcing reload​

This occurs during a workflow configuration reload, due to a configuration change when there were workflows executing when the reload was triggered, and the running workflows had not completed within the reload timeout. The reload occured regardless and this might have resulted in two versions of the workflow running for a period of time.

To prevent this happening, increase the reloadTimeout in the bootstrap.

CRW1: Exception while checking license​

This error occurs when there is a problem with the license checking mechanism.

Please see additional messages and contact opscotch if necessary.

CW1: Malformed url​

The Url specified in the bootstrap remoteConfiguration is an invalid Url.

CW2: Invalid bootstrap remote configuration settings.​

The remoteConfigurationTimeout must be less than frequency

FC1: Can not read file​

The requested file could not be read. Check file system permissions

FC2: Not permitted to read file​

The requested file is not permitted to be read. Check bootstrap.allowFileAccess.READ.

FC3: Can not read file​

The requested file could not be read. Check file system permissions

FC4: Not permitted to read file​

The requested file is not permitted to be read. Check bootstrap.allowFileAccess.READ.

FC5: Not permitted to read file​

The requested file is not permitted to be read. Check bootstrap.allowFileAccess.READ.

FC6: Not permitted to move file without delete permission​

The requested file is not permitted to be deleted while moving. Check bootstrap.allowFileAccess.DELETE.

FC7: Source file does not exist​

The requested file does not exist

FC8: Not permitted to write​

The requested destination file is not permitted to be written. Check bootstrap.allowFileAccess.WRITE.

FC9: Destination file exists and no overwrite​

The destination file exists and overwrite was specified.

FC10: Not permitted to write​

The requested destination file is not permitted to be written. Check bootstrap.allowFileAccess.WRITE.

FC11: Can not write file​

The requested destination file could not be written to. Check file system properties.

FC12: Can not write file​

The requested destination file could not be written to. Check file system properties.

FC13: Can not delete file​

The requested destination file could not be deleted. Check file system properties.

FW1: Failed to load file​

This error occurs when bootstrap remoteConfiguration refers to a file, and the file could not be loaded.

Please report this to opscotch support.

FW2: Failed to read file​

This error occurs when bootstrap remoteConfiguration refers to a file, and the file could not be read.

Check the file permission on the file

FW3: Failed to load file​

This error occurs when bootstrap remoteConfiguration refers to a file, the file was read but the contents could not be processed.

Check the file contents.

FW4: Can not load zero length file. Check packaging.​

This error occurs when bootstrap remoteConfiguration refers to a file of zero length.

Check the file contents and the packaging process.

HF6: Requires bootstrap.allowFileAccess​

Http file serving steps requires bootstrap.allowFileAccess

HP1: Not on allow list​

This error occurs when attempting to call an HTTP address that is not on the host.allowList for the path and method combination.

Ensure that calls to HTTP addresses have an allowList entry.

HP2: Authentication Failed​

This occurs when the authenticationProcessor fails to complete execution.

Please see additional messages.

HP3: Url not set​

This error occurs when the step defines a urlGenerator but a url was not set.

When using a urlGenerator ensure that a url is being set.

HP4: Error processing HTTP response​

This error occurs when the processing of the HTTP response caused an exception.

Please see additional messages.

HP5: Error while making http request from step​

This error occurs when the making the HTTP request failed.

Please see additional messages.

HW1: Could not get Workflow configuration​

This error occurs when bootstrap remoteConfiguration refers to an HTTP resource which failed during fetch. Check that the file is where you expect it to be.

Please see additional messages.

HW2: Could not get Workflow configuration​

This error occurs when bootstrap remoteConfiguration refers to an HTTP resource which failed after fetch but during decode. Check that the file is in the correct packaged format.

Please see additional messages.

IT2: Path is invalid​

Check the step.trigger.fileWatcher paths

Please see additional messages.

IT2: Path is invalid​

Check the step.trigger.fileWatcher paths

IT3: eventSplitter pattern is invalid​

Check the step.trigger.fileWatcher.eventSplitter pattern

IT4: unable to load files​

An error occurred while starting the file water for the step.trigger.fileWatcher

IT5: No bootstrap.allowFileAccess with id​

Check the bootstrap.allowFileAccess pattern for the step.trigger.fileWatcher.bootstrapFileId

IT6: bootstrap.allowFileAccess has an invalid pattern​

Check the bootstrap.allowFileAccess pattern for the step.trigger.fileWatcher.bootstrapFileId

IT7: fileTrigger.pattern has an invalid pattern​

Check the step.trigger.fileWatcher.patterns

IT8: eventSplitter has an invalid pattern​

Check the step.trigger.fileWatcher.eventSplitter pattern

IT9: unable to load file​

An error occurred while loading a file from the file water for the step.trigger.fileWatcher

IT11: Can not start file watcher​

File watching is not supported on this OS or filesystem

IT12: unable to load file​

An error occurred while loading a file from the file water for the step.trigger.fileWatcher

IT13: unable to load file​

An error occurred while loading a file from the file water for the step.trigger.fileWatcher

JC1: Messages failed to load​

This error occurs when the workflow configuration messages fail to load.

Please see additional messages (logs).

JC2: getPassedMessageAsString​

This error occurs during the calling of context.getPassedMessageAsString()

Please see additional messages.

JC3: getMessageBodyAsString​

This error occurs during the calling of context.getMessageBodyAsString()

Please see additional messages.

JC4: sendToStep​

This error occurs during the calling of context.sendToStep()

Please see additional messages.

JC5, JC7, JC8 : sendMetric​

This error occurs during the calling of context.sendMetric()

Please see additional messages.

JC6: getAnomalyDetection​

This error occurs during the calling of context.getAnomalyDetection()

Please see additional messages.

JC9: diagnosticLog​

This error occurs during the calling of context.diagnosticLog()

Please see additional messages.

JC10: getTimestamp​

This error occurs during the calling of context.getTimestamp()

Please see additional messages.

JC11: getTimestampManager​

This error occurs during the calling of context.getTimestampManager()

Please see additional messages.

JC12: setProperty​

This error occurs during the calling of context.setProperty()

Please see additional messages.

JC13: getStepProperties​

This error occurs during the calling of context.getStepProperties()

Please see additional messages.

JC14: getProperty​

This error occurs during the calling of context.getProperty()

Please see additional messages.

JC15: setPersistedItem​

This error occurs during the calling of context.setPersistedItem()

Please see additional messages.

JC16: getPersistedItem​

This error occurs during the calling of context.getPersistedItem()

Please see additional messages.

JC17: setMessage​

This error occurs during the calling of context.setMessage()

Please see additional messages.

JC18: addSplitReturnItem​

This error occurs during the calling of context.addSplitReturnItem()

Please see additional messages.

JC19: getData​

This error occurs during the calling of context.getData()

Please see additional messages.

JC20: setHeader​

This error occurs during the calling of context.setHeader()

Please see additional messages.

JC21: getHeader​

This error occurs during the calling of context.getHeader()

Please see additional messages.

JC27: removeHeader​

This error occurs during the calling of context.removeHeader()

Please see additional messages.

JC29: Malformed URL​

This error occurs when context.setUrl() is called with an invalid URL.

Ensure that only valid URLs are set.

JC30: Host not found​

This error occurs when specifying a hostref that does not exist in the boostrap host property.

Ensure that the hostref matches a host in the bootstrap.

JC31: Host not permitted​

This error occurs when referencing a host from the wrong context.

Ensure only hosts with authenticationHost: true are referenced from an authentication context. Ensure only hosts with authenticationHost: false (or not set) are referenced from a non-authentication context.

JC32: host-ref can not be null​

This error occurs when not specifying a hostref when calling context.setUrl().

Ensure that the hostref argument to context.setUrl() is not null.

OHC1: Unhandled Exception​

The error occurs when an exception was not handled during HTTP response processing.

Please report this error to opscotch.

OHC2: Unhandled Exception​

The error occurs when an exception was not handled during HTTP failure.

Please report this error to opscotch.

OMR1: Error while closing Agent Metric Sender​

Please report this error to opscotch.

SE1: Could not create persistence​

This error occurs when the step defines a persistenceFile but the file could not be created.

See additional messages. Check the permissions for the file and directory tree.

SE2: Ending Run​

The run has been ended prematurely and was not completed.

See additional messages.

SE3: Unhandled Exception - Please report this to opscotch support​

The run has been ended prematurely and was not completed. Additionally, the exception handler also failed. This is an important problem to diagnose. Please contact opscotch support.

SE4: Workflow configuration was updated​

The workflow configuration was updated and the previous configuration was unloaded. This message is printed when a workflow was running while the configuration was updated and this run did not complete.

SE5: Ending Run​

An error occurred while running this workflow. The cause of this error is likely in the Javascript of the executing step processor.

See additional messages.

SE6: Persistence requested but not setup​

A step processor made a call to persistent storage, but it has not been setup for this step

SE7: TimestampManager requires persistence​

TimestampManager requires persistence, but it was not set up