API Reference
This documentation describes the functionality that is available for configuration authors.
Documentation describing the functionality and schema of configuration (workflow, bootstrap etc.) is referred to as the model and is available below.
Documentation describing the functionality and schema of the testing harness is available below.
Packages available:
- Model
- Testing
opscotch configuration model
The opscotch model is relatively straight forward, and uses repeated patterns - once you grasp these you should be able to understand the entire structure.
Here is the outline of the entire structure of the two configurations: the bootstrap and the workflow. Note that not every field is required, so be sure to check the detailed descriptions.
The bootstrap schema
[
{
"agentPrivateKey": "",
"deploymentId": "",
"remoteConfiguration": "",
"remoteConfigurationAuth": "",
"frequency": 0,
"errorHandling": {
"enableLocalLogging": true,
"metrics": {
"enabled": true,
"routingToken": "",
"outputUrl": "",
"outputAuthorization": ""
},
"logs": {
"enabled": true,
"routingToken": "",
"outputUrl": "",
"outputAuthorization": ""
},
"redactionPatterns": [
""
]
},
"workflow" : {
"metricOutput": {
"enabled": true,
"routingToken": "",
"outputUrl": "",
"outputAuthorization": ""
},
"errorHandling": {
"enableLocalLogging": true,
"metrics": {
"enabled": true,
"routingToken": "",
"outputUrl": "",
"outputAuthorization": ""
},
"logs": {
"enabled": true,
"routingToken": "",
"outputUrl": "",
"outputAuthorization": ""
}
}
},
"hosts": {
"": {
"authenticationHost" : false,
"host": "",
"headers": {
"": ""
},
"allowlist": [
["",""]
],
"data": {}
}
},
"persistenceRoot" : "",
"data": {}
}
]
The workflow schema
{
"workflows": [
{
"enabled": true,
"name": "",
"steps": [
{
"enabled": true,
"debug": true,
"type": "",
"stepId": "",
"trigger" : {
"timer" : {
"delay" : 0,
"period" : 0
},
"runOnce" : true
},
"authenticationProcessor": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"splitGenerator": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"urlGenerator": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"payloadGenerator": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"itemResultProcessor": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"resultsProcessor": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"httpStatusHandling": {
"" : {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
}
},
"httpConnectFailedProcessor": {
"script": "",
"resource": "",
"processors" : [
{
"script": "",
"resource": ""
"data": {}
}
],
"data": {}
},
"timer": "",
"persistenceFile" : "",
"data": {}
}
],
"data" : {},
}
]
}
JSON Schema in the Bootstrap Package:
Bootstrap JSON Schemas
Bootstrap
A configuration that is loaded into the agent at startup and determines how the agent loads the main configuration is loaded.
Please note that the bootstap file that is authored is an ARRAY of these objects ie:
[
{
"agentPrivateKey": ...
"deploymentId": ...
}
]
It will be provided to the agent as an argument, either via a base64 encoded json text or a path to a json file
JSON Properties
The following properties are available to set on the JSON schema.
agentPrivateKey
Type: String
Required The private key for the agent to decrypt the remote configuration.
This should be a base64 encoded private key in pkcs8 format
This is how to create an appropriate key
1. Create key pair
openssl genrsa -out keypair.pem 2048
2. Extract public part
openssl rsa -in keypair.pem -pubout -out public.key
3. Extract private part
openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in keypair.pem -out private.key
4. Base64 the contents of this file
cat private.key | base64
{
"agentPrivateKey": "LS0tLS1CRUdJTiBQUklWQV...."
}
This property is required
deploymentId
Type: String
Required A Service Provider defined value that must be unique per deployed agent configuration.
This property is used by Service Provider downstream processor to enrich the transmitted data.
{
"deploymentId": "dogurj"
}
This property is required
remoteConfiguration
Type: String
Required A URL or file path to the main configuration.
The main configuration does not exist, the agent will wait until it becomes available.
{
"remoteConfiguration": "http://www.example.com",
"remoteConfigurationAuth" : "xodsjfujrhdjfk",
"frequency" : 60000
}
This property is required
remoteConfigurationAuth
Type: String
When using a URL remoteConfiguration, setting this property will set the "Authorization" header
{
"remoteConfiguration": "http://www.example.com",
"remoteConfigurationAuth" : "xodsjfujrhdjfk",
"frequency" : 60000
}
This property is optional
frequency
Type: Integer
When using a URL Bootstrap.remoteConfiguration, setting this property will set the polling period in milliseconds
{
"remoteConfiguration": "http://www.example.com",
"remoteConfigurationAuth" : "xodsjfujrhdjfk",
"frequency" : 60000
}
This property is optional
errorHandling
Type: ErrorHandling
Determines the behavior of error handling while loading the main configuration
{
"errorHandling": {
...
}
}
This property is optional
workflow
Type: WorkflowOutputs
Required
Determines metric and log outputs for the workflows
{
"workflow": {
...
}
}
This property is required
hosts
Type: Object of Host
A set of named Host objects. This allows the bootstrap to define http(s) host that can be called from the main configuration.
{
"hosts": {
"mynamedhost": {
"host": "https://www.example.com",
"headers": {
"Content-Type": "application/json;charset=UTF-8",
"Accept": "application/json, text/plain, *\/*"
},
"allowlist" : [
["GET", "/this/is/allowed.*"],
["POST", "/this/.*\/allowed"]
],
"data" : {
}
}
}
}
This property is optional
persistenceRoot
Type: String
Defines the filesystem storage root for persistence
{
"persistenceRoot": "/storage"
}
This property is optional
data
Type: Object
A JSON Object for adding properties.
These properties will be merged with the WorkflowConfiguration.data configuration field and be available to all child elements in the main configuration tree.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3]
}
}
This property is optional
ErrorHandling
A configuration defining if and how to send error metrics or log to a remote system
JSON Properties
The following properties are available to set on the JSON schema.
enableLocalLogging
Type: Boolean
Enable or disables logging of errors and information to the agent standard out
Defaults to false
{
"enableLocalLogging": true
}
This property is optional
metrics
Type: Output
Defines how error metrics are sent to a remote system.
Error metrics represent a fault in the agent and are sent to a remote system for attention.
Error metric configuration only apply to the process configuration they are defined for. For example the Bootstrap.errorHandling defines error handling while loading configurations, and the WorkflowConfiguration.errorHandling defines error handling during the execution of configuration steps.
{
"metrics": {
...
}
}
This property is optional
logs
Type: Output
Defines how error logs are sent to a remote system.
Error logs are brief messages from a fault in the agent and are sent to a remote system for attention.
Error log configuration only apply to the process configuration they are defined for. For example the Bootstrap.errorHandling defines error handling while loading configurations, and the WorkflowConfiguration.errorHandling defines error handling during the execution of configuration steps.
{
"logs": {
...
}
}
This property is optional
redactionPatterns
Type: Array of String
Defines regex patterns to apply redaction to log statements
Log statements from will have these redaction patterns applied before logging.
{
"redactionPatterns": [
"[0-9]+"
]
}
This property is optional
Host
A HTTP(S) host that the agent is expected to communicate with.
This only applies to customer specific hosts, not to Service Provider hosts.
Hosts are defined in Bootstrap.data and WorkflowConfiguration.data
JSON Properties
The following properties are available to set on the JSON schema.
host
Type: String
Required A URL that is used as the base for constructing URLs to call.
{
"host": "https://www.example.com:8080"
}
This property is required
headers
Type: Object of Strings
A object of HTTP headers to be sent with every call.
{
"headers": {
"Content-Type": "application/json;charset=UTF-8",
"Accept": "application/json, text/plain, *\/*"
}
}
This property is optional
allowlist
Type: Array of String tuple
A list of allowed paths that may be called on this host.
A list of regex that will be matched to URL path. Matching results will be allowed, non-matching results will be blocked and logged.
{
"allowlist": [
["GET","/this/is/allowed"],
["POST","/this/.*\/allowed"]
]
}
This property is optional
authenticationHost
Type: Boolean
When set to true the host can only be used by an authentication context,
when set to false (default) the host can only be used by a non-authentication context.
{
"authenticationHost": true
}
Defaults to false
This property is optional
data
Type: Object
A JSON Object with Additional properties that will accessible during authentication steps.
You can use environment variables in the form of ${NAME} anywhere in the data structure:
the environment variable will be substituted prior to parsing.
Strings in the data structure (after environment variable substitution) can be encrypted using the opscotch packager. Values will be decrypted at access time.
Encryption is done via the packaging tool and requires the agent public key.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3],
"<some encrypted string>" : {
"abc" : "<some encrypted value>"
}
}
}
This property is optional
Output
Configuration for how to send data to a remote system.
JSON Properties
The following properties are available to set on the JSON schema.
enabled
Type: Boolean
Enables or disables the sending of data.
{
"enabled": true
}
This property is optional
routingToken
Type: String
Required A Service Provider defined value that must be unique per deployed agent.
This property is used by Service Provider downstream processor to direct and enrich the transmitted data.
{
"routingToken": "dogurj"
}
This property is required
outputUrl
Type: String
Required The URL that data is transmitted to.
{
"outputUrl": "http://www.example.com"
}
This property is required
outputAuthorization
Type: String
Set the "Authorization" header.
{
"outputAuthorization" : "xodsjfujrhdjfk"
}
This property is optional
WorkflowConfiguration
The configuration that describes the main activities of the agent
JSON Properties
The following properties are available to set on the JSON schema.
workflows
Type: Array of Workflow
Required A list of Workflow describing the agent tasks
{
"workflows" [
...
]
}
This property is required
data
Type: Object
A JSON Object for adding properties.
These properties will be merged with the Bootstrap.date configuration field and be available to all child elements.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3]
}
}
This property is optional
WorkflowOutputs
JSON Properties
The following properties are available to set on the JSON schema.
metricOutput
Type: Output
Required Defines how metrics are sent to a remote system.
Metrics represent a data point that is interesting, generated by the agent and are sent to a remote system for analysis.
{
"metricOutput": {
...
}
}
This property is required
errorHandling
Type: ErrorHandling
Required Determines the behavior of error handling while running the main configuration
{
"errorHandling": {
...
}
}
This property is required
JSON Schema in the Workflow Package:
Scripting context objects in the Workflow Package:
Workflow JSON Schemas
JavaScriptSource
A JavaScript that is executed during the processing of a Step
JSON Properties
The following properties are available to set on the JSON schema.
script
Type: String
The script to execute.
A json-escaped script. Must be supplied if JavaScriptSource.resource is not supplied
{
"script": "console.log(\\"hello world\\");"
}
This property is optional
resource
Type: String
A file path to the script to execute.
Must be supplied if JavaScriptSource.script is not supplied
{
"resource": "scripts/myscript.js"
}
This property is optional
data
Type: Object
A JSON Object for adding properties.
Used to pass parameters and data to the script.
These properties will be merged with all other data fields in the json path, available to the step script elements.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3],
}
}
This property is optional
JavascriptProcessor
JSON Properties
The following properties are available to set on the JSON schema.
processors
Type: List of JavaScriptSource
A list of JavaScriptSource that will be executed in order
{
"processors" : [
{
"resource" : "..."
},
{
"resource" : "..."
}
]
}
This property is optional
script
Depricated: use processors
Type: String
The script to execute.
A json-escaped script. Must be supplied if JavaScriptSource.resource is not supplied
{
"script": "console.log(\\"hello world\\");"
}
This property is optional
resource
Depricated: use processors
Type: String
A file path to the script to execute.
Must be supplied if JavaScriptSource.script is not supplied
{
"resource": "scripts/myscript.js"
}
This property is optional
data
Type: Object
A JSON Object for adding properties.
Used to pass parameters and data to the script.
These properties will be merged with all other data fields in the json path, available to the step script elements.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3],
}
}
This property is optional
Step
The Step is the primary configuration item for the agent configuration.
This configuration describes how to perform a single step or action in a Workflow.
There are different types of steps, however they all have the same execution structure:
- Prepare
- Execute
- Process
Steps take three forms of input:
- The response from a call to a Host (
context.getMessageBodyAsString()) - The message passed from another Step (
context.getPassedMessageAsString()) - Data from the configuration (
context.getData())
The different types of Steps allow for specialised behaviour:
-
"scripted" type: this is the default type with the following flow:
-
Optionally generate a URL (urlGenerator)
-
Optionally (requires a url to have been set) generate a payload (payloadGenerator)
-
Optionally perform authentication steps (authenticationProcessor, call other steps to perform authentication)
-
Optionally call out to a named Bootstrap Host with the generated URL and payload
-
Optionally handle specific HTTP status codes with httpStatusHandling
OR
-
Process the response or perform after-call processing (resultsProcessor)
Perhaps:
- Send a message to another Step
- Produce some metrics
-
-
"scripted-split-aggregate" type: this is a modified "scripted" type that has the following workflow of:
-
transforming the input into a list (splitGenerator)
-
calling a URL with each item in the list:
-
Optionally generate a URL (urlGenerator)
-
Optionally (requires a URL to have been set) generate a payload (payloadGenerator)
-
Optionally perform authentication steps (authenticationProcessor, call other steps to perform authentication)
-
Optionally call out to a named Bootstrap Host with the generated URL and payload
-
Optionally handle specific HTTP status codes with httpStatusHandling
OR
-
Optionally Process each item's response (itemResultProcessor)
-
Collect the result into a response list
-
-
The list of collected responses can then be processed and per normal (resultsProcessor)
Perhaps:
- Send a message to another Step
- Produce some metrics
Please note that in the case that one of the HTTP requests fail, it is your responsibility to handle this case by using the
httpStatusHandling,itemResultProcessor, andresultsProcessor
-
-
"scripted-auth" type: this is identical to "scripted" except that it has an alternative javascript context - it is not allowed to send metrics or call to non-authentication steps, but has access to the bootstrap data which may contain secrets.
At least one step in a Workflow will define a Step.trigger that will trigger the start of Workflow processing.
These forms of input and different type allows for the construction of flexible pipelines that should be sufficient to perform most tasks.
JSON Properties
The following properties are available to set on the JSON schema.
enabled
Type: Boolean
Enables or disables the execution of this step. Defaults to true.
{
"enabled": true
}
This property is optional
debug
Type: Boolean
Enables or disables the debug logs for this step. Defaults to false.
{
"debug": true
}
This property is optional
type
Type: String
Sets the type of the step.
Either "scripted" or "scripted-split-aggregate" or "scripted-auth" as defined above.
Defaults to "scripted"
{
"type": "scripted-split-aggregate"
}
This property is optional
stepId
Type: String
Required Sets the id of the step.
Must be unique.
{
"stepId": "the-first-step-in-my-route"
}
This property is required
urlGenerator
Type: JavascriptProcessor
Required Defines how to generate the URL
This processor should not attempt to set the payload (the current internal logic may result in unexpected results). To set the payload, use the payloadGenerator
{
"urlGenerator": {
...
}
}
This property is optional
resultsProcessor
Type: JavascriptProcessor
Required Defines how to process the result of the call to the host.
{
"resultsProcessor": {
...
}
}
This property is required
authenticationProcessor
Type: JavascriptProcessor
Performs authentication tasks. Generally this step might call to authentication steps and/or apply headers.
{
"authenticationProcessor": {
...
}
}
This property is optional
payloadGenerator
Type: JavascriptProcessor
Defines how to generate the call payload, including setting headers.
It is expected that the context.setMessage(..) is called.
This requires the urlGenerator to be present and will throw a validation exception.
If this processor is not used, then the HTTP body will be removed.
{
"payloadGenerator": {
...
}
}
This property is optional
splitGenerator
Type: JavascriptProcessor
When using Step.type "scripted-split-aggregate", defines how to generate the list to be operated on.
{
"splitGenerator": {
...
}
}
This property is optional
itemResultProcessor
Type: JavascriptProcessor
When using Step.type "scripted-split-aggregate", defines if and how to process items before they're added to the list.
{
"itemResultProcessor": {
...
}
}
This property is optional
httpConnectFailedProcessor
Type: JavascriptProcessor
Defines how to handle HTTP connection failures
{
"httpConnectFailedProcessor": {
...
}
}
This property is optional
httpStatusHandling
Type: Object of JavascriptProcessor
A set of JavascriptProcessor to process depending on the http status.
The key is the HTTP status either as a discrete number ie "401" or a range "400-499"
When a status code matches a status handler and the handler is executed the resultProcessor or itemProcessor
will not be executed.
{
"httpStatusHandling": {
"301-302" : {
...
},
"401" : {
...
}
}
}
This property is optional
timer
Depricated: use Trigger
Type: String
Defines a timer to trigger the execution of a step.
There will likely only be one timer in a Workflow.
Accepts Apache Camel Timer query string parameters
{
"timer": "fixedRate=true&period=3600000"
}
This property is optional
trigger
Type: Trigger
Defines how a workflow is started.
{
"trigger": {
...
}
}
This property is optional
persistenceFile
Type: String
Defines a cache/memory/key-value store that the Step has access to.
Required if using various functions. If required and not defined an error will be logged
{
"persistenceFile": "persistenceFile.data"
}
This property is optional
data
Type: Object
A JSON Object for adding properties.
Used to pass parameters and data to the JavaScriptSource elements.
These properties will be merged with all other data fields in the json path, available to the step JavaScriptSource elements.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3],
}
}
This property is optional
Timer
Defines a repeating, fixed-rate period trigger
JSON Properties
The following properties are available to set on the JSON schema.
delay
Type: Long
Defines a delay in milliseconds before the first invocation.
Defaults to 1000.
{
"delay": 60000
}
This property is optional
period
Type: Long
Defines a delay in milliseconds between invocations.
Defaults to 1000.
{
"period": 60000
}
This property is optional
Trigger
Defines the starting condition for a workflow
JSON Properties
The following properties are available to set on the JSON schema.
timer
Type: Timer
Defines a repeating period trigger
{
"timer": {
...
}
}
This property is optional
runOnce
Type: Boolean
Sets trigger to run only once.
Defaults to false
{
"runOnce": true
}
This property is optional
Workflow
Defines a set of Steps that the agent will perform in order to generate metrics
JSON Properties
The following properties are available to set on the JSON schema.
enabled
Type: Boolean
Enable or disables the loading of the route.
Defaults to true
{
"enabled": false
}
This property is optional
name
Type: String
Required The user friendly name of the route.
{
"name": "my route"
}
This property is required
steps
Type: Array of Step
Required The list of Steps or actions to take to perform the intention of the Workflow.
Please note that steps are not executed in order, rather they are a group of actions that can send messages to one and other. The order of execution is defined by the actions taken by the Step
{
"steps": [
...
]
}
This property is required
data
Type: Object
A JSON Object for adding properties.
Used to pass parameters and data to the JavaScriptSource elements.
These properties will be merged with all other data fields in the json path, available to the step JavaScriptSource elements.
{
"data": {
"somedata" : {
"abc: 123
},
"someotherdata" : [ 1, 2, 3],
}
}
This property is optional
Workflow Javascript Context
AuthenticationJavascriptContext
This is a specialised Javascript Context for authentication steps.
All the methods on Javascript Context are still available, the methods listed below are specialisations.
Methods
The following methods are available to call on the context object.
-
getAuthenticationPropertiesFromStep(String stepId, String key)
-
sendMetric(String routingToken, long timestamp, String key, double value, Map metadata)
-
sendMetric(long timestamp, String key, double value, Map metadata)
String getRestrictedDataFromHost(String host)
Returns: json object string of host data
hostData = JSON.parse(context.getRestrictedDataFromHost("myhost"));
ACCESSES RESTRICTED DATA
Gets the json string of restricted data for a host name
String getAuthenticationPropertiesFromStep(String stepId, String key)
Returns: json string of authentication properties value or null if expired
ACCESSES RESTRICTED DATA
Gets the authentication properties from another step. These properties can expire.
void setAuthenticationPropertiesOnStep(String stepName, Long expiresInMs, String key, String authenticationProperties)
ACCESSES RESTRICTED DATA
Sets the authentication properties from another step. These properties can expire.
void sendMetric(String routingToken, long timestamp, String key, double value, Map metadata)
This method is not allowed in the authentication context
void sendToStep(String stepId, String body, Map headers)
This method is not allowed in the authentication context
void sendMetric(String key, double value)
This method is not allowed in the authentication context
void sendMetric(long timestamp, String key, double value)
This method is not allowed in the authentication context
void sendMetric(long timestamp, String key, double value, Map metadata)
This method is not allowed in the authentication context
JavascriptContext
This is a really important class. It is the only class that javascript files can (should) interact with. Anything that a javascript needs to do must be referenced here.
This class is referenced from javascript as context
Methods
The following methods are available to call on the context object.
-
sendMetric(String routingToken, long timestamp, String key, double value, Map metadata)
-
sendMetric(long timestamp, String key, double value, Map metadata)
String getPassedMessageAsString()
Returns: the message body that was passed to this step.
passedMessage = context.getPassedMessageAsString();
Note that is will likely be in JSON format and need to be parsed to be useful
String getMessageBodyAsString()
Returns: the current message body, this could be the response from an http call for example
body = context.getMessageBodyAsString();
void sendToStep(String stepName, String body, Map headers)
Send a message to another step
context.sendToStep("myStep", "theBody", { "header1" : "abc"});
void sendToStep(String stepName, String body)
Send a message to another step
context.sendToStep("myStep", "theBody");
void sendMetric(String key, double value)
Send a metric
context.sendMetric("theKey", 123);
void sendMetric(String routingToken, long timestamp, String key, double value, Map metadata)
Send a metric
context.sendMetric("theRoutingToken", theTimestamp, "theKey", 123, { "someMeta" : "Data"});
void sendMetric(long timestamp, String key, double value)
Send a metric
context.sendMetric(theTimestamp, "theKey", 123);
void sendMetric(long timestamp, String key, double value, Map metadata)
Send a metric
context.sendMetric(theTimestamp, "theKey", 123, { "someMeta" : "Data"});
void diagnosticLog(String message)
Write a diagnostic log. This will only be emitted in development mode, or in Production mode if the step is enabled via a "enableDiagnostics" message
context.diagnosticLog("Hello World");
long getTimestamp()
Returns: the current timestamp (note that the timestamp might be set by a test or other mechanism)
ts = context.getTimestamp();
workflow.TimestampManager getTimestampManager()
Returns: the TimestampManager for the step
tm = context.getTimestampManager();
The timestamp manager allows for persistence of timestamps
void setProperty(String key, Object value)
Sets a property onto this running exchange, to be used later
context.setProperty("aKey", "aValue");
Object getProperty(String key)
Returns: the propery for the key
property = context.getProperty("aKey");
Gets a property from this running step
void setPersistedItem(String key, String value)
Sets an item to the step persistence
context.setPersistedItem("myItem", "theValue");
String getPersistedItem(String key)
Returns: the string value
item = context.getPersistedItem("myItem");
Gets an item from the step persistence
Map getStepProperties()
Returns: the propery object
property = context.getStepProperties().put("myProperty", "ok");
Returns the set properties for the Step. These properties will persist after requests.
void setMessage(Object message)
Sets the message body for this running exchange
context.setMessage("hello world");
void addSplitReturnItem(String item)
When the current step is a "scripted-split-aggregate" type (see Step the processed item must be added to the aggregate using this method.
context.addSplitReturnItem("hello world");
String getData()
Returns: the JSON string for the data for this step
data = JSON.parse(context.getData());
String getData(String key)
Returns: the string for the key in the data for this step
data = context.getData(key);
void setData(String data)
Set the JSON data for this step
context.setData(JSON.stringify(data));
void setHeader(String name, Object value)
Add a header to the next http request
context.setHeader("aHeader", "aValue");
String getHeader(String name)
Returns: the header value as a json array, or an empty array
headerArray = context.getHeader("aHeader");
header = headerArray[0];
Get a header from the previous request/response
void removeHeader(String name)
Remove a header from the current state
context.removeHeader("aHeader");
void setUrl(String hostref, String path)
Set the url for the next http action with reference to a named host
context.setUrl("myHost", "/a/path");
void setHttpMethod(String method)
Set the HTTP method for the next http action
This is generally used when a method other than the default is required.
The HTTP method will default to GET unless there is a payload body then it will default to POST
context.setHttpMethod("POST");
String hash(String toHash)
Returns: the hashed value
hashedValue = context.hash("hello world");
When hashing a value, the original and the hash are emitted to the log to be referred to at a later date. The hash can be used in metrics etc so that the original value is not transmitted
double delta(String key, Double currentValue)
Returns: currentValue - lastValue
delta = context.delta("myTs", 12);
Returns the differences between the current value and the last value stored for given key
List regexMatch(String regex, String input)
Returns: array of matches
Returns the regex matches
String mergeJsonStrings(String one, String two)
Returns: the merged json object
Returns a merged json string
String xmlToJson(String xml)
Returns: json string
Returns a json string from an xml string
String jsonPath(String json, String expression)
Returns: json string
Returns the result from a JSONPath expression
void sleep(long ms)
Force the currently running script to sleep for the given milliseconds
TimestampManager
Looks after timestamps for metrics, for example recording the last known timestamp for a metric
TimestampManager is local to the step.
Timestamps can have a named key, or if not provided with a key, will just store with a default key.
Methods
The following methods are available to call on the context object.
Long get()
Returns: the last timestamp set without a key
Long get(String key)
Returns: the last timestamp set for the given key
void set(String timestamp)
Sets the timestamp without a key see: set(String key, String timestamp)
void set(String key, String timestamp)
Sets the timestamp for the given key
void setIfLastest(String key, String timestamp)
Only performs the set operation if the passed timestamp is later than the previously stored.
Long firstTimestamp()
Returns: the earliest timestamp stored
Integer minutesAgo(Long timestamp)
Returns: the number of minutes since the given timestamp
opscotch testing model
The opscotch testing model describes how to define tests that are executed against a running opscotch agent.
The test runner configuration is passed to the test runner and describes the locations of tests and resource files. The test configuration describes each individual test.
Here is the outline of the entire structure of the opscotch testing models: the runner configuration and the test configuration.
Note that not every field is required, so be sure to check the detailed descriptions.
The test runner schema
{
"resourceDir": "",
"resourceDirs": [
""
],
"testDir": "",
"testDirs": [
""
],
"license": ""
}
The test schema
{
"ignore": true,
"fromDirectory": "",
"testThisStep": "",
"useThisTestConfigFile": "",
"useThisBootstrapFile": "",
"mockEndpointResponses" : [
{
"whenThisIsCalled": "",
"expectedPayload": "",
"expectedHeaders": {
"": [""]
},
"returnThisFile": "",
"mockedHeaders" : {
"": [""]
},
"thisManyTimes" : 0
}
],
"theseMetricsShouldHaveBeenSent" : [
{
"timestamp": 0,
"key": "",
"value": 0.0,
"dimensionMap" : {
"": ""
}
}
],
"theseLogsShouldHaveBeenSent" : [
""
],
"logsShouldNotContain" : [
""
],
"timestampOverride": 0
}
JSON Schema in the Test Package:
Test JSON Schemas
MockHttpResponse
Instructs the mock http server what urls to expect and how to respond.
These requests and responses are required to have been called by the test, otherwise the test will fail.
JSON Properties
The following properties are available to set on the JSON schema.
whenThisIsCalled
Type: String
Required The complete url that the mock server will listen for.
The "mockserver" host is replaced out to the actual host and port for the mock server.
{
"whenThisIsCalled" : "http://mockserver/controller/restui/licenseRule/getAllRulesSummary"
}
This property is required
expectedPayload
Type: String
If set will assert that the exact payload was received.
{
"expectedPayload": "{\"some\":\"thing\"}"
}
This property is optional
expectedHeaders
Type: Object with string key with array of string property
If set, will assert that expectedHeader's key/value pair is found within the HTTP header. As headers can have multiple values the value is represented as an array.
{
"expectedHeaders" : {"some": ["thing"] }
}
This property is optional
returnThisFile
Type: String
The file path of the response to return.
{
"returnThisFile" : "test.response.summary.json"
}
This property is optional
returnThisStatusCode
Type: Integer
The status code to return
{
"returnThisStatusCode" : 302
}
This property is optional
mockedHeaders
Type: Object with string key with array of string property
If set will return these headers with the response. As headers can have multiple values the value is represented as an array.
{
"mockedHeaders" : {
"Set-Cookie": ["JSESSIONID=A","X-CSRF-TOKEN=B"]
}
}
This property is optional
withMethod
Type: String
The method expected ie GET, POST etc
This will default to GET and if an expectedPayload is provided it will default to POST
{
"withMethod" : "GET"
}
This property is optional
thisManyTimes
Type: Integer
The number of times this endpoint should be invoked This will default to 1
{
"thisManyTimes" : 2
}
This property is optional
SendMetricVerification
Defines expected metrics to have been published from the test. If these metrics are not received the test will fail.
Pay special note to the timestamp field.
{
"theseMetricsShouldHaveBeenSent": [
{
"key": "mymetric",
"value": 67.0
}
]
}
JSON Properties
The following properties are available to set on the JSON schema.
timestamp
Type: Long
If set will assert the exact timestamp on the metric, otherwise the timestamp will be ignored during assertion.
{
"timestamp" : 1593658546131
}
This property is optional
key
Type: String
Required Assert the exact key on the metric.
{
"key" : "abcde"
}
This property is required
value
Type: Double
Required Assert the exact value on the metric.
{
"value" : "1234.56"
}
This property is required
dimensionMap
Type: Object
Assert that received and expected dimensions are exact.
{
"dimensionMap" : {
"ad" : "ad_mh"
}
}
This property is optional
TestConfig
Describes how to execute a test and verify results
JSON Properties
The following properties are available to set on the JSON schema.
ignore
Type: Boolean
If true the test will not be run.
{
"ignore" : true
}
This property is optional
fromDirectory
Type: String
Required The relative path from the test directory to the test resources (test workflow configurations, mock response files etc).
{
"fromDirectory" : "/mytest/"
}
This property is required
testThisStep
Type: String
Required The name of the step in the workflow configuration to trigger to start the test.
{
"testThisStep" : "start-application"
}
This property is required
useThisTestConfigFile
Type: String
Required The workflow configuration file to load for the test.
{
"useThisTestConfigFile" : "test.config.json"
}
This property is required
useThisTestBootstrapFile
Type: String
The bootstrap configuration to include for the test.
{
"useThisBootstrapFile" : "test.bootstrap.json"
}
This property is optional
mockEndpointResponses
Type: Array of MockHttpResponse
Required Sets up a http server with mocked responses.
{
"mockEndpointResponses" : [
...
]
}
This property is required
theseMetricsShouldHaveBeenSent
Type: Array of SendMetricVerification
Required These metrics are expected and the test will fail if they are not generated by the workflow configuration
{
"theseMetricsShouldHaveBeenSent" : [
...
]
}
This property is required
theseLogsShouldHaveBeenSent
Type: Array of String
These logs are expected and the test will fail if they are not generated by the workflow configuration
{
"theseLogsShouldHaveBeenSent" : [
...
]
}
This property is optional
logsShouldNotContain
Type: Array of String
These logs are not expected and the test will fail if they are generated by the workflow configuration.
This is useful to ensure that secrets are not logged.
{
"logsShouldNotContain" : [
...
]
}
This property is optional
timestampOverride
Type: Long
If set (to a millisecond timestamp) the system clock will be set to this time.
{
"timestampOverride" : 1593658546131
}
This property is optional
TestRunnerConfig
JSON configuration required for defining test configurations against agents.
This configuration file is passed to the test runner as a command line argument and tells it where to find the tests and resource files.
JSON Properties
The following properties are available to set on the JSON schema.
resourceDir
Deprecated: use resourceDirs
Type: String
Required The path to the configuration resource directory
This property is optional
resourceDirs
Type: List of String
Required The paths to the configuration resource directories
This property is required
testDir
Deprecated: use testDirs
Type: String
Required The path to the configuration test directory
This property is required
testDirs
Type: List of String
Required The paths to the configuration test directories
This property is required
license
Type: String
Required The license key
This property is required
debug
Opscotch Error Codes
Many errors will have an Opscotch Error Code that may be referenced in logs, metrics or messages. Please see the list below for descriptions of error codes:
AJC1: Can not decrypt encrypted data
This error occurs when the host data property accessed in an authentication context contains encrypted data but failed to decrypt.
Check that the encrypted data is complete. Check that the data was encrypted with a compatible packaging tool version. Check that the matching bootstrap agentPrivateKey public key was used for encryption.
BL1: Failed to load bootstrap
This is a generic error code that indicates that the specific individual bootstrap failed to load.
Check the error message for additional Opscotch Error Codes
BL2: Duplicate deploymentId
This error occurs when a bootstrap is loaded with a deploymentId that is already loaded.
Check the bootstrap file for duplicated deploymentId properties.
BL3: Failed to load bootstrap - Decryption
This is a generic error code that indicates that the specific individual bootstrap failed to decrypting using the internal opscotch keys.
Check that the version of the packaging tool matches the opscotch agent.
BS1: Malformed url
This error occurs when the url specified in a Output.outputUrl is malformed.
Check the outputUrl property is correct.
CD1: The agent private key is invalid
The agentPrivateKey was read and Base64 decoded from the bootstrap record, but could not read as an RSA PKCS8 private key.
Check that the bootstrap agentPrivateKey is a Base64 encoded private key in pkcs8 format
CD2: Encoding versions do not match
This error occurs when the Workflow configuration is packaged with an old packaging tool version.
Check that the version of the packaging tool matches the opscotch agent.
CD3: Could not decrypt config
This error code occurs when the Workflow configuration can not be decrypted.
Check that the matching public and private keys are used. Check that the version of the packaging tool matches the opscotch agent. Check that you're deploying the intended Workflow configuration to the correct agent.
CD4: Workflow JSON file format is not supported
This error occurs when attempting to load a raw workflow json file.
Ensure that the bootstrap remoteConfiguration property refers to a packaged workflow configuration.
CD5: Failed to decode workflow
Check that the version of the packaging tool matches the opscotch agent.
CD6: Failed to decode workflow
Check that the version of the packaging tool matches the opscotch agent.
CD7: Failed to decode workflow
Check that the version of the packaging tool matches the opscotch agent.
CD8: Workflow Configuration not loaded
Check that the version of the packaging tool matches the opscotch agent.
CDE1: Workflow Configuration not loaded
This error occurs when the Workflow configuration signature can not be validated, even after decryption.
This implies that the content of the encrypted Workflow configuration was manipulated before encryption. This is a very strange state and you should contact opscotch immediately.
CP1:
This is a generic error that occurs when the Workflow configuration fails to load.
Please see additional messages
CP2:
This is a generic error that occurs when the Workflow configuration fails to load.
Please see additional messages
CP3: Invalid agentPrivateKey in bootstrap
This error occurs when the Base64 decode of the bootstrap agentPrivateKey property fails.
Ensure that the bootstrap agentPrivateKey property is a Base64 encoded string
CR1: Failed loading configuration
This is a generic error that occurs after json validation when trying to load the steps. Please see additional messages.
CR2: Requested step does not exist
A request to a step that does not exist. Check the configuration.
CRW1: Exception while checking license
This error occurs when there is a problem with the license checking mechanism.
Please see additional messages and contact opscotch if necessary.
CW1: Malformed url
The Url specified in the bootstrap remoteConfiguration is an invalid Url.
FW1: Failed to read file
This error occurs when bootstrap remoteConfiguration refers to a file, and the file could not be read.
Check the file referred to by the bootstrap remoteConfiguration exists and has appropriate permissions.
HP1: Not on allow list
This error occurs when attempting to call an HTTP address that is not on the host.allowList for the path and method combination.
Ensure that calls to HTTP addresses have an allowList entry.
HP2: Authentication Failed
This occurs when the authenticationProcessor fails to complete execution.
Please see additional messages.
HP3: Url not set
This error occurs when the step defines a urlGenerator but a url was not set.
When using a urlGenerator ensure that a url is being set.
HW1: Could not get Workflow configuration
This error occurs when bootstrap remoteConfiguration refers to an HTTP resource which could not be read.
Please see additional messages.
JC1: Messages failed to load
This error occurs when the workflow configuration messages fail to load.
Please see additional messages (logs).
JC2: getPassedMessageAsString
This error occurs during the calling of `context.getPassedMessageAsString()
Please see additional messages.
JC3: getMessageBodyAsString
This error occurs during the calling of `context.getMessageBodyAsString()
Please see additional messages.
JC4: sendToStep
This error occurs during the calling of `context.sendToStep()
Please see additional messages.
JC5, JC7, JC8 : sendMetric
This error occurs during the calling of `context.sendMetric()
Please see additional messages.
JC6: getAnomalyDetection
This error occurs during the calling of `context.getAnomalyDetection()
Please see additional messages.
JC9: diagnosticLog
This error occurs during the calling of `context.diagnosticLog()
Please see additional messages.
JC10: getTimestamp
This error occurs during the calling of `context.getTimestamp()
Please see additional messages.
JC11: getTimestampManager
This error occurs during the calling of `context.getTimestampManager()
Please see additional messages.
JC12: setProperty
This error occurs during the calling of `context.setProperty()
Please see additional messages.
JC13: getStepProperties
This error occurs during the calling of `context.getStepProperties()
Please see additional messages.
JC14: getProperty
This error occurs during the calling of `context.getProperty()
Please see additional messages.
JC15: setPersistedItem
This error occurs during the calling of `context.setPersistedItem()
Please see additional messages.
JC16: getPersistedItem
This error occurs during the calling of `context.getPersistedItem()
Please see additional messages.
JC17: setMessage
This error occurs during the calling of `context.setMessage()
Please see additional messages.
JC18: addSplitReturnItem
This error occurs during the calling of `context.addSplitReturnItem()
Please see additional messages.
JC19: getData
This error occurs during the calling of `context.getData()
Please see additional messages.
JC20: setHeader
This error occurs during the calling of `context.setHeader()
Please see additional messages.
JC21: getHeader
This error occurs during the calling of `context.getHeader()
Please see additional messages.
JC27: removeHeader
This error occurs during the calling of `context.removeHeader()
Please see additional messages.
JC29: Malformed URL
This error occurs when context.setUrl() is called with an invalid URL.
Ensure that only valid URLs are set.
JC30: Host not found
This error occurs when specifying a hostref that does not exist in the boostrap host property.
Ensure that the hostref matches a host in the bootstrap.
JC31: Host not permitted
This error occurs when referencing a host from the wrong context.
Ensure only hosts with authenticationHost: true are referenced from an authentication context. Ensure only hosts with authenticationHost: false (or not set) are referenced from a non-authentication context.
JC32: host-ref can not be null
This error occurs when not specifying a hostref when calling context.setUrl().
Ensure that the hostref argument to context.setUrl() is not null.
SE1: Could not create persistence
This error occurs when the step defines a persistenceFile but the file could not be created.
See additional messages. Check the permissions for the file and directory tree.
SE2: Ending Run
The run has been ended prematurely and was not completed.
See additional messages.
SE3: Unhandled Exception - Please report this to opscotch support
The run has been ended prematurely and was not completed. Additionally, the exception handler also failed. This is an important problem to diagnose. Please contact opscotch support.
SE4: Workflow configuration was updated
The workflow configuration was updated and the previous configuration was unloaded. This message is printed when a workflow was running while the configuration was updated and this run did not complete.
SE5: Ending Run
An error occurred while running this workflow. The cause of this error is likely in the Javascript of the executing step processor.
See additional messages.
SE6: Persistence requested but not setup
A step processor made a call to persistent storage, but it has not been setup for this step