Skip to main content
Version: 3.0.0

Testing

opscotch was built to be reliable, and that includes workflow authoring.

opscotch has built-in support for authoring workflow test cases which is essential for bullet proof upgrades and workflow development.

When authoring a workflow ensure that you also author a test case. This has the following benefits:

The test case:

  • documents the expected flow of the workflow.
  • can detect faults caused by a change of agent behavior.
  • is portable and can be given to opscotch consultants to diagnose a problem.
  • mocks out services used in the workflow and during development: prevents bad requests sent to the services; prevents rate limiting from services; prevents additional load on the services.

How opscotch workflow testing works​

opscotch workflow tests approximate the concept of "mocked integration tests", whereby the agent has no indication that it is interacting with a testing harness: as far as the agent is concerned, it is interacting with real services.

The opscotch testing harness is a program that pretends to be a HTTP server (web/api). It writes out a bootstrap file that an agent is observing, with all the host records for the tests pointing to the test harness program. The testing harness loads a set of test configurations and writes out workflow configurations, which the agent detects and executes. The testing harness listens for HTTP requests from the agent and sends replies for the workflow to consume. It also listens for metrics and logs sent from the agent. Each of these interactions (URLs, metics, logs) are observed and can be asserted on.

A full example is available here

Test case configuration​

Test cases are defined using a JSON file (schema reference). The general nature of these tests will be:

  • trigger a step to run.
  • expect some URLs to be called by the agent with appropriate responses sent back to the agent.
  • expect some metrics or logs to be sent.

Test harness configuration​

To select the test cases to run, a configuration is passed to the Test harness. See the schema here

This configuration requires:

  • the path to the opscotch resource directories for javascript to include.
  • the path to the test directories for tests to run.
  • the path to your license file.

These are entered into the Test Harness configuration.

Running tests​

The test harness configuration file is passed as an argument to the test harness program:

opscotch-testrunner testrunner.config.json

The Test harness will scan the test directories configured in the configuration file for files ending in .test.json and will schedule them for execution. It will output a message like this:

Start the opscotch agent with this argument: <your working directory>/test.bootstrap.json

Start your agent with the argument to the test harness bootstrap file:

opscotch-agent <your working directory>/test.bootstrap.json

The tests will run and give details on the results.

The following outlines the recommended running environment for the test runner. This is recommended so that when seeking support this is the common setup that everyone uses.

  1. Clone the opscotch community repository (this repository) to a directory. This directory will now be known as OPSCOTCH_COMMUNITY_ROOT.

  2. In the community root directory, add your license.txt

  3. Ensure your testrunner.config.json looks something like this

    {   
        "license" : "license.txt",  
        "resourceDirs" : [ "resources" ],   
        "testDirs" : [ "tests" ]
    }

  4. It is important to understand that the directories mentioned in the testrunner.config.json: resourceDirs and testDirs are relative to the opscotch-testrunner executable WORKING DIRECTORY ie. the directory where you START the agent:

    • NOT the directory the opscotch-testrunner file is in
    • NOT the direcory the testrunner.config.json is in
    • NOT the OPSCOTCH_COMMUNITY_ROOT
    • However, that all being said, in this example we will be starting the opscotch-testrunner from the OPSCOTCH_COMMUNITY_ROOT, so that will be our working direcory, AND both resources and tests exist and are relative to OPSCOTCH_COMMUNITY_ROOT.

  5. Set an environment variable OPSCOTCH_COMMUNITY_ROOT to the path to your community root directory.

  6. From a terminal, ensure that OPSCOTCH_COMMUNITY_ROOT is your working directory. Start the opscotch-testrunner with testrunner.config.json as the first argument.

  7. The opscotch-testrunner will start and will provide the arguments to start the agent with.

  8. In another terminal start the opscotch-agent:

    1. Make sure to set the environment variable OPSCOTCH_COMMUNITY_ROOT for the agent also.
    2. Start the opscotch-agent with the arguments provided by the opscotch-testrunner.
    3. Once the opscotch-agent starts engaging with the opscotch-testrunner the opscotch-testrunner with start loading the tests.