Version: Current (3.1.x)

Testing

opscotch was built to be reliable, and that includes workflow authoring.

opscotch has built-in support for authoring workflow test cases which is essential for bullet proof upgrades and workflow development.

When authoring a workflow ensure that you also author a test case. This has the following benefits:

The test case:

documents the expected flow of the workflow.
can detect faults caused by a change of agent behavior.
is portable and can be given to opscotch consultants to diagnose a problem.
mocks out services used in the workflow and during development: prevents bad requests sent to the services; prevents rate limiting from services; prevents additional load on the services.

How opscotch workflow testing works

opscotch workflow tests approximate the concept of "mocked integration tests", whereby the agent has no indication that it is interacting with a testing harness: as far as the agent is concerned, it is interacting with real services.

The opscotch testing harness is a program that pretends to be a HTTP server (web/api). It writes out a bootstrap file that an agent is observing, with all the host records for the tests pointing to the test harness program. The testing harness loads a set of test configurations and writes out workflow configurations, which the agent detects and executes. The testing harness listens for HTTP requests from the agent and sends replies for the workflow to consume. It also listens for metrics and logs sent from the agent. Each of these interactions (URLs, metics, logs) are observed and can be asserted on.

A full example is available here

Test case configuration

Test cases are defined using a JSON file (schema reference). The general nature of these tests will be:

trigger a step to run.
expect some URLs to be called by the agent with appropriate responses sent back to the agent.
expect some metrics or logs to be sent.

Test harness configuration

To select the test cases to run, a configuration is passed to the Test harness. See the schema here

This configuration requires:

the path to the opscotch resource directories for javascript to include.
the path to the test directories for tests to run.
the path to your license file.

These are entered into the Test Harness configuration.

Running tests

The test harness configuration file is passed as an argument to the test harness program:

Linux/macOS
Windows

opscotch-testrunner testrunner.config.json

opscotch-testrunner.exe testrunner.jar testrunner.config.json

The Test harness will scan the test directories configured in the configuration file for files ending in .test.json and will schedule them for execution. It will output a message like this:

Start the opscotch agent with this argument: <your working directory>/test.bootstrap.json

Start your agent with the argument to the test harness bootstrap file:

Linux/macOS
Windows

opscotch-agent <your working directory>/test.bootstrap.json

opscotch-agent.exe <your working directory>/test.bootstrap.json

The tests will run and give details on the results.

Running tests with opscotch-testrunner-single

If you want a single-container test harness, use the official all-in-one image ghcr.io/opscotch/opscotch-testrunner-single:latest. This image contains both the test runner and the agent runtime, waits for the generated bootstrap, and exits with the test runner status code.

Note: this container only runs one test.

The container expects:

OPSCOTCH_LEGAL_ACCEPTED
a mounted license file at /license/license.txt
a mounted test directory at /tests
TEST_PATH pointing at the test file to run

Example:

export OPSCOTCH_LEGAL_ACCEPTED='<base64-acceptance-blob>'

docker run --rm \
  -e OPSCOTCH_LEGAL_ACCEPTED \
  -e TEST_PATH=/tests/general/httpserver.test.json \
  -v /path/to/tests:/tests:ro \
  -v /path/to/license.txt:/license/license.txt:ro \
  ghcr.io/opscotch/opscotch-testrunner-single:latest

For AI clients and other automation, capture stdout and stderr to a file and inspect only the summary line first:

docker run --rm \
  -e OPSCOTCH_LEGAL_ACCEPTED \
  -e TEST_PATH=/tests/general/httpserver.test.json \
  -v /path/to/tests:/tests:ro \
  -v /path/to/license.txt:/license/license.txt:ro \
  ghcr.io/opscotch/opscotch-testrunner-single:latest \
  > testrunner-single.log 2>&1

grep -q "1 succeeded" testrunner-single.log && echo "success"
grep -q "1 failed" testrunner-single.log && echo "failure"

This keeps token use low. On failure, inspect testrunner-single.log for the detailed trace.

Optional extra resource overlays can be mounted at:

/local-resources1
/local-resources2

Example with overlays:

docker run --rm \
  -e OPSCOTCH_LEGAL_ACCEPTED \
  -e TEST_PATH=/tests/general/httpserver.test.json \
  -v /path/to/tests:/tests:ro \
  -v /path/to/license.txt:/license/license.txt:ro \
  -v /path/to/resources1:/local-resources1:ro \
  -v /path/to/resources2:/local-resources2:ro \
  ghcr.io/opscotch/opscotch-testrunner-single:latest

When a test uses fromDirectory, preserve that directory structure under /tests. For example, the community httpserver test should be mounted so that the container sees /tests/general/httpserver.test.json.

Unit testing JavaScript resources

For fast feedback on resource files themselves, use the published GHCR unit test image ghcr.io/opscotch/opscotch-resource-testkit:latest.

This image is intended for resource unit tests, not full workflow integration tests. It provides a JavaScript runtime shim for the Opscotch doc and context globals, a real in-memory byte buffer implementation, and stubbed or mockable complex contexts such as crypto().

Recommended layout in the community repository:

resources/ for resource files
tests/ for existing Opscotch integration harness tests
unit-tests/ for Vitest-based resource unit tests

Example:

docker run --rm \
  -v /path/to/opscotch-community:/workspace:ro \
  ghcr.io/opscotch/opscotch-resource-testkit:latest

By default the image runs tests from /workspace/unit-tests.

Running all unit tests

Mount the Opscotch community repository at /workspace and run the image with no extra arguments:

docker run --rm \
  -v /path/to/opscotch-community:/workspace:ro \
  ghcr.io/opscotch/opscotch-resource-testkit:latest

Running a single unit test file

Point UNIT_TEST_PATH at a single .test.ts file:

docker run --rm \
  -e UNIT_TEST_PATH=/workspace/unit-tests/general/standard-clear-body.test.ts \
  -v /path/to/opscotch-community:/workspace:ro \
  ghcr.io/opscotch/opscotch-resource-testkit:latest

Running a set of unit tests

Point UNIT_TEST_PATH at a folder to run a group of tests:

docker run --rm \
  -e UNIT_TEST_PATH=/workspace/unit-tests/general \
  -v /path/to/opscotch-community:/workspace:ro \
  ghcr.io/opscotch/opscotch-resource-testkit:latest

You can also run a narrower subset by passing Vitest arguments after the image name. For example, this filters by test name:

docker run --rm \
  -e UNIT_TEST_PATH=/workspace/unit-tests/general \
  -v /path/to/opscotch-community:/workspace:ro \
  ghcr.io/opscotch/opscotch-resource-testkit:latest \
  --testNamePattern="clears the current body"

Running unit tests locally with the published npm package

The local package-based route uses the published npmjs package @opscotch/resource-testkit@0.1.2.

Install the local test dependencies:

cd /path/to/opscotch-community/unit-tests
npm install

Run all unit tests:

cd /path/to/opscotch-community/unit-tests
npm test

Run a single test file:

cd /path/to/opscotch-community/unit-tests
npx vitest run general/standard-clear-body.test.ts

Run a folder of tests:

cd /path/to/opscotch-community/unit-tests
npx vitest run httpserver

Recommended layout in the mounted repository:

resources/ for resource files
tests/ for existing Opscotch integration harness tests
unit-tests/ for Vitest-based resource unit tests

Recommended running environment

The following outlines the recommended running environment for the test runner. This is recommended so that when seeking support this is the common setup that everyone uses.

Clone the opscotch community repository (this repository) to a directory. This directory will now be known as OPSCOTCH_COMMUNITY_ROOT.
In the community root directory, add your license.txt

Ensure your testrunner.config.json looks something like this

{ 	
    "license" : "license.txt",	
    "resourceDirs" : [ "resources" ],	
    "testDirs" : [ "tests" ]
}

It is important to understand that the directories mentioned in the testrunner.config.json: resourceDirs and testDirs are relative to the opscotch-testrunner executable WORKING DIRECTORY ie. the directory where you START the agent:
- NOT the directory the opscotch-testrunner file is in
- NOT the direcory the testrunner.config.json is in
- NOT the OPSCOTCH_COMMUNITY_ROOT
- However, that all being said, in this example we will be starting the opscotch-testrunner from the OPSCOTCH_COMMUNITY_ROOT, so that will be our working direcory, AND both resources and tests exist and are relative to OPSCOTCH_COMMUNITY_ROOT.
Set an environment variable OPSCOTCH_COMMUNITY_ROOT to the path to your community root directory.
From a terminal, ensure that OPSCOTCH_COMMUNITY_ROOT is your working directory. Start the opscotch-testrunner with testrunner.config.json as the first argument.
The opscotch-testrunner will start and will provide the arguments to start the agent with.
In another terminal start the opscotch-agent:
1. Make sure to set the environment variable OPSCOTCH_COMMUNITY_ROOT for the agent also.
2. Start the opscotch-agent with the arguments provided by the opscotch-testrunner.
3. Once the opscotch-agent starts engaging with the opscotch-testrunner the opscotch-testrunner with start loading the tests.

How opscotch workflow testing works​

Test case configuration​

Test harness configuration​

Running tests​

Running tests with opscotch-testrunner-single​

Unit testing JavaScript resources​

Running all unit tests​

Running a single unit test file​

Running a set of unit tests​

Running unit tests locally with the published npm package​

Recommended running environment​