From Problem Statement to Implementation Plan

This guide walks you through the process of taking a problem statement and turning it into a concrete plan for an opscotch workflow. Whether you're automating a manual process, building an integration, or creating a monitoring system, this structured approach will help you design effective solutions.

Understanding Your Problem Statement

Before diving into implementation, you need a clear understanding of what you're trying to achieve. A good problem statement answers three questions:

1. What - What is happening or what needs to happen?
2. Why - Why does this need to be automated or improved?
3. When/Where - When does this need to occur and where does the data come from/go?

Example Problem Statements

Integration scenario: "We need to receive webhook notifications from our payment provider and update our internal records with the transaction status, then send a notification to the customer."

Monitoring scenario: "We need to poll our production services every 5 minutes to check for health status, and alert the on-call team if any service becomes unavailable."

Data processing scenario: "We need to watch a folder for new CSV files, validate the data structure, transform each row into our internal format, and upload to our analytics system."

Identify the Trigger

The trigger determines how and when your workflow starts. Understanding this is foundational to the entire design.

Available Trigger Types

Trigger	When to Use
http	External systems need to initiate the workflow via an HTTP request
timer	Workflow needs to run on a schedule (fixed interval or cron-like)
tcp	Data arrives on a TCP socket (useful for legacy systems, sensors)
fileWatcher	New or changed files in a directory should trigger processing
runOnce	Workflow runs once at startup (initialization, data loading)
deploymentAccess	Another deployment triggers this workflow

Decision Questions

• Does an external system push data to you, or do you need to poll for it?
• Is the trigger event-based or time-based?
• What's the expected frequency of triggers?

Example: Choosing Your Trigger

Problem	Trigger Choice	Reasoning
Receive webhooks from SaaS	http	External system pushes data to you
Daily report generation	timer with period	Scheduled task, not event-driven
Process new files in SFTP	fileWatcher	Event-driven when files appear
Health check on services	timer with delay	Run once at startup

Map the Data Flow

Understanding how data enters, moves through, and exits your workflow is critical.

Inputs - How Data Gets In

Data can enter your workflow through multiple paths:

1. Trigger input - Data embedded in the trigger event itself (HTTP body, TCP message, file content)
2. Bootstrap data - Static configuration data defined in the bootstrap file
3. Host configuration - Data retrieved from external services via HTTP
4. File system - Data read from files via file access configuration

Outputs - Where Data Goes

Think about where results need to go:

1. HTTP response - Send data back to a caller (for HTTP-triggered workflows)
2. External service - Call another API or service via configured hosts
3. File system - Write results to files
4. Metrics - Send telemetry data for monitoring
5. Logs - Generate diagnostic output
6. Persistence - Store state for later retrieval

Decision Questions

• What format does the input data come in?
• Do you need to transform the data before processing?
• Where should the results go?
• Do you need to handle errors or retries?

What Data Is Available in a Step

When a workflow runs, processing is always scoped to a specific step at any given moment. The data available to processors on a step comes from multiple sources with different lifetimes and scopes:

Data Source	Description	Scope	Lifetime
data property	Configuration merged from bootstrap → workflow → step → processor	Processor on this specific step	Consistent across all invocations of this step
Trigger input	Data from the trigger event (HTTP body, file content, timer tick)	This specific step invocation	Varies per invocation
Context (body/properties)	Data passed between steps via sendToStep	The running workflow execution	Changes as workflow progresses through steps
Step local data	Data stored via setPersistedItem, queue operations, stepProperties	The specific step	Persists across invocations

For more discussion on this, see Understanding Step Scope and Context.

Design Your Steps

With the trigger and data flow understood, you can design the processing steps.

Common Step Patterns

1. Route and Validate

Trigger -> Validate Input -> Route to Handler

Use when: Input can be one of several types requiring different handling.

2. Transform and Enrich

Input -> Transform -> Enrich with external data -> Output

Use when: Data needs formatting or additional context from other systems.

3. Batch Process

Trigger -> Split into items -> Process each -> Aggregate results

Use when: Handling multiple records in a single trigger event.

4. Chain of Operations

Step 1 -> Step 2 -> Step 3 -> Output

Use when: Operations must happen sequentially with each step depending on the previous.

5. Synthesized Storage

Worker Step -> Storage Step (persisted data) -> Worker Step

Use when: Data needs to be shared across multiple steps or persist across workflow executions.

For detailed implementation, see Synthesized Storage Pattern.

6. Synthesized Controller

Worker Step -> Controller Step (centralized logic) -> Worker Step

Use when: You need centralized validation, business rules, or logic that multiple steps share.

For detailed implementation, see Synthesized Controller Pattern.

7. Multiple Triggers

Step with HTTP + Timer trigger -> Different handling per trigger

Use when: You need both event-driven (HTTP) and scheduled (timer) processing on the same step.

For detailed implementation, see Multiple Triggers Pattern.

Step Composition

Consider these aspects for each step:

• Type - What kind of processing does this step do?
• Processor - What JavaScript logic runs in this step?
• URL generator - For HTTP calls, how is the URL determined?
• Payload generator - For HTTP calls, what body gets sent?
• Results processor - How is the response handled?
• Split generator - Does output need to be split into multiple items?

Check Feasibility

Before proceeding to implementation, verify that your design is achievable within opscotch's capabilities.

Trigger Feasibility

• Is your trigger type supported? (http, timer, tcp, fileWatcher, runOnce, deploymentAccess)
• Does the trigger provide sufficient data? Can you extract what's needed from the trigger event?

Input Feasibility

• Can you access the input data? Check bootstrap configuration for appropriate permissions
• Do you need external services? Ensure hosts are configured in the bootstrap

Output Feasibility

• Can you reach the destination? Verify host configurations and permissions
• Is the output format correct? Plan for any necessary transformations

Processing Feasibility

• Can it be done in plain JavaScript? No external libraries or Node.js APIs
• Is it synchronous? Remember - no async/await, no callbacks, no setTimeout/setInterval
• Use sendToStep to yield to the global event loop for sequential processing
• Use sendToStepAndForget for parallel, fire-and-forget operations

Common Infeasibility Patterns

If you encounter these, you'll need to adjust your approach:

• Requires Node.js modules - Cannot use fs, path, crypto directly
• Requires npm packages - No require() or import allowed
• Requires async patterns - Must restructure using sendToStep instead
• Complex state management - May need multiple steps with persistence

Map to Configuration

Once your design is complete, you can translate it to opscotch configuration. Here's the mapping:

Bootstrap Elements

{
  "workflows": [{
    "name": "workflow-name",
    "steps": [
      // Your step definitions go here
    ]
  }],
  "hosts": [
    // External services you need to call
  ],
  "allowFileAccess": {
    // File permissions if needed
  }
}

Step Configuration Template

{
  "stepId": "unique-step-name",
  "type": "javascript",
  "trigger": {
    // Your trigger type and configuration
  },
  "urlGenerator": {
    "script": "return 'https://api.example.com/endpoint';"
  },
  "resultsProcessor": {
    "processors": [
      {
        "script": "// Your processing logic here"
      }
    ]
  }
}

Common Template Patterns

Non-valid JSON examples

The JSON examples below contain multi-line strings for prose clarity. These are not valid JSON but demonstrate the structure and intent clearly.

HTTP Webhook Handler

{
  "stepId": "webhook-handler",
  "trigger": {
    "http": {
      "method": "POST",
      "path": "/webhook"
    }
  },
  "resultsProcessor": {
    "script": "
      var body = context.getBody();
      // Process webhook payload
      context.setData('processed', true);
    "
  }
}

Scheduled Poller

{
  "stepId": "health-check",
  "trigger": {
    "timer": {
      "period": 300000
    }
  },
  "urlGenerator": {
    "script": "return context.getProperty('healthEndpoint');"
  },
  "resultsProcessor": {
    "script": "
      var status = context.getBody();
      if (status !== 'healthy') {
        context.addUserError('Service unhealthy: ' + status);
      }
    "
  }
}

File Processor

{
  "stepId": "file-ingest",
  "trigger": {
    "fileWatcher": {
      "patterns": ["**/*.csv"]
    }
  },
  "resultsProcessor": {
    "script": "
      var content = context.getMessageBodyAsString();
      // Parse and process CSV content
    "
  }
}

Next Steps

With a clear plan in place, you're ready to:

1. Create your bootstrap configuration - Set up hosts, permissions, and workflow definitions
2. Implement your processors - Write the JavaScript logic for each step
3. Test locally - Use the opscotch test framework to verify behavior
4. Deploy and monitor - Run in production and observe via logs and metrics

Remember: Start simple, validate each piece, then compose into more complex workflows. The structured approach outlined here will help you build reliable, maintainable automation with opscotch.