How can I watch log file?

In opscotch version 3, file access was introduced, along with the fileWatcher step trigger. File watching is a capability that allows you to request the agent to call a step when a file is changed—for example, when a log file is updated, you can receive the new lines as a step payload. As security is always a top priority, file access is strictly controlled in several ways.

Watching a file utilizes the bootstrap file access property, which already provides the necessary file security mechanisms.

To watch a file or files in a directory, add a file access record to the bootstrap with the READ and LIST permissions.

Bootstrap:

{
    ...
    "allowFileAccess": [
        {
            "id": "watchMyDirectory",
            "directoryOrFile": "/a/path",
            "LIST": true,
            "READ": true
        }
    ],
    ...
}

Next add a step with the fileWatcher trigger. The following properties are required:

• bootstrapFileId: the id of the bootstrap allowFileAccess record
• eventSplitter: A (escaped) regex pattern to use to split the file contents into records.
  • Take note of the splitAtEnd property; this can be used to change the behavior of event splitting. For example, if the start of the record is a date, set the pattern to match the date and splitAtEnd: false - now the file will be split such that the date is at the start of the record.

Workflow:

{
    ...
    "workflows" : [
        {
            ...
            "steps" : [
                {
                    ...
                    "trigger" : {
                        "fileWatcher" : {
                            "bootstrapFileId" : "watchMyDirectory",
                            "eventSplitter" : "\\n" 
                        }
                    },
                    "resultsProcessor" : {
                        "processors" : [
                            {
                                "script" : "console.log(context.getMessageBodyAsString());"
                            }
                        ]
                    }
                    ...
                }
            ]
            ...
        }
    ]
    ...
}

When the workflow is loaded, opscotch will place a watch on the files. The watch covers all file operations in the watched file or directory scope. When a matching change is made to the files, the step's resultsProcessor will be called with the following payload structure.

If appended content does not yet contain the next matching eventSplitter, processing blocks until the next matching splitter is seen. This prevents incomplete records from being emitted before their configured boundary is available.

{
   "log": {
     "file": {
       "path": "/path/to/file.txt"
     },
     "offset": 4130
   },
   "message": "this is a line from a file",
   "input": {
     "type": "log"
   },
   "host": {
     "name": "hostname",
     "ip": "[fe80:0:0:0:a6d7:feea:f601:902%wlp1s0, 192.168.0.27]"
   },
   "agent": {
     "type": "opscotch",
     "version": "3.0.0"
   },
   "ecs": {
     "version": "1.12"
   }
}

You can then use workflows to process the file contents as you like.