Skip to main content

How can I use the internal queue?

· 2 min read

An exciting change in version 3 of opscotch is the introduction of workflow-available high-performance persistent FIFO queues. This was the missing functionality that allows opscotch workflows to robustly buffer outbound data - essentially acting as a log aggregator, among other uses.

Version 3 of opscotch changed the way that internal opscotch logs and metrics are prepared and sent to monitoring platforms: opscotch now uses (internal system) opscotch workflows to send logs and metrics—opscotch using opscotch! In doing so, a high-performing persistent FIFO queue was implemented and made available to workflows.

How does it work?

The queuing mechanism is ready to use in the JavaScript context: just use the context.queue() to get a queue context. Refer to the documentation to see that there are two main functions:

  • push(...): Adds an item to the queue.
  • take(...): Removes items from the queue.

So the basic usage in any processor is:

context.queue().push("Hello World");
var item = context.queue().take(1); // item is "Hello World"

When you use push(...), the item is added to the end of the queue. Depending on the size in bytes of the queue data, the item might be added to a memory queue, or if the queue is larger, then the item might be written to a file on disk.

When you use take(N), you are requesting "at most" N items from the queue. The key here is "at most": opscotch only holds a small amount of the queue in memory and will only serve items from memory. When the items stored in memory have been consumed, the take(...) method will return what was available and request the memory portion to be loaded from disk. It's important to note that the take(...) function will not block while data is being loaded; workflows are encouraged to work with the data that you've been given, rather than waiting for a specific amount (imagine you ask for 1000 items, but there are only 10 available: you could be waiting forever!).

As of version 3.0.0, the queue mechanics are hard-coded to:

  • Maximum in-memory queue size is 1MB
  • Maximum file size before file rolling is 5MB