Monitoring the Crosser Node

Introduction

Once you have your first flows deployed, you might think about how to integrate the Crosser Node and Flows into your existing monitoring solution.

In this article we describe what options you have and how to utilize provided interfaces for integration into your monitoring solution.

Local log files

The logs of the Node are stored locally and roll-over once per day assuming they do not reach the size limit, in which case additional logfiles with a sequence number appended will be created. A timestamp is added to the log name to make it easy to identify.

You can find the logs in the installation folder of your Node ./data/logs.

We decided to create one log per process running on the device which results in one log for the Node itself ./data/logs/host and separated logs for each flow ./data/logs/runtime/<Flow ID>. Assuming you had executed Remote Sessions at some point on that Node, you will also see a folder ./data/logs/runtime/remotesession/<Flow ID>.

Since you most likely do not want to integrate this one into your monitoring solution, we leave this folder out of scope.

Once you have located the logs you can use third-party tools like 'datadog' or 'filebeat' to catch and upload the logs into your monitoring solution.

Node API

General information

The Node offers a local API interface (version 2.5.x and above) which you can use to extract information about the Node, Flows and Modules.

A documentation about this API is integrated into the Node. The API is exposed to port 9191 (default) for both Windows and Docker installations.

Assuming your port is exposed and not blocked by a firewall, you should be able to access your Node's local API and documentation via port 9191.

http://<Your Node IP>:9191/swagger/index.html

You should be able to access the Web UI with any browser:

Once you know that the API is accessible, check the endpoints and think about what information you would like to extract.

You can use the exposed Web UI to run API commands immediately to see how it works.

Now it is up to you how to utilize the API.

Consume the API from external systems

Assuming your monitoring solution has access to the local Node's API, you can think about implementing them directly (if your monitoring system does support that).

This approach is often seen in environments where customer have monitoring solutions on-premise or agents of their monitoring solution that are capable of consuming APIs.

Create a monitoring flow

Another way to consume the local API is to create a Flow that uses the API itself. You can simply use the HTTP Request module to call the API and extract the relevant information with the low-code approach. Once you have done that you can react on certain events and send out notifications to external Brokers, APIs, HTTP endpoints or notify a user with the extracted and relevant information.

With this setup you would push the data out to external systems. This is useful if your monitoring solution does not have access to the Node's local API but the Monitoring System's endpoint is exposed so we can i.e. use HTTP Request module to push data to it. You can deploy this flow on the same Node as your other flows are running.

The downside of this implementation though is that if the entire Node fails for some reason, also your monitoring flow will die.

To get around that, you can think about setting up a dedicated Node which only runs monitoring flows in a dedicated environment.

This setup can also help bridging network gaps between the Nodes (production network) and a monitoring system (office network/cloud).

MQTT (Node version 2.6+)

With Node 2.6 Crosser has introduced the possibility to utilize the Node’s internal MQTT Broker to get events when a flow/module status/state changes. This allows you to create an event driven notification architecture with external monitoring tools.

Similar to the HTTP approach, you can either connect to the Node’s local broker from an external MQTT client, or create a flow which subscribes to the relevant topics and send out notification in case something goes wrong.
The Node will publish messages when things occur that someone might want to know. The root topic for all the messages will be $crosser/. So for example $crosser/flows/{flowDefinitionId}/started will be published when a flow starts.

The $crosser topic will be forbidden by clients to publish to, only subscriptions are allowed. If a client tries to publish with a topic that starts with $crosser/ the client will be disconnected.

Note that MQTT Notifications are disabled for remote sessions. Only deployed flows will trigger MQTT Notifications.

Flow Topics

Topic: $crosser/flows/{flowDefinitionId}/state [RETAINED]
When a flow changes state (started, stopped) this topic will be triggered with the information:

{
    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
    "flowName": "MQTT Notifications",
    "version": 9,
    "running": true,
    "status": "Ok",
    "reason": "Flow started",
    "origin": "Cloud",
    "timestamp": "2023-07-04T10:40:47.601+00:00",
    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
    "nodeName": "Machine123",
    "timestamp": "2023-09-28T13:22:51.743+00:00"
}

Topic: $crosser/flows/{flowDefinitionId}/status [RETAINED]
When a flow change status to Ok, Warning or Error:

    {
  "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
  "flowDefinitionId": "6faf4080-c7ce-e642-37a0-43e965904899",
  "flowName": "Example MQTT Notifications",
  "version": 9,
  "status": "Warning",
  "reason": "Flow status changed to Warning",
  "reasons": [
    {
      "moduleId": "9c3727c6-7d64-4cb3-95eb-a597596068b2",
      "moduleName": "OPC UA Reader",
      "moduleType": "Crosser.EdgeNode.Modules.OPC.UA.Reader",
      "version": "2.3.0",
      "status": "Warning",
      "reason": "Module OPC UA Reader, 2.3.0 changed to status Warning: This is a test",
      "timestamp": "2023-09-28T13:22:51.743+00:00"
    }
  ],
  "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
  "nodeName": "Machine123",
  "timestamp": "2023-09-28T13:22:51.743+00:00"
}

Topic: $crosser/modules/{moduleId}/status [RETAINED]
When a module changes status (Ok, Warning, Error):

    {
    "moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",
    "moduleName": "Queue & Persistence",
    "version": "2.3.0",
    "status": "Warning",
    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
    "flowName": "MQTT Notifications",
    "reason": "Module Queue & Persistence, 2.3.0 changed to status Warning: Queue is full  (100), message was dropped, fullMode=DropWrite, persistent=True",
    "queue": {
        "mode": "DropWrite",
        "size": 100,
        "retries": 3,
        "retryDelay": 100,
        "delay": 0,
        "persistence": true
    },
    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
    "nodeName": "Machine123",
    "timestamp": "2023-09-28T13:22:51.743+00:00"
}

Topic: $crosser/modules/{moduleId}/messages/dropped

When the module queue is full and the strategy is not wait.

    {
    "moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",
    "moduleName": "Queue & Persistence",
    "version": "2.3.0",
    "status": "Warning",
    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
    "flowName": "MQTT Notifications",
    "reason": "Queue is full",
    "queue": {
        "mode": "DropWrite",
        "size": 100,
        "persistence": {
            "enabled": true,
            "retries": 3,
            "retryDelay": 100
        }
    },
    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
    "nodeName": "Machine123",
    "timestamp": "2023-09-28T13:22:51.743+00:00"
}

Topic: $crosser/modules/{moduleId}/messages/deadletters

When persistence is on and a message has failed maximum number of times it will be moved to the deadletter queue. You will then get a message similar to this:

    {
    "moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",
    "moduleName": "Queue & Persistence",
    "version": "2.3.0",
    "status": "Ok",
    "flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
    "flowName": "MQTT Notifications",
    "reason": "Retries maxed out",
    "queue": {
        "mode": "DropWrite",
        "size": 100,
        "persistence": {
            "enabled": true,
            "retries": 3,
            "retryDelay": 100
        }
    },
    "link":     "http://localhost:9191/api/flows/2220e7a7-9e98-4af3-afd4-a1b309128540/deadletters/043e9604-837d-41b7-8a63-ad85321bfe32",
    "nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
    "nodeName": "Machine123",
    "timestamp": "2023-09-28T13:22:51.743+00:00"
}

Subscribing

By subscribing to $crosser/# you would get all messages
Subscribing to $crosser/flows/# would get you all flow messagee
Subscribing to $crosser/modules/# would get you all module messages

Of course you can also use other combinations and use + as a single level wildcard

Subscribing to $crosser/flows/+/stopped would get you only stopped flow messages
Subscribing to $crosser/modules/+/status would get you module status messages for all modules

Get the current status for all deployed flows. When a MQTT client creates a subscription for the topic $crosser/flows/# or $crosser/flows/+/status the broker will send out the current status for all deployed flows.

01 Jun 2024

About the author

Goran Appelquist

Göran has 20 years experience in leading technology teams. He’s the lead architect of our end-to-end solution and is extremely focused in securing the lowest possible Total Cost of Ownership for our customers.

"Hidden Lifecycle (employee) cost can account for 5-10 times the purchase price of software. Our goal is to offer a solution that automates and removes most of the tasks that is costly over the lifecycle.

My career started in the academic world where I got a PhD in physics by researching large scale data acquisition systems for physics experiments, such as the LHC at CERN. After leaving academia I have been working in several tech startups in different management positions over the last 20 years.

In most of these positions I have stood with one foot in the R&D team and another in the product/business teams. My passion is learning new technologies, use it to develop innovative products and explain the solutions to end users, technical or non-technical."