Monitoring the Crosser Node
Introduction
Once you have your first flows deployed, you might think about how to integrate the Crosser Node and Flows into your existing monitoring solution.
In this article we describe what options you have and how to utilize provided interfaces for integration into your monitoring solution.
Local log files
The logs of the Node are stored locally and roll-over once per day assuming they do not reach the size limit, in which case additional logfiles with a sequence number appended will be created. A timestamp is added to the log name to make it easy to identify.
You can find the logs in the installation folder of your Node ./data/logs.
We decided to create one log per process running on the device which results in one log for the Node itself ./data/logs/host and separated logs for each flow ./data/logs/runtime/<Flow ID>. Assuming you had executed Remote Sessions at some point on that Node, you will also see a folder ./data/logs/runtime/remotesession/<Flow ID>.
Since you most likely do not want to integrate this one into your monitoring solution, we leave this folder out of scope.
Once you have located the logs you can use third-party tools like 'datadog' or 'filebeat' to catch and upload the logs into your monitoring solution.
Node API
General information
The Node offers a local API interface (version 2.5.x and above) which you can use to extract information about the Node, Flows and Modules.
A documentation about this API is integrated into the Node. The API is exposed to port 9191 (default) for both Windows and Docker installations.
Assuming your port is exposed and not blocked by a firewall, you should be able to access your Node's local API and documentation via port 9191.
i.e. http://<Your Node IP>:9191/swagger/index.html
You should be able to access the Web UI with any browser:
Once you know that the API is accessible, check the endpoints and think about what information you would like to extract.
You can use the exposed Web UI to run API commands immediately to see how it works.
Now it is up to you how to utilize the API.
Consume the API from external systems
Assuming your monitoring solution has access to the local Node's API, you can think about implementing them directly (if your monitoring system does support that).
This approach is often seen in environments where customer have monitoring solutions on-premise or agents of their monitoring solution that are capable of consuming APIs.
Create a monitoring flow
Another way to consume the local API is to create a Flow that uses the API itself. You can simply use the HTTP Request module to call the API and extract the relevant information with the low-code approach. Once you have done that you can react on certain events and send out notifications to external Brokers, APIs, HTTP endpoints or notify a user with the extracted and relevant information.
With this setup you would push the data out to external systems. This is useful if your monitoring solution does not have access to the Node's local API but the Monitoring System's endpoint is exposed so we can i.e. use HTTP Request module to push data to it. You can deploy this flow on the same Node as your other flows are running.
The downside of this implementation though is that if the entire Node fails for some reason, also your monitoring flow will die.
To get around that, you can think about setting up a dedicated Node which only runs monitoring flows in a dedicated environment.
This setup can also help bridging network gaps between the Nodes (production network) and a monitoring system (office network/cloud).
MQTT (Node version 2.6+)
With Node 2.6 Crosser has introduced the possibility to utilize the Node’s internal MQTT Broker to get events when a flow/module status/state changes. This allows you to create an event driven notification architecture with external monitoring tools.
Similar to the HTTP approach, you can either connect to the Node’s local broker from an external MQTT client, or create a flow which subscribes to the relevant topics and send out notification in case something goes wrong.
The Node will publish messages when things occur that someone might want to know. The root topic for all the messages will be $crosser/. So for example $crosser/flows/{flowDefinitionId}/started will be published when a flow starts.
The $crosser topic will be forbidden by clients to publish to, only subscriptions are allowed. If a client tries to publish with a topic that starts with $crosser/ the client will be disconnected.
Note that MQTT Notifications are disabled for remote sessions. Only deployed flows will trigger MQTT Notifications.
Flow Topics
Topic: $crosser/flows/{flowDefinitionId}/state [RETAINED]
When a flow changes state (started, stopped) this topic will be triggered with the information:
{
"flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
"flowName": "MQTT Notifications",
"version": 9,
"running": true,
"status": "Ok",
"reason": "Flow started",
"origin": "Cloud",
"timestamp": "2023-07-04T10:40:47.601+00:00",
"nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
"nodeName": "Machine123",
"timestamp": "2023-09-28T13:22:51.743+00:00"
}
Topic: $crosser/flows/{flowDefinitionId}/status [RETAINED]
When a flow change status to Ok, Warning or Error:
{
"flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
"flowDefinitionId": "6faf4080-c7ce-e642-37a0-43e965904899",
"flowName": "Example MQTT Notifications",
"version": 9,
"status": "Warning",
"reason": "Flow status changed to Warning",
"reasons": [
{
"moduleId": "9c3727c6-7d64-4cb3-95eb-a597596068b2",
"moduleName": "OPC UA Reader",
"moduleType": "Crosser.EdgeNode.Modules.OPC.UA.Reader",
"version": "2.3.0",
"status": "Warning",
"reason": "Module OPC UA Reader, 2.3.0 changed to status Warning: This is a test",
"timestamp": "2023-09-28T13:22:51.743+00:00"
}
],
"nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
"nodeName": "Machine123",
"timestamp": "2023-09-28T13:22:51.743+00:00"
}
Topic: $crosser/modules/{moduleId}/status [RETAINED]
When a module changes status (Ok, Warning, Error):
{
"moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",
"moduleName": "Queue & Persistence",
"version": "2.3.0",
"status": "Warning",
"flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
"flowName": "MQTT Notifications",
"reason": "Module Queue & Persistence, 2.3.0 changed to status Warning: Queue is full (100), message was dropped, fullMode=DropWrite, persistent=True",
"queue": {
"mode": "DropWrite",
"size": 100,
"retries": 3,
"retryDelay": 100,
"delay": 0,
"persistence": true
},
"nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
"nodeName": "Machine123",
"timestamp": "2023-09-28T13:22:51.743+00:00"
}
Topic: $crosser/modules/{moduleId}/messages/dropped
When the module queue is full and the strategy is not wait.
{
"moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",
"moduleName": "Queue & Persistence",
"version": "2.3.0",
"status": "Warning",
"flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
"flowName": "MQTT Notifications",
"reason": "Queue is full",
"queue": {
"mode": "DropWrite",
"size": 100,
"persistence": {
"enabled": true,
"retries": 3,
"retryDelay": 100
}
},
"nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
"nodeName": "Machine123",
"timestamp": "2023-09-28T13:22:51.743+00:00"
}
Topic: $crosser/modules/{moduleId}/messages/deadletters
When persistence is on and a message has failed maximum number of times it will be moved to the deadletter queue. You will then get a message similar to this:
{
"moduleId": "9f5f82a2-1dc0-433d-805e-182be417fdef",
"moduleName": "Queue & Persistence",
"version": "2.3.0",
"status": "Ok",
"flowId": "2220e7a7-9e98-4af3-afd4-a1b309128540",
"flowName": "MQTT Notifications",
"reason": "Retries maxed out",
"queue": {
"mode": "DropWrite",
"size": 100,
"persistence": {
"enabled": true,
"retries": 3,
"retryDelay": 100
}
},
"link": "http://localhost:9191/api/flows/2220e7a7-9e98-4af3-afd4-a1b309128540/deadletters/043e9604-837d-41b7-8a63-ad85321bfe32",
"nodeId": "27929e60-a586-4f51-a512-0583391b0c86",
"nodeName": "Machine123",
"timestamp": "2023-09-28T13:22:51.743+00:00"
}
Subscribing
- By subscribing to $crosser/# you would get all messages
- Subscribing to $crosser/flows/# would get you all flow messages
- Subscribing to $crosser/modules/# would get you all module messages
Of course you can also use other combinations and use + as a single level wildcard
- Subscribing to $crosser/flows/+/stopped would get you only stopped flow messages
- Subscribing to $crosser/modules/+/status would get you module status messages for all modules
Get the current status for all deployed flows. When a MQTT client creates a subscription for the topic $crosser/flows/# or $crosser/flows/+/status the broker will send out the current status for all deployed flows.
Search Documentation
Page Sections