Analytics Module Overview

This document presents the analytics modules available in the Crosser Module library. Analytics modules are used to transform and harmonize message formats from different sources, as well as operate on the actual message content. The modules are presented in groups with related functionality.

Data Transformation

Property Mapper

The Property Mapper module is the Swiss army knife for modifying message structure.

Use it to:

Rename properties
Move properties to new hierarchy levels
Remove properties
Add new properties with static values
Copy existing property values to new properties

Working with Arrays

Arrays are common both on inputs and outputs, but may not be the optimal internal format when applying streaming processing. For example, to apply processing on individual array elements in a streaming pipeline you need to split up the array into individual messages. Another example is when you want to write data into columns in a database, then you need an object with key/value pairs that can be mapped against the database columns.

There are several modules in the Crosser Module library that help you convert back and forth between these different formats. In this environment an array is typically an array of objects, i.e. each element contains multiple values and even hierarchical message structures.

Names in bold indicate the most common array modules.

Name	Description	Input	Output
Array Split	Breaks up an array into individual messages	Array	Messages
Array Join	Combine a stream of messages into an array, by time or message count (opposite of Array Split)	Messages	Array
Array to Object	Convert an array into an object with key/value pairs. Each item in the array must have a property holding the key value and another property with the value for that key	Array	Message
Object to Array	Convert an object with key/value pairs into an array. Each element in the array will have a property containing the name of the key and another with the value belonging to that key (opposite of Array To Object)	Message	Array
Array Filter	Filter out some elements from an array by matching against one ore several conditions.	Array	Array
Array GroupBy	Split an array into sub-arrays by grouping elements based on the value of a property.	Array	Arrays
Array Property Pick	Selects some of the properties available in each array element. A list of properties to pick must be provided.	Array	Array
Array Property Omit	Remove some of the properties available in each array element. A list of properties to remove must be provided.	Array	Array
Array Sort By List	Sort array elements by comparing the values on a specified property against an ordered list of values provided. Typically used together with Array Property Get to create lists of values where the position in the array is used to identify a source.	Array	Array
Array Property Get	Select the values from a selected property. The output is an array with only values. Useful when feeding e.g. ML models where the position in the array is used to identify a source, rather than key/value mappings.	Array	Array of Values
Join	This is a variant of the Array Join module that ensures that data for a specified number of sources are always available in each output. If no data has been received for a specific source within the given time period an output will be generated based on the strategy selected. This module is more complex to set up but is useful in cases where you must ensure that each output has a value for each source, e.g. when feeding an ML model.	Messages	Array

Working with Strings

Name	Description	Input	Output
Text Template	Generates text messages based on a template provided in the settings. Template syntax can be used to insert data from the message together with static text.	Message	String
String Replace	Replace a substring with another substring.	String	String
String Substring	Select a substring based on start and stop positions	String	String
CSV Line Parser	Break up a line of text (string) into substrings based on a delimiter, e.g. “,”. The output is an object with one property per substring. These properties are named based on the position in the string, e.g. col1, col2...	String	Object with substrings
CSV Text Parser	Same operation as the CSV Line Parser but operating on multi-line strings. The output is an array with one element per line. Each element has the same format as with the CSV Line Parser.	String	Array of objects with substrings

Format Conversions

Messages sent between modules use a custom .NET data type (FlowMessage) that supports dynamic structures with hierarchies of objects and arrays which can contain basic .NET data types. Sometimes, e.g. in the debug window, messages may look like Javascript objects (JSON) but that is only for presentation purposes. JSON is never used internally. When communicating with external systems you may get data in JSON or XML format, or the output is expected to be in one of these formats. To work with these types of data you must convert between the internal format and the external formats. To do that you use these conversion modules.

Name	Description
JSON	Converts to and from JSON strings. If the input is a string it will be interpreted as JSON and the output is a FlowMessage. If the input is a FlowMessage it will be converted into a JSON string. This module operates on a selected property of the incoming message and the result can be assigned to a new property.
XML	Same as the JSON module but working with XML text.
Time Stamp	Convert DateTime values into another format. Predefined formats are ISO8601 and Unix timestamp in seconds or milliseconds. Custom output formats can also be defined. If no input is specified the module will add the current system time on the output.
Base64 Encode	Convert a binary array or a string into a base64 encoded string. For string inputs the output can be set to use base64url format, to create URL safe strings.
Base64 Decode	Decode a base64 encoded string into a byte array or string

Intelligent Logic

Operating on message Data

The following modules will create new data by processing the values in messages. They all work on numerical values and the result can either be assigned to a new property or the original value can be overwritten.

Name	Description
Aggregate	Calculate average, min and max on a selected property from a group of messages. The group of messages to use can either be selected by specifying a time period, or a number of messages. Calculations can be performed independently for messages coming from different sources by specifying a property that indicates the source. At the end of each interval a message is delivered for each source with the calculated statistics.
Math	Execute generic mathematical expressions on message data. The expressions are specified using template syntax where message data can be inserted into the mathematical expression, e.g. ‘Abs({data.cur_temp} - {data.prev_temp})’.
Range Classifier	Add a text classification based on ranges of values on a selected property, e.g. ‘Low’, ‘Med’, ‘High’
Scale	Scale a value on a selected property ‘(value*scale)-offset’, e.g. to convert temperature values from Farenheit to Celsius. Note: This operation can also be performed with the Math Expression module.
Smooth	Apply an exponential smoothing filter on a property value, e.g. to remove noise.
Statistics	Same as Aggregate but operating on a rolling window of messages, i.e. an output message is generated for each input message. This module will also calculate the standard deviation of the values.
Toggle	Toggle the value of a boolean property, or toggle the value of an internal variable and add to the output message

Code Modules

With the code modules you can apply custom processing of messages using either C# or Python code. The code can either be entered directly in the settings UI, or code files can be uploaded to the Resource library and then referenced from within the modules.

Name	Description
Csharp	Execute C# code. The code is compiled at runtime. Standard .NET libraries can be used but not 3rd party libraries.
IronPython	Run Python code using the IronPython interpreter which runs in .NET. Standard libraries can be used but not 3rd party libraries. Supports Python 2.7 code.
Python Bridge	Run python code in a standard Python 3.7 environment outside of .NET. On docker this module is only supported with images that includes the external python environment (tagged with ‘latest-python’). On Windows the python environment must be installed separately and the local node configuration must be updated accordingly. Standard and 3rd party libraries can be used, including ML frameworks.

Counter Modules

Name	Description
Message Counter	Counts the values seen in a selected message property over a time period. At the end of the period the number of occurrences of each value is listed together with the relative count of each value. Useful for KPI calculations, such as Yield. If no source is specified the total number of messages received is reported.
Time Counter	Counts the time spent in different states by looking at the values received on a specified property and the time between value changes. At the end of the period the total time spent in each state (value) is reported together with the relative times. Useful for KPI calculations such as Availability.
Timeout	Measures the time since the last update on a selected property. If no update is seen within a specified timeout period a message is sent out. Use this module to monitor expected traffic patterns. For example, if you know that data should arrive every second this module can be used to trigger an alert if no update is seen within 2 seconds. By keeping track of individual property values the module can operate on multi-source streams and trigger an alert as soon as one a value is missing from one of the sources.

Storage Modules

Sometimes you need to store message data temporarily and then these modules will come in handy.

Name	Description
Key Value Get/Set/Delete	The Node has internal key/value stores which can be accessed using the Key Value modules. You can have multiple named stores and they can either be kept in memory, or on disk (persistent). If stored on disk these stores can also be shared between Flows.
Memory Buffer	This module will keep messages until acknowledged by an external signal. It is designed to work with any output module to keep messages until successfully delivered. Output modules produce a ‘crosser.success’ value indicating if the external delivery was successful or not. This value can be fed back to the Memory Buffer module. The size of the buffer as well as retry strategies can be defined. Note: All modules have built-in store-and-forward capabilities similar to what the Memory Buffer module can do. You only need to use this module when custom logic is required for controlling the resend operation.

Name

Description

Key Value Get/Set/Delete

The Node has internal key/value stores which can be accessed using the Key Value modules. You can have multiple named stores and they can either be kept in memory, or on disk (persistent). If stored on disk these stores can also be shared between Flows.

Memory Buffer

This module will keep messages until acknowledged by an external signal. It is designed to work with any output module to keep messages until successfully delivered. Output modules produce a ‘crosser.success’ value indicating if the external delivery was successful or not. This value can be fed back to the Memory Buffer module. The size of the buffer as well as retry strategies can be defined.

Note: All modules have built-in store-and-forward capabilities similar to what the Memory Buffer module can do. You only need to use this module when custom logic is required for controlling the resend operation.

Condition Logic

Message filters

In addition to the below modules each module has a configurable message filter, which can be found on the “Common” tab. By default modules will process every message they receive. By adding a message filter the module will only process messages matching some criteria. Messages not matching the filter criteria can either be dropped, or passed on untouched.

Message filters can be set to match conditions on numeric property values (=, <, >), string property values (contains, does not contain, equal/not equal, isNull, isNotNull, regex) and boolean values (isTrue, isFalse). Multiple conditions can be used.

Filtering modules

In addition to the message filters above the following modules can be used to control which messages that are passed on to the next module(s), based on different criteria.

Name	Description
Range Filter	Only let through messages where a selected value is either within or outside of a specified range. The same effect can be achieved by using message filters.
Report By Exception	Only let through messages when the value of a specified property changes. Can be used to remove duplicates.
Split	Split a stream of messages into multiple paths by defining conditions on message data. Multiple conditions can be specified for each path. Each path will get a separate output on the module.
Throttle	Limit the rate of messages, e.g. only let through a maximum of one message per second. Can also be used to spread out messages in time by enforcing a minimum delay between messages, which can be used to smooth out bursty traffic.

29 Jul 2024

Modules

About the author

Goran Appelquist

Göran has 20 years experience in leading technology teams. He’s the lead architect of our end-to-end solution and is extremely focused in securing the lowest possible Total Cost of Ownership for our customers.

"Hidden Lifecycle (employee) cost can account for 5-10 times the purchase price of software. Our goal is to offer a solution that automates and removes most of the tasks that is costly over the lifecycle.

My career started in the academic world where I got a PhD in physics by researching large scale data acquisition systems for physics experiments, such as the LHC at CERN. After leaving academia I have been working in several tech startups in different management positions over the last 20 years.

In most of these positions I have stood with one foot in the R&D team and another in the product/business teams. My passion is learning new technologies, use it to develop innovative products and explain the solutions to end users, technical or non-technical."