Search Crosser Knowledge Base

The Array Modules

A common scenario is that data arrives as an array of objects, e.g. from a database query or through an API call. The Crosser module library has many modules to work with arrays, in this article we will go through the main use cases and the modules to use.

When working with arrays there are two main classes of use cases:

  1. You want to apply some processing to each of the elements in the array
  2. You want to operate on the whole array

Iterating over array elements

Let’s start with the first class of use cases: Operating on individual array elements. For this use case the ‘Array Split’ module is your best friend. It will break up an array into individual messages, one per element. This will allow you to iterate through the elements and apply whatever processing you like, think a “for loop” in programming languages. When the ‘Array Split’ module receives a message with an array, it will generate a bunch of messages on the output, one for each element. After the ‘Array Split’ module you can then apply any type of processing, by using other modules.

If you want to get the processed data back into an array there is a companion module named ‘Array Join’. This module will collect a number of messages and combine them into a single message with an array. You can configure how to combine messages, e.g. by specifying the number of messages you want to combine, or use a time window to collect messages over. When used in combination with the ‘Array Split’ module there is also an ‘auto’ mode that uses information from the ‘Array Split’ module to recombine the same messages. To make this work you must enable ‘Add array information’ on the ‘Array Split’ module.

Note: If you just want to remove some properties from each element you don’t need to use the above concept, with a Property Mapper in between the ‘Split’ and ‘Join’ modules. There are dedicated modules for that purpose: ‘Array Property Omit/Pick’.

Crosser Example Flow | Array Modules

Operating on arrays

The other class of use cases is when you want to operate on the whole array and either produce new arrays or some value calculated over data in the array elements. Here are some modules that can help you with that:

  • Array Filter - Produce a new array with a subset of the elements in the original array, by selecting elements to keep using filter conditions.
  • Array GroupBy - Create multiple arrays by grouping array elements with the same value on some property. Similar to SQL Group By.
  • Array Statistics - Calculate statistics (min, max, count…) based on the values on some property.
  • Array to Object - Combine array elements into an object where ‘key’ and ‘value’ are taken from each element. For example if you want to store an array of data from a PLC module into a database, where tag names are matched against database columns.
  • Object to Array - Create an array from an object by converting each property into an array element. Useful for example if you want to apply processing to all properties in an object.

About the author

Goran Appelquist

Göran has 20 years experience in leading technology teams. He’s the lead architect of our end-to-end solution and is extremely focused in securing the lowest possible Total Cost of Ownership for our customers.

"Hidden Lifecycle (employee) cost can account for 5-10 times the purchase price of software. Our goal is to offer a solution that automates and removes most of the tasks that is costly over the lifecycle.

My career started in the academic world where I got a PhD in physics by researching large scale data acquisition systems for physics experiments, such as the LHC at CERN. After leaving academia I have been working in several tech startups in different management positions over the last 20 years.

In most of these positions I have stood with one foot in the R&D team and another in the product/business teams. My passion is learning new technologies, use it to develop innovative products and explain the solutions to end users, technical or non-technical."