Connect to data sources

You can use data from a large variety of data sources in Crosser Flows. This section will introduce you to some common use cases.

Machine data

The most common sources of machine data are PLCs, OPC UA servers and Historians. There are corresponding modules available for the most common systems. Also the MQTT modules can be useful in some cases.

When getting data from PLCs there's typically no metadata available, only the raw bytes/words. To make the data useful we must add relevant information to allow the input modules to provide useful data. This is done by defining tags, e.g. by specifying address, data type and name. This can be done in the module settings, or by providing a list of tags as a resource file referenced from the module. Each module can read any number of tags from the same PLC, but since data is pulled from the PLCs, these modules must be triggered each time you want to read data, e.g. by using an Interval module.

OPC UA server provides structured data with corresponding metadata and you reference the tags by specifying node IDs, either in the module settings or through a resource file. With OPC UA you also have the option to trigger reads from the Flow with the OPC UA Reader module, or let the OPC UA server push data to your Flow when using the OPC UA Subscriber module. Each time you trigger the Reader module it will get all configured tags in a single message. With the Subscriber module you will get individual messages for each tag when the value change, but not more often than the configured publishing interval. You can also subscribe to events from the OPC UA server with the OPC UA Events module.

The available machine connectors are found in the Sources/Industrial category.

APIs

Getting data from an API is typically done by sending a HTTP request. This can be done with the HTTP Request module. This is a generic module where you need to provide the full URL and optional payload and headers. These can either be static or provided as input to the module. The module can convert common data formats, such as JSON, into the internal format used in Flows. Basic an Bearer (token) authentication can be used, though the Credentials library.

If you are using the same API in many Flows it may make sense to create a custom module for this API. You can then build your own connector using the Universal Connector tool. This is a wizard-based tool where you can parameterize a request using both module settings and Flow data. In the end you will get a new module in your library that looks like any other module, with settings, a custom icon and documentation. This makes using an API much easier, since you don't need to remember all the details of the requests. Universal Connectors also supports OAuth credentials.

You can also define your own 'API endpoint', to let other HTTP clients publish data into your Flows. The Node has a built-in HTTP server and by using the HTTP Listener module you can enable an endpoint on this server. When data is published by an external client you will get an output from this module. If you need to create a custom response, you can combine the Listener module with the HTTP Response module.

Databases

You can get data from most common databases: SQL, Non-SQL and timeseries. When working with SQL databases you have the option of using our Select modules where you configure your query without using SQL. For full flexibility, including combining data from multiple tables, the Executer modules are available where you can enter your own SQL statements.

The database modules are found in the Sources/Databases category.

Files

Data in text files can be used as input to Flows. Any text file can be read with the File TextReader module. The content can then be converted into useful data with the JSON, XML or CSV TextParser modules. For CSV files there's also a dedicated CSV Reader module. If files are added to a watch folder you can monitor this folder with the File Watcher module and then read any new files with the File TextReader module, or you can use the File TailReader to get a message each time data is appended to a file. There is also a similar set of modules for getting files from SFTP server.

The file-related modules are found in the Sources/Files category.