Introducing Crosser FlowWatch for Data Observability

Data Observability in a Hybrid World. Get on-top of the Health of Your Data and Data Integration Flows wherever they run. From the edge to the cloud.

As data is becoming more and more critical for organizations, the need to know the health of the data and the data integration flows is increasing rapidly. There are a number of data observability, data quality and data trust topics that data teams need to be on-top of. 

The Crosser Platform is uniquely positioned for companies with a combination of streaming data and batch data integration in a hybrid and distributed architecture. As a real-time integration platform Crosser is perfectly positioned to Monitor, Validate and Control that the data flows deliver what you expect.

Are you On-Top of your Data Flows?

The concept of Data Observability is broad. At Crosser we prefer to think of it as capabilities that answers the following questions:

1. Are my integration flows working as they should?
2. How fresh is the data? Did the data arrive when it should?
3. Does the data have the right data distribution? Is the data matching expected values?
4. Is the quality of the data from my sources as expected or is there data drift (E.g. unexpected changes to schemas)
5. Is the volume from my sources as expected?
6. Is my data team the first to know if there is a problem?
7. If there is a problem, how fast can we identify the root cause?

Introducing Crosser FlowWatch

Monitor, Validate and Control your Data and Data Integration Flows. Crosser FlowWatch is a suite of capabilities for Data Observability in a world of streaming data and event-driven data integration.

It's 2am. Do you know what your Data Integration Flows are doing?

We are very excited to announce several key features and capabilities including a recent major update of the monitoring capabilities of the Crosser Platform designed to deliver Data Trust and Observability. Crosser FlowWatch is the first solution for monitoring of data flows in hybrid and distributed cloud, on-premise and edge environments. 

Crosser has, unlike other integration and ETL platforms, advanced real-time processing capabilities ideal for running advanced validation logic, pattern recognition, anomaly detection and alerting conditions, including AI/ML model inference. 

As traditional integration and ETL platforms lack these capabilities, companies have been blind or forced to purchase another platform just to address Data Observability with high cost as a consequence.

With the Crosser Platform we include the Crosser FlowWatch capabilities for Free to all our users and our goal is that you should not be needing an external Data Observability solution in order to trust your data and data integration flows.

First solution for Data Observability of Streaming IoT data

Crosser is the leading platform for processing of streaming IoT data in hybrid environments and the first solution provider to launch a full suite of capabilities for Data Observability for industrial, asset and production data. This further strengthens the value of the platform for our customers.

Crosser FlowWatch Capabilities

With the latest releases of modules and the Crosser Control Center we have expanded the capabilities and dramatically simplified for our users to be on-top of their data.

• Flow Monitoring & Alerts - Crosser Control Center
  The new monitoring page is designed to give you a detailed view of all deployed Flows and to highlight any potential issues with the status of the Flows. Alerts in the Control Center or via email warn you of issues of failed Flows or Crosser Nodes.
• Data Freshness Monitoring - Time-out Module
  The Time-out Module lets you define the expected frequency of data which you can use to build alerts and notifications via your channels of choice; SMS, email, Slack, Microsoft Teams etc. 
• Data Validation - JSON Schema Module
  Validate the quality of the incoming data against your expected Schema. Check both the structure of your data and the actual values. Make sure data that doesn’t meet your criteria enters your processing pipeline, creating issues further down the line.
• Streaming Data Validation - Data Trigger Module
  Secure data quality and detect anomalies in your streaming data by defining conditions that can combine multiple data points and check conditions over time. For example, “check that the temperature is not above some limit for more than x minutes when the machine is running”, or “check that the pressure values from two identical machines never deviates more than y”. Within a single module you can define any number of conditions, checking any number of data points. Fully configurable metadata can be associated with each trigger, to drive downstream actions, such as creating a work order in a maintenance system.
• Data Volume Validation
  Make sure you get data from the expected number of tags in your machines, or that the volume of records received from your database queries falls within reasonable ranges and trigger actions/notifications when this is not the case.
• AI/ML model Monitoring
  If your AI/ML models that are deployed with Crosser provide hooks to monitor the performance you can use the above tools to take actions on this data, just like you do with the data that enters the model.

FlowWatch - powered by an intuitive user experience in the Flow Studio and the Monitoring page

The Flow Studio makes it very easy to build, monitor and troubleshoot data flows using a combination of the modules available.

In this example we get data (records) from an external API. We check that the received data matches our expectations by validating it against a JSON schema. If the data is ok we add the received records to a database, if not ok we send a Slack message to the data engineering team with information about the issues that were found.

The Monitor tool in the Control Center collects information from all deployed Flows and provides both high-level overviews and drill-down capabilities to quickly steer the attention to the most critical issues.

Crosser FlowWatch makes you sleep great again

Your data is critical and with Crosser FlowWatch you can sleep great at night. As your data is becoming increasingly critical for your business you need to know you can trust that all integrations are working and delivering the expected results.

Data Observability is another reason why DIY direct integrations with code/scripting is a bad idea

The majority of data integrations within enterprises are not monitored today. In fact, most of them are point-to-point direct integrations, manually coded or scripted. The result is that the data teams are always the last to know when something goes wrong and a significant amount of time needs to be spent firefighting integration issues and data quality problems. 

Crosser FlowWatch lets you monitor, validate and control the quality of your data and data flows so you can sleep great at night knowing you can fully trust your data.