How to Implement and Scale Python projects for Industrial IoT

Python has gained immense popularity as a programming language among data scientists and machine learning developers worldwide. Its less complex coding syntax makes it accessible for anyone looking to apply logic to data streams. 

In this article, we explain how and when you can use Python for advanced analytics when building data pipelines and flows within the Crosser Solution. And how you can save time and effort by simplifying data preparation, testing, deployment, and scaling of your advanced analytics projects.

There are currently over 200 modules in the Crosser Module Library that you can combine with your Python functionality. There are also options available for other language preferences when creating data pipelines and flows, such as C# and JavaScript.

Hybrid runtime and custom code modules

Crosser Node, the hybrid and generic runtime, is where your data pipeline or flow is executed. The Crosser Node is built in .NET and execute code as following:

• C# - The native language in Crosser the environment. Your code is compiled at runtime and executed inside .NET.
• Python - 2 options available:
• Python Bridge - Uses a standard Python 3.9 environment that is included in the Crosser Node container, but runs outside .NET. Your Python code is executed in this external environment, but data to and from the Python code comes through the module inside the flow. You can install any third-party library into the Python environment through configuration of the module in the Flow.
• IronPython - Uses the IronPython interpreter that runs inside .NET. Supports Python 2.7 and can operate on .NET data types without translation. A subset of the standard libraries are available, but no additional libraries can be installed by users.
• Javascript - Javascript interpreter running inside .NET. Supports most of the features of the latest ECMAScript standards. External libraries not supported.

Standard Python Support in the Crosser Platform:

One of the use cases for Python in the Crosser Platform is analyzing machine data, e.g. from an OPC UA server, with a machine learning (ML) model. Since many of today’s ML models are developed using one of the Python ML frameworks, like TensorFlow and PyTorch, it’s easy to use the same framework in Crosser and load your trained model for inference. The machine data can be transformed using standard modules to align the data with the requirements of your machine learning models. The result of the ML analysis can then be delivered to the desired destinations using other standard modules.

Common scenarios where machine data can be utilized include process optimization for energy consumption or chemical usage, and anomaly detection to trigger actions when specific events occur.

How does this work in practice?

1. 1. Design: Add designated code modules for executing your code with no limitations:
      • Standard environment with no limitations on the type of code that can be executed
      • Data in and out must be JSON serializable (if using the PythonBridge module)
   2. Design: Add designated libraries and resources:
      • Installed when flows are deployed (using a standard pip tool)
      • Support for custom repositories, full flexibility to install any needed libraries
   3. Design: Add designated connectors and pre-processing modules for transforming data accordingly.
   4. Test: Interactive testing with real data in the Flow Studio design tool.
   5. FlowCode Sync Deployment:
      • Select node(s) and deploy
      • Upload Python code and additional files (ex. ML models) as resources. Deployed dynamically together with the flow.
      • Mass-deploy to any number of nodes if you want to replicate the same use case over multiple instances
   6. Management: Use Flow versioning to manage updates.

Best Practices

• Use standard modules to communicate with external systems and prepare data. There is an extensive library of connectors for different systems available in the Crosser Platform. 
• Separate data ingest and communication from algorithm (ML) code.
• Build a thin ‘Crosser specific’ layer and call reusable functions (in resource files), then you don’t need to create special versions of your code to run in the Crosser environment.
• Test and verify the algorithm code outside Crosser.

Challenges and Use Case Examples:

One of the main challenges is to get your data scientists to understand streaming data. In Crosser you don’t work with stored data, you work with streaming data. Your code will receive one sample at a time. If your model is not designed for operating on time series data you may need to create a window of samples to allow your model to treat each update as an independent sample.

Use case examples for Python implementation include:

• Process message data with custom algorithms and ML. You run an advanced algorithm that the standard modules don’t support and you want to use Python code.
• Implement functionality missing in the standard library. Add any type of functionality that you need for a specific case.
• Deploy distributed Python code
  When you already have some kind of Python code running and distributed in multiple locations but is complex to manage. You can use the Crosser tool as a way to distribute the Python code.
• Build custom connectors
  Use Python to add connectors to other systems not supported by the standard library. 
• Build custom data simulators
  Build simulators to create more complex data to be used for testing.

Alternatives to Code Modules:

There are some limitations to consider when building flows with custom code modules in the Crosser Platform. These include the absence of a configurable UI, challenges in reusing modules, the module not appearing in the library for future flows, and potential performance limitations. 

If your project needs some or all of these features, there are alternative options, such as using the open SDK, which allows you to build modules and connectors with a configurable UI and relevant settings that can be included in the library for easier reuse.

Conclusion:

Python's versatility and popularity as a programming language make it a powerful tool for implementing and scaling industrial IoT projects in the Crosser Platform. With support for various Python options, extensive module library, and flexibility in custom code implementation, Crosser provides a robust environment for leveraging Python in data pipelines and advanced analytics projects for IoT applications. Following best practices and being aware of potential challenges and alternatives, can help you ensure successful implementation and scalability of Python projects in the Crosser Platform.

You can learn more about the Crosser Module Library here → 

Watch the webinar with Crosser’s CTO Dr. Göran Appelquist or Schedule a Demo with a Crosser Expert to learn more.