Search Crosser Knowledge Base

Best Practice File Upload

We see many use cases where a large volume of data is to be stored in cloud storages with focus on cost efficiency rather than latency. In these cases it is usually more efficient to upload files directly into a Blob Storage rather than streaming it to a cloud service and creating files from that stream which are then stored in the Blob. To come up with the most stable approach we recommend to separate data acquisition from data upload.

The example below shows a setup where we create parquet files with data from on-prem systems, store those files on the local filesystem and upload them with a different flow to Azure. Once the files are uploaded they are deleted from the local filesystem.

Flow 1: Data to parquet file

Crosser flow example - Data to parquet file

Crosser Example Flow

Flow 2: File upload

Crosser flow example - File upload

 

Crosser Example Flow

With this concept you can upload multiple files in one go and also make sure that no files will be left. Even if the upload fails, the process will try to upload the files in the next iteration.

About the author

David Nienhaus | Senior Solution Engineer

David is a Senior Solution Engineer at Crosser. He has over 10 years experience working with software integration and digitization projects for critical infrastructure.
His engineering background gives him the understanding and focus needed to solve customer use cases in the most efficient and successful way.