Google Cloud Storage Writer Documentation

Version: 1.0.0 

Retrieved: 2025-10-09 15:15:54

Google Cloud Storage Writer

The module uploads data to a bucket in Google Cloud Storage. Google Cloud Storage is a cloud storage service that stores objects(binary data) identified by a unique key in a bucket. To use this module an Google Cloud account is needed and a storage bucket must have been created. Even though Google Cloud stores object in a flat structure, a logical hierarchy can be inferred using key prefixes and / as a delimiter character. This module makes use of this and makes it easy to simulate a folder like structure in the bucket. To facilitate different use cases this module can operate in different modes, which are selected by the Upload Type setting.

MultipleFiles mode

The is the default mode.

In this mode multiple files are uploaded to Google Cloud for each message received by the module. When a new message is received, its value(Source Property) must contain a list of filepaths. The module then uploads these files one after another to Google Cloud. If any of the uploads fails, the module will continue to upload the remaining files, but log an error about the file that couldn't be uploaded. If Delete File(s) After Upload is set, then the successfully uploaded files will also be removed from disk. The files will be stored in the Google Cloud bucket using their filename(without directory) as the key. The Prefix setting can be used to add a common prefix to all keys of the uploaded files, to simulate a folder structure in the bucket.

SingleFile mode

In this mode a single file is uploaded to Google Cloud for each message received by the module. Works the same as Multiple Files with the difference that the Source Property must only contain a single filepath and not a list.

ByteArray mode

In this mode a byte array from the incoming message is uploaded to Google Cloud for each message received by the module. When receiving a message in this mode the Source Property should contain the actual data to be uploaded and the data should be a byte array. Since the data is not a file, the key can't be inferred from a filename and must instead be provided with the incoming message. The key to store the data at is created from combining Prefix with the value of Key Property.

String mode

In this mode a string from the incoming message is uploaded to Google Cloud for each message received by the module. Works the same as ByteArray mode except that the data must be a string.

A note on how keys are calculated

The object key(s) are calculated by combining Prefix and either a filename or the value of the Key Property depending on the upload type. The key(s) are always created to simulate a folder like structure, which means that depending on the prefix, sometimes an / is added or removed when calculating the key. The following table has some examples to show this.

Settings

Note that not all settings are used for all upload types.

Credential

This module contains an option to select credentials to use in the module. All credentials supported by the module are presented in a drop-down.

Input

Output

One output message is sent for every incoming message. Note that in MultipleFiles mode, all files will be uploaded before the message is sent from the module. The output message will retain all message properties from the incoming message. It will also add a crosser result object to the message if one does not already exist in the incoming message. If the upload succeeds then the success property of the crosser result object is set to true. If the upload fails the success property is set to false and the message property of the crosser result object is set to a message describing why the upload failed. Note that if any of the uploads fail in MultipleFiles, success is set to false, even though some files may have been successfully uploaded.

Examples

Uploading multiple files

# Settings:
Source Property = "data"
Upload Type = "MultipleFiles"
Bucket Name = "my-bucket"
Delete File(s) After Upload = false
Prefix = ""
# Incoming message
{
   "data": ["file1.txt",
            "my-folder/file2.json",
            "another_folder/that/is/nested/file-3.xml"]
}
# Outgoing message
{
   "data": ["file1.txt",
            "my-folder/file2.json",
            "another_folder/that/is/nested/file-3.xml"]
   "crosser": {
     "success": true
   }
}

Result in Google Cloud:

• Contents of file1.txt is stored in my-bucket with key file.txt.
• Contents of my-folder/file2.json is stored in my-bucket with key file2.json
• Contents of another_folder/that/is/nested/file-3.xml is stored in my-bucket with key file-3.xml.

Uploading single file

# Settings:
Source Property = "data"
Upload Type = "SingleFile"
Bucket Name = "my-bucket"
Delete File(s) After Upload = true
Prefix = "tests/storageModule/files/"
# Incoming message
{
   "data": "my-folder/single_file.xml",
}
# Outgoing message
{
   "data": ["file1.txt", "file2.json"],
   "crosser": {
     "success": true
   }
}

Result in Google Cloud:

• Contents of my-folder/single_file.xml is stored in my-bycket at key tests/storageModule/files/single_file.xml. Result on disk:
• The file my-folder/single_file.xml has been removed.

Uploading Binary Data(ByteArray)

# Settings:
Source Property = "myByteArray"
Upload Type = "ByteArray"
Bucket Name = "my-bucket"
Delete File(s) After Upload = true
Prefix = "my_arrays/"
Key = "keyProperty"
# Incoming message
{
   "myByteArray": "dGhpcyBpcyBiaW5hcnkgZGF0YSBiYXNlNjQgZW5jb2RlZAo="
}
# Outgoing message
{
   "myByteArray": "dGhpcyBpcyBiaW5hcnkgZGF0YSBiYXNlNjQgZW5jb2RlZAo="
   "keyProperty": "aArray.bin",
   "crosser": {
     "success": true
   }
}

Result in Google Cloud:

• The byteArray(shown base64 encoded) dGhpcyBpcyBiaW5hcnkgZGF0YSBiYXNlNjQgZW5jb2RlZAo= is stored in my-bucket at key my_arrays/aArray.bin

Uploading String/Text

# Settings:
Source Property = "myString"
Upload Type = "String"
Bucket Name = "my-bucket"
Delete File(s) After Upload = true
Prefix = "my_data/textfiles"
Key = "keyProperty"
# Incoming message
{
   "myString": "Lorem ipsum"
}
# Outgoing message
{
   "myString": "Lorem ipsum"
   "keyProperty": "textfile.txt",
   "crosser": {
     "success": true
   }
}

Result in Google Cloud:

• The string Lorem ipsum is stored in my-bycket at key textfiles/textfile.txt.