Creating Google Cloud Pub/Sub Input Data Source

The Google Cloud Pub/Sub connector allows connection to Google Cloud Pub/Sub’s message bus on a real-time streaming basis. Specifically, the connector allows Panopticon Streams to subscribe to XML, JSON, TEXT or FIX based messages that are published on particular topics. The data format itself is arbitrary, and consequently, the connection includes the message definition.

Steps:

1.     In the New Data Source page, select Input > Google Cloud PubSub in the Connector drop-down list.

 

2.     Enter the Service Account Credential JSON Text with the generated JSON key (contains the private key) in the following format:

NOTE	Ensure that when parameterizing the values in the Credential JSON Text, there is no white space as a single line content.

 

3.     Click  to populate the Topic drop-down list. Initially, the first topic in the list is displayed in the Topic drop-down box.

Select a topic.

4.     Click  to populate the Subscription Name drop-down list and select a subscription name.

You can also opt to create a subscription by manually entering the value into the Subscription Name list box.

 

NOTE

·         A subscription name will be automatically generated when it is not entered or selected in the drop-down list. This subscription will be created for connection and will be deleted as soon as its work is done. For example, when starting a presentation mode, a subscription will be created. Upon quitting the presentation mode, the subscription will then be deleted. ·         Pub/Sub can automatically delete inactive subscriptions. This can be done by configuring the minimum required time of inactivity to schedule a subscription for deletion. This time must be longer than the message retention duration.

 

5.     Select the Message Type.

6.     Select either the period (.) or comma (,) as the Decimal Separator.

NOTE	Prepend 'default:' for the elements falling under default namespace.

 

7.     Click  to the fetch the schema based on the connection details. Consequently, the list of columns with the data type found from inspecting the first ‘n’ rows of the input data source is populated and the Save button is enabled.

8.     You can also opt to load or save a copy of the column definition.

9.    Click . This adds columns to the Google Cloud Pub/Sub connection that will represent sections of the message.

10.   Provide the following information:

Property	Description
Name	The column name of the source schema.
Fix Tag/XPath/Json Path	The Fix Tag/XPath/Json Path of the source schema.
Type	The data type of the column. Can be a Text, Numeric, or Time
Date Format	The format when the data type is Time.
Filter	Defined parameters that can be used as filter. Only available for JSON, Text, and XML message types.
Enabled	Determines whether the message field should be processed.

 

To delete a column, check its  or all the column entries, check the topmost , then click .

11.   Google Cloud Pub/Sub messages can have additional metadata as custom attributes.

Panopticon Google Cloud Pub/Sub connector supports reading these attributes as column values. The generate column logic automatically checks and generates attribute columns if messages received contain attributes.

Additionally, like columns from message data, you can manually add them by clicking . A new entry displays.

Name can be any unique column name within the data source. The attribute name must match to an attribute name in message otherwise it will be treated as null value. Currently all attribute columns are treated as Text columns, we can't change column type.

Check the Enabled box to enable an attribute column.

To delete an attribute column, check its  or all the column entries, check the topmost , then click .

12.   If the Type of the generated columns is set to Text, they will be listed in the ID column drop-down list box and can be used to select a key column to manage data updates and inserts.

Every message definition needs a Text column to be defined as the ID column.

Furthermore, streaming time series window can be generated by creating a compound key with the ID column, plus a separately specified Time ID Column. The Time ID Column can be from the source data set, or alternatively automatically generated.

If the Time ID Column is selected, a scrolling time window can be specified.

For Automatic Time Id, define the Time Id Column Name.

As new data arrives from the subscription, new time slices will be automatically added and the old ones will be deleted.

If a new ID is received, a new row is added to the in-memory data set representing the Google Cloud Pub/Sub topic subscription. However, if an existing ID is received, an existing row is updated.

13.   Check the Reset Data on Reconnect box to flush out the stale data and reload data after reconnection.

14.   Click . The new data source is added in the Data Sources list.