PubSub For DevOps¶

Overview¶

Send messages metadata of job to BigQuery by PubSub

Deployment¶

The deployment has come through:

• [1] Create table to pull data to from PubSub

• [2] Create PubSub topic: basement_metadata

• [3] Create subcription from topic job-metadata

• [4] Using Python to publish to the topic.

Data Format¶

Overview¶

Our team use various of tools to transition data from place to place, so it should

• Standard used for Cloud Run / Functions to not if/else based on data-types

• Easy to remember the way to interactive between services

Components¶

For serverless oschestration, using likely Pubsub format¶

From ¹, the description of PubSub message format is

A message consists of fields with the message data and metadata.

Specify at least one of the following in the message:

• The message data

• An ordering key

• Attributes with additional metadata

If you're using the REST API, the message data must be base64-encoded.

The Pub/Sub service adds the following fields to the message:

A message ID unique to the topic

A timestamp for when the Pub/Sub service receives the message

Default extractor from message, ref from ²

envelope = json.loads(request.data.decode('utf-8'))
payload = base64.b64decode(envelope['message']['data'])

From Cloud Storage¶

Reference¶

[^4] What's pubsub? https://cloud.google.com/storage/docs/pubsub-notifications