Publish non-string message in Cloud PubSub

Question

Almost all of the examples of pub/sub available over web uses String messages.

How do I publish message other than text messages like Java Object, JSON or AVRO to topic and then consume it through subscription.

Many Thanks Pari

Answer 1

You cannot really do that directly, as you can see here and here the published message has to be a bytestring. What you can do instead is load your file, encode it to utf-8 and then publish it. Afterwards, when pulling, you can dump the message data to a file. For instance, using python for json, you would need to publish like so:

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project, topic_name)

with open(json_path) as f:
    data = str(json.load(f))

data = data.encode('utf-8')
publisher.publish(topic_path, data=data)

Then you can pull and dump like so:

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    project, subscription_name)

def callback(message):
    with open('received_message.json', 'w') as outfile:
        json.dump(message.data, outfile)

subscriber.subscribe(subscription_path, callback=callback)

Answer 2

publish JSON with Node - publishMessage:

const {PubSub} = require('@google-cloud/pubsub')
const pubsub = new PubSub()

const json = {
  foo: 'bar'
}

await pubsub.topic('my-topic').publishMessage({json})

Answer 3

The following code lets you publish a message in Pub/Sub using JSON:

topic_path = 'your-topic-id'

publisher = pubsub_v1.PublisherClient()

record = {
    'Key1': 'Value1',
    'Key2': 'Value2',
    'Key3': 'Value3',
    'Key4': 'Value4'
}

data = json.dumps(record).encode("utf-8")
future = publisher.publish(topic_path, data)
print(f'published message id {future.result()}')

However, you must encode the JSON to UTF-8.

I hope that It helps you.