Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Publish non-string message in Cloud PubSub

Almost all of the examples of pub/sub available over web uses String messages.

How do I publish message other than text messages like Java Object, JSON or AVRO to topic and then consume it through subscription.

Many Thanks Pari

like image 311
Pari Avatar asked Jul 03 '18 07:07

Pari


3 Answers

You cannot really do that directly, as you can see here and here the published message has to be a bytestring. What you can do instead is load your file, encode it to utf-8 and then publish it. Afterwards, when pulling, you can dump the message data to a file. For instance, using python for json, you would need to publish like so:

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project, topic_name)

with open(json_path) as f:
    data = str(json.load(f))

data = data.encode('utf-8')
publisher.publish(topic_path, data=data)

Then you can pull and dump like so:

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
    project, subscription_name)

def callback(message):
    with open('received_message.json', 'w') as outfile:
        json.dump(message.data, outfile)

subscriber.subscribe(subscription_path, callback=callback)
like image 160
Lefteris S Avatar answered Oct 17 '22 06:10

Lefteris S


publish JSON with Node - publishMessage:

const {PubSub} = require('@google-cloud/pubsub')
const pubsub = new PubSub()

const json = {
  foo: 'bar'
}

await pubsub.topic('my-topic').publishMessage({json})
like image 30
vitaliytv Avatar answered Oct 17 '22 05:10

vitaliytv


The following code lets you publish a message in Pub/Sub using JSON:

topic_path = 'your-topic-id'

publisher = pubsub_v1.PublisherClient()

record = {
    'Key1': 'Value1',
    'Key2': 'Value2',
    'Key3': 'Value3',
    'Key4': 'Value4'
}

data = json.dumps(record).encode("utf-8")
future = publisher.publish(topic_path, data)
print(f'published message id {future.result()}')

However, you must encode the JSON to UTF-8.

I hope that It helps you.

like image 1
Javier Muñoz Avatar answered Oct 17 '22 06:10

Javier Muñoz