How to read and manipulate a Json file with Apache beam in Python

Question

I have a .txt file which has the JSON format. I want to read, manipulate and restructure the file (change the fields name...) How Can I do this in Python with Apache Beam?

Answer 1

To be able to read a Json File with Apache Beam on Python, you can make a Custom Coder:

CF : https://beam.apache.org/documentation/programming-guide/#specifying-coders

class JsonCoder(object):
"""A JSON coder interpreting each line as a JSON string."""

def encode(self, x):
    return json.dumps(x)

def decode(self, x):
    return json.loads(x)

And then you have to specify it when you read or write your data, for instance :

lines = p | 'read_data' >> ReadFromText(known_args.input, coder=JsonCoder())

Best regards, work well ;)