convert large binary file to hexadecimal (binascii.hexlify returns MemoryError)

Question

I am trying to convert a large file (~1GB) into hexadecimal string with binascii (which works great on smaller files), but it causing a MemoryError.

this is the code I am using:

import binascii

filePath = "demo/11.mp4.zip"

file = open(filePath, "rb")
with file:
    byte = file.read()
    hexa = binascii.hexlify(byte)

hexa_string = hexa.decode('ascii');

any advice?

Answer 1

Read your file in chucks:

import binascii

file_path = 'demo/11.mp4.zip'

chunk_size = 1024
with open(file_path, 'rb') as f:
    while True:
        data = f.read(chunk_size)
        if not data:
            break
        hexa = binascii.hexlify(data)
        hexa_string = hexa.decode('ascii')
        # work with hex string

Answer 2

Actually You are approximately using 3 GB of your memory with storing the file and its hex within byte and hexa variables, instead of that you can iterate within your file and process the it based on chunks .