How to write a program in python that will process a text stream?

Question

I am sorry if this is a repeat question. How do I write a python script to process data as a stream of line? I need to do this because the files that I am processing are huge, and I would rather not read the file into the memory.

I know that you can potentially read one line of the file at a time, but I want something that will process a text stream.

Answer 1

You could just read the data from stdin, as described in this answer. This would look like that in code:

for line in sys.stdin:
    # do suff

If you want to process a file, then just call the script like this (on Unix platforms):

cat file.txt | python script.py

You can of course pipe the output of any other program in there too.