Python: How do I go through multiple lines in a file at once?

Question

I have created a class which loops through a file and after checking if a line is valid, it'll write that line to another file. Every line it checks is a lengthy process making it very slow. I need to implement either threading/multiprocessing at the process_file function; I do not know which library is best suited for speeding this function up or how to implement it.

class FileProcessor:
    def process_file(self):
        with open('file.txt', 'r') as f:
            with open('outfile.txt', 'w') as output:
                for line in f:
                    # There's some string manipulation code here...
                    validate = FileProcessor.do_stuff(self, line)
                    # If true write line to output.txt
    def do_stuff(self, line)
        # Does stuff...
        pass 

Extra Information: The code goes through a proxy list checking whether it is online. This is a lengthy and time consuming process.

Thank you for any insight or help!

Answer 1

The code goes through a proxy list checking whether it is online

It sounds like what takes a long time is connecting to the internet, meaning your task is IO bound and thus threads can help speed it up. Multiple processes are always applicable but can be harder to use.