Python: something faster than not in for large lists?

Question

I'm doing a project with word lists. I want to combine two word lists, but only store the unique words.

I'm reading the words from a file and it seems to take a long time to read the file and store it as a list. I intend to copy the same block of code and run it using the second (or any subsequent) word files. The slow part of the code looks like this:

    while inLine!= "":
        inLine = inLine.strip()
        if inLine not in inList:
            inList.append(inLine)
        inLine = inFile.readline()

Please correct me if I'm wrong, but I think the slow(est) part of the program is the "not in" comparison. What are ways I can rewrite this to make it faster?

Answer 1

Judging by this line:

if inLine not in inList:
    inList.append(inLine)

It looks like you are enforcing uniqueness in the inList container. You should consider to use a more efficient data structure, such as an inSet set. Then the not in check can be discarded as redundant, because duplicates will be prevented by the container anyway.

If insertion ordering must be preserved, then you can achieve a similar result by using an OrderedDict with null values.