Python edit distance

Question

I am a molecular biologist using Biopython to analyze mutations in genes and my problem is this:

I have a file containing many different sequences (millions), most of which are duplicates. I need to find the duplicates and discard them, keeping one copy of each unique sequence. I was planning on using the module editdist to calculate the edit distance between them all to determine which ones the duplicates are, but editdist can only work with 2 strings, not files.

Anyone know how I can use that module with files instead of strings?

Answer 1

Assuming your file consists solely of sequences arranged one sequence per line, I would suggest the following:

seq_file = open(#your file)

sequences = [seq for seq in seq_file]

uniques = list(set(sequences))

Assuming you have the memory for it. How many millions?

ETA:

Was reading the comments above (but don't have comment privs) - assuming the sequence IDs are the same for any duplicates, this will work. If duplicate sequences can different sequence IDs, then would to know which comes first and what is between them in the file.