How to remove duplicate phrases in Python?

Question

Suppose I have a string such as

'I hate *some* kinds of duplicate. This string has a duplicate phrase, duplicate phrase.'

I want to remove the second occurrence of duplicate phrase without removing other occurrences of its constituent parts, such as the other use of duplicate.

Moreover, I need to remove all potential duplicate phrases, not just the duplicates of some specific phrase that I know in advance.

I have found several posts on similar problems, but none that have helped me solve my particular issue:

• Removing duplicate words
• Removing duplicate lines
• Removing repeated words within a phrase

I had hoped to adapt the approach from the last link there (re.sub(r'\b(.+)(\s+\1\b)+', r'\1', s)) for my purposes, but could not figure out how to do so.

How do I remove all arbitrary duplicate phrases of two or more words from a string in Python?

Answer 1

Thanks everyone for your attempts and comments. I have finally found a solution:

s = 'I hate *some* kinds of duplicate. This string has a duplicate phrase, duplicate phrase.'
re.sub(r'((\b\w+\b.{1,2}\w+\b)+).+\1', r'\1', s, flags = re.I)
# 'I hate *some* kinds of duplicate. This string has a duplicate phrase.'

Explanation

The regular expression

r'((\b\w+\b.{1,2}\w+\b)+).+\1'

finds every occurrence of multiple runs of alphanumeric characters separated by one or two [any character] (to cover the case where words are separated not just by a space, but perhaps a period or comma and a space), and then repeated following some run of [any character] of indeterminate length. Then

re.sub(r'((\b\w+\b.{1,2}\w+\b)+).+\1', r'\1', s, flags = re.I)

replaces such occurrences with the first multiple run of alphanumeric characters separated by one or two [any character], being sure to ignore case (since the duplicate phrase could sometimes occur at the beginning of a sentence).