How can I mark/highlight duplicate lines in VI editor?

Question

How would you go about marking all of the lines in a buffer that are exact duplicates of other lines? By marking them, I mean highlighting them or adding a character or something. I want to retain the order of the lines in the buffer.

Before:

foo bar foo baz 

After:

foo* bar foo* baz 

Answer 1

As an ex one-liner:

:syn clear Repeat | g/^\(.*\)\n\ze\%(.*\n\)*\1$/exe 'syn match Repeat "^' . escape(getline('.'), '".\^$*[]') . '$"' | nohlsearch 

This uses the Repeat group to highlight the repeated lines.

Breaking it down:

• syn clear Repeat :: remove any previously found repeats
• g/^\(.*\)\n\ze\%(.*\n\)*\1$/ :: for any line that is repeated later in the file
  • the regex
    • ^\(.*\)\n :: a full line
    • \ze :: end of match - verify the rest of the pattern, but don't consume the matched text (positive lookahead)
    • \%(.*\n\)* :: any number of full lines
    • \1$ :: a full line repeat of the matched full line
  • exe 'syn match Repeat "^' . escape(getline('.'), '".\^$*[]') . '$"' :: add full lines that match this to the Repeat syntax group
    • exe :: execute the given string as an ex command
    • getline('.') :: the contents of the current line matched by g//
    • escape(..., '".\^$*[]') :: escape the given characters with backslashes to make a legit regex
    • syn match Repeat "^...$" :: add the given string to the Repeat syntax group
• nohlsearch :: remove highlighting from the search done for g//

Justin's non-regex method is probably faster:

function! HighlightRepeats() range   let lineCounts = {}   let lineNum = a:firstline   while lineNum <= a:lastline     let lineText = getline(lineNum)     if lineText != ""       let lineCounts[lineText] = (has_key(lineCounts, lineText) ? lineCounts[lineText] : 0) + 1     endif     let lineNum = lineNum + 1   endwhile   exe 'syn clear Repeat'   for lineText in keys(lineCounts)     if lineCounts[lineText] >= 2       exe 'syn match Repeat "^' . escape(lineText, '".\^$*[]') . '$"'     endif   endfor endfunction  command! -range=% HighlightRepeats <line1>,<line2>call HighlightRepeats() 

Answer 2

None of the answers above worked for me so this is what I do:

1. Sort the file using :sort
2. Execute command :g/^\(.*\)$\n\1$/p