how to delete the repeat lines in emacs

Question

I have a text with a lots of lines, my question is how to delete the repeat lines in emacs? using the command in emacs or elisp packages without external utils.

for example:

this is line a this is line b this is line a 

to remove the 3rd line (same as 1st line)

this is line a this is line b 

Answer 1

If you have Emacs 24.4 or newer, the cleanest way to do it would be the new delete-duplicate-lines function. Note that

• this works on a region, not a buffer, so select the desired text first
• it maintains the relative order of the originals, killing the duplicates

For example, if your input is

test dup dup one two one three one test five 

M-x delete-duplicate-lines would make it

test dup one two three five 

You've the option of searching from backwards by prefixing it with the universal argument (C-u). The result would then be

dup two three one test five 

Credit goes to emacsredux.com.

Other roundabout options, not giving quite the same result, available via Eshell:

1. sort -u; doesn't maintain the relative order of the originals
2. uniq; worse it needs its input to be sorted

Answer 2

Put this code to your .emacs:

(defun uniq-lines (beg end)   "Unique lines in region. Called from a program, there are two arguments: BEG and END (region to sort)."   (interactive "r")   (save-excursion     (save-restriction       (narrow-to-region beg end)       (goto-char (point-min))       (while (not (eobp))         (kill-line 1)         (yank)         (let ((next-line (point)))           (while               (re-search-forward                (format "^%s" (regexp-quote (car kill-ring))) nil t)             (replace-match "" nil nil))           (goto-char next-line)))))) 

Usage:

M-x uniq-lines