Remove Duplicate Line in Vim?

Question

I'm trying to use VIM to remove a duplicate line in an XML file I created. (I can't recreate the file because the ID numbers will change.)

The file looks something like this:

    <tag k="natural" v="water"/>
    <tag k="nhd:fcode" v="39004"/>
    <tag k="natural" v="water"/>

I'm trying to remove one of the duplicate k="natural" v="water" lines. When I try to use the \_ modifier to include newlines in my regex replaces, VIM doesn't seem to find anything.

Any tips on what regex or tool to use?

Answer 1

First of all, you can use awk to remove all duplicate lines, keeping their order.

:%!awk '\!_[$0]++'

If you not sure if there are some other duplicate lines you don't want remove, then just add conditions.

:%!awk '\!(_[$0]++ && /tag/ && /natural/ && /water/)'

But, parsing a nested structure like xml with regex is a bad idea, IMHO. You are going to care them not to be screwed up all the time. xmllint gives you a list of specific elements in the file:

:!echo "cat //tag[@k='natural' and @v='water']" | xmllint --shell %

You can slash duplicate lines step by step.