This question has a great answer saying you can use awk '!seen[$0]++' file.txt
to delete non-consecutive duplicate lines from a file. How can I delete non-consecutive duplicate lines from a file only if they match a pattern? e.g. only if they contain the string "#####"
Example input
deleteme.txt ##########
1219: 'PCM BE PTP'
deleteme.txt ##########
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1222: , 'PCM BE PTP UT'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1223: , 'PCM BE PTP'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1225: , 'PCM FE/MID PTP'
Desired output
deleteme.txt ##########
1219: 'PCM BE PTP'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1222: , 'PCM BE PTP UT'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
1223: , 'PCM BE PTP'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
1225: , 'PCM FE/MID PTP'
You may use
awk '!/#####/ || !seen[$0]++'
Or, as Ed Morton suggests, a synonymical
awk '!(/#####/ && seen[$0]++)'
Here, !seen[$0]++
does the same thing as usual, it will remove any duplicated line. The !/#####/
part matches lines that contain a #####
pattern and negates the match. The two patterns combined with ||
will remove all duplicate lines having #####
pattern inside them.
See an online awk
demo:
s="deleteme.txt ##########
1219: 'PCM BE PTP'
deleteme.txt ##########
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1222: , 'PCM BE PTP UT'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1223 #####: , 'PCM BE PTP'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1225: , 'PCM FE/MID PTP'"
awk '!/#####/ || !seen[$0]++' <<< "$s"
Output:
deleteme.txt ##########
1219: 'PCM BE PTP'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
deleteme2.txt ##########
1222: , 'PCM BE PTP UT'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
1223 #####: , 'PCM BE PTP'
1221: , 'PCM FE/MID PTP UT','PCM IA 1 PTP'
1225: , 'PCM FE/MID PTP'
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With