Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to join two files with bash & remove duplicates?

I would like to know how to join two files on one column and remove the duplicates. Some examples first.

File1:

SERVER1; Deployed; Infrastructure
SERVER2; Deployed; Infrastructure
SERVER3; Deployed; Infrastructure
SERVER4; Deployed; Infrastructure
SERVER5; Deployed; Infrastructure

File2:

SERVER1;
SERVER2;
SERVER5;

Expectation:

SERVER3; Deployed; Infrastructure
SERVER4; Deployed; Infrastructure

Tried commands like: sort File1 File2 | uniq > File3, but it only returns me the joined output, as it does see every column as unique, output is like:

 SERVER1;
 SERVER1; Deployed; Infrastructure
 SERVER2;
 SERVER2; Deployed; Infrastructure
 SERVER3; Deployed; Infrastructure
 SERVER4; Deployed; Infrastructure
 SERVER5;
 SERVER5; Deployed; Infrastructure

Then tried to remove the duplicates from what I got above with command awk -F";" '!_[$1]++' File3, but seems like it only remove one duplicate line and left the other be:

SERVER1;
SERVER2;
SERVER3; Deployed; Infrastructure
SERVER4; Deployed; Infrastructure
SERVER5;

I would like to check on duplicates and remove both duplicate and the server itself, do you have any suggestions?

like image 305
Robba Avatar asked Dec 20 '25 17:12

Robba


1 Answers

Following awk may help you on same.

awk 'FNR==NR{a[$0];next} !($1 in a)' File2  File1
like image 57
RavinderSingh13 Avatar answered Dec 23 '25 09:12

RavinderSingh13



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!