Filter a large file based on a specific column in bash

Question

I have a file with >100m lines and wanted to filter it by some criteria:

$ wc -l s1bam.bed 
104797540 s1bam.bed

$ head -n 3 s1bam.bed 
chr1    14688   14979   NB501800:50:H3NW5BGX3:2:22310:19560:5036/1  3   +
chr1    14688   14979   NB501800:50:H3NW5BGX3:3:13501:3458:17919/1  3   +
chr1    14727   15018   NB501800:50:H3NW5BGX3:2:22310:19560:5036/2  3   -

What I wanted to output is the fifth field value >=20 and !=255 (e.g. the second field from right). How to realize it in Bash?

Answer 1

You can use awk for this:

awk '$5 >= 20 && $5 != 255' file