Delete duplicate string with PowerShell

Question

I have got text file:

1 2 4 5 6 7
1 3 5 6 7 8
1 2 3 4 5 6
1 2 4 5 6 7

Here first and last line are simmilar. I have a lot of files that have double lines. I need to delete all dublicate.

Answer 1

All these seem really complicated. It is as simple as:

gc $filename | sort | get-unique > $output

Using actual file names instead of variables:

gc test.txt| sort | get-unique > unique.txt

Answer 2

To get unique lines:

PS > Get-Content test.txt | Select-Object -Unique
1 2 4 5 6 7
1 3 5 6 7 8
1 2 3 4 5 6

To remove the duplicate

PS >  Get-Content test.txt | group -noelement | `
      where {$_.count -eq 1} | select -expand name

1 3 5 6 7 8
1 2 3 4 5 6

Answer 3

If order is not important:

Get-Content test.txt | Sort-Object -Unique | Set-Content test-1.txt

If order is important:

$set = @{}
Get-Content test.txt | %{
    if (!$set.Contains($_)) {
        $set.Add($_, $null)
        $_
    }
} | Set-Content test-2.txt