Using Windows/DOS shell/batch commands, how do I take a file and only keep unique lines?

Question

Say I have a file like:

apple
pear
lemon
lemon
pear
orange
lemon

How do I make it so that I only keep the unique lines, so I get:

apple
pear
lemon
orange

I can either modify the original file or create a new one.

I'm thinking there's a way to scan the original file a line at a time, check whether or not the line exists in the new file, and then append if it doesn't. I'm not dealing with really large files here.

Answer 1

@echo off
setlocal disabledelayedexpansion
set "prev="
for /f "delims=" %%F in ('sort uniqinput.txt') do (
  set "curr=%%F"
  setlocal enabledelayedexpansion
  if "!prev!" neq "!curr!" echo !curr!
  endlocal
  set "prev=%%F"
)

What it does: sorts the input first, and then goes though it sequentially and outputs only if current line is different to previous one. It could have been even simpler if not for need to handle special characters (that's why those setlocal/endlocal are for).
It just echoes lines to stdout, if you want to write to file do (assuming you named your batch myUniq.bat) myUniq >>output.txt

Answer 2

Run PowerShell from the command prompt.

Assuming the items are in a file call fruits.txt, the following will put the unique lines in uniques.txt:

type fruits.txt | Sort-Object -unique | Out-File uniques.txt

Answer 3

In Windows 10 sort.exe has a hidden flag called /unique that you can use

C:\Users>sort fruits.txt
apple
lemon
lemon
lemon
orange
pear
pear

C:\Users>sort /unique fruits.txt
apple
lemon
orange
pear