BASH: How to remove all files except those named in a manifest?

Question

I have a manifest file which is just a list of newline separated filenames. How can I remove all files that are not named in the manifest from a folder?

I've tried to build a find ./ ! -name "filename" command dynamically:

command="find ./ ! -name \"MANIFEST\" "
for line in `cat MANIFEST`; do
    command=${command}"! -name \"${line}\" " 
done
command=${command} -exec echo {} \;
$command

But the files remain.

[Note:] I know this uses echo. I want to check what my command does before using it.

Solution:(thanks PixelBeat)

ls -1 > ALLFILES
sort MANIFEST MANIFEST ALLFILES | uniq -u | xargs rm

Without temp file:

ls -1 | sort MANIFEST MANIFEST - | uniq -u | xargs rm

Both Ignores whether the files are sorted/not.

Answer 1

For each file in current directory grep filename in MANIFEST file and rm file if not matched.

for file in *
  do grep -q -F "$file" PATH_TO_YOUR_MANIFIST ||  rm "$file" 
done

Answer 2

Using the "set difference" pattern from http://www.pixelbeat.org/cmdline.html#sets

(find ./ -type f -printf "%P\n"; cat MANIFEST MANIFEST; echo MANIFEST) |
  sort | uniq -u | xargs -r rm

Note I list MANIFEST twice in case there are files listed there that are not actually present. Also note the above supports files in subdirectories