RewriteCond in .htaccess with negated regex condition doesn't work?

Question

I'm trying to prevent, in this case WordPress, from rewriting certain URLs. In this case I'm trying to prevent it from ever handling a request in the uploads directory, and instead leave those to the server's 404 page. So I'm assuming it's as simple as adding the rule:

RewriteCond %{REQUEST_URI} !^/wp-content/uploads/

This rule should evaluate to false and make the chain of rules fail for those requests, thus stopping the rewrite. But no... Perhaps I need to match the cover the full string in my expression?

RewriteCond %{REQUEST_URI} !^/wp-content/uploads/.*$

Nope, that's not it either. So after scratching my head I do a check of sanity. Perhaps something is wrong with the actual pattern. So I make a simple test case.

RewriteCond %{REQUEST_URI} ^/xyz/$

In this case, the rewrite happens if and only if the requested URL is /xyz/ and shows the server's 404 page for any other page. This is exactly what I expected. So I'll just stick in a ! to negate that pattern.

RewriteCond %{REQUEST_URI} !^/xyz/$

Now I'm expecting to see the exact opposite of the above condition. The rewrite should not happen for /xyz/ but for every other possible URL. Instead, the rewrite happens for every URL, both /xyz/ and others.

So, either the use of negated regexes in RewriteConds is broken in Apache, or there's something fundamental I don't understand about it. Which one is it?

The server is Apache2.

The file in its entirety:

<IfModule mod_rewrite.c>
RewriteEngine On

RewriteBase /
RewriteRule ^index\.php$ - [L]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !^/wp-content/uploads/
RewriteRule . /index.php [L]
</IfModule>

WordPress's default file plus my rule.

Answer 1

<IfModule mod_rewrite.c>
RewriteEngine On

RewriteBase /
RewriteRule ^index\.php$ - [L]

RewriteCond %{REQUEST_URI} !^/wp-content/uploads/ [OR]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule . /index.php [L]
</IfModule>