Purging nginx cache files does not always work

Question

I run an nginx server + PHP webservices API. I use nginx's fastcgi_cache to cache all GET requests, and when certain resources are updated, I purge one or more related cached resources.

The method I'm using to do this is to calculate the nginx cache file name for each resource I want to purge, and then deleting that file. For the most part, this works well.

However, I've found that sometimes, even after the cache file is deleted, nginx will still return data from cache.

This is not a problem with selecting the correct cache file to delete -- as part of my testing, I've deleted the entire cache directory, and nginx still returns HIT responses

Is anyone aware of why this might be happening? Is it possible that another cache is involved? E.g., could it be that the OS is returning a cached version of the cache file to nginx, so nginx is not aware that it's been deleted?

I'm running this on CentOS, and with this nginx config (minus irrelevant parts):

user nginx;

# Let nginx figure out the best value
worker_processes auto;

events {
    worker_connections  10240;
    multi_accept        on;
    use                 epoll;
}

# Maximum number of open files should be at least worker_connections * 2
worker_rlimit_nofile 40960;

# Enable regex JIT compiler
pcre_jit on;

http {

    # TCP optimisation
    sendfile        on;
    tcp_nodelay     on;
    tcp_nopush      on;

    # Configure keep alive
    keepalive_requests  1000;
    keepalive_timeout   120s 120s;

    # Configure SPDY
    spdy_headers_comp 2;

    # Configure global PHP cache
    fastcgi_cache_path /var/nginx/cache levels=1:2 keys_zone=xxx:100m inactive=24h;

    # Enable open file caching
    open_file_cache max=10000 inactive=120s;
    open_file_cache_valid 120s;
    open_file_cache_min_uses 5;
    open_file_cache_errors off;

    server {
        server_name xxx;
        listen 8080;

        # Send all dynamic content requests to the main app handler
        if (!-f $document_root$uri) {

            rewrite ^/(.+)    /index.php/$1         last;
            rewrite ^/        /index.php            last;
        }

        # Proxy PHP requests to php-fpm
        location ~ [^/]\.php(/|$) {

            # Enable caching
            fastcgi_cache xxx;

            # Only cache GET and HEAD responses
            fastcgi_cache_methods GET HEAD;

            # Caching is off by default, an can only be enabled using Cache-Control response headers
            fastcgi_cache_valid 0;

            # Allow only one identical request to be forwarded (others will get a stale response)
            fastcgi_cache_lock on;

            # Define conditions for which stale content will be returned
            fastcgi_cache_use_stale error timeout invalid_header updating http_500 http_503;

            # Define cache key to uniquely identify cached objects
            fastcgi_cache_key "$scheme$request_method$host$request_uri";

            # Add a header to response to indicate cache results
            add_header X-Cache $upstream_cache_status;

            # Configure standard server parameters
            fastcgi_split_path_info ^(.+\.php)(/.+)$;
            include fastcgi_params;

            # php-fpm config
            fastcgi_param SCRIPT_URL        $fastcgi_path_info;
            fastcgi_param PATH_INFO         $fastcgi_path_info;
            fastcgi_param SCRIPT_FILENAME   $document_root$fastcgi_script_name;
            fastcgi_param REQUEST_SCHEME    $scheme;
            fastcgi_param REMOTE_USER       $remote_user;

            # Read buffer sizes
            fastcgi_buffer_size 128k;
            fastcgi_buffers 256 16k;
            fastcgi_busy_buffers_size 256k;
            fastcgi_temp_file_write_size 256k;

            # Keep connection open to enable keep-alive
            fastcgi_keep_conn on;

            # Proxy to PHP
            fastcgi_pass unix:/var/run/php-fpm/fpm.sock;
        }
    }
}

Now that I look at this, could the open_file_cache parameters be affecting the cache files?

Any ideas?

Answer 1

No, the OS does not cache files.

However, the reason this might be happening is that files are not actually fully deleted until both the link count and the number of processes that have the file open both go down to zero.

The unlink(2) manual page, which documents the system call used by tools like rm, reads as follows:

The unlink() function removes the link named by path from its directory and decrements the link count of the file which was referenced by the link. If that decrement reduces the link count of the file to zero, and no process has the file open, then all resources associated with the file are reclaimed. If one or more processes have the file open when the last link is removed, the link is removed, but the removal of the file is delayed until all references to it have been closed.

Depending on the system, you can actually still recover such open files fully without any data loss, for example, see https://unix.stackexchange.com/questions/61820/how-can-i-access-a-deleted-open-file-on-linux-output-of-a-running-crontab-task.

So, indeed, open_file_cache would effectively preclude your deletion from having any effect within the processes that still have relevant file descriptors in their cache. You may want to use a shorter open_file_cache_valid if urgent purging after deletion is very important to you.