How to cache file handles?

Question

I have an application that wants to keep open many files: periodically it receives a client request saying "add some data to file X", and it would be ideal to have that file already opened, and the file's header section already parsed, so that this write is fast. However, keeping open this many files is not very nice to the operating system, and could become impossible if our data-storage needs grow.

So I would like a "give me this file handle, opening if it's not cached" function, and some process for automatically closing files which have not been written to for, say, five minutes. For the specific case of caching file handles which are written to in short spurts, this is probably enough, but this seems a general enough problem that there ought to be functions like "give me the object named X, from cache if possible" and "I'm now done with object X, so make it eligible for eviction five minutes from now".

core.cache looks like it might be suitable for this purpose, but the documentation is quite lacking and the source provides no particular clues about how to use it. Its TTLCache looks promising, but as well as being unclear how to use it relies on garbage collection to evict items, so I can't cleanly close a resource when I'm ready to expire it.

I could roll my own, of course, but there are a number of tricky spots and I'm sure I would get some things subtly wrong, so I'm hoping someone out there can point me to an implementation of this functionality. My code is in clojure, but of course using a java library would be perfectly fine if that's where the best implementation can be found.

Answer 1

Check out Guava's cache implementation.

• You can supply a Callable (or a CacheLoader) to the get method for "if handle is cached, return it, otherwise open, cache and return it" semantics
• You can configure timed eviction such as expireAfterAccess
• You can register a RemovalListener to close the handles on removal

Modifying the code examples from the linked Guava page slightly, using CacheLoader:

LoadingCache<Key, Handle> graphs = CacheBuilder.newBuilder()
   .maximumSize(100) // sensible value for open handles?
   .expireAfterAccess(5, TimeUnit.MINUTES)
   .removalListener(removalListener)
   .build(
       new CacheLoader<Key, Handle>() {
         public Handle load(Key key) throws AnyException {
           return openHandle(key);
         }
       });

RemovalListener<Key, Handle> removalListener = 
  new RemovalListener<Key, Handle>() {
    public void onRemoval(RemovalNotification<Key, Handle> removal) {
      Handle h = removal.getValue();
      h.close(); // tear down properly
    }
  };

* DISCLAIMER * I have not used the cache myself this way, ensure you test this sensibly.

Answer 2

If you don't mind some java, see http://ehcache.org/ and http://ehcache.org/apidocs/net/sf/ehcache/event/CacheEventListener.html.