SOLR slave is doing full Copy as it was not able to Delete unused Index dir

Question

When a slave replicates the index it creates a index directory with name format as index.timestamp next time when it replicates it tries to cleanup and create a new folder with new time stamp but in my case its not happening and everytime i see this warning and Slave trigger a fullCopy

Unable to cleanup unused lucene index files so we must do a full copy instead

Could anyone why slave is not able to clean up the unused index files.

thanks

Answer 1

It is very difficult to answer without knowing your solr version.

From my experience, solr copies index files that is different (newer then timestamp). However, if your indexes are merged, then it will trigger full copy.

Here is some related tickets

https://issues.apache.org/jira/browse/SOLR-6640

** update 11/27/2017 **

this is a relevant part in 5x branch,

https://github.com/apache/lucene-solr/blob/branch_5x/solr/core/src/java/org/apache/solr/handler/IndexFetcher.java#L398-L418

try {
    IndexWriter indexWriter = writer.get();
    int c = 0;
    indexWriter.deleteUnusedFiles();
    while (hasUnusedFiles(indexDir, commit)) {
        indexWriter.deleteUnusedFiles();
        LOG.info("Sleeping for 1000ms to wait for unused lucene index files to be delete-able");
        Thread.sleep(1000);
        c++;
        if (c >= 30) {
            LOG.warn("IndexFetcher unable to cleanup unused lucene index files so we must do a full copy instead");
            isFullCopyNeeded = true;
            break;
        }
    }
    if (c > 0)  {
        LOG.info("IndexFetcher slept for " + (c * 1000) + "ms for unused lucene index files to be delete-able");
    }
} finally {
    writer.decref();
}

So, you have more than 30 unused index files and this will trigger the warning message & full copy. I would try to optimize or merge indexes from the master and see whether fullCopy is replicated.