Teamcity NuGet Repository corruption

Question

So, my google-fu is weak... I could not find another instance of my errors.

I've been having problems with my teamcity nuget repository since a day. NuGet downloads from the repo fail with unexpected EOF's or corrupt package warnings.

As far as I can tell this is not a hardware failure, the vm, and vm-host do not report disk errors.

To add insult to injury the teamcity logfile 'teamcity-javaLogging-2013-07-17.log' grows unbounded (+3GB in the time to type this, where 10 MB a day is normal) with the stacktraces like the ones below.

My Teamcity version is 7.1.5 (build 24400)

Anyone know how to recover from this failure?

I've not yet summoned up enough courage to just clear all the caches I can find on the teamcity admin page ( Administration > Diagnostics > Caches) because there is a warning in scary yellow on that page not to do that.

Below is a sample of the stacktraces I'm getting.

17-jul-2013 3:00:02 net.sf.ehcache.store.DiskStore get
SEVERE: provider-nugetCache: Could not read disk store element for key 2731. Error was unexpected EOF in middle of data block
java.io.StreamCorruptedException: unexpected EOF in middle of data block
        at java.io.ObjectInputStream$BlockDataInputStream.refill(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.read(Unknown Source)
        at java.io.DataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream.readInt(Unknown Source)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readSplitted(SerializableEntry.java:5)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.EntryImpl.readObjectInternal(EntryImpl.java:34)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readExternal(SerializableEntry.java:16)
        at java.io.ObjectInputStream.readExternalData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.readObject(Unknown Source)
        at net.sf.ehcache.store.DiskStore.loadElementFromDiskElement(DiskStore.java:313)
        at net.sf.ehcache.store.DiskStore.get(DiskStore.java:268)
        at net.sf.ehcache.Cache.searchInDiskStore(Cache.java:1290)
        at net.sf.ehcache.Cache.get(Cache.java:904)
        at net.sf.ehcache.Cache.get(Cache.java:879)
        at jetbrains.buildServer.serverSide.metadata.impl.cache.TypedCacheImpl.getValue(TypedCacheImpl.java:3)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.MetadataStorageImpl.getReportedKeys(MetadataStorageImpl.java:7)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.MetadataStorageImpl.removeBuild(MetadataStorageImpl.java:45)
        at jetbrains.buildServer.serverSide.metadata.impl.indexer.BuildIndexCleaner.performCleanup(BuildIndexCleaner.java:16)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.cleanupExtensionsData(HistoryEntryCleaner.java:38)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:138)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:132)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$3.performCleanup(ServerCleanupManagerImpl.java)
        at jetbrains.buildServer.serverSide.db.DBFacade$1$1.doInConnection(DBFacade.java:178)
        at jetbrains.buildServer.serverSide.db.DBFacade$6.doInConnection(DBFacade.java:415)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
        at jetbrains.buildServer.serverSide.db.DBFacade._runSql(DBFacade.java:411)
        at jetbrains.buildServer.serverSide.db.DBFacade.access$000(DBFacade.java:33)
        at jetbrains.buildServer.serverSide.db.DBFacade$1.doInTransaction(DBFacade.java:174)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at jetbrains.buildServer.serverSide.db.DBFacade.compact(DBFacade.java:171)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:74)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:0)
        at java.util.TimerThread.mainLoop(Unknown Source)
        at java.util.TimerThread.run(Unknown Source)
17-jul-2013 3:00:02 net.sf.ehcache.store.DiskStore remove
SEVERE: provider-nugetCache: Could not remove disk store entry for key 2731. Error was unexpected EOF in middle of data block
java.io.StreamCorruptedException: unexpected EOF in middle of data block
        at java.io.ObjectInputStream$BlockDataInputStream.refill(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.read(Unknown Source)
        at java.io.DataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream$BlockDataInputStream.readInt(Unknown Source)
        at java.io.ObjectInputStream.readInt(Unknown Source)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readSplitted(SerializableEntry.java:5)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.EntryImpl.readObjectInternal(EntryImpl.java:34)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.SerializableEntry.readExternal(SerializableEntry.java:16)
        at java.io.ObjectInputStream.readExternalData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.defaultReadFields(Unknown Source)
        at java.io.ObjectInputStream.readSerialData(Unknown Source)
        at java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
        at java.io.ObjectInputStream.readObject0(Unknown Source)
        at java.io.ObjectInputStream.readObject(Unknown Source)
        at net.sf.ehcache.store.DiskStore.loadElementFromDiskElement(DiskStore.java:313)
        at net.sf.ehcache.store.DiskStore.remove(DiskStore.java:483)
        at net.sf.ehcache.Cache.remove(Cache.java:1465)
        at net.sf.ehcache.Cache.remove(Cache.java:1392)
        at net.sf.ehcache.Cache.remove(Cache.java:1350)
        at net.sf.ehcache.Cache.remove(Cache.java:1328)
        at jetbrains.buildServer.serverSide.metadata.impl.cache.TypedCacheImpl.remove(TypedCacheImpl.java:16)
        at jetbrains.buildServer.serverSide.metadata.impl.metadata.MetadataStorageImpl.removeBuild(MetadataStorageImpl.java:30)
        at jetbrains.buildServer.serverSide.metadata.impl.indexer.BuildIndexCleaner.performCleanup(BuildIndexCleaner.java:16)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.cleanupExtensionsData(HistoryEntryCleaner.java:38)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:138)
        at jetbrains.buildServer.serverSide.impl.cleanup.HistoryEntryCleaner.performCleanup(HistoryEntryCleaner.java:132)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$3.performCleanup(ServerCleanupManagerImpl.java)
        at jetbrains.buildServer.serverSide.db.DBFacade$1$1.doInConnection(DBFacade.java:178)
        at jetbrains.buildServer.serverSide.db.DBFacade$6.doInConnection(DBFacade.java:415)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:341)
        at jetbrains.buildServer.serverSide.db.DBFacade._runSql(DBFacade.java:411)
        at jetbrains.buildServer.serverSide.db.DBFacade.access$000(DBFacade.java:33)
        at jetbrains.buildServer.serverSide.db.DBFacade$1.doInTransaction(DBFacade.java:174)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:130)
        at jetbrains.buildServer.serverSide.db.DBFacade.compact(DBFacade.java:171)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.startCleanup(ServerCleanupManagerImpl.java:74)
        at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$2.run(ServerCleanupManagerImpl.java:0)
        at java.util.TimerThread.mainLoop(Unknown Source)
        at java.util.TimerThread.run(Unknown Source)

Answer 1

As far as I know, TeamCity NuGet Server is subject to the artifact cleanup policy defined in "Administration | Project-related Settings | Build History Clean-up", so make sure your package is still there!!

My personal advise is ALWAYS setup a dedicated nuget server. You can setup one for free simply cloning the official NugetGallery project on GitHub: that's the same codebase used by nuget.org so you'll have a familiar UI and increased performances (NugetGallery leverage Lucene.NET indexing capabilities).