Mysterious disappearing Azure development storage assets

Question

We're building a site backed by Azure Storage. One worker role has a few files that it downloads from the blob when it's starting up. The files are never modified once they're in storage, we just pull them down and use them.

On occasion, when trying to download these files from development storage the Storage Emulator service returns 500 errors. We can list the files in the blob and get metadata, but not download the file itself. The only solution we've found is to delete the blob and reupload.

Has anyone else run into this?

Update: 1.7 SDK

Answer 1

Possible Solution (Work Around)

1. Exit Storage Emulator;
2. Open up administrative Sql Server Management Studio 2012;
3. Attach C:\Users\<username>\DevelopmentStorageDb201206.mdf file (where <username>is the name of the affected user);
4. If it doesn't allow to attach, copy mdf and log files to some other drive and then attach, otherwise you can access to LocalDB ((localdb)\v11.0);
5. Find stored procedure CommitBlockList;
6. Change SET UncommittedBlockIdLength = NULL to SET UncommittedBlockIdLength = 0;
7. Execute it;
8. Close Management Studio;
9. Copy these edited mdf, log files to original location;
10. Start Storage Emulator;

How I got there

I found that about every seven days ONLY BLOCK blobs got deleted.
Creating those blobs again for testing purposes was painful while in the middle of development/testing.
Tried to find Storage Emulator source code but couldn't find it.
Turned on logging at C:\Users\<username>\AppData\Local\DevelopmentStorage by adding following to DevelopmentStorage.201206.config

<LogPath>C:\Users\<username>\AppData\Local\DevelopmentStorage\Logs</LogPath>  
<LoggingEnabled>true</LoggingEnabled>  

After painful waiting found following in logs:

DefragmentBlobFiles BlobInfo Name 40f5e12f-65a5-4a3a-ae46-41c71c8514c0/file1.txt, ContainerName storage1, Directory c:\users\username\appdata\local\developmentstorage\ldb\blockblobroot\1\12735b4b-f9ed-481b-a091-78387facf05b, ROFile , RWFile c:\users\username\appdata\local\developmentstorage\ldb\blockblobroot\1\12735b4b-f9ed-481b-a091-78387facf05b\1, Size5

I don't think above defragmentation causing any problems.
Found another log:

BlockBlob: Load Interval failed. IsGC: True, Exception at System.Number.ParseDouble(String value, NumberStyles options, NumberFormatInfo numfmt) at Microsoft.WindowsAzure.DevelopmentStorage.Store.BlockBlobGarbageCollector.GetTimerIntervalOrDefault(Boolean isGC)

So for BlockBlobs Uncommitted blocks are garbage-collected by this BlockBlobGarbageCollector. Nowhere I could find how often this uncommitted blocks are garbage-collected. I don't think even this is causing the problem.

Another log:

BlockBlob: Checking Directory C:\Users\username\AppData\Local\DevelopmentStorage\LDB\BlockBlobRoot\1\0477877c-4cb3-4ddb-a035-14a5cf52d86f in the list of valid directories
BlockBlob: Deleting Directory C:\Users\username\AppData\Local\DevelopmentStorage\LDB\BlockBlobRoot\1\0477877c-4cb3-4ddb-a035-14a5cf52d86f

THIS ABOVE LOG SHOWS THE PROBLEM. The emulator must be determining valid blockblob directories.

Checked schema of database DevelopmentStorageDb201206. Found few columns like IsCommitted and UncommittedBlockIdLength. Found that ClearUncommittedBlocks is setting UncommittedBlockIdLength to null. Any Blobs which were not being deleted were having UncommittedBlockIdLength value 0. So checked stored procedure CommitBlockList and changed UncommittedBlockIdLength to 0 instead of null. I think emulator in previous version must be checking IsCommitted and UncommittedBlockIdLength both to determine valid blockblob directories, while in this version it might be checking only UncommittedBlockIdLength as null and deleting all those block blob files.

As I said, it takes about seven days to find out whether this solution permanently fixes it. I have 4 more days to go to validate it.

If this is a workaround that works,... Microsoft owes me 6 hours;)

Answer 2

I've ran into the same issue. I suspect there is something wrong with Azure storage emulator when it is run on top of LocalDB. Here's why:

• when you got 500 error, open Blob table using Management Studio or Server Explorer, and find value of DirectoryPath field for blob in question (it will be something like c:\users\username\appdata\local\developmentstorage\ldb\blockblobroot\1\305469d0-7b68-4b1e-a41a-a429c21b6b9d

• navigate to that path using Explorer and notice that this directory is empty

• now reupload your file and navigate to a new directory it was uploaded to, notice that there is your file

So, the question now is why do blob files disappear?

Answer 3

I don't have a solution, but I can add that this behavior also occurs when the the storage emulator's backing store is SQL Server 2012. We've seen this repeatedly. All is well for a while, then blobs disappear. Our experience is that all, or nearly all blobs vanish from the file system while the database references persist. No clue why this happens - there is no obvious precipitating event.