Windows Azure Table Storage row size limit is less than the stated 1MB

Question

I have done a lot of research on table storage and its size limitations, overhead etc. using these sources:

• http://blogs.msdn.com/b/windowsazurestorage/archive/2010/07/09/understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx
• http://msdn.microsoft.com/en-us/library/dd179338.aspx

Using this information I wrote up some code to effectively store binary data across multiple properties, calculating any row and property overhead and staying within the 64KB property limit and 1MB row limit.

Unfortunately it just doesn't work. As an example storing around 0.5MB returns a 400 Bad Request stating the entity is too large - which I can't see why it would be given the 1MB row size limit.

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata">
  <code>EntityTooLarge</code>
  <message xml:lang="en-GB">The entity is larger than allowed by the Table Service.</message>
</error>

The code I've been using is fairly straight forward, but I might have made mistakes in estimating the overhead - however I doubt it would be off by 100% of the data size..

class Program
{
    static void Main(string[] args)
    {
        var client = CloudStorageAccount.DevelopmentStorageAccount.CreateCloudTableClient();
        var table = client.GetTableReference("sometable");
        table.CreateIfNotExists();

        const int rowOverhead = 4;
        const int maxRowSize = 1024 * 1024; // 1MB row size limit
        const int maxProperties = 252; // 255 less 3 system properties
        const int maxPropertySize = 64 * 1024; // 64KB property size limit

        var stream = new MemoryStream(new byte[512 * 1024]); // 0.5MB of data
        var entity = new DynamicTableEntity("pk", "rk");
        var buffer = new byte[maxPropertySize];
        var keySize = (entity.PartitionKey.Length + entity.RowKey.Length) * 2;
        var used = rowOverhead + keySize;

        for (var i = 0; i < maxProperties + 1; i++)
        {
            if (i > maxProperties)
            {
                throw new ArgumentException(string.Format("You have exceeded the column limit of {0}.", maxProperties));
            }

            var name = string.Concat("d", i);
            var overhead = CalculatePropertyOverhead(name, EdmType.Binary);
            var read = stream.Read(buffer, 0, maxPropertySize - overhead);
            used += read + overhead;

            if (used > maxRowSize)
            {
                throw new ArgumentException(string.Format("You have exceeded the max row size of {0} bytes.", maxRowSize));
            }

            if (read > 0)
            {
                var data = new byte[read];
                Array.Copy(buffer, 0, data, 0, read);
                entity.Properties.Add(name, new EntityProperty(data));
            }
            else
            {
                break;
            }
        }

        Console.WriteLine("Total entity size: {0}", used);
        table.Execute(TableOperation.InsertOrReplace(entity));
    }

    static int CalculatePropertyOverhead(string name, EdmType type)
    {
        const int propertyOverhead = 8;
        int propertyNameSize = name.Length * 2;
        int propertyTypeSize;

        switch (type)
        {
            case EdmType.Binary:
            case EdmType.Int32:
            case EdmType.String:
                propertyTypeSize = 4;
                break;
            case EdmType.Boolean:
                propertyTypeSize = 1;
                break;
            case EdmType.DateTime:
            case EdmType.Double:
            case EdmType.Int64:
                propertyTypeSize = 8;
                break;
            case EdmType.Guid:
                propertyTypeSize = 16;
                break;
            default:
                throw new NotSupportedException();
        }

        return propertyOverhead + propertyNameSize + propertyTypeSize;
    }
}

Any help in explaining what I'm missing is appreciated!

Thanks,

Mattias

Answer 1

Mattias, the limits you are refering to are for the actual Storage Service, but you are targeting the local Storage Emulator. The emulator uses your local SQL Server as its backing store and has different limitations than the actual storage service. See http://msdn.microsoft.com/en-us/library/windowsazure/gg433135.aspx for more information, in particular this line:

* The total size of a row in a table in the storage emulator is limited to less than 1 MB.