Azure Append Blobs Slow Read Performance

Question

I have observed that reading from Azure Append Blob is very slow when a blob is appended to a few thousand times or more. Writing/appending is fast, but reading a typical log blob with a few thousand appends each of size a few KB, for total size of a few MB takes more than a minute! Reading standard blog or page blob of similar size takes milliseconds. Is there some way to speed up reading from append blob, i.e. by flattening internal structure?

So far it looks like that it may be best to periodically "archive" / convert append blobs to block blobs, and then process them. Any suggestion?

If not, what would be recommended alternative storage for logs? Azure table can be used, but it would take many more reads, even when bulk operations are used.

Answer 1

I did switch to Azure Tables and read performance there is reasonable, about 1 second for 1.5K items, read in bulk mode. Still, reading a block or page blob with same content is in milliseconds, much faster. It would be quite effective if there was a way to append to a page blob. That can be done manually, so maybe Append (or some other type of) Blob can automate that in some future version. Append Blobs are quite complex inside Deep dive in Append Blob, that is likely reason for slow read.

Answer 2

This is because many small appends will cause the blob to become heavily fragmented, trading fast writes for slow reads.

Currently your best option is, like you wrote, to periodically copy the append blob into a block blob and read from that instead.

However, the Azure storage team have plans (since june 2018) to add a de-fragmentation feature. With that in place we may finally be able to get fast reads for small appends too.