Storing many small files (on S3)?

Question

I have 2 million zipped HTML files (100-150KB) being added each day that I need to store for a long time. Hot data (70-150 million) is accessed semi regularly, anything older than that is barely ever accessed.

This means each day I'm storing an additional 200-300GB worth of files.

Now, Standard storage costs $0.023 per GB and $0.004 for Glacier.

While Glacier is cheap, the problem with it is that it has additional costs, so it would be a bad idea to dump 2 million files into Glacier:

PUT requests to Glacier $0.05 per 1,000 requests

Lifecycle Transition Requests into Glacier $0.05 per 1,000 requests

Is there a way of gluing the files together, but keeping them accessible individually?

Answer 1

An important point, that if you need to provide quick access to these files, then Glacier can give you access to the file in up to 12 hours. So the best you can do is to use S3 Standard – Infrequent Access (0,0125 USD per GB with millisecond access) instead of S3 Standard. And maybe for some really not using data Glacier. But it still depends on how fast do you need that data.

Having that I'd suggest following:

• as html (text) files have a good level of compression, you can compress historical data in big zip files (daily, weekly or monthly) as together they can have even better compression;
• make some index file or database to know where each html-file is stored;
• read only desired html-files from archives without unpacking whole zip-file. See example in python how to implement that.