Export large amounts of data to client in asp.net

Question

I need to export a large amount of data (~100mb) from a sql table to a user via the web. What would be the best solution for doing so? One thought was to export the data to a folder on the db server, compress it (by some means) and then provide a download link for the user. Any other methods for doing so? Also, can we compress data from within sql server?

Any approaches are welcome.

Answer 1

I wouldn't tie up the database waiting for the user to download 100Mb, even for a high speed user. When the user requests the file have them specify an email address. Then call an asynch process to pull the data, write it to a temp file (don't want > 100mb in memory after all), then zip the temp file to a storage location, then send the user an email with a link to download the file.

Answer 2

You can respond to a page request with a file:

Response.AddHeader("Content-Disposition", 
    "attachment; filename=yourfile.csv");
Response.ContentType = "text/plain";

Be sure to turn buffering off, so IIS can start sending the first part of the file while you are building the second:

Response.BufferOutput = false;

After that, you can start writing the file like:

Response.Write("field1,field2,field3\r\n");

When the file is completely written, end the response, so ASP.NET doesn't append a web page to your file:

Response.End();

This way, you don't have to write files on your web servers, you just create the files in memory and send them to your users.

If compression is required, you can write a ZIP file in the same way. This is a nice free library to create ZIP files.