Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Download multiple files vs single one big file&unzip via socket

I need my client to download 30Mb worth of files.

Following is the setup.

  1. They are comprised of 3000 small files.
  2. They are downloaded through tcp bsd socket.
  3. They are stored in client's DB as they get downloaded.
  4. Server can store all necessary files in memory.(no file access on server side)

I've not seen many cases where client downloads such large number of files which I suspect due to server side's file access.
I'm also worried if multiplexer(select/epoll) will be overwhelmed by excessive network request handling.(Do I need to worry about this?)

With the above suspicions, I zipped up 3000 files to 30 files. (overall size doesn't change much because the files are already compressed files(png))

Test shows, 3000 files downloading is 25% faster than 30files downloading & unzipping.
I suspect it's because client device's is unable to download while unzipping & inserting into DB, I'm testing on handheld devices.. iPhone..
(I've threaded unzipping+DB operation separate from networking code, but DB operation seems to take over the whole system. I profiled a bit, and unzipping doesn't take long, DB insertion does. On server-side, files are zipped and placed in memory beforehand.)

I'm contemplating on switching back to 3000 files downloading because it's faster for clients.

I wonder what other experienced network people will say over the two strategies,
1. many small data
2. small number of big data & unzipping.

EDIT

For experienced iphone developers, I'm threading out the DB operation using NSOperationQueue.
Does NSOperationQueue actually threads out well?
I'm very suspicious on its performance.
-- I tried posix thread, no significant difference..

like image 401
eugene Avatar asked Oct 21 '11 02:10

eugene


1 Answers

I'm answering my own question.

It turned out that inserting many images into sqlite DB at once in a client takes long time, as a result, network packet in transit is not delivered to client fast enough.

http://www.sqlite.org/faq.html#q19

After I adopted the suggestion in the faq to speed up "many insert", it actually outperforms the "many files download individually strategy".

like image 174
eugene Avatar answered Oct 04 '22 20:10

eugene