Why is NSURLSession slower than cURL when downloading many files?

Question

I've been using cURL to download about 1700+ files -- which total to about ~290MB -- in my iOS app. It takes about 5-7 minutes on my Internet connection to download all of them using cURL. But since not everyone has fast internet connection (especially when on the go), I decided to allow the files to be downloaded in the background, so that the user can do other things while waiting for the download to finish. This is where NSURLSession comes in.

Using NSURLSession, it takes about 20+ minutes on my Internet connection to download all of them while the app is in foreground. I don't mind it being slow when the app is in background, because I understand that it is up to the OS to schedule the downloads. But it's a problem when it's slow even when it's in foreground. Is this the expected behaviour? Is it because of the quantity of the files?

In case I'm not using NSURLSession correctly, here's a snippet of how I'm using it:

// Initialization

NSURLSessionConfiguration *sessionConfiguration = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier:@"<my-identifier>"];
sessionConfiguration.HTTPMaximumConnectionsPerHost = 40;

backgroundSession = [NSURLSession sessionWithConfiguration:sessionConfiguration
                                                  delegate:self
                                             delegateQueue:nil];

// ...

// Creating the tasks and starting the download
for (int i = 0; i < 20 && queuedRequests.count > 0; i++) {
    NSDictionary *requestInfo = [queuedRequests lastObject];
    NSURLSessionDownloadTask *downloadTask = [backgroundSession downloadTaskWithURL:[NSURL URLWithString:requestInfo[@"url"]]];
    ongoingRequests[@(downloadTask.taskIdentifier)] = requestInfo;
    [downloadTask resume];
    [queuedRequests removeLastObject];
    NSLog(@"Begin download file %d/%d: %@", allRequests.count - queuedRequests.count, allRequests.count, requestInfo[@"url"]);
}

// ...

// Somewhere in (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didFinishDownloadingToURL:(NSURL *)location

// After each download task is completed, grab a file to download from
// queuedRequests, and create another task

if (queuedRequests.count > 0) {
    requestInfo = [queuedRequests lastObject];
    NSURLSessionDownloadTask *newDownloadTask = [backgroundSession downloadTaskWithURL:[NSURL URLWithString:requestInfo[@"url"]]];
    ongoingRequests[@(newDownloadTask.taskIdentifier)] = requestInfo;

    [newDownloadTask resume];
    [queuedRequests removeLastObject];
    NSLog(@"Begin download file %d/%d: %@", allRequests.count - queuedRequests.count, allRequests.count, requestInfo[@"url"]);
}

I've also tried using multiple NSURLSession, but it's still slow. The reason I tried that is because when using cURL, I create multiple threads (around 20), and each thread will download a single file at a time.

It's also not possible for me to reduce the number of files by zipping it, because I need the app to be able to download individual files since I will update them from time to time. Basically, when the app starts, it will check if there are any files that have been updated, and only download those files. Since the files are stored in S3, and S3 doesn't have zipping service, I could not zip them into a single file on the fly.

Answer 1

As mentioned by Filip and Rob in the comments, the slowness is because when NSURLSession is initialized with backgroundSessionConfigurationWithIdentifier:, the download tasks will be executed in the background regardless if the app is in the foreground. So I solved this issue by having 2 instances of NSURLSession: one for foreground download, and one for background download:

NSURLSessionConfiguration *foregroundSessionConfig = [NSURLSessionConfiguration defaultSessionConfiguration];
foregroundSessionConfig.HTTPMaximumConnectionsPerHost = 40;

foregroundSession = [NSURLSession sessionWithConfiguration:foregroundSessionConfig
                                                  delegate:self
                                             delegateQueue:nil];
[foregroundSession retain];

NSURLSessionConfiguration *backgroundSessionConfig = [NSURLSessionConfiguration backgroundSessionConfigurationWithIdentifier:@"com.terato.darknessfallen.BackgroundDownload"];
backgroundSessionConfig.HTTPMaximumConnectionsPerHost = 40;

backgroundSession = [NSURLSession sessionWithConfiguration:backgroundSessionConfig
                                                  delegate:self
                                             delegateQueue:nil];
[backgroundSession retain];

When the app is switched to background, I simply call cancelByProducingResumeData: on each of the download tasks that's still running, and then pass it to downloadTaskWithResumeData::

- (void)switchToBackground
{
    if (state == kDownloadManagerStateForeground) {
        [foregroundSession getTasksWithCompletionHandler:^(NSArray *dataTasks, NSArray *uploadTasks, NSArray *downloadTasks) {
            for (NSURLSessionDownloadTask *downloadTask in downloadTasks) {
                [downloadTask cancelByProducingResumeData:^(NSData *resumeData) {
                    NSURLSessionDownloadTask *downloadTask = [backgroundSession downloadTaskWithResumeData:resumeData];
                    [downloadTask resume];
                }];
            }
        }];

        state = kDownloadManagerStateBackground;
    }
}

Likewise, when the app is switched to foreground, I do the same but switched foregroundSession with backgroundSession:

- (void)switchToForeground
{
    if (state == kDownloadManagerStateBackground) {
        [backgroundSession getTasksWithCompletionHandler:^(NSArray *dataTasks, NSArray *uploadTasks, NSArray *downloadTasks) {
            for (NSURLSessionDownloadTask *downloadTask in downloadTasks) {
                [downloadTask cancelByProducingResumeData:^(NSData *resumeData) {
                    NSURLSessionDownloadTask *downloadTask = [foregroundSession downloadTaskWithResumeData:resumeData];
                    [downloadTask resume];
                }];
            }
        }];

        state = kDownloadManagerStateForeground;
    }
}

Also, don't forget to call beginBackgroundTaskWithExpirationHandler: before calling switchToBackground when the app is switched to background. This is to ensure that the method is allowed to complete while in background. Otherwise, it will only be called once the app enters foreground again.