AFNetworking 2.0 Converting NSURLSessionDataTask to NSURLSessionDownloadTask doesn't write all file data to disk

Question

While converting a NSURLSessionDataTask to a NSURLSessionDownloadTask we are experiencing data loss. Specifically, on files that are larger than 16K we are losing the first 16K bytes (exactly 16384 bytes). The file that is written to disk is short by the length of the initial response....

Long post, thanks for reading and any suggestions.

UPDATE 2014-09-30 - Final Fix

So I encountered this same behaviour again recently and decided to figure dig deeper. As it turns out, Matt T (author of AFNetworking) posted a commit that modifies the AFURLSessionManager -respondsToSelector method and it will return NO if any of the OPTIONAL delegate calls are not set as Blocks. The commit is here (Issue #1779): https://github.com/AFNetworking/AFNetworking/commit/6951a26ada965edc6e43cf83a4985b88b0f514d2.

So, the way you are SUPPOSED to use the optional delegates is to call -setTaskDidReceiveAuthenticationChallengeBlock: method (call the one for ythe optional delegate you want to use) with your block INSTEAD of overriding the -URLSession:dataTask:didReceiveResponse:completionHandler: method in your subclass. Doing this yields the expected result.

Setup:

We are writing an iOS app that downloads files from a web server. The files are protected by a php script that authenticates the request from the iOS client.

We are using AFNetworking 2.0+ and are performing an initial POST (NSURLSessionDataTask) operation to the API sending the user credentials and such. here is the initial request:

NSURLSessionDataTask *task = [self POST:API_FULL_SYNC_GETFILE_PATH parameters:body success:^(NSURLSessionDataTask *task, id responseObject){ .. }];

We have a custom class that inherits from the AFHTTPSessionManager class where all the iOS code in this problem is contained.

The server receives this request and authenticates the user. One of the POST parameters is the file that the client is trying to download. The server locates the file and spits it out. To keep things simple to start I've removed the authentication and some cache control headers but here is the server php script that runs:

$file_name = $callparams['FILENAME'];
$requested_file = "$sync_data_dir/$file_name";

@apache_setenv('no-gzip', 1);
@ini_set('zlib.output_compression', 'Off');
set_time_limit(0);`

$file_size = filesize($requested_file);
header("Content-Type: application/gzip");
header("Content-Transfer-Encoding: Binary");
header("Content-Length: {$file_size}");
header("Content-Disposition: attachment; filename=\"{$file_name}\"");

$read_bytes = readfile($requested_file);

The files are always .gz files.

Back on the client, the response is received and the -URLSession:dataTask:didReceiveResponse:completionHandler: method of the NSURLSessionDataDelegate is called. We detect the MIME type and switch the task to a download task:

-(void)URLSession:(NSURLSession *)session
         dataTask:(NSURLSessionDataTask *)dataTask
didReceiveResponse:(NSURLResponse *)response
completionHandler:(void (^)(NSURLSessionResponseDisposition))completionHandler
{
    [super URLSession:session dataTask:dataTask didReceiveResponse:response completionHandler:completionHandler];

    /*
     This transforms a data task into a download taks for certain API calls. Check the headers to determine what to do
     */
    if ([response.MIMEType isEqualToString:@"application/gzip"]) {
        // Convert to download task
        completionHandler(NSURLSessionResponseBecomeDownload);
        return;
    }
    // continue as-is
    completionHandler(NSURLSessionResponseAllow);

}

The -URLSession:dataTask:didBecomeDownloadTask: method is called. We Use this method to relate the data task and download task using the id. This is done to track the results of the download task in the data task completion handler. Not super important for the problem, but here is the code:

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didBecomeDownloadTask:(NSURLSessionDownloadTask *)downloadTask
{
    [super URLSession:session dataTask:dataTask didBecomeDownloadTask:downloadTask];

    // Relate the data task with the download task.
    if (!_downloadTaskIdToDownloadIdTaskMap) {
        _downloadTaskIdToDownloadIdTaskMap = [NSMutableDictionary dictionary];
    }
    [_downloadTaskIdToDownloadIdTaskMap setObject:@(dataTask.taskIdentifier) forKey:@(downloadTask.taskIdentifier)];
}

Where the problem shows up:

In the -URLSession:downloadTask:didFinishDownloadingToURL: method, the size of the temp file written is smaller than the content-length.

What we Discovered:

A) If we implement the URLSession:dataTask:didReceiveData: method of the NSURLSessionTaskDelegate class, we observe exactly 1 call for each file we are trying to download. If the file is larger than 16384 bytes, then the resulting temp file will be short by that amount. Placing a log entry into this method, we see that the length of the data parameter is 16384 bytes for files larger than that.

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveData:(NSData *)data
{
    [super URLSession:session dataTask:dataTask didReceiveData:data];

    NSMutableDictionary *dataTaskDetails = [_dataTaskDetails objectForKey:@(dataTask.taskIdentifier)];
    NSString *fileName  = dataTaskDetails[@"FILENAME"];

    DDLogDebug(@"Data recieved for file '%@'. Data length %d",fileName,data.length);
}

B) Placing a log entry into the URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite: method of the NSURLSessionDownloadDelegate class, we observe 1 or more calls to this method for each file we try to download. If the file is <16K then only single call appears. If the file is >16K we get more calls. Here is that method:

- (void)URLSession:(NSURLSession *)session
      downloadTask:(NSURLSessionDownloadTask *)downloadTask
      didWriteData:(int64_t)bytesWritten
 totalBytesWritten:(int64_t)totalBytesWritten
totalBytesExpectedToWrite:(int64_t)totalBytesExpectedToWrite
{
    [super URLSession:session downloadTask:downloadTask didWriteData:bytesWritten totalBytesWritten:totalBytesWritten totalBytesExpectedToWrite:totalBytesExpectedToWrite];

    id dataTaskId = [_downloadTaskIdToDownloadIdTaskMap objectForKey:@(downloadTask.taskIdentifier)];
    NSMutableDictionary *dataTaskDetails = [_dataTaskDetails objectForKey:dataTaskId];
    NSString *fileName  = dataTaskDetails[@"FILENAME"];
    DDLogDebug(@"File '%@': Wrote %lld bytes. Total %lld of %lld bytes written.",fileName,bytesWritten,totalBytesWritten,totalBytesExpectedToWrite);
}

As an example, below is the console output for a single file 'members.json.gz'. I have added comments to highlight the important line.

[2014-02-24 00:54:16:290][main][I][APIClient.m:syncFullGetFile:withSyncToken:andUserName:andPassword:andCompletedBlock:][Line: 184] API Client requesting file 'members.json.gz' for session with token 'MToxMzkzMjIxMjM4'. <-- This is the initial request for the file.
[2014-02-24 00:54:17:448][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:dataTask:didReceiveData:][Line: 542] Data recieved for file 'members.json.gz'. Data length 16384 <-- Initial response, seems to fire BEFORE the conversion to a download task.
[2014-02-24 00:54:17:487][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite:][Line: 521] File 'members.json.gz': Wrote 16384 bytes. Total 16384 of 92447 bytes written. <-- Now the data task is a download task.
[2014-02-24 00:54:17:517][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite:][Line: 521] File 'members.json.gz': Wrote 16384 bytes. Total 32768 of 92447 bytes written.
[2014-02-24 00:54:17:533][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite:][Line: 521] File 'members.json.gz': Wrote 16384 bytes. Total 49152 of 92447 bytes written.
[2014-02-24 00:54:17:550][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite:][Line: 521] File 'members.json.gz': Wrote 16384 bytes. Total 65536 of 92447 bytes written.
[2014-02-24 00:54:17:568][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didWriteData:totalBytesWritten:totalBytesExpectedToWrite:][Line: 521] File 'members.json.gz': Wrote 10527 bytes. Total 76063 of 92447 bytes written. <-- Total is short by same 16384 - same number as the initial response.
[2014-02-24 00:54:17:573][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didFinishDownloadingToURL:][Line: 472] Temp file size for 'members.json.gz' is 76063
[2014-02-24 00:54:17:573][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didFinishDownloadingToURL:][Line: 485] File 'members.json.gz' downloaded. Reported 92447 of 92447 bytes received.
[2014-02-24 00:54:17:574][NSOperationQueue 0x14eb6380][?][APIClient.m:URLSession:downloadTask:didFinishDownloadingToURL:][Line: 490] File size after move for 'members.json.gz' is 76063
[2014-02-24 00:54:17:574][NSOperationQueue 0x14eb6380][E][APIClient.m:URLSession:downloadTask:didFinishDownloadingToURL:][Line: 497] Expected size of file 'members.json.gz' is 92447 but size on disk is 76063. Temp file size is 0.

Help:

We figure we are doing something wrong. Perhaps the headers we are sending from the server don't play nice with the data-to-download-task switch. Perhaps we are not using AFNetworking correctly.

Does anyone have a clue about this behaviour? Are we supposed to capture the initial response body in the URLSession:dataTask:didReceiveData: before the task is switched to a download task?

The really odd part is that if the file is under 16K then there are no issues. The entire file is written.

ALL requests for files start as a data task and are converted to download tasks.

Answer 1

I can convert a NSURLSessionDataTask to a NSURLSessionBackgroundTask, and (a) the file is the right size, and (b) I don't see any calls to didReceiveData.

I notice that you're calling super instances of these various delegate methods. That's a little curious. I wonder if your super implementation of didReceiveResponse is calling the completion handler, itself, resulting in your calling this completion handler twice. Notably, I can reproduce your problem if I deliberately call the handler twice, once with NSURLSessionResponseAllow before I call it again with NSURLSessionResponseBecomeDownload.

Make sure you call your completion handler only once and be very careful about what you have in these super methods (or just remove the reference to them altogether).