iOS Chunked Upload

Question

I'm trying to stream contacts from the user's address book to our server. Pulling all the contacts into memory at once can crash or make the device unresponsive. I don't want to incur the overhead of writing all the contacts to a file and uploading a file. I can see the data being sent across the wire, but it looks like it's in an invalid format. The server doesn't recognize a request body.

I'm reading contacts from the address book and writing them to an NSOutputStream. This NSOutputStream shares a buffer with an NSInputStream via this code

Buffering NSOutputStream used as NSInputStream?

//
//  NSStream+BoundPairAdditions.m
//  WAControls
//
//

#import "NSStream+BoundPairAdditions.h"
#include <sys/socket.h>

static void CFStreamCreateBoundPairCompat(
                                          CFAllocatorRef      alloc,
                                          CFReadStreamRef *   readStreamPtr,
                                          CFWriteStreamRef *  writeStreamPtr,
                                          CFIndex             transferBufferSize
                                          )
// This is a drop-in replacement for CFStreamCreateBoundPair that is necessary because that
// code is broken on iOS versions prior to iOS 5.0 <rdar://problem/7027394> <rdar://problem/7027406>.
// This emulates a bound pair by creating a pair of UNIX domain sockets and wrapper each end in a
// CFSocketStream.  This won't give great performance, but it doesn't crash!
{
#pragma unused(transferBufferSize)
    int                 err;
    Boolean             success;
    CFReadStreamRef     readStream;
    CFWriteStreamRef    writeStream;
    int                 fds[2];
    
    assert(readStreamPtr != NULL);
    assert(writeStreamPtr != NULL);
    
    readStream = NULL;
    writeStream = NULL;
    
    // Create the UNIX domain socket pair.
    
    err = socketpair(AF_UNIX, SOCK_STREAM, 0, fds);
    if (err == 0) {
        CFStreamCreatePairWithSocket(alloc, fds[0], &readStream,  NULL);
        CFStreamCreatePairWithSocket(alloc, fds[1], NULL, &writeStream);
        
        // If we failed to create one of the streams, ignore them both.
        
        if ( (readStream == NULL) || (writeStream == NULL) ) {
            if (readStream != NULL) {
                CFRelease(readStream);
                readStream = NULL;
            }
            if (writeStream != NULL) {
                CFRelease(writeStream);
                writeStream = NULL;
            }
        }
        assert( (readStream == NULL) == (writeStream == NULL) );
        
        // Make sure that the sockets get closed (by us in the case of an error,
        // or by the stream if we managed to create them successfull).
        
        if (readStream == NULL) {
            err = close(fds[0]);
            assert(err == 0);
            err = close(fds[1]);
            assert(err == 0);
        } else {
            success = CFReadStreamSetProperty(readStream, kCFStreamPropertyShouldCloseNativeSocket, kCFBooleanTrue);
            assert(success);
            success = CFWriteStreamSetProperty(writeStream, kCFStreamPropertyShouldCloseNativeSocket, kCFBooleanTrue);
            assert(success);
        }
    }
    
    *readStreamPtr = readStream;
    *writeStreamPtr = writeStream;
}

// A category on NSStream that provides a nice, Objective-C friendly way to create
// bound pairs of streams.

@implementation NSStream (BoundPairAdditions)

+ (void)createBoundInputStream:(NSInputStream **)inputStreamPtr outputStream:(NSOutputStream **)outputStreamPtr bufferSize:(NSUInteger)bufferSize
{
    CFReadStreamRef     readStream;
    CFWriteStreamRef    writeStream;
    
    assert( (inputStreamPtr != NULL) || (outputStreamPtr != NULL) );
    
    readStream = NULL;
    writeStream = NULL;
    
#if defined(__MAC_OS_X_VERSION_MIN_REQUIRED) && (__MAC_OS_X_VERSION_MIN_REQUIRED < 1070)
#error If you support Mac OS X prior to 10.7, you must re-enable CFStreamCreateBoundPairCompat.
#endif
#if defined(__IPHONE_OS_VERSION_MIN_REQUIRED) && (__IPHONE_OS_VERSION_MIN_REQUIRED < 50000)
#error If you support iOS prior to 5.0, you must re-enable CFStreamCreateBoundPairCompat.
#endif
    
    if (NO) {
        CFStreamCreateBoundPairCompat(
                                      NULL,
                                      ((inputStreamPtr  != nil) ? &readStream : NULL),
                                      ((outputStreamPtr != nil) ? &writeStream : NULL),
                                      (CFIndex) bufferSize
                                      );
    } else {
        CFStreamCreateBoundPair(
                                NULL,
                                ((inputStreamPtr  != nil) ? &readStream : NULL),
                                ((outputStreamPtr != nil) ? &writeStream : NULL), 
                                (CFIndex) bufferSize
                                );
    }
    
    if (inputStreamPtr != NULL) {
        *inputStreamPtr  = CFBridgingRelease(readStream);
    }
    if (outputStreamPtr != NULL) {
        *outputStreamPtr = CFBridgingRelease(writeStream);
    }
}

@end

Here I build the request body by handling the NSOutputStream delegation.

- (void)stream:(NSStream *)stream handleEvent:(NSStreamEvent)eventCode {
    
    switch(eventCode) {
        case NSStreamEventHasSpaceAvailable: {
            
            if(self.contactIndex == 0 && [self.producerStream hasSpaceAvailable]) {
                 NSMutableData *data = [[NSMutableData alloc] init];
                [data appendData:[@"\r\n\r\n\r\n" dataUsingEncoding:NSUTF8StringEncoding]];
                [data appendData:[@"{\"contacts\": [" dataUsingEncoding:NSUTF8StringEncoding]];
                [self.producerStream write:[data bytes] maxLength:[data length]];
            }
            
            while([self.producerStream hasSpaceAvailable] &&  self.contactIndex < [self.dataContactIDs count]) {
                NSMutableData *contactData = [[[self getNextContact] dataUsingEncoding:NSUTF8StringEncoding] mutableCopy];
                if(self.contactIndex < [self.dataContactIDs count]) {
                    [contactData appendData:[@"," dataUsingEncoding:NSUTF8StringEncoding]];
                }
                
                [self.producerStream write:[contactData bytes] maxLength:[contactData length]];
            }
            
            if(self.contactIndex == self.dataContactIDs.count) {
                 NSMutableData *data = [[NSMutableData alloc] init];
                [data appendData:[@"]}" dataUsingEncoding:NSUTF8StringEncoding]];
                [data appendData:[@"\r\n\r\n\r\n" dataUsingEncoding:NSUTF8StringEncoding]];
                [self.producerStream write:[data bytes] maxLength:[data length]];
               
                [stream close];
                [stream removeFromRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
                stream = nil;
            }
        } break;
        case NSStreamEventHasBytesAvailable: {
        } break;
        case NSStreamEventErrorOccurred: {
        } break;
        case NSStreamEventEndEncountered: {
            [stream close];
            [stream removeFromRunLoop:[NSRunLoop currentRunLoop] forMode:NSDefaultRunLoopMode];
            stream = nil;
        } break;
        default: {
        } break;
    }
}

I'm using AFNetworking to do the networking. I set the request body stream to the NSInputStream.

NSMutableURLRequest *request = [[NSMutableURLRequest alloc] initWithURL:[NSURL URLWithString:url]];
[request setHTTPMethod:@"POST"];
[request setValue:@"application/json; charset=UTF-8" forHTTPHeaderField:@"Content-Type"];
[request setHTTPBodyStream:inputStream];

AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
op.responseSerializer = [AFHTTPResponseSerializer serializer];

[op setUploadProgressBlock:^(NSUInteger bytesWritten, long long totalBytesWritten, long long totalBytesExpectedToWrite) {
    NSLog(@"PROGRESS %d %lld %lld", bytesWritten, totalBytesWritten, totalBytesExpectedToWrite);
}];

[op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject) {
    [self processResponse:responseObject success:success error:error log:log];
 } failure:^(AFHTTPRequestOperation *operation, NSError *e) {
     [self processError:e op:operation error:error log:log];
 }];


[[NSOperationQueue mainQueue] addOperation:op];

Then network request comes across like so: (captured using Wireshark)

POST /upload?token=dd224bceb02929b36d35&agent=iPhone%20Simulator&v=1.0 HTTP/1.1
Host: localhost:6547
Transfer-Encoding: Chunked
Accept-Encoding: gzip, deflate
Content-Type: application/json; charset=UTF-8
Accept-Language: en-us
Connection: keep-alive
Accept: */*
User-Agent: MyApp/2.0 CFNetwork/672.0.8 Darwin/13.0.0

9BD



{"contacts": [(valid json array)]}



0

I'm not sure why the 9BD and 0 are included in the request body. I think it's an error with how the buffers are setup and I believe this causes the server to disregard the http body because it's invalid. Does it look like I'm building the request correctly? Is there a better way to do this? I'm using pyramid/python to handle the request. The server receives the request okay, but the request body is empty.

Edit

If I don't send any contacts, the "9BD" goes away. If I change the contact data, the "9BD" changes to different characters. The "0" is always at the bottom.

Edit 2

Jim pointed out that the request is in a valid format. That means the server isn't handling the stream properly. The request is hitting the server okay, and the server is replying okay. However, I'm not seeing any of the request body. The server is running pyramid/python. On the server, request.body is empty.

Answer 1

That request is fine. Your request is chunked:

Transfer-Encoding: Chunked

The 9BD indicates the length of the next chunk. The zero at the end indicates that there are no more chunks.

See section 3.6.1 of RFC 2616 for details.

Your problem is probably that your server doesn't understand chunked requests.