iOS how to upload a large asset file into sever by streaming

Question

I am new iOS programmer.
I want to upload a large file(video or image) from asset library into sever, My original way is just use NSMutableURLRequest and append NSData(large video or large image) to it, and crash happened in the following code:

    ALAssetsLibraryAssetForURLResultBlock resultblock = ^(ALAsset *asset){
        //.......some code I just skip it...
        ALAssetRepresentation *rep = [asset defaultRepresentation];
        void *buffer = [rawData mutableBytes];
        [rep getBytes:buffer fromOffset:0 length:size error:nil];
        NSData *videoData = [[NSData alloc] initWithBytes:buffer length:size];//crash here
        [self startUploading:videoData];
    }

I know this crash is because memory is not enough, video file can't just alloc to NSData.
I've google this for 2 days, and sounds there are several ways to solve this.

1. use third party library: such like AFNetworking, ASIHTTPRequest(but I don't want to use it,because don't know when will this stop maintaining or updating )
2. use streaming to upload large file

and I want to use streaming way(point 2) to do upload stuff,
I found this link: http://zh.scribd.com/doc/51504708/10/Upload-Files-Over-HTTP
Look like can solve my problem, but still not very clear to know how to do

Question1: there is an example in that link, the upload file is from bundle
how to make asset into stream? or copy asset to APP's folder?
I found this link Copy image from assets-library to an app folder
but still can not find the way.

Question2: or Is there any other more clear streaming example for uploading large file?

Thank you for your passion

updated1: after I implemented needNewBodyStream delegate, "request stream exhaustion" message seemed to solved, but instead, meet another "Error Domain=kCFErrorDomainCFNetwork Code=303 "The operation couldn't be completed." how to solve it?

-(NSInputStream *)connection:(NSURLConnection *)connection needNewBodyStream:(NSURLRequest *)request
{
    [NSThread sleepForTimeInterval:2];
    NSInputStream *fileStream = [NSInputStream inputStreamWithFileAtPath:pathToBodyFile];
    if (fileStream == nil) {
        NSLog(@"NSURLConnection was asked to retransmit a new body stream for a request. returning nil!");
    }
    return fileStream;
}

Answer 1

Assuming your data is too big to fit into memory:

An effective and reliable approach would utilize a bounded pair of CFStreams (see CFStreamCreateBoundPair).

The bounded stream pair's input stream is set to the NSMutableURLRequest's HTTPBodyStream property. The bounded stream pair's output stream will be used to write bytes obtained from a fixed size memory buffer which has been filled with ALAssetRepresentation's getBytes:fromOffset:length:error: method.

The size of the bounded stream pair's transfer-buffer should be the same as the size of the buffer for the asset representation.

Setting up the code will require a couple lines of code and some experience with NSStreams and handling the events (there are often a few subtleties with NSStreams).

This approach works as follows:

1. Create a stream's delegate which handles all stream events.

2. Setup the paired streams with a particular size for the transfer buffer, setup the delegate and schedule them on a run loop.

3. Setup a memory buffer for the asset data with the same size.

4. When you opened the streams, you get a NSStreamEventHasSpaceAvailable event. You handle that event by reading from the asset data via getBytes:fromOffset:length:error:, and write into the memory buffer. When you filled your buffer with a chunk of the asset data, write this buffer into the bounded stream pair's output stream. Track offsets appropriately!

5. Now, the bounded streams pair's input stream well be extracted by the underlaying connection (which moves bytes from the internal transfer-buffer to the network socket), and you will get another NSStreamEventHasSpaceAvailable event, since now space is available in the internal transfer-buffer. Write as much bytes as fit into the bounded stream pair's output stream from your asset data buffer to output stream and as much bytes are available in your asset data buffer. If the asset data buffer has been written completely, refill it. Carefully track offsets and ranges!

6. You handle the events until the the whole asset data has been written. Then close the output stream.

7. You will need to handle other stream events as well, see: Stream Programming Guide

Note: you may notice that your memory buffer can only be partially written to the output stream. Take care of this by tracking offsets so that you always keep a continuos stream of asset data in your buffer and write the appropriate range of data from your buffer to the output stream!

Caveat:

Setting up a correct implementation withe a bounded pair of streams may be tricky and is perhaps error prone. I do have a generic version of a "InputStreamSource" (which exposes an ordinary NSInputStream which will be used to setup the HTTPBodyStream property) which can be easily extended to be used with any source of input - e.g. the asset data. I may put this code on gist, if you are interested.

AFNetworking or any other network library will not solve that for you. And to be honest, I wouldn't recommend to use AFNetworking in conjunction with a stream as the body part - since the implementation of AFNetworking is still suspect in this regard. I would recommend to use NSURLConnection with implementing the delegates yourself, or use another third party library which handles input body streams for POST requests correctly.

A Short (incomplete) Example

The idea is, to create some sort of "asset input source" class which exposes a NSInputStream (which can be used to set the HTTPBodyStream property of the NSURLRequest) and provides the asset data.

If the "asset input source" were a file, the task would be easy: just create a NSInputStream object associated to that file. However, our asset can be accessed only via a range of bytes in a certain representation, which resides in some temporary buffer.

So, the task is to fill that temporary buffer with the appropriate range of bytes. Then, piecewise, write those bytes into a private output stream which is bound to an input stream. This input stream and output stream pair will be created via function CFStreamCreateBoundPair.

The input stream will become our exposed NSInputStream of the "asset input source".

The output stream is used internally only. The "asset input source" will be initialized with the asset.

Our "asset input source" class needs to handle stream events, thus it will become a stream delegate.

Now, we have everything to implement it.

The CFStreamCreateBoundPair function creates CFStream objects. However, since NSStreams are toll-free bridged we can easily "convert" them to NSStreams.

Part of a start or an init method of the "asset input source" class can be implemented as follows:

    _buffer = (uint8_t)malloc(_bufferSize);
    _buffer_end = _buffer + _bufferSize;
    _p = _buffer;
    CFReadStreamRef readStream = NULL;
    CFWriteStreamRef writeStream = NULL;
    CFStreamCreateBoundPair(NULL, &readStream, &writeStream, _bufferSize);
    self.inputStream = CFBridgingRelease(readStream);
    self.internalOutputStream = CFBridgingRelease(writeStream);        

    [self.internalOutputStream scheduleInRunLoop:[NSRunLoop currentRunLoop] forMode:self.runLoopMode];
    [self.internalOutputStream open];
    // (Note: inputStream will be opened and scheduled by the request!)

inputStream is a public @property of that class (the exposed input stream).

internalOutputStream is a private property of the class.

_buffer is the internal buffer holding a range of bytes of the asset representation.

Note that the bounded stream pair's internal buffer size is equal to the buffer holding the asset data.

The stream delegate method stream:handleEvent: can be implemented as shown below:

- (void)stream:(NSStream *)theStream handleEvent:(NSStreamEvent)streamEvent
{
    if (_isCancelled) {
        return;
    }
    switch (streamEvent) {
        case NSStreamEventNone:
            break;
        case NSStreamEventOpenCompleted:
            DLogInfo(@"internal output stream: open completed");
            break;
        case NSStreamEventHasBytesAvailable:
            // n.a.
            NSAssert(0, @"bogus stream event");
            break;
        case NSStreamEventHasSpaceAvailable:
            NSAssert(theStream == _internalOutputStream, @"bogus stream event");
            DLogInfo(@"destination stream: has space available");
            [self write];
            break;
        case NSStreamEventErrorOccurred:
            DLogInfo(@"destination stream: error occurred");
            [self finish];
            break;
        case NSStreamEventEndEncountered:
            // weird error: the output stream is full or closed prematurely, or canceled.
            DLogWarn(@"destination stream: EOF encountered");
            if (_result == nil) {
                self.result = [NSError errorWithDomain:NSStringFromClass([self class])
                                                  code:-2
                                              userInfo:@{NSLocalizedDescriptionKey: @"output stream EOF encountered"}];
            }
            [self finish];
            break;
    }
}

As you can see, the secret is in method write. There is also a finish method and a cancel method.

Basically, method write copies from the _buffer into the internal output stream, as much as fit into the stream. When _buffer is completely written into the output stream, it will be filled again from the asset data.

When there is no more data available to write from the asset to the output stream, method finish is invoked.

Method finish closes the internal output stream and unschedules the stream.

A complete and reliable implementation may be a bit tricky. The "asset input source" should also be cancelable.

As mentioned, I do have an "abstract input source" class, which implements everything except the filling of _buffer with asset data, which I can provide as a code snippet on Gist, if you want.