Memory issues when encrypting/decrypting a large file with RNCryptor on iOS

Question

I'm trying to use RNCryptor to encrypt and decrypt large files (600+MB) on iOS. On the github I found example code on how to use the library asynchronously on streams. This code is similar to the answer of Rob Napier on a question about this same subject.

However, although I think I implemented the code correctly, the app uses up to 1.5 GB of memory (in the iPad 6.1 simulator). I thought the code was supposed to prevent the app from keeping more than one block of data in-memory? So what is going wrong?

In my controller, I create a 'CryptController' which I message with encrypt/decrypt requests.

  // Controller.m
  NSString *password = @"pw123";
  self.cryptor = [[CryptController alloc] initWithPassword:password];

  //start encrypting file
  [self.cryptor streamEncryptRequest:self.fileName andExtension:@"pdf" withURL:[self samplesURL]];

  //wait for encryption to finish
  NSDate *timeout = [NSDate dateWithTimeIntervalSinceNow:1];
  do {
    [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode
                             beforeDate:timeout];
  } while (![self.cryptor isFinished]);

In CryptController I have:

- (void)streamEncryptionDidFinish {
  if (self.cryptor.error) {
    NSLog(@"An error occurred. You cannot trust decryptedData at this point");
  }
  else {
    NSLog(@"%@ is complete. Use it as you like", [self.tempURL lastPathComponent]);
  }
  self.cryptor = nil;
  self.isFinished = YES;
}

- (void) streamEncryptRequest:(NSString *)fileName andExtension:(NSString *)ext withURL:(NSURL *)directory {

  //Make sure that this number is larger than the header + 1 block.
  int blockSize = 32 * 1024;

  NSString *encryptedFileName = [NSString stringWithFormat:@"streamEnc_%@", fileName];
  self.tempURL = [[NSFileManager defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateForURL:nil create:NO error:nil];
  self.tempURL = [self.tempURL URLByAppendingPathComponent:encryptedFileName isDirectory:NO];
  self.tempURL = [self.tempURL URLByAppendingPathExtension:@"crypt"];

  NSInputStream *decryptedStream = [NSInputStream inputStreamWithURL:[[directory URLByAppendingPathComponent:fileName isDirectory:NO] URLByAppendingPathExtension:ext]];
  NSOutputStream *cryptedStream = [NSOutputStream outputStreamWithURL:self.tempURL append:NO];

  [cryptedStream open];
  [decryptedStream open];

  __block NSMutableData *data = [NSMutableData dataWithLength:blockSize];
  __block RNEncryptor *encryptor = nil;

  dispatch_block_t readStreamBlock = ^{
    [data setLength:blockSize];
    NSInteger bytesRead = [decryptedStream read:[data mutableBytes] maxLength:blockSize];
    if (bytesRead < 0) {
      //Throw an error
    }
    else if (bytesRead == 0) {
      [encryptor finish];
    }
    else {
      [data setLength:bytesRead];
      [encryptor addData:data];
      //NSLog(@"Sent %ld bytes to encryptor", (unsigned long)bytesRead);
    }
  };

  encryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                           password:self.password
                                            handler:^(RNCryptor *cryptor, NSData *data) {
                                              //NSLog(@"Encryptor received %ld bytes", (unsigned long)data.length);
                                              [cryptedStream write:data.bytes maxLength:data.length];
                                              if (cryptor.isFinished) {
                                                [decryptedStream close];
                                                //call my delegate that i'm finished with decrypting
                                                [self streamEncryptionDidFinish];
                                              }
                                              else {
                                                readStreamBlock();
                                              }
                                            }];

  // Read the first block to kick things off
  self.isFinished = NO;
  readStreamBlock();
}

When I profile using the Allocation Instrument, the allocation categories I see consistently growing are malloc 32.50 KB, malloc 4.00 KB, NSConcreteData and NSSubrangeData. Especially the malloc 32.50 KB grows big, over 1 GB. The responsible caller is [NSConcreteData initWithBytes:length:copy:freeWhenDone:bytesAreVM:] For NSConcreteData the responsible caller is -[NSData(NSData) copyWithZone:].

When I profile using the Leaks Instrument, no leaks are found.

I'm new to Objective-C, and from what I understood, the new ARC is supposed to handle allocation and deallocation of memory. When googling on anything memory related, all the information I find is assuming you don't use ARC (or it didn't exist at time of writing). I sure am using ARC, since I get compile errors saying so when I try to manually deallocate memory.

If anyone can help me with this, it would be greatly appreciated! If any more information is needed, I'll be happy to provide it :) Also, I'm new to StackOverflow, so if there's anything I've overlooked that I should have done, kindly inform me!

Answer 1

I finally tried the solution given here, which uses semaphores instead of depending on the callback to wait for the stream. This works perfectly :) The memory usage hovers around 1.1 MB according to the Allocations Instrument. It may not look as neat because of the semaphore syntax, but at least it does what I need it to do.

Other suggestions are still welcome of course :)

- (void)encryptWithSemaphore:(NSURL *)url {
  dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);

  __block int total = 0;
  int blockSize = 32 * 1024;

  NSString *encryptedFile = [[url lastPathComponent] stringByDeletingPathExtension];
  NSURL *docsURL = [[NSFileManager defaultManager] URLForDirectory:NSDocumentDirectory inDomain:NSUserDomainMask appropriateForURL:nil create:NO error:nil];
  self.tempURL = [[docsURL URLByAppendingPathComponent:encryptedFile isDirectory:NO] URLByAppendingPathExtension:@"crypt"];

  NSInputStream *inputStream = [NSInputStream inputStreamWithURL:url];
  __block NSOutputStream *outputStream = [NSOutputStream outputStreamWithURL:self.tempURL append:NO];
  __block NSError *encryptionError = nil;

  [inputStream open];
  [outputStream open];

  RNEncryptor *encryptor = [[RNEncryptor alloc] initWithSettings:kRNCryptorAES256Settings
                                                        password:self.password
                                                         handler:^(RNCryptor *cryptor, NSData *data) {
                                                           @autoreleasepool {
                                                             [outputStream write:data.bytes maxLength:data.length];
                                                             dispatch_semaphore_signal(semaphore);

                                                             data = nil;
                                                             if (cryptor.isFinished) {
                                                               [outputStream close];
                                                               encryptionError = cryptor.error;
                                                               // call my delegate that I'm finished with decrypting
                                                             }
                                                           }
                                                         }];
  while (inputStream.hasBytesAvailable) {
    @autoreleasepool {
      uint8_t buf[blockSize];
      NSUInteger bytesRead = [inputStream read:buf maxLength:blockSize];
      if (bytesRead > 0) {
        NSData *data = [NSData dataWithBytes:buf length:bytesRead];

        total = total + bytesRead;
        [encryptor addData:data];

        dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);
      }
    }
  }

  [inputStream close];
  [encryptor finish];  
}

Answer 2

Just run:

self.cryptorQueue = dispatch_queue_create([queueName cStringUsingEncoding:NSUTF8StringEncoding], NULL);

dispatch_async(self.cryptorQueue, ^{
        readStreamBlock();
    });

trouble: stack is growing and autorelease pull will not execute release for buffer.

solution: add async in the same queue.. this will let current block to finish execution.

Here is the code:

__block NSMutableData *data = [NSMutableData dataWithLength:blockSize];
__block RNDecryptor *decryptor = nil;


dispatch_block_t readStreamBlock = ^{

    [data setLength:blockSize];

    NSInteger bytesRead = [inputStream read:[data mutableBytes] maxLength:blockSize];
    if (bytesRead < 0) {
        // Throw an error
    }
    else if (bytesRead == 0) {
        [decryptor finish];
    }
    else {

        [data setLength:bytesRead];
        [decryptor addData:data];
    }
};

decryptor = [[RNDecryptor alloc] initWithPassword:@"blah" handler:^(RNCryptor *cryptor, NSData *data) {

        [decryptedStream write:data.bytes maxLength:data.length];
        _percentStatus = (CGFloat)[[decryptedStream propertyForKey:NSStreamFileCurrentOffsetKey] intValue] / (CGFloat)_inputFileSize;
        if (cryptor.isFinished)
        {
            [decryptedStream close];
            [self decryptFinish];
        }
        else
        {
            dispatch_async(cryptor.responseQueue, ^{
                readStreamBlock();
            });
            [self decryptStatusChange];
        }

}];


// Read the first block to kick things off

decryptor.responseQueue = self.cryptorQueue;
[self decryptStart];
dispatch_async(decryptor.cryptorQueue, ^{
    readStreamBlock();
});