Rotating video without rotating AVCaptureConnection and in the middle of AVAssetWriter session

Question

I'm using PBJVision to implement tap-to-record video functionality. The library doesn't support orientation yet so I'm in the process of trying to engineer it in. From what I see, there are three ways to rotate the video - I need help on deciding the best way forward and how to implement it. Note that rotation can happen between tap-to-record segments. So in a recording session, the orientation is locked to what it was when the user tapped the button. The next time the user taps the button to record, it should re-set the orientation to whatever the device's orientation is (so the resulting video shows right-side-up).

The approaches are outlined in the issue page on GitHub as well

Method 1 Rotate the AVCaptureConnection using setVideoOrientation: - this causes the video preview to flicker every time it's switched, since this switches the actual hardware it seems. Not cool, not acceptable.

Method 2 Set the transform property on the AVAssetWriterInput object used to write the video. The problem is, once the asset writer starts writing, the transform property can't be changed, so this only works for the first segment of the video.

Method 3 Rotate the image buffer being appended using something like this: How to directly rotate CVImageBuffer image in IOS 4 without converting to UIImage? but it keeps crashing and I'm not even sure if I'm barking up the right tree. There's an exception that is thrown and I can't really trace it back to much more than the fact that I'm using the vImageRotate90_ARGB8888 function incorrectly.

The explanation is a bit more detailed on the GitHub issue page I linked to above. Any suggestions would be welcome - to be honest, I'm not hugely experienced at AVFoundation and so I'm hoping that there's some miraculous way to do this that I don't even know about!

Answer 1

There is an easy and safe way.

#define degreeToRadian(x) (Double.pi * x / 180.0)

self.assetWriterInputVideo.transform = 
CGAffineTransformMakeRotation(CGFloat(degreeToRadian(-90))) ;

Answer 2

Method 1 isn't the preferred method according to Apple's documentation ("Physically rotating buffers does come with a performance cost, so only request rotation if it's necessary"). Method 2 worked for me but if I played my video on an app that doesn't support the transformation "metadata", the video isn't rotated properly. Method 3 is what I did.

I think it's crashing for you before you're trying to pass the image data directly from vImageRotate... to the AVAssetWriterInputPixelBufferAdaptor. You have to create a CVPixelBufferRef first. Here's my code:

Inside of captureOutput:didOutputSampleBuffer:fromConnection: I rotate the frame before writing it into the adaptor:

if ([self.videoWriterInput isReadyForMoreMediaData])
{
    // Rotate buffer first and then write to adaptor
    CMTime sampleTime = CMSampleBufferGetPresentationTimeStamp(sampleBuffer);
    CVPixelBufferRef rotatedBuffer = [self correctBufferOrientation:sampleBuffer];
    [self.videoWriterInputAdaptor appendPixelBuffer:rotatedBuffer withPresentationTime:sampleTime];
    CVBufferRelease(rotatedBuffer);
}

The referenced function that performs the vImage rotation is:

/* rotationConstant:
 *  0 -- rotate 0 degrees (simply copy the data from src to dest)
 *  1 -- rotate 90 degrees counterclockwise
 *  2 -- rotate 180 degress
 *  3 -- rotate 270 degrees counterclockwise
 */

- (CVPixelBufferRef)rotateBuffer:(CMSampleBufferRef)sampleBuffer withConstant:(uint8_t)rotationConstant
{
    CVImageBufferRef imageBuffer        = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);        

    OSType pixelFormatType              = CVPixelBufferGetPixelFormatType(imageBuffer);
    NSAssert(pixelFormatType == kCVPixelFormatType_32ARGB, @"Code works only with 32ARGB format. Test/adapt for other formats!");

    const size_t kAlignment_32ARGB      = 32;
    const size_t kBytesPerPixel_32ARGB  = 4;

    size_t bytesPerRow                  = CVPixelBufferGetBytesPerRow(imageBuffer);
    size_t width                        = CVPixelBufferGetWidth(imageBuffer);
    size_t height                       = CVPixelBufferGetHeight(imageBuffer);

    BOOL rotatePerpendicular            = (rotateDirection == 1) || (rotateDirection == 3); // Use enumeration values here
    const size_t outWidth               = rotatePerpendicular ? height : width;
    const size_t outHeight              = rotatePerpendicular ? width  : height;

    size_t bytesPerRowOut               = kBytesPerPixel_32ARGB * ceil(outWidth * 1.0 / kAlignment_32ARGB) * kAlignment_32ARGB;

    const size_t dstSize                = bytesPerRowOut * outHeight * sizeof(unsigned char);

    void *srcBuff                       = CVPixelBufferGetBaseAddress(imageBuffer);

    unsigned char *dstBuff              = (unsigned char *)malloc(dstSize);

    vImage_Buffer inbuff                = {srcBuff, height, width, bytesPerRow};
    vImage_Buffer outbuff               = {dstBuff, outHeight, outWidth, bytesPerRowOut};

    uint8_t bgColor[4]                  = {0, 0, 0, 0};

    vImage_Error err                    = vImageRotate90_ARGB8888(&inbuff, &outbuff, rotationConstant, bgColor, 0);
    if (err != kvImageNoError) 
    {
        NSLog(@"%ld", err);
    }

    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

    CVPixelBufferRef rotatedBuffer      = NULL;
    CVPixelBufferCreateWithBytes(NULL,
                                 outWidth,
                                 outHeight,
                                 pixelFormatType,
                                 outbuff.data,
                                 bytesPerRowOut,
                                 freePixelBufferDataAfterRelease,
                                 NULL,
                                 NULL,
                                 &rotatedBuffer);

    return rotatedBuffer;
}

void freePixelBufferDataAfterRelease(void *releaseRefCon, const void *baseAddress)
{
    // Free the memory we malloced for the vImage rotation
    free((void *)baseAddress);
}

Note: You may like to use enumeration for rotationConstant. Something like that (don't call this function with MOVRotateDirectionUnknown):

typedef NS_ENUM(uint8_t, MOVRotateDirection)
{
    MOVRotateDirectionNone = 0,
    MOVRotateDirectionCounterclockwise90,
    MOVRotateDirectionCounterclockwise180,
    MOVRotateDirectionCounterclockwise270,
    MOVRotateDirectionUnknown
};

Note: If you need IOSurface support, you should use CVPixelBufferCreate instead of CVPixelBufferCreateWithBytes and pass bytes data into it directly:

NSDictionary *pixelBufferAttributes = @{ (NSString *)kCVPixelBufferIOSurfacePropertiesKey : @{} };
CVPixelBufferCreate(kCFAllocatorDefault,
                    outWidth,
                    outHeight,
                    pixelFormatType,
                    (__bridge CFDictionaryRef)(pixelBufferAttributes),
                    &rotatedBuffer);

CVPixelBufferLockBaseAddress(rotatedBuffer, 0);
uint8_t *dest = CVPixelBufferGetBaseAddress(rotatedBuffer);
memcpy(dest, outbuff.data, bytesPerRowOut * outHeight);

CVPixelBufferUnlockBaseAddress(rotatedBuffer, 0);

Answer 3

method 3 does work to rotate the frame of video。 But I found out it can cause the MM leak. in order to it, I try to move the funcation in the same thread as the merging the frame of video. it does work. When you meet the issue, Please take care.