Problem setting video frame rate using AVAssetWriter/AVAssetReader

Question

Situation:

I am trying to export video with some parameters like video bit rate, audio bit rate, frame rate, changing video resolution, etc. Note that I am letting the user set the video frame rate in fractions; like user can set the video frame rate say, 23.98.

I use AVAssetWriter and AVAssetReader for this operation. I use AVAssetWriterInputPixelBufferAdaptor for writing the sample buffers.

Everything else works just fine, except the video frame rate.

What I have tried:

1. Setting the AVAssetWriter.movieTimeScale as suggested here. Which does change the video frame rate but also makes the video sluggish. (gist here)

---

2. Setting AVVideoExpectedSourceFrameRateKey. Which does not help. (gist here)

---

3. Setting AVAssetWriterInput.mediaTimeScale. Again, it changes the video frame rate but makes the video sluggish as AVAssetWriter.movieTimeScale does. The video shows different frames at some point and sometimes it sticks and resumes again. (gist here)

---

4. Using AVAssetReaderVideoCompositionOutput and setting AVMutableVideoComposition.frameDuration; just like SDAVAssetExportSession does. Ironically with SDAVAssetExportSession code, the video is being exported just at the right frame rate that I want, but it just does not work in my code. gist here

I am not sure why it won't work with my code. The issue with this approach is it always returns nil from AVAssetReaderVideoCompositionOutput.copyNextSampleBuffer().

---

5. Manually changing the timestamp of the frame with CMSampleTimingInfo, as suggested here Something like:

var sampleTimingInfo = CMSampleTimingInfo()
var sampleBufferToWrite: CMSampleBuffer?

CMSampleBufferGetSampleTimingInfo(vBuffer, at: 0, timingInfoOut: &sampleTimingInfo)

sampleTimingInfo.duration = CMTimeMake(value: 100, timescale: Int32(videoConfig.videoFrameRate * 100))

sampleTimingInfo.presentationTimeStamp = CMTimeAdd(previousPresentationTimeStamp, sampleTimingInfo.duration)

previousPresentationTimeStamp = sampleTimingInfo.presentationTimeStamp

let status = CMSampleBufferCreateCopyWithNewTiming(allocator: kCFAllocatorDefault, sampleBuffer: vBuffer,sampleTimingEntryCount: 1, sampleTimingArray: &sampleTimingInfo, sampleBufferOut: &sampleBufferToWrite)

With this approach, I do get the frame rate set just right, but it increases the video duration (as mentioned in the comment of that question’s answer). I think at some point I may have to discard some frames (if the target frame rate is lower; I need to lower the frame rate in most of the cases).

If I know that if I want 30fps, and my current frame rate is 60fps, it's simple to discard every second frame and setting the SampleBuffer time accordingly.

If I go with this approach(i.e. setting 23.98 fps), how do I decide which frame to discard and if the target frame rate is higher, which frame to duplicate? Reminder: the frame rate could be in fractions.

---

Answer 1

Here is an idea to select frames. Suppose the fps of source video is F and target fps is TF. rate = TF/F

Initiate a variable n equal to -rate and add rate each time, when the integer part of n changed, select the frame.

e.g. rate = 0.3
          n: -0.3 0 0.3 0.6 0.9 1.2 1.5 1.8 2.1
                  ^              ^           ^
frame index:      0  1   2   3   4   5   6   7
select 0 4 7

float rate = 0.39999f; // TF/F 
float n =  -rate; // to make sure first frame will be selected
for (int i = 0; i < 100; ++i, n += rate) { // i stands for frame index, take a video with 100 frames as an example
    int m = floor(n);
    int tmp = n+rate;
    // if rate > 1.0 repeat i
    // if rate < 1.0 some of the frames will be dropped
    for (int j = 0; m+j < tmp; ++j) {
        // Use this frame
        printf("%d ", i);
    }
}