I'm modifying an Android Framework example to package the elementary AAC streams produced by MediaCodec into a standalone .mp4 file. I'm using a single MediaMuxer
instance containing one AAC track generated by a MediaCodec
instance.
However I always eventually get an error message on a call to mMediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo)
:
E/MPEG4Writer﹕timestampUs 0 < lastTimestampUs XXXXX for Audio track
When I queue the raw input data in mCodec.queueInputBuffer(...)
I provide 0 as the timestamp value per the Framework Example (I've also tried using monotonically increasing timestamp values with the same result. I've successfully encoded raw camera frames to h264/mp4 files with this same method).
Check out the full source
Most relevant snippet:
private static void testEncoder(String componentName, MediaFormat format, Context c) {
int trackIndex = 0;
boolean mMuxerStarted = false;
File f = FileUtils.createTempFileInRootAppStorage(c, "aac_test_" + new Date().getTime() + ".mp4");
MediaCodec codec = MediaCodec.createByCodecName(componentName);
try {
codec.configure(
format,
null /* surface */,
null /* crypto */,
MediaCodec.CONFIGURE_FLAG_ENCODE);
} catch (IllegalStateException e) {
Log.e(TAG, "codec '" + componentName + "' failed configuration.");
}
codec.start();
try {
mMediaMuxer = new MediaMuxer(f.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
} catch (IOException ioe) {
throw new RuntimeException("MediaMuxer creation failed", ioe);
}
ByteBuffer[] codecInputBuffers = codec.getInputBuffers();
ByteBuffer[] codecOutputBuffers = codec.getOutputBuffers();
int numBytesSubmitted = 0;
boolean doneSubmittingInput = false;
int numBytesDequeued = 0;
while (true) {
int index;
if (!doneSubmittingInput) {
index = codec.dequeueInputBuffer(kTimeoutUs /* timeoutUs */);
if (index != MediaCodec.INFO_TRY_AGAIN_LATER) {
if (numBytesSubmitted >= kNumInputBytes) {
Log.i(TAG, "queueing EOS to inputBuffer");
codec.queueInputBuffer(
index,
0 /* offset */,
0 /* size */,
0 /* timeUs */,
MediaCodec.BUFFER_FLAG_END_OF_STREAM);
if (VERBOSE) {
Log.d(TAG, "queued input EOS.");
}
doneSubmittingInput = true;
} else {
int size = queueInputBuffer(
codec, codecInputBuffers, index);
numBytesSubmitted += size;
if (VERBOSE) {
Log.d(TAG, "queued " + size + " bytes of input data.");
}
}
}
}
MediaCodec.BufferInfo info = new MediaCodec.BufferInfo();
index = codec.dequeueOutputBuffer(info, kTimeoutUs /* timeoutUs */);
if (index == MediaCodec.INFO_TRY_AGAIN_LATER) {
} else if (index == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
MediaFormat newFormat = codec.getOutputFormat();
trackIndex = mMediaMuxer.addTrack(newFormat);
mMediaMuxer.start();
mMuxerStarted = true;
} else if (index == MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED) {
codecOutputBuffers = codec.getOutputBuffers();
} else {
// Write to muxer
ByteBuffer encodedData = codecOutputBuffers[index];
if (encodedData == null) {
throw new RuntimeException("encoderOutputBuffer " + index +
" was null");
}
if ((info.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) {
// The codec config data was pulled out and fed to the muxer when we got
// the INFO_OUTPUT_FORMAT_CHANGED status. Ignore it.
if (VERBOSE) Log.d(TAG, "ignoring BUFFER_FLAG_CODEC_CONFIG");
info.size = 0;
}
if (info.size != 0) {
if (!mMuxerStarted) {
throw new RuntimeException("muxer hasn't started");
}
// adjust the ByteBuffer values to match BufferInfo (not needed?)
encodedData.position(info.offset);
encodedData.limit(info.offset + info.size);
mMediaMuxer.writeSampleData(trackIndex, encodedData, info);
if (VERBOSE) Log.d(TAG, "sent " + info.size + " audio bytes to muxer with pts " + info.presentationTimeUs);
}
codec.releaseOutputBuffer(index, false);
// End write to muxer
numBytesDequeued += info.size;
if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {
if (VERBOSE) {
Log.d(TAG, "dequeued output EOS.");
}
break;
}
if (VERBOSE) {
Log.d(TAG, "dequeued " + info.size + " bytes of output data.");
}
}
}
if (VERBOSE) {
Log.d(TAG, "queued a total of " + numBytesSubmitted + "bytes, "
+ "dequeued " + numBytesDequeued + " bytes.");
}
int sampleRate = format.getInteger(MediaFormat.KEY_SAMPLE_RATE);
int channelCount = format.getInteger(MediaFormat.KEY_CHANNEL_COUNT);
int inBitrate = sampleRate * channelCount * 16; // bit/sec
int outBitrate = format.getInteger(MediaFormat.KEY_BIT_RATE);
float desiredRatio = (float)outBitrate / (float)inBitrate;
float actualRatio = (float)numBytesDequeued / (float)numBytesSubmitted;
if (actualRatio < 0.9 * desiredRatio || actualRatio > 1.1 * desiredRatio) {
Log.w(TAG, "desiredRatio = " + desiredRatio
+ ", actualRatio = " + actualRatio);
}
codec.release();
mMediaMuxer.stop();
mMediaMuxer.release();
codec = null;
}
Update: I've found a root symptom I think lies within MediaCodec
.:
I send presentationTimeUs=1000
to queueInputBuffer(...)
but receive info.presentationTimeUs= 33219
after calling MediaCodec.dequeueOutputBuffer(info, timeoutUs)
. fadden left a helpful comment related to this behavior.
Thanks to fadden's help I've got a proof-of-concept audio encoder and video+audio encoder on Github. In summary:
Send AudioRecord
's samples to a MediaCodec
+ MediaMuxer
wrapper. Using the system time at audioRecord.read(...)
works sufficiently well as an audio timestamp, provided you poll often enough to avoid filling up AudioRecord's internal buffer (to avoid drift between the time you call read and the time AudioRecord recorded the samples). Too bad AudioRecord doesn't directly communicate timestamps...
// Setup AudioRecord
while (isRecording) {
audioPresentationTimeNs = System.nanoTime();
audioRecord.read(dataBuffer, 0, samplesPerFrame);
hwEncoder.offerAudioEncoder(dataBuffer.clone(), audioPresentationTimeNs);
}
Note that AudioRecord only guarantees support for 16 bit PCM samples, though MediaCodec.queueInputBuffer
takes input as byte[]
. Passing a byte[]
to audioRecord.read(dataBuffer,...)
will truncate split the 16 bit samples into 8 bit for you.
I found that polling in this way still occasionally generated a timestampUs XXX < lastTimestampUs XXX for Audio track
error, so I included some logic to keep track of the bufferInfo.presentationTimeUs
reported by mediaCodec.dequeueOutputBuffer(bufferInfo, timeoutMs)
and adjust if necessary before calling mediaMuxer.writeSampleData(trackIndex, encodedData, bufferInfo)
.
The code from above answer https://stackoverflow.com/a/18966374/6463821 also provides timestampUs XXX < lastTimestampUs XXX for Audio track
error, because if you read from AudioRecord`s buffer faster then need, duration between generated timstamps will smaller than real duration between audio samples.
So my solution for this issue is generate first timstamp and each next sample increase timestamp by duration of your sample (depends on bit-rate, audio format, channel config).
BUFFER_DURATION_US = 1_000_000 * (ARR_SIZE / AUDIO_CHANNELS) / SAMPLE_AUDIO_RATE_IN_HZ;
...
long firstPresentationTimeUs = System.nanoTime() / 1000;
...
audioRecord.read(shortBuffer, OFFSET, ARR_SIZE);
long presentationTimeUs = count++ * BUFFER_DURATION + firstPresentationTimeUs;
Reading from AudioRecord should be in separate thread, and all read buffers should be added to queue without waiting for encoding or any other actions with them, to prevent losing of audio samples.
worker =
new Thread() {
@Override
public void run() {
try {
AudioFrameReader reader =
new AudioFrameReader(audioRecord);
while (!isInterrupted()) {
Thread.sleep(10);
addToQueue(
reader
.read());
}
} catch (InterruptedException e) {
Log.w(TAG, "run: ", e);
}
}
};
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With