Capturing RTP Timestamps

Question

I was trying a little experiment in order to get the timestamps of the RTP packets using the VideoCapture class from Opencv's source code in python, also had to modify FFmpeg to accommodate the changes in Opencv.

Since I read about the RTP packet format.Wanted to fiddle around and see if I could manage to find a way to get the NTP timestamps. Was unable to find any reliable help in trying to get RTP timestamps. So tried out this little hack.

Credits to ryantheseer on github for the modified code.

Version of FFmpeg: 3.2.3 Version of Opencv: 3.2.0

In Opencv source code:

modules/videoio/include/opencv2/videoio.hpp:

Added two getters for the RTP timestamp:

.....   
    /** @brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
    */
    CV_WRAP virtual int64 getRTPTimeStampSeconds() const;
    
    /** @brief Gets the lower bytes of the RTP time stamp in NTP format (fraction of seconds).
    */
    CV_WRAP virtual int64 getRTPTimeStampFraction() const;
.....

modules/videoio/src/cap.cpp:

Added an import and added the implementation of the timestamp getter:

....
#include <cstdint>
....
....
static inline uint64_t icvGetRTPTimeStamp(const CvCapture* capture)
{
  return capture ? capture->getRTPTimeStamp() : 0;
}
...

Added the C++ timestamp getters in the VideoCapture class:

 ....
/**@brief Gets the upper bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampSeconds() const
{
    int64 seconds = 0;
    uint64_t timestamp = 0;
    //Get the time stamp from the capture object
    if (!icap.empty())
        timestamp = icap->getRTPTimeStamp();
    else
        timestamp = icvGetRTPTimeStamp(cap);
    //Take the top 32 bytes of the time stamp
    seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);
    return seconds;
}

/**@brief Gets the lower bytes of the RTP time stamp in NTP format (seconds).
*/
int64 VideoCapture::getRTPTimeStampFraction() const
{
    int64 fraction = 0;
    uint64_t timestamp = 0;
    //Get the time stamp from the capture object
    if (!icap.empty())
        timestamp = icap->getRTPTimeStamp();
    else
        timestamp = icvGetRTPTimeStamp(cap);
    //Take the bottom 32 bytes of the time stamp
    fraction = (int64)((timestamp & 0xFFFFFFFF));
    return fraction;
}
...

modules/videoio/src/cap_ffmpeg.cpp:

Added an import:

...
#include <cstdint>
...

Added a method reference definition:

...
static CvGetRTPTimeStamp_Plugin icvGetRTPTimeStamp_FFMPEG_p = 0;
...

Added the method to the module initializer method:

...
if( icvFFOpenCV )
...
...
  icvGetRTPTimeStamp_FFMPEG_p =
                (CvGetRTPTimeStamp_Plugin)GetProcAddress(icvFFOpenCV, "cvGetRTPTimeStamp_FFMPEG");
...
...
icvWriteFrame_FFMPEG_p != 0 &&
icvGetRTPTimeStamp_FFMPEG_p !=0)
...

icvGetRTPTimeStamp_FFMPEG_p = (CvGetRTPTimeStamp_Plugin)cvGetRTPTimeStamp_FFMPEG;

Implemented the getter interface:

...
virtual uint64_t getRTPTimeStamp() const
    {
        return ffmpegCapture ? icvGetRTPTimeStamp_FFMPEG_p(ffmpegCapture) : 0;
    } 
...

In FFmpeg's source code:

libavcodec/avcodec.h:

Added the NTP timestamp definition to the AVPacket struct:

typedef struct AVPacket {
...
...
uint64_t rtp_ntp_time_stamp;
}

libavformat/rtpdec.c:

Store the ntp time stamp in the struct in the finalize_packet method:

static void finalize_packet(RTPDemuxContext *s, AVPacket *pkt, uint32_t timestamp)
{
    uint64_t offsetTime = 0;
    uint64_t rtp_ntp_time_stamp = timestamp;
...
...
/*RM: Sets the RTP time stamp in the AVPacket */
    if (!s->last_rtcp_ntp_time || !s->last_rtcp_timestamp)
        offsetTime = 0;
    else
        offsetTime = s->last_rtcp_ntp_time - ((uint64_t)(s->last_rtcp_timestamp) * 65536);
    rtp_ntp_time_stamp = ((uint64_t)(timestamp) * 65536) + offsetTime;
    pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp;

libavformat/utils.c:

Copy the ntp time stamp from the packet to the frame in the read_frame_internal method:

static int read_frame_internal(AVFormatContext *s, AVPacket *pkt)
{
    ...
    uint64_t rtp_ntp_time_stamp = 0;
...
    while (!got_packet && !s->internal->parse_queue) {
          ...
          //COPY OVER the RTP time stamp TODO: just create a local copy
          rtp_ntp_time_stamp = cur_pkt.rtp_ntp_time_stamp;


          ...


  #if FF_API_LAVF_AVCTX
    update_stream_avctx(s);
  #endif

  if (s->debug & FF_FDEBUG_TS)
      av_log(s, AV_LOG_DEBUG,
           "read_frame_internal stream=%d, pts=%s, dts=%s, "
           "size=%d, duration=%"PRId64", flags=%d\n",
           pkt->stream_index,
           av_ts2str(pkt->pts),
           av_ts2str(pkt->dts),
           pkt->size, pkt->duration, pkt->flags);
pkt->rtp_ntp_time_stamp = rtp_ntp_time_stamp; #Just added this line in the if statement.
return ret;

My python code to utilise these changes:

import cv2

uri = 'rtsp://admin:[email protected]:554'
cap = cv2.VideoCapture(uri)

while True:
    frame_exists, curr_frame = cap.read()
    # if frame_exists:
    k = cap.getRTPTimeStampSeconds()
    l = cap.getRTPTimeStampFraction()
    time_shift = 0x100000000
    #because in the getRTPTimeStampSeconds() 
    #function, seconds was multiplied by 0x10000000 
    seconds = time_shift * k
    m = (time_shift * k) + l
    print("Imagetimestamp: %i" % m)
cap.release()

What I am getting as my output:

    Imagetimestamp: 0
    Imagetimestamp: 212041451700224
    Imagetimestamp: 212041687629824
    Imagetimestamp: 212041923559424
    Imagetimestamp: 212042159489024
    Imagetimestamp: 212042395418624
    Imagetimestamp: 212042631348224
    ...

What astounded me the most was that when i powered off the ip camera and powered it back on, timestamp would start from 0 then quickly increments. I read NTP time format is relative to January 1, 1900 00:00. Even when I tried calculating the offset, and accounting between now and 01-01-1900, I still ended up getting a crazy high number for the date.

Don't know if I calculated it wrong. I have a feeling it's very off or what I am getting is not the timestamp.

Answer 1

As I see it, you receive a timestamp of type uint64 which contains to values uint32 in the high and low bits. I see that in a part of the code you use:

seconds = (int64)((timestamp & 0xFFFFFFFF00000000) / 0x100000000);

Which basically removes the lower bits and shifts the high bits to be in the lower bits. Then you cast it to int64. Here I only consider that it should be unsigned first of all, since it should not be negative in any case (seconds since epoch is always positive) and it should be uint32, since it is guarantee it is not bigger (you are taking only 32 bits). Also, this can be achieved (probably faster) with bitshifts like this:

auto seconds = static_cast<uint32>(timestamp >> 32);

Another error I spotted was in this part:

time_shift = 0x100000000
seconds = time_shift * k
m = (time_shift * k) + l

Here you are basically reconstructing the 64 bit timestamp, instead of creating the timestamp usable in other contexts. This means, you are shifting the lower bits in seconds to higher bits and adding the fraction part as the lower bits... This will end in a really big number which may not be useful always. You can still use it for comparison, but then all the conversions done in the C++ part are not needed. I think a more normal timestamp, which you can use with python datetime would be like this:

timestamp = float(str(k) + "." + str(l)) # don't know if there is a better way
date = datetime.fromtimestamp(timestamp)

If you don't care of the fractional part you can just use the seconds directly.

Another thing to consider is, that the timestamp of RTP protocols depends on the camera/server... They may use the clock timestamp or just some other clock like start of the streaming of start of the system. So it may or not be from epoch.