Texture streaming in 4K (multi-res)

Question

My goal is to stream single channel images in 4K at the highest possible frame rate using OpenGL. This means I have 1 texture which I'm going to update with new content very often.

As for now I'm using PBO's to take advantage of the asynchronous texture download on the GPU. The code is basically:

// EACH TIME TEXTURE NEEDS UPDATE
const int pboSize = 4096 * 4096 * 4;

// Bind PBO to texture data source
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pboHandle);
// Discard data in PBO - already sent to GPU
glBufferData(GL_PIXEL_UNPACK_BUFFER, pboSize, NULL, GL_STREAM_DRAW);
// Map buffer to client memory
float* data = static_cast<float*>(glMapBuffer(GL_PIXEL_UNPACK_BUFFER, GL_WRITE_ONLY));
if (data) {
    std::valarray<IMG_PRECISION>& newImageContent = myImageArray[currentImage];
    // This seems too slow
    std::memcpy(data, &newImageContent[0], newImageContent.size() * sizeof(IMG_PRECISION));
} else {
    std::err << "Failed to map PBO to client memory";
}

// Release the mapped buffer
glUnmapBuffer(GL_PIXEL_UNPACK_BUFFER);

const glm::uvec3& dimensions = _texture->dimensions();
_texture->bind();
// Send async to GPU
glTexSubImage2D(
    _texture->type(),
    0,
    0,
    0,
    GLsizei(dimensions.x),
    GLsizei(dimensions.y),
    GLint(_texture->format()),
    _texture->dataType(),
    nullptr
);

// Set back to normal texture data source
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

This works faster, but not fast enough, than just using glTexSubImage() without PBO's. The bottleneck seems to be the std::memcpy that takes around 0.0211s each frame. I create the texture with glTexImage2D() using GL_R16F as internal format, GL_RED as format and GL_FLOAT as data type.

However, I really only need 4K resolution when the texture is very close to the camera, but generating mipmaps at run-time seems too slow. Precomputing them manually would also be pretty annoying since I have lots of these images already in System RAM, loading the mipmaps together with the normal images would be too slow on the CPU side.

Is there any other multi-resolution approach I can't think of right now, or is the only way to get a decent frame rate to use some video codec such as ffmpeg? I have a lot of metadata in these images when I load them from disc and would ideally want to avoid that.

EDIT: Target is cross-platform desktop environment using OpenGL 4.5

Answer 1

You want to avoid blocking. Have one thread upload while the other one draws. You should have multiple buffers.

There is a good example on this site that ping-pongs two buffers. http://www.songho.ca/opengl/gl_pbo.html

A more advanced multi-threaded approach is found here.

http://on-demand.gputechconf.com/gtc/2012/presentations/S0356-GTC2012-Texture-Transfers.pdf

Regarding Mipmaps: Don't generate them if you don't need them. Only upload the 0 level.

Additional advice:

• Keep your decoder in another thread.
• Have the decoder buffer a few frames in main memory so the uploads aren't starved waiting for new frames.
• Benchmark. PBO's might not be any faster than glTexSubImage for your case if you end up blocking anyway.
• Watch out for internal conversions. If the internal format is GL_RGB and the glTexSubImage says GL_BGR (for example), it will be doing another copy on the cpu.