Vulkan API: moving rendering work into its own thread

Question

I'm learning the Khronos Vulkan API, and I'm having a hard time getting synchronization to work across multiple threads.

I'm trying to break the Rendering work into its own thread. My idea is to have a main thread, which will be in charged of high level app management stuff and, importantly, it will call VkQueuePresentKHR() to present the surface. However, the actual VkCmd* rendering work is to be done on a dedicated thread.

I want my app to use triple buffering, as it's very important for me to have a smooth 60fps. On a video presentation, NVIDIA mentioned that you'll want to have 1 VkCommandPool per framebuffer, and to just clear the entire pool after each use; that is more efficient than trying to clean individual Command Buffers. I'm going a step further: I'm trying to break up the rendering work into 6 Queues:

• 3 graphic queues: each frame gets its own dedicated queue.
• 1 for presentation.
• 1 for misc one time graphic commands.
• 1 for dedicated transfer work

My GTX770 video card has 16 general purpose queues + 1 dedicated transfer queue, so this should be no problem. Each queue would have its own VkCommandPool. The Queues, CommandPool, and CommandBuffer objects are created in the main thread, and then used in the Render thread.

I'm running into the following issues:

• VkAcquireNextImageKHR is called in the Render thread. If the thread acquires 3 images, then I can't just call the function again and rely on it to block until the next image is available. Instead, I get a validation layer error that I've grabbed all the images that are available. And, anyway, I would prefer to detect that no image is currently available, so that I can do other work in the thread, such as defragmenting memory. HOW CAN I TELL WHEN ALL THE IMAGES ARE CURRENTLY ACQUIRED, AND THAT I SHOULDN'T YET CALL vkAcquireNextImageKHR.
• How can I find out if vkQueuePresentKHR() is done presenting, and that I can send another frame to be presented? I don't know how to tell when I can start using that framebuffer again.
• It's not safe to clear the CommandPool right after vkQueueSubmit() returns, correct? To clear a CommandPool, Do I need to monitor the fence that is given to vkQueueSubmit()? It's not entirely clear to me from reading the spec.
• I know it's not safe to mess with a CommandPool from multiple threads, but is it ok to work with multiple Queues/CommandPools from one thread?
• I might be doing something wrong, but it looks like after calling vkResetFence() on some fence, you can't call vkGetFenceStatus() on it, or you'll get a validation layer error. Is this correct?

Answer 1

1. vkAcquireNextImageKHR acquires ownership of an image, and vkQueuePresentKHR releases ownership of it back to the presentation engine. The PE needs to own the image it's currently displaying, and sometimes others. Acquire will block until an image becomes available or the timeout expires. You get the validation error when there is no way you can rely on Acquire to return, since you haven't presented enough images. The spec says:

   Let n be the total number of images in the swapchain, m be the value of VkSurfaceCapabilitiesKHR::minImageCount, and a be the number of presentable images that the application has currently acquired (i.e. images acquired with vkAcquireNextImageKHR, but not yet presented with vkQueuePresentKHR). vkAcquireNextImageKHR can always succeed if a ≤ n - m at the time vkAcquireNextImageKHR is called. vkAcquireNextImageKHR should not be called if a > n - m with a timeout of UINT64_MAX; in such a case, vkAcquireNextImageKHR may block indefinitely.

2. The semaphore/fence you pass in to vkAcquireNextImageKHR will signal when the PE is done reading from the image. At that point, you can start writing to it again, and present it when ready.

3. You can't clear the command pool as long as any of the command buffers in it are still executing. You know when they're done executing when the fence passed to vkQueueSubmit signals.

4. Yes, you can use any number of objects of any type from a single thread. Nothing in the Vulkan implementation is bound to a particular thread it's used on. The restrictions on accessing objects concurrently from multiple threads are similar to what you'd have for your own objects: some object are serialized internally so concurrent access is okay, other objects are not.

5. That seems wrong. You should be able to call vkGetFenceStatus on any valid fence at any time.