Is it possible to wait for a transfer from the staging buffer to complete without calling vkQueueWaitIdle

Question

The following piece of code show you how i transfer a vertex buffer data from the staging buffer to a local memory buffer :

bool Vulkan::UpdateVertexBuffer(std::vector<VERTEX>& data, VULKAN_BUFFER& vertex_buffer)
{
    std::memcpy(this->staging_buffer.pointer, &data[0], vertex_buffer.size);

    size_t flush_size = static_cast<size_t>(vertex_buffer.size);
    unsigned int multiple = static_cast<unsigned int>(flush_size / this->physical_device.properties.limits.nonCoherentAtomSize);
    flush_size = this->physical_device.properties.limits.nonCoherentAtomSize * ((uint64_t)multiple + 1);

    VkMappedMemoryRange flush_range = {};
    flush_range.sType = VK_STRUCTURE_TYPE_MAPPED_MEMORY_RANGE;
    flush_range.pNext = nullptr;
    flush_range.memory = this->staging_buffer.memory;
    flush_range.offset = 0;
    flush_range.size = flush_size;
    vkFlushMappedMemoryRanges(this->device, 1, &flush_range);

    VkResult result = vkWaitForFences(this->device, 1, &this->transfer.fence, VK_FALSE, 1000000000);
    if(result != VK_SUCCESS) {
        #if defined(_DEBUG)
        std::cout << "UpdateVertexBuffer => vkWaitForFences : Timeout" << std::endl;
        #endif
        return false;
    }
    vkResetFences(this->device, 1, &this->transfer.fence);

    VkCommandBufferBeginInfo command_buffer_begin_info = {};
    command_buffer_begin_info.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
    command_buffer_begin_info.pNext = nullptr; 
    command_buffer_begin_info.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT;
    command_buffer_begin_info.pInheritanceInfo = nullptr;

    vkBeginCommandBuffer(this->transfer.command_buffer, &command_buffer_begin_info);

    VkBufferCopy buffer_copy_info = {};
    buffer_copy_info.srcOffset = 0;
    buffer_copy_info.dstOffset = 0;
    buffer_copy_info.size = vertex_buffer.size;

    vkCmdCopyBuffer(this->transfer.command_buffer, this->staging_buffer.handle, vertex_buffer.handle, 1, &buffer_copy_info);

    VkBufferMemoryBarrier buffer_memory_barrier = {};
    buffer_memory_barrier.sType = VK_STRUCTURE_TYPE_BUFFER_MEMORY_BARRIER;
    buffer_memory_barrier.pNext = nullptr;
    buffer_memory_barrier.srcAccessMask = VK_ACCESS_MEMORY_WRITE_BIT;
    buffer_memory_barrier.dstAccessMask = VK_ACCESS_VERTEX_ATTRIBUTE_READ_BIT;
    buffer_memory_barrier.srcQueueFamilyIndex = this->queue_stack[this->transfer_stack_index].index;
    buffer_memory_barrier.dstQueueFamilyIndex = this->queue_stack[this->graphics_stack_index].index;
    buffer_memory_barrier.buffer = vertex_buffer.handle;
    buffer_memory_barrier.offset = 0;
    buffer_memory_barrier.size = VK_WHOLE_SIZE;

    vkCmdPipelineBarrier(this->transfer.command_buffer, VK_PIPELINE_STAGE_TRANSFER_BIT, VK_PIPELINE_STAGE_VERTEX_INPUT_BIT, 0, 0, nullptr, 1, &buffer_memory_barrier, 0, nullptr);

    vkEndCommandBuffer(this->transfer.command_buffer);

    VkSubmitInfo submit_info = {};
    submit_info.sType = VK_STRUCTURE_TYPE_SUBMIT_INFO;
    submit_info.pNext = nullptr;
    submit_info.waitSemaphoreCount = 0;
    submit_info.pWaitSemaphores = nullptr;
    submit_info.pWaitDstStageMask = nullptr;
    submit_info.commandBufferCount = 1;
    submit_info.pCommandBuffers = &this->transfer.command_buffer;
    submit_info.signalSemaphoreCount = 0;
    submit_info.pSignalSemaphores = nullptr;

    VkResult result = vkQueueSubmit(this->queue_stack[this->transfer_stack_index].handle, 1, &submit_info, this->transfer.fence);
    if(result != VK_SUCCESS) {
        #if defined(_DEBUG)
        std::cout << "UpdateVertexBuffer => vkQueueSubmit : Failed" << std::endl;
        #endif
        return false;
    }

    #if defined(_DEBUG)
    std::cout << "UpdateVertexBuffer : Success" << std::endl;
    #endif
    return true;
}

It works perfectly without any validation layer warning. But when i call i twice, both buffers contains the same data, from the second call. For example :

UpdateVertexBuffer(cube_data, cube_buffer);
UpdateVertexBuffer(prism_data, prism_buffer);

This will result in having a prism inside both cube_buffer and prism_buffer. To fix this, i can simply wait for a few milliseconds between the two calls :

UpdateVertexBuffer(cube_data, cube_buffer);
std::this_thread::sleep_for(std::chrono::milliseconds(100));
UpdateVertexBuffer(prism_data, prism_buffer);

or preferably, i can replace the fence by a call to vkQueueWaitIdle(this->queue_stack[this->transfer_stack_index].handle);

In my opinion this will result in performance loss and the fence is supposed to be the optimal way to wait for transfer operation to complete properly, so why is my first buffer filled by second when i'm using a fence. And is there a way to do this properly without using vkQueueWaitIdle.

Thanks for your help.

Answer 1

You wait for the fence for the previous upload after you have already written the data to the staging buffer. That's too late; the fence is there to prevent you from writing data to memory that's being read.

But really, your problem is that your design is wrong. Your design is such that sequential updates all use the same memory. They shouldn't. Instead, sequential updates should use different regions of the same memory, so that they cannot overlap. That way, you can perform the transfers and not have to wait on fences at all (or at least, not until next frame).

Basically, you should treat your staging buffer like a ring buffer. Every operation that wants to do some staged transfer work should "allocate" X bytes of memory from the staging ring buffer. The staging buffer system allocates memory sequentially, wrapping around if there is insufficient space. But it also remembers where the last memory region is that it synchronized with. If you try to stage too much work, then it has to synchronize.

Also, one of the purposes behind mapping memory is that you can write directly to that memory, rather than writing to some other CPU memory and copying it in. So instead of passing in a VULKAN_BUFFER (whatever that is), the process that generated that data should have fetched a pointer to a region of the active staging buffer and written its data into that.

Oh, and one more thing: never, ever create a command buffer and immediately submit it. Just don't do it. There's a reason why vkQueueSubmit can take multiple command buffers, and multiple batches of command buffers. For any one queue, you should never be submitting more than once (or maybe twice) per frame.