ffmpeg - cuda encode - OpenEncodeSessionEx failed: out of memory

Question

I'm having a problem with ffmpeg video encoding using GPU (CUDA).

I have 2x nVidia GTX 1050 Ti

The problem comes when i try to do multiple parallel encodings. More than 2 processes and ffmpeg dies like this:

[h264_nvenc @ 0xcc1cc0] OpenEncodeSessionEx failed: out of memory (10)

The problem is nvidia-smi shows there are a lot of resources available on the gpu:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.66                 Driver Version: 384.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 00000000:41:00.0 Off |                  N/A |
| 40%   37C    P0    42W /  75W |    177MiB /  4038MiB |     30%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 105...  Off  | 00000000:42:00.0 Off |                  N/A |
| 40%   21C    P8    35W /  75W |     10MiB /  4038MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

the second GPU doesn't seem to be used at all, and there's more than enough memory left on the first one, to support the 3rd file.

Any ideas would be extremely helpful!

Answer 1

Actually your card is 'non-qualified' (in terms of NVIDIA) and supports only 2 simultaneous sessions. You could consult with https://developer.nvidia.com/video-encode-decode-gpu-support-matrix#Encoder or download NVENC SDK, which contains pdf with license terms for qualified and non-qualified GPUs. There are some patches for drivers which disables session count checking, you could try them https://github.com/keylase/nvidia-patch

Answer 2

1. Since there's no codes about how you apply the encoding context, I can't tell why the second gpu is not used. Have you specified using it in av_opt_set() or command line argument?
2. The more important problem here is geforce cards cannot own more than 2 encoding sessions in one system. If you need more, you have to use those expensive ones like quadro, tesla etc.