"UserWarning: An input could not be retrieved. It could be because a worker has died. We do not have any information on the lost sample."

Question

While training model I got this warning "UserWarning: An input could not be retrieved. It could be because a worker has died.We do not have any information on the lost sample.)", after showing this warning, model starts training. What does this warning means? Is it something that will affect my training and I need to worry about?

Answer 1

If you are running the training in GPU, the Warning will occur. You have to know that there are two running progress during the fit_generator running.

1. GPU, trains the IMAGE DATASETS with each steps in each epoch.
2. CPU, prepares the IMAGE DATASETS with each batch size.

While, they are parallel tasks. So if CPU's compute is lower than GPUs', the Warning occurs.

Solution:

Just set your batch_size smaller or upgrade your CPU config.

Answer 2

This is just a user warning that will be usually thrown when you try to fetch the inputs,targets during training. This is because a timeout is set for the queuing mechanism which will be specified inside the data_utils.py.

For more details you can refer the data_utils.py file which will be inside the keras/utils folder.

https://github.com/keras-team/keras/blob/master/keras/utils/data_utils.py