Dataloader freezes

Question

My Pytorch (1.11.0) dataloader on a custom dataset freezes occasionally.

I cannot reproduce the freezing, it seems random: it usually "runs" without issues, but sometimes it gets stuck. When I interrupt it (ctrl+c), I read this:

   idx, data = self._get_data()
  File "/opt/conda/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1163, in _get_data
    success, data = self._try_get_data()
  File "/opt/conda/envs/torch/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1011, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/opt/conda/envs/torch/lib/python3.8/queue.py", line 179, in get
    self.not_empty.wait(remaining)
  File "/opt/conda/envs/torch/lib/python3.8/threading.py", line 306, in wait
    gotit = waiter.acquire(True, timeout)
KeyboardInterrupt

Asking here because this issue has been raised quite a few times on the official forum, but there are no replies.

I tried to cycle through the dataset to catch errors, but I had no problems: when I cycle through the dataloader it never freezes. I am working on an Ubuntu 20.04 linux pod on Kubernetes.

I am aware that concurrency in python is quite a mess, but is there anyone who can give me a suggestion on what to check?

CUSTOM DATASET:

class MultiModalDataset(Dataset):
    def __init__(self, img_dataset: pd.DataFrame, text_dataset: pd.DataFrame, 
            img_fld: str, img_transforms=None, n_classes=None, img_size=224,
            n_sentences=1, n_tokens=12, collate_fn=None, l1normalization=False, verbose=False):
        super().__init__()
        self.n_classes = n_classes or img_dataset.shape[1]
        assert self.n_classes == img_dataset.shape[1]
        self.img_ds = img_dataset
        # print(text_dataset.head())
        self.text_ds = text_dataset.set_index("image_filename")
        self.img_fld = img_fld
        self.transforms = img_transforms
        self.img_size = img_size
        self.n_sentences = n_sentences
        self.n_tokens = n_tokens
        self.collate_fn = collate_fn
        self.l1normalization = l1normalization
        self.verbose = verbose

    def __len__(self):
        return len(self.img_ds)
    
    def __getitem__(self, idx):
        assert (idx >=0) and (idx < len(self.img_ds))
        item = self.img_ds.iloc[idx]
        filename = item.name
        labels = item.values
        if self.l1normalization:
            nlabs = sum(labels)
            assert nlabs > 0, f"dataset, at index {idx}, no labels found"
            labels = labels / nlabs
        
        text = self.text_ds.loc[filename, "enc_text"]

        if self.collate_fn is not None:
            padded_text = self.collate_fn(text, n_sents=self.n_sentences, max_tokens=self.n_tokens, verbose=self.verbose)
        else:
            padded_text = text

        return self.load_image(filename), torch.tensor(labels.astype(np.float32)), torch.tensor(padded_text)

    def load_image(self, img_filename):
        fn = join(self.img_fld, img_filename)
        img = Image.open(fn)
        if self.transforms is not None:
            img = self.transforms(img)
        return img

DATA LOADER:

DataLoader(dataset, batch_size=128, shuffle=True, num_workers=4, drop_last=[False,False,False], pin_memory=False)

I iterate over the dataset/dataloder with:

for bi, (_, _, _) in enumerate(dataloader):
 ...

• switching pin_memory to True does not solve the issue.
• some suggest to set num_workers to zero, but I cannot: it become too slow. Changing the number of workers to any other value > 0 has no effects on the freezing (it still freezes).

Answer 1

I don't fully understand why but for me this solved the issue:

if __name__ == '__main__':
    import torch
    torch.multiprocessing.set_start_method('spawn')
    main()