AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>' when implementing Pytorch framework

Question

I was trying to implement a pytorch framework on CNN.
I'm sure the code is right because it's from a tutorial and it works when I ran it on Jupyter Notebook on GoogleDrive.
But when I tried to localize it as a .py file, it suggest an error:
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>' I know it's about inferencing objects outside a function, but what was the exact matter about this error?
And how should I fix it?

Here's the major part of the code.

def pre_datasets():
    TRAIN_TFM = transforms.Compose(
        [
            transforms.Resize(size=(128, 128)),
            # TODO
            transforms.ToTensor(),
        ]
    )
    train_set = DatasetFolder(
        root=CONFIG["train_set_path"],
        loader=lambda x: Image.open(x),
        extensions="jpg",
        transform=TRAIN_TFM,
    )
    train_loader = DataLoader(
        dataset=train_set,
        batch_size=CONFIG["batch_size"],
        shuffle=True,
        num_workers=CONFIG["num_workers"],
        pin_memory=True,
    )
    return train_loader

def train(train_loader):
    ...
    for epoch in range(CONFIG["num_epochs"]):
    ...
        for batch in train_loader: # error happened here
    ...

if __name__ == "__main__":
    train_loader = pre_datasets()
    train(train_loader)

Here's the error message:

Traceback (most recent call last):
  File "HW03_byCRZ.py", line 197, in <module>
    train(train_loader, valid_loader)
  File "HW03_byCRZ.py", line 157, in train
    for batch in train_loader:
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 355, in __iter__
    return self._get_iterator()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 301, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 914, in __init__
    w.start()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'

Answer 1

I had a similar issue and I used dill like this:

import dill as pickle

and it worked out of the box!