PyCharm IDE is very slow on debug Python reading small pickle file

Question

Debug takes sometimes 10 seconds. The problem starts after reading 30mega + 5 mega pickles, with with pdb debugging is fine. Even on lines of 'print'

any suggestions?

if __name__ == "__main__":
    print 'loading files...',
    with open(fname1, 'rb') as handle: items = pickle.load(handle)
    with open(fname2, 'rb') as handle: sentences_by_id= pickle.load(handle)
    print ' done!'

Answer 1

I had a similar problem with a dict of lists of ndarrays. when the list was too long, the pycharm debugger would freeze, top would show cpu for the pycharm at 102% ( I guess several cores were active) and it would take a very long time to unfreeze. Running without breakpoints was fine. My solution was to convert all my lists on ndarrays to ndarrays, e.g

<pseudo python>
for k in dict_of_lists_of_ndarrays:
     dict_of_lists_of_ndarrays[k]=np.array(dict_of_lists_of_ndarrays[k])
</pseudo python>

now things are much improved. I have no idea what the issue was

Answer 2

What help for me was to change the "variable loading policy" to "on demand". This is done by clicking the small cog wheel in the debug menu. For a picture check the link:

https://intellij-support.jetbrains.com/hc/en-us/community/posts/206601625-pycharm-debugger-slow-due-to-collecting-data-