PyCharm 3.1 hangs forever during indexing and unusable

Question

After updating to 3.1, PyCharm hangs forever (on OSX 10.9.1, Python 2.7.5) during the "indexing" of packages.

For me this occurs while indexing scipy (0.13.3). If I unistall scipy, indexing appears to complete, but then hangs again on "pythonstubs". The UI becomes unresponsive, CPU use is maxed, and I'm unable to do anything and have to force-quit the app.

If I reinstall scipy, PyCharm hangs again at the same spot in the scipy scan (see screen capture of dialog):

FWIW, I can run Python scripts from the system command line (including some that use scipy and many other packages recently updated or installed) without issue, so the Python installation is sound.

Has anyone had a similar problem or found a way around this one?

Answer 1

The problem lies with any regular expression matches that may have been defined to identify TODO items. The Java standard regular expression library used by PyCharm to match these items uses an algorithm of exponential complexity to search for '*.a' and similar patterns.

Theoretically, it is possible to match any regexp very fast (a linear algorithm exists), > but many developers of regexp libs simply don't bother implementing it.

The same problem exists for the Python re module:

>>> from timeit import timeit
>>> timeit("import re; list(re.finditer('.*a', 'foo' * 10000))", number=1)
0.6927990913391113
>>> timeit("import re; list(re.finditer('.*a', 'foo' * 50000))", number=1)
17.076900005340576

In general, if indexing is taking a long time, or hanging, look to the RegEx in your TODO items and see if you can narrow the scope of matches in order to improve performance.