matplotlib error when running plotting in multiprocess

Question

I am using python's Multiprocess.Pool to plot some data using multiple processes as follows:

class plotDriver:
    def plot(self, parameterList):
        numberOfWorkers = len(parameterList)
        pool = Pool(numberOfWorkers)
        pool.map(plotWorkerFunction, parameterList)
        pool.close()
        pool.join()

this is a simplified version of my class, the driver also contains other stuffs I choose to omit. The plotWorkderFunction is a single threaded function, which imports matplotlib and does all the plotting and setting figure styles and save the plots to one pdf file, and each worker is not interacting with the other.

I need to call this plot function multiple times since I have many parameterList, like following:

parameters = [parameterList0, parameterList1, ... parameterListn]
for param in parameters:
    driver = PlotDriver()
    driver.plot(param)

If parameters only contains one parameterList (the for loop only runs once), the code seems working fine. But it consistently fails whenever parameters contains more than one element, with the following error message happening on the second time in the loop.

Traceback (most recent call last):
File "plot.py", line 59, in <module>
  plottingDriver.plot(outputFile_handle)
File "/home/yingryic/PlotDriver.py", line 69, in plot
  pool.map(plotWrapper, workerParamList)
File "/home/yingryic/.conda/envs/pp/lib/python2.7/multiprocessing/pool.py", line 251, in map
  return self.map_async(func.iterable, chunksize).get()
File "/home/yingryic/.conda/envs/pp/python2.7/multiprocessing/pool.py", line 567, in get
  raise self._value
RuntimeError: In set_text: could not load glyph
X Error: BadIDChoice (invalid resouce ID chosen for this connection) 14
  Extension: 138 (RENDER)
  Minor opcode: 17 (RenderCreateGlyphSet)
  Resouce id: 0xe00002
 : Fatal IO error: client killed

any idea what is going wrong and how should I fix?

Answer 1

You can try placing import matplotlib into plotWorkerFunction() so that child processes will have their own copy of the module.