Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Processes not joining

I'm unable to paste my code snippet, so I'll explain the scenario 1. I have a list of around 50000 stocks that need some valuation 2. These stocks are in a MultiProcess.Queue 3. I create multiple processes, each process will take a batch of 50 from the queue and do something. 4. in the main thread, I have a check which looks like this

 anymore_to_process = True

 while anymore_to_process:
     if (stock_queue.qsize() == 0):
         anymore_to_process = False

for jobs in stock_jobs:
    jobs.join()
  1. however, this doesn't seem to work when i process 50000 records. If I process 500 stocks, this works fine.

what am I doing wrong? Why are the processes not joining when I process a lot of stocks.

I know this is difficult to answer without looking at my code.....but if you can give me some pointers, that would be very helpful.

like image 400
user2097496 Avatar asked Sep 24 '14 20:09

user2097496


1 Answers

Problem resolved - took a cue from @dano's question. I was indeed writing to another queue, which was blocking the processes. I took it out and problem resolved.

like image 51
user2097496 Avatar answered Oct 26 '22 06:10

user2097496