I have a dict to store objects:
jobs = {}
job = Job()
jobs[job.name] = job
now I want to convert it to use manager dict because I want to use multiprocessing and need to share this dict amonst processes
mgr = multiprocessing.Manager()
jobs = mgr.dict()
job = Job()
jobs[job.name] = job
just by converting to use manager.dict() things got extremely slow.
For example, if using native dict, it only took .65 seconds to create 625 objects and store it into the dict.
The very same task now takes 126 seconds!
Any optimization i can do to keep manager.dict() on par with python {}?
The problem is that each insert is quite slow for some reason (117x slower on my machine), but if you update your manager.dict()
with a normal dict, it will be a single and fast operation.
jobs = {}
job = Job()
jobs[job.name] = job
# insert other jobs in the normal dictionary
mgr = multiprocessing.Manager()
mgr_jobs = mgr.dict()
mgr_jobs.update(jobs)
Then use the mgr_jobs
variable.
Another option is to use the widely adopted multiprocessing.Queue
class.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With