deepcopy() is extremely slow

Question

I have a game state in Python with about 1000 objects (planetary systems + stars + planets), and I need to copy it and apply a bunch of transformations to it when requested. However, at about 1 request/second, this is taking up 24.63% of my runtime. How can I make it go fast? Note that copying less is not an option, since the transforms touch just about everything.

EDIT: got it down to 8% with judicious implementation of __deepcopy__ on things. Still, not good enough. (Good enough is 1% or less, I plan on throwing many more things at this.) timeit says 41.8ms per deepcopy().

Answer 1

Actually, deepcopy is very slow. But we can use json, ujson, or cPickle. we can use json/cPickle to dump an object, and load it later. This is my test:

Total time: 3.46068 s File: test_deepcopy.py Function: test at line 15 Line #   Hits          Time Per Hit   % Time  Line Contents ============================================================== 15                                             @profile 16                                             def test(): 17       100       957585   9575.9     27.7        b = deepcopy(a) 18       100          862      8.6      0.0        c = copy(a) 19       100        42295    422.9      1.2        d = ujson.loads(ujson.dumps(a)) 20       100        85040    850.4      2.5        e = json.loads(json.dumps(a)) 21       100      2323465  23234.7     67.1        f = pickle.loads(pickle.dumps(a, -1)) 22       100        51434    514.3      1.5        g = cPickle.loads(cPickle.dumps(a, -1)) 

as what we can see, json/ujson/cPickle is faster than deepcopy, but pickle...

Answer 2

If you create your own class to hold these objects you can create your own methods that work with copy and deep copy. http://www.rafekettler.com/magicmethods.html#copying (Broken Link)

New Link for a github repository https://github.com/RafeKettler/magicmethods

class MyClass():     def __copy__(self):         copy_object = MyClass()         return copy_object      def __deepcopy__(self, memodict={}):         copy_object = MyClass()         copy_object.value = self.value         return copy_object  if __name__ == "__main__":     my_inst = MyClass()     print(copy.deepcopy(my_inst)) 

Here is a similar description from the previous broken link.

Copying

Sometimes, particularly when dealing with mutable objects, you want to be able to copy an object and make changes without affecting what you copied from. This is where Python's copy comes into play. However (fortunately), Python modules are not sentient, so we don't have to worry about a Linux-based robot uprising, but we do have to tell Python how to efficiently copy things.

__copy__(self)

Defines behavior for copy.copy() for instances of your class. copy.copy() returns a shallow copy of your object -- this means that, while the instance itself is a new instance, all of its data is referenced -- i.e., the object itself is copied, but its data is still referenced (and hence changes to data in a shallow copy may cause changes in the original).

__deepcopy__(self, memodict={})

Defines behavior for copy.deepcopy() for instances of your class. copy.deepcopy() returns a deep copy of your object -- the object and its data are both copied. memodict is a cache of previously copied objects -- this optimizes copying and prevents infinite recursion when copying recursive data structures. When you want to deep copy an individual attribute, call copy.deepcopy() on that attribute with memodict as the first argument. What are some use cases for these magic methods? As always, in any case where you need more fine-grained control than what the default behavior gives you. For instance, if you are attempting to copy an object that stores a cache as a dictionary (which might be large), it might not make sense to copy the cache as well -- if the cache can be shared in memory between instances, then it should be.