deep copy with filtering in python

Question

I want to make a deep copy of a python data structure, ignoring some of the elements based on some criteria.

E.g., the input could be arbitrary json, and I'd like to copy everything but dictionaries with 'ignore' key in them. Like this:

def my_filter(entry):
    # return true if we need to skip this entry
    if not isinstance(entry, dict): return False
    return ('ignore' in entry)

a = [
    {"free": "yourself", "ignore": ""},
    [
        {"two": "words" },
        {"one": "finger"},
        [{"apples": 2}, {"array": [1,2,3,4], "ignore": ""}],
    ],
    {"hi": "there", "ignore": ""},
]

print copy_with_filter(a, my_filter)

and it would output

[[{'two': 'words'}, {'one': 'finger'}, [{'apples': 2}]]]

I've implemented this code which does the job for json input, that is when only list, dict, or literals can be present, and it works well:

def copy_with_filter(a, filter_func):
    # assume input is either list, or dict, or "literal" (i.e. string/int/float)
    if isinstance(a, list):
        out = [ copy_with_filter(x, filter_func) for x in a if not filter_func(x) ]
    elif isinstance(a, dict):
        out = a.copy() # shallow copy first
        for k in a.iterkeys():
            # replace values with filtered deep copies
            out[k] = copy_with_filter(a[k], filter_func)
    else:
        out = a
    return out

Though my question is whether there's more generic/better/built-in way of doing a deep copy with filtering in python?

Answer 1

Use the Python deepcopy builtin with memoization - that works better when deep copying a structure which has cross-referenced resources - your approach will create duplicates of things which are referenced multiple times in the children of the root object.

The somewhat terrible Python documentation:

https://docs.python.org/2/library/copy.html

... but there are good tutorials out there. One SO post on overloading the copy constructors for custom classes:

Python: Implementation of shallow and deep copy constructors

Adding filtering is a little more fiddly - but for custom objects it's easy enough (just early out of the deepcopy constructor, without adding a new object to the memo dict - the only real difficulty is plumbing in custom filtering rules, but that's just plumbing).