Removing mulitple key items with dict comprehension python

Question

I want to remove multiple key from my json and I'm using dictionary comprehensions like this

remove_key = ['name', 'description']

    data = {'full-workflow': {'task_defaults': {'retry': {'count': 3, 'delay': 2}}, 'tasks': {'t1': {'action': 'nn.postgres.export-db', 'description': 'Dump prod DB with default settings', 'input': {'database': 'prod', 'filepath': '/var/tmp/prod-dump.pgsql', 'host': 'postgres.local', 'password': 'mypass', 'username': 'myuser'}, 'name': 'db export', 'on-success': ['t2']}, 't2': {'action': 'nn.aws.upload-to-s3', 'description': 'Upload to S3 bucket for development', 'input': {'sourcepath': '{{ tasks(t1).result.filepath }}', 'targetpath': 's3://mybucket/prod-dump.pgsql'}, 'name': 'Upload to S3', 'on-success': ['t3'], 'retry': {'count': 5, 'delay': 5}}, 't3': {'action': 'nn.shell.command', 'description': 'Remove temp file from batch folder ', 'input': {'cmd': 'rm {{ tasks(t1).result.filepath }}'}, 'name': 'Remove temp file', 'on-complete': ['t4']}, 't4': {'action': 'nn.notify.send-mail', 'description': 'Send email to admin containing target path', 'input': {'from': '[email protected]', 'message': 'DB Dump {{ tasks(t1).result.filepath }} was stored to S3', 'subject': 'Prod DB Backup', 'to': '[email protected]'}, 'name': 'Send email', 'target': 'nn'}}}, 'version': '2'}

    def remove_additional_key(data):
        return {
            key: data[key] for key in data if key not in remove_key
        }

then just

new_data = remove_additional_key(data)

Because this is nested dict, I want to remove_key from tasks dict, so what am I doing wrong?

Answer 1

Your data are nested dictionaries. If you want to remove any data with a key contained in remove_key, then I suggest a recursive approach. This can be achieved, based on your exiting function remove_additional_key, with ease:

def remove_additional_key(data):

    sub_data = {} 
    for key in data:
        if key not in remove_key:
            if isinstance(data[key], dict): # if is dictionary
                sub_data[key] = remove_additional_key(data[key])
            else:
                sub_data[key] = data[key]
    return sub_data

new_data = remove_additional_key(data)

Note, if a entry is a dictionary can be checked by isinstance(data[key], dict). See How to check if a variable is a dictionary in Python?

Answer 2

You have a dictionary with a few nested dictionaries. If you know exactly which subdictionary you have those keys to remove, you can use:

data['full-workflow']['tasks']['t1'].pop('name')

Using the lookup approach (key: data[key]) in a dictionary comprehension is inefficient however, on such a small data amount you won't notice a difference.

If you don't know the exact path to your nested dictionary, you can use a function (posting another answer for your convenience)

def delete_keys_from_dict(d, lst_keys):
    for k in lst_keys:
        try:
            del dict_del[k]
        except KeyError:
            pass
    for v in dict_del.values():
        if isinstance(v, dict):
            delete_keys_from_dict(v, lst_keys)

    return dict_del

Then you could call

delete_keys_from_dict(data, ['name', 'description'])

Needless to say, should you have name key in multiple nested dictionaries, all of them would be deleted, so be careful.