Finding difference between two list of dictionary in Python

Question

I am trying to find the difference between 2 list of dictionary. I found some information in this forum but did not serve my purpose.

incoming_rows = [{'column_name': 'LOAD_ID', 'data_type': 'int', 'table_name': 'CONFIG'},
            {'column_name': 'ROW_NUMBER', 'data_type': 'int', 'table_name': 'CONFIG'},
            {'column_name': 'CREATE_DATE', 'data_type': 'VARCHAR(20)', 'table_name': 'CONFIG'},
            {'column_name': 'CONFIG_TYPE', 'data_type': 'varchar(1)', 'table_name': 'CONFIG'},
            {'column_name': 'CONFIG_ID', 'data_type': 'numeric(10,0)', 'table_name': 'CONFIG'}
            ]

available_row = [{'column_name': 'LOAD_ID', 'data_type': 'int', 'table_name': 'CONFIG'},
             {'column_name': 'ROW_NUMBER', 'data_type': 'int', 'table_name': 'CONFIG'},
             {'column_name': 'CREATE_DATE', 'data_type': 'date', 'table_name': 'CONFIG'}
             ]

Here I need to compare the incoming_rows with the available_row list of dictionary and the difference want to list in another list of dict format.Here my table name is unique. Conditions: 1. Any new addition of columns. 2. Any change in data type If these two conditions are true then the the expected_row should contain only these changed rows only.

# expected output
expected_row=[{'column_name': 'CONFIG_TYPE', 'data_type': 'varchar(1)', 'table_name': 'CONFIG'},
          {'column_name': 'CONFIG_ID', 'data_type': 'numeric(10,0)', 'table_name': 'CONFIG'},
          {'column_name': 'CREATE_DATE', 'data_type': 'VARCHAR(20)', 'table_name': 'CONFIG'}
        ]

Answer 1

A set is the perfect solution for this problem. Unfortunately, python will not let you add dictionaries to a set, because they are mutable and their hashcode could change between insert and lookup.

If you "freeze" the items to make them immutable, you can then add them to set objects instead of a list; and then take a set difference using the minus operator:

In [20]: i_set = { frozenset(row.items()) for row in incoming_rows }

In [21]: a_set = { frozenset(row.items())  for row in available_row }

In [22]: (i_set - a_set)
Out[22]: 
{frozenset({('column_name', 'CONFIG_ID'),
            ('data_type', 'numeric(10,0)'),
            ('table_name', 'CONFIG')}),
 frozenset({('column_name', 'CREATE_DATE'),
            ('data_type', 'VARCHAR(20)'),
            ('table_name', 'CONFIG')}),
 frozenset({('column_name', 'CONFIG_TYPE'),
            ('data_type', 'varchar(1)'),
            ('table_name', 'CONFIG')})}

Edit: To unfreeze:

In [25]: [dict(i) for i in i_set - a_set]
Out[25]: 
[{'column_name': 'CONFIG_ID',
  'data_type': 'numeric(10,0)',
  'table_name': 'CONFIG'},
 {'column_name': 'CREATE_DATE',
  'data_type': 'VARCHAR(20)',
  'table_name': 'CONFIG'},
 {'column_name': 'CONFIG_TYPE',
  'data_type': 'varchar(1)',
  'table_name': 'CONFIG'}]