selective printing from a list

Question

Say for an example, I've a list like this:

q_list = [('94.vmtest2', 'sgmops', 'Andy Hays', '27/04 00:27:26', 'C', '27/04 00:28:31', 'vmtest1.hep', 'express', '00:00:10', '00:01:04'), 
          ('96.vmtest2', 'sgmops', 'John Dee', '27/04 01:27:26', 'C', '27/04 01:28:33', 'vmtest1.hep', 'short', '00:00:09', '00:01:06'),
          ('99.vmtest2', 'sgmops', 'Andy Hays', '27/04 07:19:56', 'C', '27/04 07:21:12', 'vmtest1.hep', 'express', '00:00:10', '00:01:14'), 
          ('103.vmtest2', 'sgmops', 'John Dee', '27/04 14:08:00', 'C', '27/04 14:09:16', 'vmtest1.hep', 'express', '00:00:10', '00:01:16'),
          ('102.vmtest2', 'sgmops', 'John Dee', '27/04 14:02:38', 'C', '27/04 14:10:12', 'vmtest1.hep', 'short', '00:00:10', '00:01:10')]

which is formed from the log files. and then I have a dictionary like this:

q_dict = {'username': 'johndee', 'queue': 'short'}

which is formed from a query string (the user input). What I want is to print [first 8 items of] the only lines from the list match with the value in the dictionary. In this case, I'll only print these two lines:

96.vmtest2   sgmops  John Dee  27/04 01:27:26  C  27/04 01:28:33  vmtest1.hep  short
102.vmtest2  sgmops  John Dee  27/04 14:02:38  C  27/04 14:10:12  vmtest1.hep  short

In fact it doesn't have to be a dictionary at all; the user input (command-line argument) is like this:

'formatDate(%m/%d) == 4/27 && username == John Dee && queue == short'

and I'm creating the dictionary out of that. Any idea how can I do that? Thanks in advance for any help. Cheers!!

Answer 1

Firstly, I would suggest that you store your data in namedtuples or a custom class, instead of just raw tuples. This way you can access the attributes by name (e.g. by using operator.attrgetter). The other option is to use a dict.

Then, you can simply do the following (assuming your items are dicts):

[item for item in q_list if all((item[attr] == q_dict[attr]) 
                                 for q_dict in filters)]

This of course only works if your conditions always follow the pattern shown in your question (no ||, no != or stuff like that).

If you can only use tuples, then you can create a dictionary that maps the names of attributes to tuples-indices.

Now, when you print them, you can just create a list of attributes that should be printed, and use it like this:

for item in filtered_items:
    print "\t".join(item[attr] for attr in attributes_to_print) + "\n"

Answer 2

The most common approach is to use a list comprehension, for instance to filter by username:

[x for x in q_list if x[2] == 'John Dee']

You can also use filter:

filter(lambda x: x[2] == 'John Dee', q_list)

You can thus use your q_dict to filter by user and/or queue easily.

[x for x in q_list if [x[2].lower().replace(' ', ''), x[7]] == q_dict.values()]

As suggested in Space_C0wb0y's answer, using namedtuple would be a good approach, you can map out the fields rather easily:

fields =  ['field1', 'field2', 'username', 'field4', 'field5', 'field6', 'field7', 'queue', 'field9', 'field10']
Item = namedtuple('Item', fields)

For the sake of the argument, let's convert the tuples in your list to namedtuples:

q_namedtuple = [Item(*x) for x in q_list]

And then you can filter dynamically based on the query params. For instance to get a list of items where all query params match the given fields:

[item for item in q_namedtuple if all(getattr(item, k) == v for k, v in q_dict.iteritems())]

That is assuming the username in q_dict actually matched the field for username...it doesn't in your example but you can work around that. Similarly you could create a list of dicts, that'd make working with your list a bit easier:

q_list_of_dicts = [dict(zip(fields, x)) for x in q_list]

Which would give you a list of dicts like:

{'field1': '102.vmtest2',
 ...etc
 'queue': 'short',
 'username': 'John Dee'}

Then you can filter similarly:

[item for item in q_list_of_dicts if all(item.get(k) == v for k, v in q_dict.iteritems())]

Personally I think I prefer the dict approach. Whole things rolled up using for loops instead (and one generator expression):

results = []
for item in q_list:
    d = dict(zip(fields, item))
    # use some other logic to filter
    if all(d.get(k) == v for k, v in q_dict.iteritems()):
         results.append(d)