How to glob for iterable element

Question

I have a python dictionary that contains iterables, some of which are lists, but most of which are other dictionaries. I'd like to do glob-style assignment similar to the following:

myiter['*']['*.txt']['name'] = 'Woot'

That is, for each element in myiter, look up all elements with keys ending in '.txt' and then set their 'name' item to 'Woot'.

I've thought about sub-classing dict and using the fnmatch module. But, it's unclear to me what the best way of accomplishing this is.

Answer 1

The best way, I think, would be not to do it -- '*' is a perfectly valid key in a dict, so myiter['*'] has a perfectly well defined meaning and usefulness, and subverting that can definitely cause problems. How to "glob" over keys which are not strings, including the exclusively integer "keys" (indices) in elements which are lists and not mappings, is also quite a design problem.

If you nevertheless must do it, I would recommend taking full control by subclassing the abstract base class collections.MutableMapping, and implement the needed methods (__len__, __iter__, __getitem__, __setitem__, __delitem__, and, for better performance, also override others such as __contains__, which the ABC does implement on the base of the others, but slowly) in terms of a contained dict. Subclassing dict instead, as per other suggestions, would require you to override a huge number of methods to avoid inconsistent behavior between the use of "keys containing wildcards" in the methods you do override, and in those you don't.

Whether you subclass collections.MutableMapping, or dict, to make your Globbable class, you have to make a core design decision: what does yourthing[somekey] return when yourthing is a Globbable?

Presumably it has to return a different type when somekey is a string containing wildcards, versus anything else. In the latter case, one would imagine, just what is actually at that entry; but in the former, it can't just return another Globbable -- otherwise, what would yourthing[somekey] = 'bah' do in the general case? For your single "slick syntax" example, you want it to set a somekey entry in each of the items of yourthing (a HUGE semantic break with the behavior of every other mapping in the universe;-) -- but then, how would you ever set an entry in yourthing itself?!

Let's see if the Zen of Python has anything to say about this "slick syntax" for which you yearn...:

>>> import this
    ...
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.

Consider for a moment the alternative of losing the "slick syntax" (and all the huge semantic headaches it necessarily implies) in favor of clarity and simplicity (using Python 2.7-and-better syntax here, just for the dict comprehension -- use an explicit dict(...) call instead if you're stuck with 2.6 or earlier), e.g.:

def match(s, pat):
    try: return fnmatch.fnmatch(s, pat)
    except TypeError: return False

def sel(ds, pat):
    return [d[k] for d in ds for k in d if match(k, pat)]

def set(ds, k, v):
    for d in ds: d[k] = v

so your assignment might become

set(sel(sel([myiter], '*')), '*.txt'), 'name', 'Woot')

(the selection with '*' being redundant if all , I'm just omitting it). Is this so horrible as to be worth the morass of issues I've mentioned above in order to use instead

myiter['*']['*.txt']['name'] = 'Woot'

...? By far the clearest and best-performing way, of course, remains the even-simpler

def match(k, v, pat):
    try:
      if fnmatch.fnmatch(k, pat):
        return isinstance(v, dict)
    except TypeError:
        return False

for k, v in myiter.items():
  if match(k, v, '*'):
    for sk, sv in v.items():
      if match(sk, sv, '*.txt'):
        sv['name'] = 'Woot'

but if you absolutely crave conciseness and compactness, despising the Zen of Python's koan "Sparse is better than dense", you can at least obtain them without the various nightmares I mentioned as needed to achieve your ideal "syntax sugar".