Python + readline + auto-completion (tab): why are dashes and question-marks treated as word-separators?

Question

Hello coders and brave GNU-readline users,
A few months ago I started using Python's (2.7.1) readline module for a shell-like application I've written. The application has nothing to do with files and file-systems - it's a tailored solution for proprietary management software.

Yesterday I found that specific text causes unexpected auto-completion behavior, and haven't found a way to resolve this in the documentation. I'm desperately asking for your help here. I'll start with an example, and follow with a code snippet that reproduces the unwanted behavior.

Providing the values for auto-completion are:

aaa0   aaa1   aaa2   bbb_0  bbb_1  bbb_2
ccc-0  ccc-1  ccc-2  ddd?0  ddd?1  ddd?2

...then the unexpected behavior is as follows (each action is followed by the resulting output, and a pipe | sign represents the cursor):

1. Type 'b'.
   Input> b|
2. Press TAB (which in my configuration is bound to the auto-completion action).
   Input> bbb_|
3. Press TAB again. Your text will remain the same, but you will receive the following hint:
   bbb_0 bbb_1 bbb_2
Input> bbb_|
4. Type '0' and press TAB.
   bbb_0 bbb_1 bbb_2
Input> bbb_0 |
   Note the space between the '0' character and the cursor (the code snippet below shall explain this).
   So far so good, and trying the same with 'a' will result in similar output, only without an underscore (aaa0, aaa1, aaa2).
5. Start over and type 'c'.
   Input> c
6. Press TAB.
   Input> ccc-
7. Press TAB again.
   aaa0 aaa1 aaa2 bbb_0 bbb_1 bbb_2 ccc-0 ccc-1 ccc-2 ddd?0 ddd?1 ddd?2
Input> ccc-|
   This here is the first half of my problem. All values are displayed, instead of only values beginning with 'ccc-'.
8. Type '0', and press TAB.
   aaa0 aaa1 aaa2 bbb_0 bbb_1 bbb_2 ccc-0 ccc-1 ccc-2 ddd?0 ddd?1 ddd?2
Input> ccc-0|
   This here is the second half of my problem, you see, there's no space between the '0' character and the cursor (again, the snippet below shall explain why a space should be there). In fact, pressing TAB neither changed the text nor displayed a hint, and further TAB presses behave the same.

In practice, what happens in step 7 is a misunderstanding. Readline "mistakes" the dash '-' character for a word-separator (and the same goes for the question-mark '?' character, if you try auto-completing 'ddd?'; other common word separators are, for example: space, tab, '='). So, since the current line buffer ends with a word separator, then it's time for a new word, right? Hence, in step 7 (that's where we are), all values are displayed upon pressing TAB.

In step 8, once the line looks like this "Input> ccc-0|", pressing TAB has no effect because the dash, being a word-separator, separates the line into two words: 'ccc', and '0'. So, the word to be completed is '0', but alas, none of the possible values start with '0', so, no effect.

Now, sadly, there's no right or wrong here. For instance, in my application, an equals-sign '=' actually is a word separator, but a dash '-' isn't. I suppose it must be a matter of configuration, but I haven't found a way to configure which characters separate words. That's what I need help with.

I'm a man of my word, so here's the code snippet I promised:

import readline
values = ['aaa0',  'aaa1',  'aaa2',  'bbb_0', 'bbb_1', 'bbb_2',
          'ccc-0', 'ccc-1', 'ccc-2', 'ddd?0', 'ddd?1', 'ddd?2']
def complete(text, state):
    matches = [v for v in values if v.startswith(text)]
    if len(matches) == 1 and matches[0] == text:
        # Add space if the current text is the same as the only match
        return "{} ".format(matches[0]) if state == 0 else None
    if state >= len(matches):
        return None
    return matches[state]
readline.set_completer(complete)
for line in ("tab: complete", "set show-all-if-unmodified on"):
    readline.parse_and_bind(line)
raw_input("Input> ")

Boys and girls, please - help! I promise to be very thankful, and even return the favor. :)

Thanks very much in advance, Amnon G

Answer 1

Just looking at the output of dir(readline), the functions get_completer_delims() and set_completer_delims() look like they might be useful. In fact, the documentation for the readline module includes:

set_completer_delims(...)
    set_completer_delims(string) -> None
    set the readline word delimiters for tab-completion

I think this describes exactly what you want. This is on Python 2.6.7; if you're running something earlier perhaps this functionality isn't available.