Sphinx Pygments lexer filter extension?

Question

I have a Lisp-like language I would like to highlight using Pygments in Sphinx code snippet documentation. My approach is to extend the existing CommonLispLexer to add the built-in names using a NameHighlightFilter. However, it is not working, so I must be missing something obvious. I have added the following to my conf.py:

def setup(app): 
    from sphinx.highlighting import lexers
    from pygments.lexers import CommonLispLexer
    from pygments.token import Name
    from pygments.filters import NameHighlightFilter
    tl_lexer = CommonLispLexer()
    tl_lexer.add_filter(NameHighlightFilter(
            names=['define-function', 'define-macro', 
                   'define-variable', 'define-constant'],
            tokentype=Name.Builtin,
            ))
    app.add_lexer('tl', tl_lexer)

highlight_language = 'tl'

But the NameHighlightFilter has no effect. Code blocks are highlighted as if they were Lisp, but my new builtin names have no special highlighting.

Answer 1

The reason is that the NameHighlighFilter only converts tokens that the lexer categorizes as Token.Name, but the CommonLispLexer categorizes almost everything as Name.Variable. This is the filter function of the NameHighlightFilter, from the Pygments source code:

def filter(self, lexer, stream):
    for ttype, value in stream:
        if ttype is Name and value in self.names:
            yield self.tokentype, value
        else:
            yield ttype, value

My only workaround was to write my own filter. This function gave me the look I wanted.

def filter(self, lexer, stream):
    define = False
    for ttype, value in stream:
        if value in self.tl_toplevel_forms:
            ttype = Name.Builtin
            define = True
        elif define and ttype == Name.Variable:
            define = False
            ttype = Name.Function
        elif value in self.tl_special_forms:
            ttype = Name.Variable
        # the Common Lisp lexer highlights everything else as
        # variables, which isn't the look I want.  Instead
        # highlight all non-special things as text.
        elif ttype == Name.Variable:
            ttype = Name.Text
        yield ttype, value

As a note to Pygments developers, perhaps the NameHighlightFilter could take an optional argument representing the token type(s) to be converted (currently it only takes the output token type).