Parsing namespaces with clang: AST differences in when including a header in another source file or parsing it directly

Question

Sorry for the verbose question, but I can't see any other way to make it clear. I am writing a tool to transform C++ header files to SWIG interface files as a starter for further fine-tuning.

In the process of doing this, I've noticed some strange behavior by clang (v3.0). If I parse the header file I get a significantly different AST than if I parse a source file that includes the header.

For purposes of illustration, here are some sample source files:

Source file:

// example.cpp: Test case for nsbug.py
//
#include "example.h"

Header:

// example.h: Test case for nsbug.py
//
namespace Geom {

struct Location
{
    double x, y;
};

class Shape
{
public:
    Shape();

    void set_location(const Location &where)
    {
        m_pos = where;
    };

    const Location &get_location() const

    // Draw it...
    virtual void draw() const = 0;

protected:
    Location m_pos;
};

class Circle : public Shape
{
    Circle();

    virtual void draw() const;
};
} // namespace Geom

I've used the following Python code to parse it and dump the AST:

# Usage: python nsbug.py <file>

import sys
import clang.cindex

def indent(level):
    """ Indentation string for pretty-printing
    """ 
    return '  '*level

def output_cursor(cursor, level):
    """ Low level cursor output
    """
    spelling = ''
    displayname = ''

    if cursor.spelling:
        spelling = cursor.spelling
    if cursor.displayname:
        displayname = cursor.displayname
    kind = cursor.kind;

    print indent(level) + spelling, '<' + str(kind) + '>'
    print indent(level+1) + '"'  + displayname + '"'

def output_cursor_and_children(cursor, level=0):
    """ Output this cursor and its children with minimal formatting.
    """
    output_cursor(cursor, level)
    if cursor.kind.is_reference():
        print indent(level) + 'reference to:'
        output_cursor(clang.cindex.Cursor_ref(cursor), level+1)

    # Recurse for children of this cursor
    has_children = False;
    for c in cursor.get_children():
        if not has_children:
            print indent(level) + '{'
            has_children = True
        output_cursor_and_children(c, level+1)

    if has_children:
        print indent(level) + '}'

index = clang.cindex.Index.create()
tu = index.parse(sys.argv[1], options=1)

output_cursor_and_children(tu.cursor)

When I run this on example.cpp I get (correctly I think):

 <CursorKind.TRANSLATION_UNIT>
  "example.cpp"
{

  (Deleted lots of clang-generated declarations such as __VERSION__)

  Geom <CursorKind.NAMESPACE>
    "Geom"
  {
    Location <CursorKind.STRUCT_DECL>
      "Location"
    {
      x <CursorKind.FIELD_DECL>
        "x"
      y <CursorKind.FIELD_DECL>
        "y"
    }
    Shape <CursorKind.CLASS_DECL>
      "Shape"
    {
       <CursorKind.CXX_ACCESS_SPEC_DECL>
        ""
       <CursorKind.CXX_ACCESS_SPEC_DECL>
        ""
      Shape <CursorKind.CONSTRUCTOR>
        "Shape()"
      set_location <CursorKind.CXX_METHOD>
        "set_location(const Geom::Location &)"
      {
        where <CursorKind.PARM_DECL>
          "where"
        {
           <CursorKind.TYPE_REF>
            "struct Geom::Location"
          reference to:
            Location <CursorKind.STRUCT_DECL>
              "Location"
        }
         <CursorKind.COMPOUND_STMT>
          ""
        {
           <CursorKind.CALL_EXPR>
            "operator="
          {
             <CursorKind.MEMBER_REF_EXPR>
              "m_pos"
             <CursorKind.UNEXPOSED_EXPR>
              "operator="
            {
               <CursorKind.DECL_REF_EXPR>
                "operator="
            }
             <CursorKind.DECL_REF_EXPR>
              "where"
          }
        }
      }
      get_location <CursorKind.CXX_METHOD>
        "get_location()"
      {
         <CursorKind.TYPE_REF>
          "struct Geom::Location"
        reference to:
          Location <CursorKind.STRUCT_DECL>
            "Location"
      }
       <CursorKind.CXX_ACCESS_SPEC_DECL>
        ""
       <CursorKind.CXX_ACCESS_SPEC_DECL>
        ""
      m_pos <CursorKind.FIELD_DECL>
        "m_pos"
      {
         <CursorKind.TYPE_REF>
          "struct Geom::Location"
        reference to:
          Location <CursorKind.STRUCT_DECL>
            "Location"
      }
    }
    Circle <CursorKind.CLASS_DECL>
      "Circle"
    {
       <CursorKind.CXX_BASE_SPECIFIER>
        "class Geom::Shape"
      reference to:
        Shape <CursorKind.CLASS_DECL>
          "Shape"
      {
         <CursorKind.TYPE_REF>
          "class Geom::Shape"
        reference to:
          Shape <CursorKind.CLASS_DECL>
            "Shape"
      }
      Circle <CursorKind.CONSTRUCTOR>
        "Circle()"
      draw <CursorKind.CXX_METHOD>
        "draw()"
    }
  }
}

But when I try it on the header file along with python nsbug.py example.py I only get:

 <CursorKind.TRANSLATION_UNIT>
  "example.h"
{

  (deleted lots of clang-generated definitions such as __VERSION__)

  Geom <CursorKind.VAR_DECL>
    "Geom"
}

Why does is the Geom name space in the AST as a VAR_DECL? I would have expected no difference, except in the preprocessor cursors.

The work-around is obvious--just make a temp file in memory that includes the header--but that isn't very satisfying. Can someone enlighten me?

Answer 1

Since you aren't explicitly specifying a language, Clang determines the language from the file extension, resulting in "example.h" being parsed as C, not C++. Thus the file is largely ill-formed, and the indexer tries to recover as well as it can. namespace Geom is being treated as a variable declaration for Geom with an unknown type namespace, and the following unexpected { ... } block is skipped.

Try:

tu = index.parse(sys.argv[1], args=['-x', 'c++'])