How do I use the Rust parser (libsyntax) myself?

Question

I want to use the Rust parser (libsyntax) to parse a Rust file and extract information like function names out of it. I started digging in the docs and code, so my first goal is a program that prints all function names of freestanding functions in a .rs file.

The program should expand all macros before it prints the function names, so functions declared via macro aren't missed. That's why I can't write some crappy little parser by myself to do the job.

I have to admit that I'm not yet perfectly good at programming Rust, so I apologize in advance for any stupid statements in this question.

How I understood it I need to do the following steps:

1. Parse the file via the Parser struct
2. Expand macros with MacroExpander
3. ???
4. Use a Visitor to walk the AST and extract the information I need (eg. via visit_fn)

So here are my questions:

1. How do I use MacroExpander?
2. How do I walk the expanded AST with a custom visitor?

I had the idea of using a custom lint check instead of a fully fledged parser. I'm investigating this option.

If it matters, I'm using rustc 0.13.0-nightly (f168c12c5 2014-10-25 20:57:10 +0000)

Answer 1

The syn crate works indeed. At the beginning I wrongly think it is for writing procedural macros (as its readme suggests), but indeed it can parse a source code file. Please look at this page: https://docs.rs/syn/1.0.77/syn/struct.File.html . It even gives an example that inputs a .rs file and output AST (of course, you can do anything with it - not just printing):

use std::env;
use std::fs::File;
use std::io::Read;
use std::process;

fn main() {
    let mut args = env::args();
    let _ = args.next(); // executable name

    let filename = match (args.next(), args.next()) {
        (Some(filename), None) => filename,
        _ => {
            eprintln!("Usage: dump-syntax path/to/filename.rs");
            process::exit(1);
        }
    };

    let mut file = File::open(&filename).expect("Unable to open file");

    let mut src = String::new();
    file.read_to_string(&mut src).expect("Unable to read file");

    let syntax = syn::parse_file(&src).expect("Unable to parse file");

    // Debug impl is available if Syn is built with "extra-traits" feature.
    println!("{:#?}", syntax);
}

Thanks @poolie for pointing out this hint (though lacking a bit of details).

Answer 2

You can use syntex to parse Rust, so you don't need to use unstable Rust.

Here's a simple example:

// Tested against syntex_syntax v0.33
extern crate syntex_syntax as syntax;

use std::rc::Rc;
use syntax::codemap::{CodeMap};
use syntax::errors::{Handler};
use syntax::errors::emitter::{ColorConfig};
use syntax::parse::{self, ParseSess};

fn main() {
    let codemap = Rc::new(CodeMap::new());
    let tty_handler =
        Handler::with_tty_emitter(ColorConfig::Auto, None, true, false, codemap.clone());
    let parse_session = ParseSess::with_span_handler(tty_handler, codemap.clone());

    let src = "fn foo(x: i64) { let y = x + 1; return y; }".to_owned();

    let result = parse::parse_crate_from_source_str(String::new(), src, Vec::new(), &parse_session);
    println!("parse result: {:?}", result);
}

This prints the whole AST:

parse result: Ok(Crate { module: Mod { inner: Span { lo: BytePos(0), hi: BytePos(43), expn_id: ExpnId(4294967295) },
items: [Item { ident: foo#0, attrs: [], id: 4294967295, node: Fn(FnDecl { inputs: [Arg { ty: type(i64), pat:
pat(4294967295: x), id: 4294967295 }], output: Default(Span { lo: BytePos(15), hi: BytePos(15), expn_id: ExpnId(4294967295) }),
variadic: false }, Normal, NotConst, Rust, Generics { lifetimes: [], ty_params: [], where_clause: WhereClause { id:
4294967295, predicates: [] } }, Block { stmts: [stmt(4294967295: let y = x + 1;), stmt(4294967295: return y;)], expr:
None, id: 4294967295, rules: Default, span: Span { lo: BytePos(15), hi: BytePos(43), expn_id: ExpnId(4294967295) } }),
vis: Inherited, span: Span { lo: BytePos(0), hi: BytePos(43), expn_id: ExpnId(4294967295) } }] }, attrs: [], config: [],
span: Span { lo: BytePos(0), hi: BytePos(42), expn_id: ExpnId(4294967295) }, exported_macros: [] })