Why can't Rust find method for enum generated using proc_macro_attribute?

Question

I am trying to write procedural macros that will accept a Rust enum like

#[repr(u8)]
enum Ty {
    A,
    B
}

and generate a method for the enum that will let me convert an u8 into an allowed variant like this

fn from_byte(byte: u8) -> Ty {
    match {
        0 => Ty::A,
        1 => Ty::B,
        _ => unreachable!()
    }
}

This is what I have implemented using proc_macro lib. (no external lib)

#![feature(proc_macro_diagnostic)]
#![feature(proc_macro_quote)]
extern crate proc_macro;

use proc_macro::{TokenStream, Diagnostic, Level, TokenTree, Ident, Group, Literal};
use proc_macro::quote;

fn report_error(tt: TokenTree, msg: &str) {
    Diagnostic::spanned(tt.span(), Level::Error, msg).emit();
}

fn variants_from_group(group: Group) -> Vec<Ident> {
    let mut iter = group.stream().into_iter();
    let mut res = vec![];
    while let Some(TokenTree::Ident(id)) = iter.next() {
        match iter.next() {
            Some(TokenTree::Punct(_)) | None => res.push(id),
            Some(tt) => {
                report_error(tt, "unexpected variant. Only unit variants accepted.");
                return res
            }
        }
    }
    res
}

#[proc_macro_attribute]
pub fn procmac(args: TokenStream, input: TokenStream) -> TokenStream {
    let _ = args;
    let mut res = TokenStream::new();
    res.extend(input.clone());
    let mut iter = input.into_iter()
        .skip_while(|tt| if let TokenTree::Punct(_) | TokenTree::Group(_) = tt {true} else {false})
        .skip_while(|tt| tt.to_string() == "pub");
    match iter.next() {
        Some(tt @ TokenTree::Ident(_)) if tt.to_string() == "enum" => (),
        Some(tt) => {
            report_error(tt, "unexpected token. this should be only used with enums");
            return res
        },
        None => return res
    }

    match iter.next() {
        Some(tt) => {
            let variants = match iter.next() {
                Some(TokenTree::Group(g)) => {
                    variants_from_group(g)
                }
                _ => return res
            };
            let mut match_arms = TokenStream::new();
            for (i, v) in variants.into_iter().enumerate() {
                let lhs = TokenTree::Literal(Literal::u8_suffixed(i as u8));
                if i >= u8::MAX as usize {
                    report_error(lhs, "enum can have only u8::MAX variants");
                    return res
                }
                let rhs = TokenTree::Ident(v);
                match_arms.extend(quote! {
                    $lhs => $tt::$rhs,
                })
            }
            res.extend(quote!(impl $tt {
                pub fn from_byte(byte: u8) -> $tt {
                    match byte {
                        $match_arms
                        _ => unreachable!()
                    }
                }
            }))
        }
        _ => ()
    }
    
    res
}

And this is how I am using it.

use helper_macros::procmac;

#[procmac]
#[derive(Debug)]
#[repr(u8)]
enum Ty {
    A,
    B
}

fn main() {
    println!("TEST - {:?}", Ty::from_byte(0))
}

The problem is this causing an error from the compiler. The exact error being

error[E0599]: no variant or associated item named `from_byte` found for enum `Ty` in the current scope
  --> main/src/main.rs:91:32
   |
85 | enum Ty {
   | ------- variant or associated item `from_byte` not found here
...
91 |     println!("TEST - {:?}", Ty::from_byte(0))
   |                                ^^^^^^^^^ variant or associated item not found in `Ty`

Running cargo expand though generate the proper code. And running that code directly works as expected. And so I am stumped. It could be I am missing something about how proc_macros should be used since this is the first time I am playing with them and I don't see anything that would cause this error. I am following the sorted portion of the proc_macro_workshop0. Only change is, I am using TokenStream directly instead of using syn and quote crates. Also, if I mistype the method name, the rust compiler does suggest that a method with similar name exists.

Answer 1

Here is a Playground repro: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=02c1ee77bcd80c68967834a53c011e41

So, indeed what you mention is true: the expanded code could be copy-pasted and it would work. When this happens (having behavior from macro expansion and "manual copy-pasted expansion" differ), there are two possibilities:

• macro_rules! metavariables

  When emitting code using macro_rules! special captures, some of these captures are wrapped with special invisible parenthesis that already tell the parser how the thing inside should be parsed, which make it illegal to use in other places (for instance, one may capture a $Trait:ty, and then doing impl $Trait for ... will fail (it will parse $Trait as a type, thus leading to it being interpreted as a trait object (old syntax)); see also https://github.com/danielhenrymantilla/rust-defile for other examples.

  This is not your case, but it's good to keep in mind (e.g. my initial hunch was that when doing $tt::$rhs if $tt was a :path-like capture, then that could fail).

• macro hygiene/transparency and Spans

  Consider, for instance:

  macro_rules! let_x_42 {() => (
      let x = 42;
  )}
  
  let_x_42!();
  let y = x;
  

  This expands to code that, if copy-pasted, does not fail to compile.

  Basically the name x that the macro uses is "tainted" to be different from any x used outside the macro body, precisely to avoid misinteractions when the macro needs to define helper stuff such as variables.

  And it turns out that this is the same thing that has happened with your from_byte identifier: your code was emitting a from_byte with private hygiene / a def_site() span, which is something that normally never happens for method names when using classic macros, or classic proc-macros (i.e., when not using the unstable ::proc_macro::quote! macro). See this comment: https://github.com/rust-lang/rust/issues/54722#issuecomment-696510769

  And so the from_byte identifier is being "tainted" in a way that allows Rust to make it invisible to code not belonging to that same macro expansion, such as the code in your fn main.

The solution, at this point, is easy: forge a from_bytes Identifier with an explicit non-def_site() Span (e.g., Span::call_site(), or even better: Span::mixed_site() to mimic the rules of macro_rules! macros) so as to prevent it from getting that default def_site() Span that ::proc_macro::quote! uses:

use ::proc_macro::Span;
// ...
let from_byte = TokenTree::from(Ident::new("from_byte", Span::mixed_site()));
res.extend(quote!(impl $tt {
//         use an interpolated ident rather than a "hardcoded one"
//         vvvvvvvvvv
    pub fn $from_byte(byte: u8) -> $tt {
        match byte {
            $match_arms
            _ => unreachable!()
        }
    }
}))

• Playground