Ruby: Are comments tokens?

Question

I've just read here (http://ruby.runpaint.org/programs#lexical) that comments are tokens. I've never thought of comments as tokens as they're either annotations or for a post-processor.

Are comments really tokens or is this source wrong?

Answer 1

Yes, they should be tokens, but ignored by the parser later on. If you do ruby --dump parsetree foo.rb with a file that looks like this

# this is a comment
1+1
# another comment

this is what you'll get:

# @ NODE_SCOPE (line: 3)
# +- nd_tbl: (empty)
# +- nd_args:
# |   (null node)
# +- nd_body:
#     @ NODE_CALL (line: 2)
#     +- nd_mid: :+
#     +- nd_recv:
#     |   @ NODE_LIT (line: 2)
#     |   +- nd_lit: 1
#     +- nd_args:
#         @ NODE_ARRAY (line: 2)
#         +- nd_alen: 1
#         +- nd_head:
#         |   @ NODE_LIT (line: 2)
#         |   +- nd_lit: 1
#         +- nd_next:
#             (null node)