Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Need clarification on the definition of C tokens

Tags:

c

token

From the K&R's "The C Programming Language" book:

There are six classes of tokens: identifiers, keywords, constants, string literals, operators, and other separators. Blanks, horizontal and vertical tabs, newlines, formfeeds, and comments as described below (collectively, "white space") are ignored except as they separate tokens.

What does it mean by "other separators"?

Suppose given a statement:

result = (4 * b - a * b) / 3;

So by the definition, result, a, and b should be identifiers, and =, (, ), *, /, and -should be operators. What about the semicolon, ;? Is it considered a token and if so, what category does it fall into?

Also, as for white spaces, are they considered the "other separators"?

like image 449
vxs8122 Avatar asked Mar 18 '23 00:03

vxs8122


1 Answers

This distinction between operators and other separators is something that existed in old versions of C, but has been removed.

The C89 standard gives this listing of operators and punctuators (punctuators being what K&R calls "other separators"):

operator: one of
        [  ]  (  )  .  ->
        ++  --  &  *  +  -  ~  !  sizeof
        /  %  <<  >>  <  >  <=  >=  ==  !=  ^  |  &&  ||
        ?  :
        =  *=  /=  %=  +=  -=  <<=  >>=  &=  ^=  |=
        ,  #  ##

punctuator: one of
        [  ]  (  )  {  }  *  ,  :  =  ;  ...  #

An operator is defined as something that specifies an operation to be performed, while a punctuator has syntactic significance but does not specify an operation that yields a value.

Note that ( ) [ ] are considered both operators (when used in an expression) or punctuators (e.g. in a function or array declaration).

The C99 standard removes this unnecessary distinction and calls all these symbols "punctuators".

Regarding white-space, it is not considered a token, so is not an operator or punctuator.

like image 160
interjay Avatar answered Mar 27 '23 07:03

interjay