C - Determining which delimiter used - strtok()

Question

Let's say I'm using strtok() like this..

char *token = strtok(input, ";-/");

Is there a way to figure out which token actually gets used? For instance, if the inputs was something like:

Hello there; How are you? / I'm good - End

Can I figure out which delimiter was used for each token? I need to be able to output a specific message, depending on the delimiter that followed the token.

Answer 1

man 3 strtok

The strtok() and strtok_r() functions return a pointer to the beginning of each subsequent token in the string, after replacing the token itself with a NUL character. When no more tokens remain, a null pointer is returned.

But with a little pointer arithmetic you can do something like:

char* string = "Hello,World!";
char* dup = strdup(string);

char* world = strtok(string, ",");
char delim_used = dup[world - string];

free(dup);

Answer 2

Important: strtok is not re-entrant, you should use strtok_r instead of it.

You can do it by saving a copy of the original string, and looking into offsets of the current token into that copy:

char str[] = "Hello there; How are you? / I'm good - End";
char *copy = strdup(str);
char *delim = ";-/";
char *res = strtok( str, delim );
while (res) {
    printf("%c\n", copy[res-str+strlen(res)]);
    res = strtok( NULL, delim );
}
free(copy);

This prints

;
/
-

Demo #1

EDIT: Handling multiple delimiters

If you need to handle multiple delimiters, determining the length of the current sequence of delimiters becomes slightly harder: now you need to find the next token before deciding how long is the sequence of delimiters. The math is not complicated, as long as you remember that NULL requires special treatment:

char str[] = "(20*(5+(7*2)))+((2+8)*(3+6*9))";
char *copy = strdup(str);
char *delim = "*+()";
char *res = strtok( str, delim );
while (res) {
    int from = res-str+strlen(res);
    res = strtok( NULL, delim );
    int to = res != NULL ? res-str : strlen(copy);
    printf("%.*s\n", to-from, copy+from);
}
free(copy);

Demo #2