How does the strtok function in C work? [duplicate]

Question

I found this sample program which explains the strtok function:

#include <stdio.h> #include <string.h>  int main () {     char str[] ="- This, a sample string.";     char * pch;     printf ("Splitting string \"%s\" into tokens:\n",str);     pch = strtok (str," ,.-");     while (pch != NULL)     {         printf ("%s\n",pch);         pch = strtok (NULL, " ,.-");     }     return 0; } 

However, I don't see how this is possible to work.

How is it possible that pch = strtok (NULL, " ,.-"); returns a new token. I mean, we are calling strtokwith NULL . This doesen't make a lot sense to me.

Answer 1

Two things to know about strtok. As was mentioned, it "maintains internal state". Also, it messes up the string you feed it. Essentially, it will write a '\0' where it finds the token you supplied, and returns a pointer to the start of the string. Internally it maintains the location of the last token; and next time you call it, it starts from there.

The important corollary is that you cannot use strtok on a const char* "hello world"; type of string, since you will get an access violation when you modify contents of a const char* string.

The "good" thing about strtok is that it doesn't actually copy strings - so you don't need to manage additional memory allocation etc. But unless you understand the above, you will have trouble using it correctly.

Example - if you have "this,is,a,string", successive calls to strtok will generate pointers as follows (the ^ is the value returned). Note that the '\0' is added where the tokens are found; this means the source string is modified:

t  h  i  s  ,  i  s  ,  a  ,  s  t  r  i  n  g \0         this,is,a,string  t  h  i  s  \0 i  s  ,  a  ,  s  t  r  i  n  g \0         this ^ t  h  i  s  \0 i  s  \0 a  ,  s  t  r  i  n  g \0         is                ^ t  h  i  s  \0 i  s  \0 a  \0 s  t  r  i  n  g \0         a                         ^ t  h  i  s  \0 i  s  \0 a  \0 s  t  r  i  n  g \0         string                               ^ 

Hope it makes sense.

Answer 2

strtok maintains internal state. When you call it with non-NULL it re-initializes itself to use the string you supply. When you call it with NULL it uses that string, and any other state its currently got to return the next token.

Because of the way strtok works you need to ensure that you link with a multithreaded version of the C runtime if you're writing a multithreaded application. This will ensure that each thread get its own internal state for strtok.