strtok segmentation fault

Question

I am trying to understand why the following snippet of code is giving a segmentation fault:

void tokenize(char* line)
{
   char* cmd = strtok(line," ");

   while (cmd != NULL)
   {
        printf ("%s\n",cmd);
        cmd = strtok(NULL, " ");
   } 
}

int main(void)
{
   tokenize("this is a test");
}

I know that strtok() does not actually tokenize on string literals, but in this case, line points directly to the string "this is a test" which is internally an array of char. Is there any of tokenizing line without copying it into an array?

Answer 1

The problem is that you're attempting to modify a string literal. Doing so causes your program's behavior to be undefined.

Saying that you're not allowed to modify a string literal is an oversimplification. Saying that string literals are const is incorrect; they're not.

WARNING : Digression follows.

The string literal "this is a test" is of an expression of type char[15] (14 for the length, plus 1 for the terminating '\0'). In most contexts, including this one, such an expression is implicitly converted to a pointer to the first element of the array, of type char*.

The behavior of attempting to modify the array referred to by a string literal is undefined -- not because it's const (it isn't), but because the C standard specifically says that it's undefined.

Some compilers might permit you to get away with this. Your code might actually modify the static array corresponding to the literal (which could cause great confusion later on).

Most modern compilers, though, will store the array in read-only memory -- not physical ROM, but in a region of memory that's protected from modification by the virtual memory system. The result of attempting to modify such memory is typically a segmentation fault and a program crash.

So why aren't string literals const? Since you really shouldn't try to modify them, it would certainly make sense -- and C++ does make string literals const. The reason is historical. The const keyword didn't exist before it was introduced by the 1989 ANSI C standard (though it was probably implemented by some compilers before that). So a pre-ANSI program might look like this:

#include <stdio.h>

print_string(s)
char *s;
{
    printf("%s\n", s);
}

main()
{
    print_string("Hello, world");
}

There was no way to enforce the fact that print_string isn't allowed to modify the string pointed to by s. Making string literals const in ANSI C would have broken existing code, which the ANSI C committee tried very hard to avoid doing. There hasn't been a good opportunity since then to make such a change to the language. (The designers of C++, mostly Bjarne Stroustrup, weren't as concerned about backward compatibility with C.)

Answer 2

There's a very good reason that trying to tokenize a compile-time constant string will cause a segmentation fault: the constant string is in read-only memory.

The C compiler bakes compile-time constant strings into the executable, and the operating system loads them into read-only memory (.rodata in a *nix ELF file). Since this memory is marked as read-only, and since strtok writes into the string that you pass into it, you get a segmentation fault for writing into read-only memory.