How is strtok removing lines it shouldn't have access to?

Question

I am a beginner to C and have been hitting my head against the same issue for a long time. I have a method that uses strtok to seperate by line, and function called inside that uses strtok to spe by argument (or " "). Calling the second method is causing the next line to become NULL ):

void processLine (const char* line) {
    char* copyLine = strdup(line);
    const char seps[] = " ";
    char* arg = strtok(copyLine, seps);

    // run some methods on first arg that dont effect issue when commented out

    token = strok(NULL, seps);

    // same for 2nd arg
}

void processInput (char* input) {
    const char seps[] = "\n";
    char* line = strtok(input, seps);
    while (line != NULL) {
        printf("%s\n", line);
        char* dummy = strdup(line);
        processLine(dummy);
        line = strtok(NULL, seps);
    }
    printf("end of input\n");
}

int main(void) {
    const char* text = "red IN\nblue IN\n";
    char* input = malloc((strlen(text) + 1) * sizeof(char));
    strcpy(input, text);
    char* line = NULL;
    processInput(input);

    free(input);
    free(line);
    return 0;
}

Output:

red IN
end of input

I have tried to stop processLine() from messing with the input. The input is constant, and I tried using copies in both functions to stop it. Is there an issue with my pointers or memory allocation that is causing this?

Answer 1

The strtok function uses internal static data to keep track of where it is in the string it's parsing. You're using the function at multiple "levels" that are interfering with each other.

You should instead use strtok_r, which takes an additional parameter to keep track of the state:

void processLine (const char* line) {
    char* copyLine = strdup(line);
    const char seps[] = " ";
    char *ptr = NULL;
    char* arg = strtok_r(copyLine, seps, &ptr);

    // run some methods on first arg that dont effect issue when commented out

    token = strok(NULL, seps, &ptr);

    // same for 2nd arg
}

void processInput (char* input) {
    const char seps[] = "\n";
    char *ptr = NULL;
    char* line = strtok_r(input, seps, &ptr);
    while (line != NULL) {
        printf("%s\n", line);
        char* dummy = strdup(line);
        processLine (dummy);
        line = strtok_r(NULL, seps, &ptr);
    }
    printf("end of input\n");
}