string tokenization in C++ including delimiter characters

Question

I have strings of following form a = x + y or abc = xyz + 5 or 6 + 5 or f(p)

What i need is to tokenize the string such that I read each operator and operand so for a = x + y tokens returns should be a,=,x,+,y and in case of abc=xyz+5 it should return abc,=,xyz,+,5. please note that there may or may not be spaces between operator and operands

this is what I have tried

void tokenize(std::vector<std::string>& tokens, const char* input, const char* delimiters) {
    const char* s = input;
    const char* e = s;
    while (*e != 0) {
        e = s;
        while (*e != 0 && strchr(delimiters, *e) == 0) {
            ++e;
        }
        if ( *e != ' ' && strchr(delimiters, *e) != 0 ){
            std::string op = "";
            op += *e;
            tokens.push_back(op);
        }
        if (e - s > 0) {
            tokens.push_back(std::string(s,e - s));
        }
        s = e + 1;
    }
}

Answer 1

You can use this implementation. First argument is the std::string you want to tokenize, second argument is the delimiter you want to use. It returns a vector of strings tokenized. Very simple yet efficient.

vector<string> tokenizeString(const string& str, const string& delimiters)
{  
   vector<string> tokens;
   // Skip delimiters at beginning.
   string::size_type lastPos = str.find_first_not_of(delimiters, 0);
   // Find first "non-delimiter".
   string::size_type pos = str.find_first_of(delimiters, lastPos);

   while (string::npos != pos || string::npos != lastPos)
    {  // Found a token, add it to the vector.
      tokens.push_back(str.substr(lastPos, pos - lastPos));
      // Skip delimiters.  Note the "not_of"
      lastPos = str.find_first_not_of(delimiters, pos);
      // Find next "non-delimiter"
      pos = str.find_first_of(delimiters, lastPos);
   }
    return tokens;
}