String Tokenizer with multiple delimiters including delimiter without Boost

Question

I need to create string parser in C++. I tried using

vector<string> Tokenize(const string& strInput, const string& strDelims)
{
 vector<string> vS;

 string strOne = strInput;
 string delimiters = strDelims;

 int startpos = 0;
 int pos = strOne.find_first_of(delimiters, startpos);

 while (string::npos != pos || string::npos != startpos)
 {
  if(strOne.substr(startpos, pos - startpos) != "")
   vS.push_back(strOne.substr(startpos, pos - startpos));

  // if delimiter is a new line (\n) then add new line
  if(strOne.substr(pos, 1) == "\n")
   vS.push_back("\\n");
  // else if the delimiter is not a space
  else if (strOne.substr(pos, 1) != " ")
   vS.push_back(strOne.substr(pos, 1));

  if( string::npos == strOne.find_first_not_of(delimiters, pos) )
   startpos = strOne.find_first_not_of(delimiters, pos);
  else
   startpos = pos + 1;

        pos = strOne.find_first_of(delimiters, startpos);

 }

 return vS;
}

This works for 2X+7cos(3Y)

(tokenizer("2X+7cos(3Y)","+-/^() \t");)

But gives a runtime error for 2X

I need non Boost solution.

I tried using C++ String Toolkit (StrTk) Tokenizer

std::vector<std::string> results;
strtk::split(delimiter, source,
             strtk::range_to_type_back_inserter(results),
             strtk::tokenize_options::include_all_delimiters);

 return results; 

but it doesn't give token as a separate string.

eg: if I give the input as 2X+3Y

output vector contains

2X+

3Y

Answer 1

What's probably happening is this is crashing when passed npos:

lastPos = str.find_first_not_of(delimiters, pos);

Just add breaks to your loop instead of relying on the while clause to break out of it.

if (pos == string::npos)
  break;
lastPos = str.find_first_not_of(delimiters, pos);

if (lastPos == string::npos)
  break;
pos = str.find_first_of(delimiters, lastPos);

Answer 2

Loop exit condition is broken:

while (string::npos != pos || string::npos != startpos)

Allows entry with, say pos = npos and startpos = 1.

So

strOne.substr(startpos, pos - startpos)
strOne.substr(1, npos - 1)

end is not npos, so substr doesn't stop where it should and BOOM!

If pos = npos and startpos = 0,

strOne.substr(startpos, pos - startpos)

lives, but

strOne.substr(pos, 1) == "\n"
strOne.substr(npos, 1) == "\n"

dies. So does

strOne.substr(pos, 1) != " "

Sadly I'm out of time and can't solve this right now, but QuestionC's got the right idea. Better filtering. Something along the lines of:

    if (string::npos != pos)
    {
        if (strOne.substr(pos, 1) == "\n") // can possibly simplify this with strOne[pos] == '\n'
            vS.push_back("\\n");
        // else if the delimiter is not a space
        else if (strOne[pos] != ' ')
            vS.push_back(strOne.substr(pos, 1));
    }