How to split a text file into words?

Question

I am working on a assignment where I am supposed to read a file and count the number of lines and at the same time count the words in it. I tried a combination of getline and strtok inside a while loop, which did not work.

file:example.txt (the file to be read).

Hi, hello what a pleasant surprise.
Welcome to this place.
May you have a pleasant stay here.
(3 lines, and some words).

Readfile.cpp

#include <iostream>
#include <fstream>
#include<string>
using namespace std;
int main()
{
  ifstream in("example.txt");
  int count = 0;

  if(!in)
  {
    cout << "Cannot open input file.\n";
    return 1;
  }

  char str[255];
  string tok;
  char * t2;

  while(in)
  {
    in.getline(str, 255);
    in>>tok;
    char *dup = strdup(tok.c_str());
    do 
    {
        t2 = strtok(dup," ");
    }while(t2 != NULL);
    cout<<t2<<endl;
    free (dup);
    count++;
  }
  in.close();
  cout<<count;
  return 0;
}

Answer 1

Just got this right!! Just removed all unnecessary code.

int main()
{    
    ifstream in("example.txt");
    int LineCount = 0;
    char* str = new char[500];

    while(in)
    {
        LineCount++;
        in.getline(str, 255);
        char * tempPtr = strtok(str," ");
        while(tempPtr)
        {
            AddWord(tempPtr, LineCount);
            tempPtr = strtok(NULL," ,.");
        }
    }
    in.close();
    delete [] str;
    cout<<"Total No of lines:"<<LineCount<<endl;
    showData();

    return 0;
}

BTW the original problem statement was to create a index program that would accept a user file and create an line-index of all words.

Answer 2

I have not tried compiling this, but here's an alternative that is nearly as simple as using Boost, but without the extra dependency.

#include <iostream>
#include <sstream>
#include <string>

int main() {
  std::string line;
  while (std::getline(std::cin, line)) {
    std::istringstream linestream(line);
    std::string word;
    while (linestream >> word) {
      std::cout << word << "\n";
    }
  }
  return 0;
 }