'string' has incorrect type (expected str, got spacy.tokens.doc.Doc)

Question

I have a dataframe:

train_review = train['review']
train_review

It looks like:

0      With all this stuff going down at the moment w...
1      \The Classic War of the Worlds\" by Timothy Hi...
2      The film starts with a manager (Nicholas Bell)...
3      It must be assumed that those who praised this...
4      Superbly trashy and wondrously unpretentious 8...

I add the tokens into a string:

train_review = train['review']
train_token = ''
for i in train['review']:
   train_token +=i

What I want is to tokenize the reviews using Spacy. Here is what I tried, but I get the following error:

Argument 'string' has incorrect type (expected str, got spacy.tokens.doc.Doc)

How can I solve that? Thanks in advance!

Answer 1

In your for loop you are taking spacy.tokens from your dataframe and appending them to a string, so you should cast it to str. Like this:

train_review = train['review']
train_token = ''
for i in train['review']:
   train_token += str(i)