save a split dataset in .txt format using pandas

Question

Trying to spit a dataset to train and test, and then need to save it as in .txt format.

Here's the code so far ,

import pandas as pd
from sklearn.model_selection import train_test_split

category=pd.read_csv('dataset.tsv',delimiter='\t',encoding='utf-8')

train, test = train_test_split(category, test_size=0.2)

test.to_csv('checkme.txt')

However, when I try to do so, it gives the error:

Traceback (most recent call last): File "splitter.py", line 8, in test.to_csv('checkme.tsv') File "/home/abc/micro/micro/local/lib/python2.7/site-packages/pandas/core/frame.py", line 1745, in to_csv formatter.save() File "/home/abc/micro/micro/local/lib/python2.7/site-packages/pandas/io/formats/csvs.py", line 171, in save self._save() File "/home/abc/micro/micro/local/lib/python2.7/site-packages/pandas/io/formats/csvs.py", line 286, in _save self._save_chunk(start_i, end_i) File "/home/abc/micro/micro/local/lib/python2.7/site-packages/pandas/io/formats/csvs.py", line 313, in _save_chunk self.cols, self.writer) File "pandas/_libs/writers.pyx", line 64, in pandas._libs.writers.write_csv_rows UnicodeEncodeError: 'ascii' codec can't encode character u'\u026a' in position 111: ordinal not in range(128)

What is possibly wrong here, and how to fix this?

Answer 1

You need to write your dataframe as unicode:

test.to_csv('checkme.txt', sep='\t', encoding='utf-8')