with open('result.txt', 'r') as f:
data = f.read()
print 'What type is my data:'
print type(data)
for i in data:
print "what is i:"
print i
print "what type is i"
print type(i)
print i.encode('utf-8')
I have file with string and I am trying to read the file and split the words by space and save them into a list. Below is my code:
Below is my error messages:
Someone please help!
Update:
I am going to describe what I am trying to do in details here so it give people more context: The goal of what I am trying to do is: 1. Take a Chinese text and break it down into sentences with detecting basic ending punctuations. 2. Take each sentence and use the tool jieba to tokenize characters into meaningful words. For instances, two Chinese character 學,生, will be group together to produce a token '學生' (meaning student). 3. Save all the tokens from the sentence into a list. So the final list will have multiple lists inside as there are multiple sentences in a paragraph.
# coding: utf-8
#encoding=utf-8
import jieba
cutlist = "。!?".decode('utf-8')
test = "【明報專訊】「吉野家」and Peter from US因被誤傳採用日本福島米而要報警澄清,並自爆用內地黑龍江米,日本料理食材來源惹關注。本報以顧客身分向6間日式食店查詢白米產地,其中出售逾200元日式豬扒飯套餐的「勝博殿日式炸豬排」也選用中國大連米,誤以為該店用日本米的食客稱「要諗吓會否再幫襯」,亦有食客稱「好食就得」;壽司店「板長」店員稱採用香港米,公關其後澄清來源地是澳洲,即與平價壽司店「爭鮮」一樣。有飲食界人士稱,雖然日本米較貴、品質較佳,但內地米品質亦有保證。"
#FindToken check whether the character has the ending punctuation
def FindToken(cutlist, char):
if char in cutlist:
return True
else:
return False
'''
cut check each item in a string list, if the item is not the ending punctuation, it will save it to a temporary list called line. When the ending punctuation is encountered it will save the complete sentence that has been collected in the list line into the final list.
'''
def cut(cutlist,test):
l = []
line = []
final = []
'''
check each item in a string list, if the item is not the ending punchuation , it
will save it to a temporary list called line. When the ending punchuation is encountered it will save the complete sentence that has been collected in the list line into the final list.
'''
for i in test:
if i == ' ':
line.append(i)
elif FindToken(cutlist,i):
line.append(i)
l.append(''.join(line))
line = []
else:
line.append(i)
temp = []
#This part iterate each complete sentence and then group characters according to its context.
for i in l:
#This is the function that break down a sentence of characters and group them into phrases
process = list(jieba.cut(i, cut_all=False))
#This is puting all the tokenized character phrases of a sentence into a list. Each sentence
#belong to one list.
for j in process:
temp.append(j.encode('utf-8'))
#temp.append(j)
print temp
final.append(temp)
temp = []
return final
cut(list(cutlist),list(test.decode('utf-8')))
Here is my problem, when I output my final list, it gives me a list of the following result:
[u'\u3010', u'\u660e\u5831', u'\u5c08\u8a0a', u'\u3011', u'\u300c', u'\u5409\u91ce\u5bb6', u'\u300d', u'and', u' ', u'Peter', u' ', u'from', u' ', u'US', u'\u56e0', u'\u88ab', u'\u8aa4\u50b3', u'\u63a1\u7528', u'\u65e5\u672c', u'\u798f\u5cf6', u'\u7c73', u'\u800c', u'\u8981', u'\u5831\u8b66', u'\u6f84\u6e05', u'\uff0c', u'\u4e26', u'\u81ea\u7206', u'\u7528\u5167', u'\u5730', u'\u9ed1\u9f8d', u'\u6c5f\u7c73', u'\uff0c', u'\u65e5\u672c\u6599\u7406', u'\u98df\u6750', u'\u4f86\u6e90', u'\u60f9', u'\u95dc\u6ce8', u'\u3002']
How can I turn a list of unicode into normal string?
Unicode is widely regarded as politically neutral, has good support for both simplified and traditional characters, and can be easily converted to and from the GB and Big5. Furthermore, Unicode has the advantage of not being limited only to Chinese, since it can also display many other character sets.
This means that UTF-8 can represent all of the printable ASCII characters, as well as the non-printable characters. UTF-8 also includes a variety of additional international characters, such as Chinese characters and Arabic characters.
Unicode currently has 74605 CJK characters. CJK characters not only includes characters used by Chinese, but also Japanese Kanji, Korean Hanja, and Vietnamese Chu Nom.
World's simplest unicode tool. This browser-based utility converts fancy Unicode text back to regular text. All Unicode glyphs that you paste or enter in the text area as the input automatically get converted to simple ASCII characters in the output.
Let me give you some hints:
decode()
.)split()
.for i in data
, you're saying you want to iterate over every byte of the file you just read. Each iteration of your loop will be a single character. I'm not sure if that's what you want, because that would mean you'd have to do UTF-8 decoding by hand (rather than using decode()
, which must operate on the entire UTF-8 string.).In other words, here's one line of code that would do it:
open('file.txt').read().decode('utf-8').split()
If this is homework, please don't turn that in. Your teacher will be onto you. ;-)
Edit: Here's an example how to encode and decode unicode characters in python:
>>> data = u"わかりません"
>>> data
u'\u308f\u304b\u308a\u307e\u305b\u3093'
>>> data_you_would_see_in_a_file = data.encode('utf-8')
>>> data_you_would_see_in_a_file
'\xe3\x82\x8f\xe3\x81\x8b\xe3\x82\x8a\xe3\x81\xbe\xe3\x81\x9b\xe3\x82\x93'
>>> for each_unicode_character in data_you_would_see_in_a_file.decode('utf-8'):
... print each_unicode_character
...
わ
か
り
ま
せ
ん
The first thing to note is that Python (well, at least Python 2) uses the u""
notation (note the u
prefix) on string constants to show that they are Unicode. In Python 3, strings are Unicode by default, but you can use b""
if you want a byte string.
As you can see, the Unicode string is composed of two-byte characters. When you read the file, you get a string of one-byte characters (which is equivalent to what you get when you call .encode()
. So if you have bytes from a file, you must call .decode()
to convert them back into Unicode. Then you can iterate over each character.
Splitting "by space" is something unique to every language, since many languages (for example, Chinese and Japanese) do not uses the ' '
character, like most European languages would. I don't know how to do that in Python off the top of my head, but I'm sure there is a way.
When you call encode
on a str
with most (all?) codecs (for which encode
really makes no sense; str
is a byte oriented type, not a true text type like unicode
that would require encoding), Python is implicitly decode
ing it as ASCII first, then encoding with your specified encoding. If you want the str
to be interpreted as something other than ASCII, you need to decode
from bytes-like str
to true text unicode
yourself.
When you do i.encode('utf-8')
when i
is a str
, you're implicitly saying i
is logically text (represented by bytes in the locale default encoding), not binary data. So in order to encode
it, it first needs to decode it to determine what the "logical" text is. Your input is probably encoded in some ASCII
superset (e.g. latin-1
, or even utf-8
), and contains non-ASCII bytes; it tries to decode
them using the ascii
codec (to figure out the true Unicode ordinals it needs to encode as utf-8
), and fails.
You need to do one of:
decode
the str
you read using the correct codec (to get a unicode
object), then encode
that back to utf-8
.open
, import io
and use io.open
(Python 2.7+ only; on Python 3+, io.open
and open
are the same function), which gets you an open
that works like Python 3's open
. You can pass this open
an encoding
argument (e.g. io.open('/path/to/file', 'r', encoding='latin-1')
) and read
ing from the resulting file object will get you already decode
-ed unicode
objects (that can then be encode
-ed to whatever you like with).Note: #1 will not work if the real encoding is something like utf-8
and you defer the work until you're iterating character by character. For non-ASCII characters, utf-8
is multibyte, so if you only have one byte, you can't decode
(because the following bytes are needed to calculate a single ordinal). This is a reason to prefer using io.open
to read as unicode
natively so you're not worrying about stuff like this.
data
is a bytestring (str
type on Python 2). Your loop looks at one byte at a time (non-ascii characters may be represented using more than one byte in utf-8).
Don't call .encode()
on bytes:
$ python2
>>> '\xe3'.enϲodе('utf˗8') #XXX don't do it
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)
I am trying to read the file and split the words by space and save them into a list.
To work with Unicode text, use unicode
type in Python 2. You could use io.open()
to read Unicode text from a file (here's the code that collects all space-separated words into a list):
#!/usr/bin/env python
import io
with io.open('result.txt', encoding='utf-8') as file:
words = [word for line in file for word in line.split()]
print "\n".join(words)
$ python
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import base64
>>> base64.b64encode("我们尊重原创。".encode('utf-8'))
b'5oiR5Lus5bCK6YeN5Y6f5Yib44CC'
>>> import base64
>>> str='5oiR5Lus5bCK6YeN5Y6f5Yib44CC'
>>> base64.b64decode(str)
b'\xe6\x88\x91\xe4\xbb\xac\xe5\xb0\x8a\xe9\x87\x8d\xe5\x8e\x9f\xe5\x88\x9b\xe3\x80\x82'
>>> base64.b64decode(str).decode('utf-8')
'我们尊重原创。'
>>>
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With