ValueError: time data does not match format '%Y-%m-%d %H:%M:%S.%f'

Question

I am facing one little problem. I am storing some date time data and the data is

# "datetime","numb","temperature"

"1998-04-18 16:48:36.76",0,38
"1998-04-18 16:48:36.8",1,42
"1998-04-18 16:48:36.88",2,23
"1998-04-18 16:48:36.92",3,24
"1998-04-18 16:48:36",4,42
"1998-04-18 16:48:37",5,33
"1998-04-18 16:48:37.08",6,25

the date time column is clearly string, so when I try to convert it , I got this error

ValueError: time data '1998-04-18 16:48:36' does not match format '%Y-%m-%d %H:%M:
%S.%f'

my code is

import time
import datetime
import calendar

for k, line in enumerate(lines):
                if k > (int(header_line)):
                    data_pre = line.strip().split(',')
                    stDate = data_pre[0].replace("\"", "")
                    print stDate  # got 1998-04-18 16:48:36.76


                    dat_time = datetime.datetime.strptime(stDate,
                                                       '%Y-%m-%d %H:%M:%S.%f')
                    mic_sec = dat_time.microsecond
                    timcon = calendar.timegm(dat_time.timetuple())*1000000 + mic_sec
                    strDate = "\"" + strDate + "\""
                    print stDate # got "1998-04-18 16:48:36.76"

because some of my datetime column is missing .%f value, so I got this error. my documents might contains a few thousands such date time values, so I came up with solution to append .0 with all such date time. so that if date time string is

"1998-04-18 16:48:36"

my code should append .0 to fulfill the format criteria. e.g

"1998-04-18 16:48:36.0"

I try to append .0 to stDate, but I get this error

AttributeError: 'str' object has no attribute 'append'

If somebody gives me a clue how to deal with such a problem. Any help would be greatly appreciated.

Answer 1

Update: I've looked through your code and found some misstypes. In addition, it looks like you didn't add in the concatenation.

I have sorted both out.

Mistypes:

You wrote:

for k, line in enumerate(lines):
                if k > (int(header_line)):
                    data_pre = line.strip().split(',')
                    stDate = data_pre[0].replace("\"", "")
                    print stDate  # got 1998-04-18 16:48:36.76


                    dat_time = datetime.datetime.strptime(stDate,
                                                   '%Y-%m-%d %H:%M:%S.%f')
                    mic_sec = dat_time.microsecond
                    timcon = calendar.timegm(dat_time.timetuple())*1000000 + mic_sec

                    strDate = "\"" + strDate + "\""
                    # ^ This line is wrong
                    # It should say: 
                    # strDate = "\"" + stDate + "\""

                    print stDate # got "1998-04-18 16:48:36.76"
                    # ^ This line is wrong
                    # It should say:
                    # print strDate

Implementing the above changes, we can now add the " + ".0" " addition to a sample of your code

(Try running this first, make sure you understand what it is doing, before moving on):

import time
import datetime
import calendar

A = "1998-04-18 16:48:36.76,0,38"
B = "1998-04-18 16:48:37,5,33"

# Run the Code for B

data_pre = B.strip().split(',')
print data_pre

stDate = data_pre[0].replace("\"", "")
print "stDate before: ", stDate  

### Addition of Addition of .0
# Here, we try to convert to datetime format using the format
# '%Y-%m-%d %H:%M:%S.%f'
try:
    dat_time = datetime.datetime.strptime(stDate,
                               '%Y-%m-%d %H:%M:%S.%f')

# If that doesn't work, we add ".4" to the end of stDate
# (You can change this to ".0")
# We then retry to convert stDate into datetime format                                   
except:
    stDate = stDate + ".4"
    dat_time = datetime.datetime.strptime(stDate,
                               '%Y-%m-%d %H:%M:%S.%f')
    print "stDate after: ", stDate

###                                
print "dat_time: ", dat_time

mic_sec = dat_time.microsecond
print "mic_sec: ", mic_sec

timcon = calendar.timegm(dat_time.timetuple())*1000000 + mic_sec
print "timecon: ", timcon

strDate = "\"" + stDate + "\""
print "strDate: ", strDate 

Therefore, for an example:

A = "1998-04-18 16:48:36.76,0,38"
B = "1998-04-18 16:48:37,5,33"
# Note the difference  ^^

# Output for B:
['1998-04-18 16:48:37', '5', '33']
stDate before:  1998-04-18 16:48:37
stDate after:  1998-04-18 16:48:37.4
dat_time:  1998-04-18 16:48:37.400000
mic_sec:  400000
timecon:  892918117400000
strDate:  "1998-04-18 16:48:37.4"

# Output for A:
['1998-04-18 16:48:36.76', '0', '38']
stDate before:  1998-04-18 16:48:36.76
dat_time:  1998-04-18 16:48:36.760000
mic_sec:  760000
timecon:  892918116760000
strDate:  "1998-04-18 16:48:36.76"

Integrated Everything into your main loop. This is what you want overall:

for k, line in enumerate(lines):
                if k > (int(header_line)):
                    data_pre = line.strip().split(',')
                    stDate = data_pre[0].replace("\"", "")
                    print stDate  

                    try:
                        dat_time = datetime.datetime.strptime(stDate,
                               '%Y-%m-%d %H:%M:%S.%f')                                  
                    except:
                        stDate = stDate + ".4"
                        dat_time = datetime.datetime.strptime(stDate,
                               '%Y-%m-%d %H:%M:%S.%f')

                    mic_sec = dat_time.microsecond
                    timcon = calendar.timegm(dat_time.timetuple())*1000000 + mic_sec

                    strDate = "\"" + stDate + "\""
                    # ^ Changed this line
                    print strDate 
                    # ^ Changed this line

Original Answer:

You can't append to a string.

One option would be to use A + B

A = "1998-04-18 16:48:36"
B = ".0"
C = A + B
C = "1998-04-18 16:48:36.0"

You can also use str.join:

D = "".join([A,B])
D = '1998-04-18 16:48:36.0'

For more info, see the answer to this question: Which is the preferred way to concatenate a string in Python?

Answer 2

Instead of formatting datetime with str function, try datetime.datetime.strftime function:

Code that does not work:

>>> import datetime
>>> import pytz
>>> jst = pytz.timezone('Asia/Tokyo')
>>> dt = jst.localize(datetime.datetime.now())
>>> dt
datetime.datetime(2018, 10, 11, 14, 42, 28, 557170, tzinfo=<DstTzInfo 'Asia/Tokyo' JST+9:00:00 STD>)
>>> str(dt)
'2018-10-11 14:42:28.557170+09:00'
>>> dt_new = datetime.datetime.strptime(str(dt), '%Y-%m-%d %H:%M:%S.%f%z')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 565, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py", line 362, in _strptime
    (data_string, format))
ValueError: time data '2018-10-11 14:42:28.557170+09:00' does not match format '%Y-%m-%d %H:%M:%S.%f%z'

Code that works:

>>> import datetime
>>> import pytz
>>> jst = pytz.timezone('Asia/Tokyo')
>>> dt = jst.localize(datetime.datetime.now())
>>> dt
datetime.datetime(2018, 10, 11, 14, 42, 28, 557170, tzinfo=<DstTzInfo 'Asia/Tokyo' JST+9:00:00 STD>)
>>> dt.strftime('%Y-%m-%d %H:%M:%S.%f%z')
'2018-10-11 14:42:28.557170+0900'
>>> dt_new = datetime.datetime.strptime(dt.strftime('%Y-%m-%d %H:%M:%S.%f%z'), '%Y-%m-%d %H:%M:%S.%f%z')
>>> dt_new
datetime.datetime(2018, 10, 11, 14, 42, 28, 557170, 
tzinfo=datetime.timezone(datetime.timedelta(0, 32400)))

So here I formatted the datetime with strftime function and parsed the formatted datetime using strptime function keeping formatter same in both cases.

There is no way to parse the datetime having timezone information and formatted using str(datetime) function.