Plotting Dates from Numpy Array Problems

Question

I'm plotting a CSV file of weather data, and I got it to import just fine in my code, but i'm trying to plot it. Here's a sample of the CSV data:

12:00am,171,6,7,52,76,77.1,63.7,28.74,0.00,0.00,0.0,0,63.7,78.1,67.4,56.0,29.96
12:01am,192,4,6,52,76,77.1,63.7,28.74,0.00,0.00,0.0,0,63.7,78.1,67.4,56.0,29.96
12:02am,197,3,6,52,76,77.1,63.7,28.74,0.00,0.00,0.0,0,63.7,78.1,67.4,56.0,29.96
12:03am,175,3,6,52,76,77.1,63.7,28.73,0.00,0.00,0.0,0,63.7,78.1,67.4,56.0,29.96
12:04am,194,4,6,52,76,77.1,63.7,28.73,0.00,0.00,0.0,0,63.7,78.1,67.4,56.0,29.96
12:05am,148,5,6,52,76,77.1,63.7,28.73,0.00,0.00,0.0,0,63.7,78.1,67.4,56.0,29.96

Anyway, I'd like the time to be on the X axis, but I can't get it to plot using matplotlib. I tried a method using xticks, and it plotted my y values, but that was it. It just gave me a thick solid line on my X axis.

import matplotlib as mpl
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cbook as cbook
from matplotlib.dates import date2num
import datetime as DT
import re

data = np.genfromtxt('FILE.csv', delimiter=',', dtype=None, skip_header=3)
length = len(data)

x = data['f0']
y = data['f7']

fig = plt.figure()
ax1 = fig.add_subplot(111)
ax1.set_title("Temperature")    
ax1.set_xlabel('Time')
ax1.set_ylabel('Degrees')


#plt.plot_date(x, y)
plt.show()
leg = ax1.legend()

plt.show()

I'm missing a few crucial parts because I honestly don't know where to go from here. I checked the data type of my numpy array, and it kept saying numpy.ndarray, and I can't find a way to convert it to a string or an int value to plot. It's a 24 hour CSV file, and I would like tick marks every 30 minutes or so. Any ideas?

Answer 1

Well, this is not very elegant, but it works. The key is to change the times stored in x, which are just strings, to datetime objects so that matploblib can plot them. I have made a function that does the conversion and called it get_datetime_from_string.

** Edited code to be compatible with Python 2.7 and work with times with single digit hours **

import matplotlib as mpl
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cbook as cbook
from matplotlib.dates import date2num
import datetime as DT
import re

def get_datetime_from_string(time_string):
    ''' Returns a datetime.datetime object

        Args
        time_string: a string of the form 'xx:xxam'
        '''

    # there's got to be a better way to do this.
    # Convert it to utf-8 so string slicing works as expected.
    time_string = unicode(time_string, 'utf-8')

    # period is either am or pm
    colon_position = time_string.find(':')
    period = time_string[-2:]
    hour = int(time_string[:colon_position])
    if period.lower() == 'pm':
        hour += 12

    minute = int(time_string[colon_position + 1:colon_position + 3])

    return DT.datetime(1,1,1,hour, minute)

data = np.genfromtxt('test.csv', delimiter=',', dtype=None, skip_header=3)
length=len(data)

x=data['f0']
y=data['f7']

datetimes = [get_datetime_from_string(t) for t in x]

fig = plt.figure()

ax1 = fig.add_subplot(111)

ax1.set_title("Temperature")    
ax1.set_xlabel('Time')
ax1.set_ylabel('Degrees')

plt.plot(datetimes, y)
leg = ax1.legend()

plt.show()

I kept getting tripped up because I was trying to do string slicing on time_string before converting it to utf-8. Before it was giving me the ASCII values or something. I'm not sure why converting it helped, but it did.

Answer 2

pandas is a very useful library for time series analysis and has some plotting features based on matplotlib.

Pandas uses dateutil internally to parse dates, however the problem is, that the date isn't included in your file. In the code below I assume, that you will know the date before parsing the file (from the file name?)

In [125]: import pandas as pd
In [126]: pd.options.display.mpl_style = 'default'
In [127]: import matplotlib.pyplot as plt

In [128]: class DateParser():                                          
   .....:     def __init__(self, datestring):
   .....:         self.datestring = datestring
   .....:     def get_datetime(self, time):    
   .....:         return dateutil.parser.parse(' '.join([self.datestring, time]))
   .....:     

In [129]: dp = DateParser('2013-01-01')

In [130]: df = pd.read_csv('weather_data.csv', sep=',', index_col=0, header=None,
                  parse_dates={'datetime':[0]}, date_parser=dp.get_datetime)

In [131]: df.ix[:, :12] # show the first columns
Out[131]: 
                      1   2   3   4   5     6     7      8   9   10  11  12  
datetime                                                                      
2013-01-01 00:00:00  171   6   7  52  76  77.1  63.7  28.74   0   0   0   0   
2013-01-01 00:01:00  192   4   6  52  76  77.1  63.7  28.74   0   0   0   0   
2013-01-01 00:02:00  197   3   6  52  76  77.1  63.7  28.74   0   0   0   0   
2013-01-01 00:03:00  175   3   6  52  76  77.1  63.7  28.73   0   0   0   0   
2013-01-01 00:04:00  194   4   6  52  76  77.1  63.7  28.73   0   0   0   0   
2013-01-01 00:05:00  148   5   6  52  76  77.1  63.7  28.73   0   0   0   0   

In [132]: ax = df.ix[:,1:3].plot(secondary_y=1)

In [133]: ax.margins(0.04)

In [134]: plt.tight_layout()

In [135]: plt.savefig('weather_data.png')