A good way to store date-based data using python? [closed]

Question

I'm working on a little pet project and I want to store data based on date, preferably using python.

The data to be stored is basically activity logs, i.e. you did a number of activities for some duration each for some date. I expect the number of 'activities' to be in the range 0 - 100 for each date.

I want to store the data using a as close to 'standard' way as possible. I'm somewhat hesitant to what format I should use.

What's a good way to store this type of data, and how can that be done using python (third party libraries and/or services included)?

Edit: I'm fully aware that there's a ton of different ways you can accomplish this, what I'm looking for is the most appropriate way to accomplish it.

Answer 1

First, you want to use ISO 8601 for your dates. It's a no-brainer because it's simple, sorts correctly, and unambiguous to people in any part of the world.

Second, I'd probably store the data in JSON. Depending on your approach, it should have about the same simplicity as pickle, but the data will be usable to programs in other languages. You don't have to worry about the specific representation in the file, by the way, just like you don't need to worry about the intricacies of pickle formats. Your data can be any serializable object (lists, dictionaries, etc.). Based on your described use case, you probably will not have an exceptionally large dataset. Reading or writing the stored data is much simpler with JSON or even pickle. If your program grows, it may be more complicated to work with SQL statements. (Also, JSON is more or less human readable if you need to dig into the file at any point. For me, this is a major plus when working with relatively small data sets.)

For example, I'd do something like this:

import json

with open('file.dat','r') as f:
  data = json.load(f)

Your JSON will always be readable by other software, including stuff not written in Python. Saving the data is just about as easy.

with open('file.dat','w') as f:
  json.dump(data, f)

Read up on the JSON libraries for Python; they're dead simple.

Answer 2

just use sqlite3 and store it as timestamps. this is how 90% (maybe exaggerated) of the applications you encounter (and develop) will store data that they later may need to report (with the exceptions of stuff like logfiles)

import sqlite3,time

db = sqlite3.connect("my_database.sql") #you can put whatever ... created if not exist
conn = db.cursor()

conn.execute("CREATE TABLE IF NOT EXISTS Activities (timestamp int, name text);")

def AddActivity(activityName):
    conn.execute("INSERT INTO Activities (timestamp,name) VALUES (?,?)",(time.time(),activityName))
    db.commit()

def GetAllActivitiesOnDate(month,day,year):
    start_time = time.mktime((year,month,day,0,0,0,0,0,0)) 
    end_time = time.mktime((year,month,day,23,59,0,0,0,0)) #use 1 for last  argument if you live somewhere with dst
    conn.execute("SELECT * FROM Activities WHERE timestamp > ? AND timestamp < ?",(start_time,end_time))
    return conn.fetchall()

then you would do something like

AddActivity("Jumping Jacks")
time.sleep(10)
AddActivity("Push Ups")

import datetime
today = datetime.datetime.now()
activities = GetAllActivitiesOnDate(today.month,today.day,today.year)
print "Found %d Entries"%len(activities)
for activity in activities:
   print "Activity %s @ %s"%(activity[1],time.strftime("%x %X",
                                                 time.gmtime(int(activity[0]))))

this type of storage is highly extensible and easy to query exactly how you need to ... and sqlite is included with python and has a straightforward interface and application. it scales well and if you reach a point where it doesnt scale replacing it with a higher end DB is simple