Python: Lazy DB from CSV data?

Question

I've got a relatively small (<100K) numerical CSV dataset that I want to process and graph with some numpy and pylab utilities, and it occurred to me that there's probably a better way of processing the data than ridiculous custom if-ladders for siphoning out the relevent experimental scenarios and comparisons.

If this data were in a DB rather than a CSV this wouldn't be a problem, but throwing together a 'real' db instance for the sake of this seems to be overkill. Is there a pythonic solution to what I'm looking for?

TL;DR Want to query CSV files like a DB / move CSV's into a mini-db.

Answer 1

Without knowing any specific details (at all) of your case, I'll expect that you'll find eventually one of the following ladders as a dominant one for your case:

• Simply just use the built in Python sqlite3.
  • However, if the relational model is not a necessity then pytables may be the way to go on.
    • Perhaps still, structured arrays can provide the necessary functionality.
      • But plenty can be still achieved with just proper usage of plain logic functions.
        • After all, get acquainted to live with your ridiculous custom if-ladders.

Obviously, any of the ladders sketched above will posses its specific pros and cons, depending on the actual case. Thus a really careful mix of them may eventually yield to best 'overall' result.