I'm starting on a new scientific project which has a lot of data (millions of entries) I'd like to store in an easily and quickly accessible format. I've come across a number of different potential options, but I'm not sure how to pick amongst them. My data can probably just be stored as a dictionary, or potentially a dictionary of dictionaries. Some potential considerations: <ul> <li>Speed. I can't load all the data off disk every time I start a new script, and I'd like as quick access to random entries as possible.</li> <li>Ease-of-use. This is python. The storage should feel like python.</li> <li>Stability/maturity. I'd like something that's currently supported, although something that works well but is still in development would be fine. </li> <li>Ease of installation. My sysadmin should be able to get this running on our cluster.</li> </ul> I don't really care that much about the size of the storage, but it could be a consideration if an option is really terrible on this front. Also, if it matters, I'll most likely be creating the database once, and thereafter only reading from it. Some potential options that I've started looking at (see this post): <ul> <li>pyTables</li> <li>ZopeDB</li> <li>shove</li> <li>shelve</li> <li>redis</li> <li>durus</li> </ul> Any suggestions on which of these might be better for my purposes? Any better ideas? Some of these have a back-end; any suggestions on which file-system back-end would be best?

A RDBMS. Nothing is more realiable than using tables on a well known RDBMS. Postgresql comes to mind. That automatically gives you some choices for the future like clustering. Also you automatically have a lot of tools to administer your database, and you can use it from other software written in virtually any language. It is really fast. In the "feel like python" point, I might add that you can use an ORM. A strong name is sqlalchemy. Maybe with the elixir "extension". Using sqlalchemy you can leave your user/sysadmin choose which database backend he wants to use. Maybe they already have MySql installed - no problem. RDBMSs are still the best choice for data storage.

Comparing persistent storage solutions in python

Tags:

python

orm

persistence

I'm starting on a new scientific project which has a lot of data (millions of entries) I'd like to store in an easily and quickly accessible format. I've come across a number of different potential options, but I'm not sure how to pick amongst them. My data can probably just be stored as a dictionary, or potentially a dictionary of dictionaries. Some potential considerations:

Speed. I can't load all the data off disk every time I start a new script, and I'd like as quick access to random entries as possible.
Ease-of-use. This is python. The storage should feel like python.
Stability/maturity. I'd like something that's currently supported, although something that works well but is still in development would be fine.
Ease of installation. My sysadmin should be able to get this running on our cluster.

I don't really care that much about the size of the storage, but it could be a consideration if an option is really terrible on this front. Also, if it matters, I'll most likely be creating the database once, and thereafter only reading from it.

Some potential options that I've started looking at (see this post):

pyTables
ZopeDB
shove
shelve
redis
durus

Any suggestions on which of these might be better for my purposes? Any better ideas? Some of these have a back-end; any suggestions on which file-system back-end would be best?

534

asked Aug 05 '09 20:08

Noah

3 Answers

Might want to give mongodb a shot - the PyMongo library works with dictionaries and supports most Python types. Easy to install, very performant + scalable. MongoDB (and PyMongo) is also used in production at some big names.

107

answered Oct 07 '22 07:10

mdirolf

A RDBMS.

Nothing is more realiable than using tables on a well known RDBMS. Postgresql comes to mind.

That automatically gives you some choices for the future like clustering. Also you automatically have a lot of tools to administer your database, and you can use it from other software written in virtually any language.

It is really fast.

In the "feel like python" point, I might add that you can use an ORM. A strong name is sqlalchemy. Maybe with the elixir "extension".

Using sqlalchemy you can leave your user/sysadmin choose which database backend he wants to use. Maybe they already have MySql installed - no problem.

RDBMSs are still the best choice for data storage.

answered Oct 07 '22 08:10

nosklo

I'm working on such a project and I'm using SQLite.

SQLite stores everything in one file and is part of Python's standard library. Hence, installation and configuration is virtually for free (ease of installation).

You can easily manage the database file with small Python scripts or via various tools. There is also a Firefox plugin (ease of installation / ease-of-use).

I find it very convenient to use SQL to filter/sort/manipulate/... the data. Although, I'm not an SQL expert. (ease-of-use)

I'm not sure if SQLite is the fastes DB system for this work and it lacks some features you might need e.g. stored procedures.

Anyway, SQLite works for me.

answered Oct 07 '22 08:10

wierob

Related questions
                            
                                Developing PyPy's Rpython as a general programming language [closed]
                            
                                Run python script without the "python" keyword
                            
                                What does the symbol "#!" mean in Python?
                            
                                Applying the Sobel filter using scipy
                            
                                Script to show progress?
                            
                                python is not recognised as an internal or external command [duplicate]
                            
                                Custom list_editable field in django admin change list, which doesn't correspond directly to a model field
                            
                                Python: Assigning "through" an iterator
                            
                                tkinter.TclError: image "pyimage3" doesn't exist
                            
                                Only download a part of the document using python requests
                            
                                Pythonic way to convert the string "None" to a proper None
                            
                                Python: Start and stop timer [duplicate]
                            
                                disable default certificate verification in python 2.7.9
                            
                                Rounding a list of floats into integers in Python
                            
                                How to change size of plot in xgboost.plot_importance?
                            
                                How to add quotes " " to every element in non numeric column of a dataframe saved as CSV
                            
                                Anaconda is not creating full environment
                            
                                Why agg() in PySpark is only able to summarize one column at a time? [duplicate]
                            
                                key error not in index while cross validation
                            
                                When using random, which form returns an equal 50% chance?

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!

Donate Us With