I am working on a Web based organisation tool. I am not aiming the same market as the wonderful Basecamp, but let's say the way users and data interact look like the same.
I will have to deal with user customisation, file uploads and graphical tweaks. There is a fora for each account as well. And I'd like to provide a way to backup easily each account.
I have been thinking how to create a reasonable architecture and have been trained to use beautifully normalized data in a single (yet distributed if needed) MySQL DB. Recently I have been wondering : is it possible to think about using one SQLITE DB to store the data for each account and only use MYSQL for the general web site management ?
The pro :
The cons :
More cons that pros, still, it makes me wonder (zepplin style).
What do you say ?
Very large datasets An SQLite database is limited in size to 281 terabytes (248 bytes, 256 tibibytes). And even if it could handle larger databases, SQLite stores the entire database in a single disk file and many filesystems limit the maximum size of files to something less than this.
MySQL has a well-constructed user management system which can handle multiple users and grant various levels of permission. SQLite is suitable for smaller databases. As the database grows the memory requirement also gets larger while using SQLite. Performance optimization is harder when using SQLite.
MySQL, while also easy to use, provides a more robust set of features, better security, better performance at scale and overall better multi-user management. SQLite lacks these features and optimizations.
SQLite database files have a maximum size of about 140 TB. On a phone, the size of the storage (a few GB) will limit your database file size, while the memory size will limit how much data you can retrieve from a query. Furthermore, Android cursors have a limit of 1 MB for the results.
Essentially what your question is is: Which of the above is better for a multi-tenancy SaaS application?
A thousand little sqlite databases may appeal, but may not scale.
I'll address your points in turn:
What happens if an update occurs during your backup? How long does your backup take (with say, a thousand accounts with a million posts each) ? Does your backup lock the database out for the duration of the backup, or does it backup a consistent view of the data, or neither? Does a database backup taken during updates restore correctly in each case? Have you tested these things?
I think backing up a sqlite database isn't as easy as you seem to think, because of the concurrent access issue.
Your implication that the MySQL connection time would be "expensive" may be false. Do you have hard data? Connecting to a server over a LAN does not take very long in practice.
Have you thought about how you'll do a migration if you ever need to change the schema on these 1,000 small databases? What impact will it have on the service?
I'll now add some of my own:
Because sqlite is not a network-based system, you can't just add more web servers. You would need to either partition your users across several servers and ensure that they only ever hit their own "home" server (which is going to introduce some issues but may be viable), or figure out some way to share a sqlite database between servers, which is not going to be pretty, and may well erase any perceived performance advantages that it ever had over (e.g.) MySQL.
I think that scaling a system with a thousand tiny sqlite databases won't scale. In particular, you will probably end up finding that instead of 1,000 tiny ones, you end up with 995 tiny ones, and 5 rather large ones.
Using a dedicated MySQL server will enable you to carry out central backups and migrations. It will enable you to use the resources on that box (i.e. RAM) to cache the most frequently used parts of the database, whichever account they happen to be in.
The RAM used to cache a large MySQL database (e.g. Innodb buffer pool) can be reused between requests and is shared between all of the data (e.g. tables, rows, columns) in it. A sqlite database reads the data from disc each time it's needed, except inside a single session.
My suggestions:
I gave it more thoughts, and I see more problems :
This was a bad idea.
Anyway, writing it on SO forced me to think seriously about it. Better to have a clear mind before starting.
If one day, somebody is as foolish as me, hope he'll hit the page, so it can help him to reconsider
Love this site (even if it's ASP :-p) : quickest way to help yourself, while still helping others.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With