Architectural design of write heavy application [closed]

Question

I am dealing with live tracking system where one device can push around 2 million GPS points every year (i.e 1 point every 5 seconds, operating for 8 hours over 365 days). If this operates on a global scale with thousands of devices, this results in billions of records per year.

I know SQL Server can handle it. But I need to be able to perform live tracking with thousands of devices performing concurrent writes. It works fine with a few devices but I can see this being CPU intensive when I open lots of tracking sites.

I am planning to try:

• Mongo DB
• Socket approach with kazzing.

Any alternative suggestions?

Answer 1

Given the information you have posted, there is nothing wrong with your architecture. The devil is in the details though. For one, alot depends on how well your DB is designed. It depends on how well written your queries are, db indexes, triggers, etc...

Also, if this is a mobile device of any type, you shouldn't be using a traditional sockets based connector. You cannot depend on a stable tcp connection to a remote server. You should use a stateless architecture like REST to expose/write your data for you. REST is very easy to implement in .NET b.t.w. This should move the scale difficulty from the db, to the web server.

Lastly, to minimize work done on the server, I would implement some sort of caching or buffer pool system to maintain the data on each device for reading, and create a write cache for sending data to the central server. The write cache will be vital seeing as how you cannot depend on a stable tcp connection with transaction management from the server. You need to maintain a cache of data to be written, i.e. (a queue) and pop the queue when you have confirmation from the server that it has received the data you have written. The queue should be popped whenever there is data and a data connection. However, I would need to know more about your requirements before I could say for sure or give more details.