Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Efficient way of handling large number of data in MySQL

I have a scenario and two options to achieve it. Which one will be more efficient?

I am using mySQL to store attendance of students (around 100 million). And later use this attendance data to plot charts and results based on user’s selection.

Approach.1) Store attendance data of student for each day in new row (which will increase the number of rows exponentially and reduce processing time)

Approach.2) Store serialized or JSON formatted row of one year’s attendance data of each student in a row (Which will increase processing time when updating attendance each day and reduce database size)

like image 672
mr. super cool Avatar asked Feb 25 '16 14:02

mr. super cool


People also ask

How does MySQL handle millions of data?

Show activity on this post. As already mentioned, fetching 2.5 mio entries requires loads of memory / cpu power. Try fetching the records in batches. If that's not solving your problem, you should consider finding a better way to not loop through such an amount of records each time.

How do you handle a large amount of data in a database?

Using cloud storage. Cloud storage is an excellent solution, but it requires the data to be easily shared between multiple servers in order to provide scaling. The NoSQL databases were specially created for using, testing and developing local hardware, and then moving the system to the cloud, where it works.


1 Answers

First I think you are confused, the number of rows will increase linear not exponential that is a big difference.

Second 100k is nothing for a database. even if you store 365 days that is only 36 million, I have that in a week,

Third Store in a JSON may complicated future query.

So I suggest go with approach 1

Using proper Index, design and a fast HDD a db can handle billion of records.

Also you may consider save historic data in a different schema so current data is a little faster, but that is just a minor tuneup

like image 75
Juan Carlos Oropeza Avatar answered Sep 21 '22 12:09

Juan Carlos Oropeza