I have a large amount of text data I need to import into MySQL. I'm doing this on a MacBook and don't have enough space for it so I want to store it in an external hard drive (I'm not really concerned about speed at this point - this is just for testing).
What's the best way to do it?
If you work with libraries of large image files, or in audio or video production, you need to be selective about your external hard drive. Reading and writing files directly to or from an external drive can incur some hefty performance demands, so it's best to determine your needs before you buy a drive.
You can create an InnoDB table in an external directory by specifying a DATA DIRECTORY clause in the CREATE TABLE statement. CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/external/directory'; The DATA DIRECTORY clause is supported for tables created in file-per-table tablespaces.
MySQL was not designed for running complicated queries against massive data volumes (which requires crunching through a lot of data on a huge scale). MySQL optimizer is quite limited, executing a single query at a time using a single thread.
One simple hack is to create an symbolic link replacing your current mysql database file location pointing to the external disk. Google symbolic link.
sample usage would be after you shutdown mysql, change the old mysql db folder name to something else, and create the symbolic link using the ln command like below
ln -s [EXTERNAL DRIVE PATH] [MYSQL DB FOLDER PATH]
Then move all the previous content of the mysql db folder to the new location.
Open /etc/mysql/my.cnf
and find the value of the datadir
. Alternatively, you can find this out in the mysql monitor with
mysql> select @@datadir;
Stop mysql
sudo systemctl stop mysql
Copy the data from there to your external drive
sudo rsync -av /var/lib/mysql /mnt/myHDD/somedir/mysql
Modify the location of the datadir
in my.cnf
.
Start mysql again
sudo systemctl start mysql
Verify that everything is still fine and remove the original data dir.
This page contains a more extensive guide but all the additional issues it warns about were not relevant for me on my raspberry PI. I.e. I skipped them and it worked.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With