Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to quickly export data from R to SQL Server

The standard RODBC package's sqlSave function even as a single INSERT statement (parameter fast = TRUE) is terribly slow for large amounts of data due to non-minimal loading. How would I write data to my SQL server with minimal logging so it writes much more quickly?

Currently trying:

toSQL = data.frame(...);
sqlSave(channel,toSQL,tablename="Table1",rownames=FALSE,colnames=FALSE,safer=FALSE,fast=TRUE);
like image 486
jpd527 Avatar asked Oct 04 '13 21:10

jpd527


People also ask

How do I export data from R server?

To download files from RStudio Workbench or RStudio Server you should take the following steps: Switch to directory you want to download files from within the Files pane. Select the file(s) and/or folder(s) you want to download. Click More -> Export on the toolbar.

How do I export SQL data to R?

Run the script in the R console client from the bin directory. It converts the file into an excel file and saves it into the source file directory. To execute the R script from the SQL Server, embed it into the sp_execute_external_script stored procedure. Here, you need to specify the path to save the excel file.

Can you connect R to SQL Server?

You can connect to SQL Server directly from R using at least 4 libraries (RODBC, rsqlserver, RSQLServer, RJDBC). As long as you have enough RAM, you can import your data into R and do your analysis there using for example amazing dplyr or data. table packages.

Can I export data from R?

There are two ways of exporting data into text files through R. One is using the base R functions and another one is using the functions from the readr package to export data into text/CSV format.


1 Answers

By writing the data to a CSV locally and then using a BULK INSERT (not readily available as a prebuilt function akin to sqlSave), the data can be written to the MS SQL Server very quickly.

toSQL = data.frame(...);
write.table(toSQL,"C:\\export\\filename.txt",quote=FALSE,sep=",",row.names=FALSE,col.names=FALSE,append=FALSE);
    sqlQuery(channel,"BULK
                INSERT Yada.dbo.yada
                FROM '\\\\<server-that-SQL-server-can-see>\\export\\filename.txt'
                WITH
                (
                FIELDTERMINATOR = ',',
                ROWTERMINATOR = '\\n'
                )");

SQL Server must have permission to access the network folder holding the CSV file, or else this process will not work. While it takes some setup with various permissions (the network folder and BULK ADMIN privileges, the reward in speed is infinitely more valuable).

like image 117
jpd527 Avatar answered Oct 29 '22 14:10

jpd527