The standard RODBC package's sqlSave
function even as a single INSERT
statement (parameter fast = TRUE
) is terribly slow for large amounts of data due to non-minimal loading. How would I write data to my SQL server with minimal logging so it writes much more quickly?
Currently trying:
toSQL = data.frame(...);
sqlSave(channel,toSQL,tablename="Table1",rownames=FALSE,colnames=FALSE,safer=FALSE,fast=TRUE);
To download files from RStudio Workbench or RStudio Server you should take the following steps: Switch to directory you want to download files from within the Files pane. Select the file(s) and/or folder(s) you want to download. Click More -> Export on the toolbar.
Run the script in the R console client from the bin directory. It converts the file into an excel file and saves it into the source file directory. To execute the R script from the SQL Server, embed it into the sp_execute_external_script stored procedure. Here, you need to specify the path to save the excel file.
You can connect to SQL Server directly from R using at least 4 libraries (RODBC, rsqlserver, RSQLServer, RJDBC). As long as you have enough RAM, you can import your data into R and do your analysis there using for example amazing dplyr or data. table packages.
There are two ways of exporting data into text files through R. One is using the base R functions and another one is using the functions from the readr package to export data into text/CSV format.
By writing the data to a CSV locally and then using a BULK INSERT
(not readily available as a prebuilt function akin to sqlSave
), the data can be written to the MS SQL Server very quickly.
toSQL = data.frame(...);
write.table(toSQL,"C:\\export\\filename.txt",quote=FALSE,sep=",",row.names=FALSE,col.names=FALSE,append=FALSE);
sqlQuery(channel,"BULK
INSERT Yada.dbo.yada
FROM '\\\\<server-that-SQL-server-can-see>\\export\\filename.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\\n'
)");
SQL Server must have permission to access the network folder holding the CSV file, or else this process will not work. While it takes some setup with various permissions (the network folder and BULK ADMIN
privileges, the reward in speed is infinitely more valuable).
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With