How do I copy a file from one server to another?

Question

I have one server which has nothing but xls log files. Each file is 5-15Mb and it is dynamic in the sense that files get added at any point of time. Now I need a way to do the following process using Ruby.

1. Copy a file by sending the filename from one server which has nothing but log files to another server.
2. I need to pass the server password as an argument.
3. Everything happens in the background, which is triggered from a Ruby script.

Answer 1

Check out the Net::SCP and Net::SSH gems. The first lets you retrieve a file using a secure copy, and the second will let you easily find the names of the files available for retrieval. In Net::SSH, ssh.exec! will be your friend.

From the Net::SCP docs:

Net::SCP implements the SCP (Secure CoPy) client protocol, allowing Ruby programs to securely and programmatically transfer individual files or entire directory trees to and from remote servers. It provides support for multiple simultaneous SCP copies working in parallel over the same connection, as well as for synchronous, serial copies.

Net::SCP also provides an open-uri tie-in, so you can use the Kernel#open method to open and read a remote file:

  # if you want to read from a URL voa SCP:
  require 'uri/open-scp'
  puts open("scp://[email protected]/path/to/file").read

From the Net::SSH docs:

require 'net/ssh'

Net::SSH.start('host', 'user', :password => "password") do |ssh|
  # capture all stderr and stdout output from a remote process
  output = ssh.exec!("hostname")

Add an end to the above code to close the block. Inside the block, output will contain the results of the command you sent.

An alternate to retrieving the files via Ruby from the machine containing the files, would be to have Ruby initiate the transfer directly from the machine hosting the files and push them via scp to the other machine.

Instead of using Net::SCP and Net::SSH, you could use Net::SFTP, to manage it all in one gem. It rides on a secure connection too, but SFTP might not be available to you. The Net::SFTP::Operations::Dir and Net::SFTP::Operations::Download classes and docs will be your friend.

Other options include using the standard rsync in a simple shell as @tadman mentioned. There are a multitude of ways of accomplishing this, and it is a common need in hosting environments.

---

any other better approach?

rsync, at the command-line. It's very smart and can move folders and deltas of files if needed. Also, "How to transfer files using ssh and Ruby" and its link to "Ruby file upload ssh intro".

Melding @tadman's rsync recommendation with Ruby, there's "Cheapest rsync replacement (with Ruby)".

Answer 2

This is how it worked

I used the net-ssh & net-scp gem as suggested by @theTinMan and i was able to copy my files.

require 'rubygems'
require 'net/ssh'
require 'net/scp'

Net::SSH.start("ip_address", "username",:password => "*********") do |session|
  session.scp.download! "/home/logfiles/2-1-2012/login.xls", "/home/anil/Downloads"
end

and to a copy a entire folder

require 'rubygems'
require 'net/ssh'
require 'net/scp'

Net::SSH.start("ip_address", "username",:password => "*********") do |session|
  session.scp.download!("/home/logfiles/2-1-2012", "/home/anil/Downloads",  :recursive => true)
end

Answer 3

You should probably just use rsync instead of rolling your own thing. Use ssh with public/private key access and you will avoid a password. Using a password at all is probably a bad idea.