fast way to network transfer a lot of small files with PHP

Question

I have 2 Linux servers on the same LAN.
Using PHP i need to copy 100000 small (10KB) files from Server A to Server B.

Now i'm using ssh2_scp_send and its very slow(10K files in 20 minutes).

How to make it faster?

Answer 1

Using gzipped TAR through an SSH tunnel is blazing fast. Magnitudes faster than pure scp, particularly with regard to many small files. Here is an example for the linux command line:

user@local# cd /source/ ; tar czf - * | ssh user@remote "cd /target/ ; tar xzf -"

---

Update: As requested, here you go with a pure PHP solution - had some fun fiddling out this tricky bit.

Note: You need PHPs libssh extension for this to work. Also, STDIN seems to be only available when using the stream wrappers for SSH.

This has almost no overhead, because it is operating on streams directly and your CPU is most likely always faster than the network link you are using for the transfer.

To trade of network vs. CPU speed, you can remove the option -z from the command line. (less CPU usage, but more data on the wire)

Code example:

<?php
$local_cmd = "cd /tmp/source && tar -czf - *";
$remote_cmd = "tar -C /tmp/target -xzf -";

$ssh = new SSH_Connection('localhost');
$auth = $ssh->auth_password('gast', 'gast');
$bytes = $ssh->command_pipe($local_cmd, $remote_cmd);
echo "finished: $bytes bytes of data transfered\n";

class SSH_Connection {
    private $link;
    private $auth;

    function __construct ($host, $port=22) {
        $this->link = @ssh2_connect('localhost', 22);
    }  

    function auth_password ($username, $password) {
        if (!is_resource($this->link))
            return false;
        $this->auth = @ssh2_auth_password($this->link, $username, $password); 
        return $this->auth;
    }   

    function command_pipe ($localcmd, $remotecmd) {
        if (!is_resource($this->link) || !$this->auth)
            return false;
        // open remote command stream (STDIN)
        $remote_stream = fopen("ssh2.exec://{$this->link}/$remotecmd", 'rw');
        if (!is_resource($remote_stream))
            return false;
        // open local command stream (STDOUT)
        $local_stream = popen($localcmd, 'r');
        if (!is_resource($local_stream))
            return false;
        // connect both, pipe data from local to remote
        $bytes = 0;
        while (!feof($local_stream)) 
            $bytes += fwrite($remote_stream,fread($local_stream,8192));
        // close streams
        fclose($local_stream);
        fclose($remote_stream);
        return $bytes; 
    }   

    function is_connected () { return is_resource($this->link); }
    function is_authenticated () { return $this->auth; }
}