How do I pipe the output of an LS on remote server to the local filesystem via SFTP?

Question

I'm logged into a remote server via SFTP at the command line. The folder I'm in contains hundreds of thousands of files. I need to get a list of these files in a text file so I can access them programmatically, as none of the PHP SFTP clients are able to return such a large list of files.

When I run an ls on the directory ( within the SFTP session ), it takes about 20 minutes for the file list to finally display.

I don't have write access on this server, so I can't pipe the output to a file on the remote server.

How can I pipe the output to a text file on my local machine ... or get a list of the files to my local machine some other way?

Answer 1

If you're willing to wait the 20 minutes for the data to scroll across your screen you can capture all the output using "script".

Call 'script' before you start your ssh or sftp session and it will capture all terminal output to your local disk. Type 'exit' to finish the capture.

NAME
     script -- make typescript of terminal session

SYNOPSIS
     script [-akq] [-t time] [file [command ...]]

DESCRIPTION
     The script utility makes a typescript of everything printed on your ter-
     minal.  It is useful for students who need a hardcopy record of an inter-
     active session as proof of an assignment, as the typescript file can be
     printed out later with lpr(1).

     If the argument file is given, script saves all dialogue in file.  If no
     file name is given, the typescript is saved in the file typescript.

     If the argument command is given, script will run the specified command
     with an optional argument vector instead of an interactive shell.

     The following options are available:

     -a      Append the output to file or typescript, retaining the prior con-
             tents.

     -k      Log keys sent to program as well as output.

     -q      Run in quiet mode, omit the start and stop status messages.

     -t time
             Specify time interval between flushing script output file.  A
             value of 0 causes script to flush for every character I/O event.
             The default interval is 30 seconds.

     The script ends when the forked shell (or command) exits (a control-D to
     exit the Bourne shell (sh(1)), and exit, logout or control-D (if
     ignoreeof is not set) for the C-shell, csh(1)).

     Certain interactive commands, such as vi(1), create garbage in the type-
     script file.  The script utility works best with commands that do not
     manipulate the screen.  The results are meant to emulate a hardcopy ter-
     minal, not an addressable one.

ENVIRONMENT
     The following environment variable is utilized by script:

     SHELL  If the variable SHELL exists, the shell forked by script will be
            that shell.  If SHELL is not set, the Bourne shell is assumed.
            (Most shells set this variable automatically).

SEE ALSO
     csh(1) (for the history mechanism).

HISTORY
     The script command appeared in 3.0BSD.

BUGS
     The script utility places everything in the log file, including linefeeds
     and backspaces.  This is not what the naive user expects.

     It is not possible to specify a command without also naming the script
     file because of argument parsing compatibility issues.

     When running in -k mode, echo cancelling is far from ideal.  The slave
     terminal mode is checked for ECHO mode to check when to avoid manual echo
     logging.  This does not work when in a raw mode where the program being
     run is doing manual echo.

Answer 2

Wu's answer is good if you do it remotely. Here is another option if you are logged onto the remote server and want to send the file back home to yourself:

Proper answer is here: http://scratching.psybermonkey.net/2011/02/ssh-how-to-pipe-output-from-local-to.html

your_command | ssh username@server "cat > filename.txt"