Python script to get files from one server into another and store them in separate directories?

Question

I am working on server 1. I need to write a Python script where I need to connect to a server 2 and get certain files (files whose name begins with the letters 'HM') from a directory and put them into another directory, which needs to be created at the run time (because for each run of the program, a new directory has to be created and the files must be dumped in there), on server 1.

I need to do this in Python and I'm relatively new to this language. I have no idea where to start with the code. Is there a solution that doesn't involve 'tarring' the files? I have looked through Paramiko but that just transfers one file at a time to my knowledge. I have even looked at glob but I cannot figure out how to use it.

Answer 1

to transfer the files you might wanna check out paramiko

import os
import paramiko

localpath = '~/pathNameForToday/'
os.system('mkdir ' + localpath)
ssh = paramiko.SSHClient() 
ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
ssh.connect(server, username=username, password=password)
sftp = ssh.open_sftp()
sftp.get(remotepath, localpath)
sftp.close()
ssh.close() 

I you wanna use glob you can do this:

import os
import re
import glob

filesiwant = re.compile('^HM.+') #if your files follow a more specific pattern and you don't know regular expressions you can give me a sample name and i'll give you the regex4it
path = '/server2/filedir/'
for infile in glob.glob( os.path.join(path, '*') ):
    if filesiwant.match(infile):
         print "current file is: " + infile

otherwise an easier alternative is to use os.listdir()

import os
for infile in os.listdir('/server2/filedir/'):
    ...`

does that answer your question? if not leave comments

Answer 2

Python wouldn't be my first choice for this task, but you can use calls to the system and run mkdir and rsync. In particular you could do

import os
os.system("mkdir DIRECTORY")
os.system("rsync -cav user@server2:/path/to/files/HM* DIRECTORY/")