Quickly get a list of all svn:externals for a remote svn repository

Question

We have an svn repository with lots of directories and files and our build system needs to be able to find all of the svn:externals properties, recursively for a branch in the repository, before checking it out. Currently we use:

svn propget svn:externals -R http://url.of.repo/Branch

This has proved extremely time consuming and is a real bandwidth hog. It appears that the client is receiving all of the props for everything in the repo and doing the filtering locally (though I haven't confirmed this with wireshark). Is there a faster way to do this? Preferably some way of getting the server to only return the desired data.

Answer 1

As you mentioned, it does consume network bandwidth. However, if you have access to the server where those repositories are hosted, you may run it via file:// protocol. It is proven to be faster and not network consuming.

svn propget svn:externals -R file:///path/to/repo/Branch

Also, if you had the entire working copy in place, you may also run it within your WC.

svn propget svn:externals -R /path/to/WC

Hope it helps you to achieve the results faster!

Answer 2

I finally came up with a solution. I decided to break up the request into multiple small svn requests and then make each of those a task to be run by a thread pool. This kind of slams the svn server, but in our case the svn server is on the LAN and this query is only made during full builds so it doesn't seem to be an issue.

import os
import sys
import threading
import ThreadPool

thread_pool = ThreadPool.ThreadPool(8)
externs_dict = {}
externs_lock = threading.Lock()

def getExternRev( path, url ):
    cmd = 'svn info "%s"' % url
    pipe = os.popen(cmd, 'r')
    data = pipe.read().splitlines()

    #Harvest last changed rev
    for line in data:
        if "Last Changed Rev" in line:
            revision = line.split(":")[1].strip()
            externs_lock.acquire()
            externs_dict[path] = (url, revision)
            externs_lock.release()

def getExterns(url, base_dir):
    cmd = 'svn propget svn:externals "%s"' % url
    pipe = os.popen(cmd, 'r')
    data = pipe.read().splitlines()
    pipe.close()

    for line in data:
        if line:
            line = line.split()
            path = base_dir + line[0]
            url = line[1]
            thread_pool.add_task( getExternRev, path, url )

def processDir(url, base_dir):
    thread_pool.add_task( getExterns, url, base_dir )

    cmd = 'svn list "%s"' % url
    pipe = os.popen(cmd, 'r')
    listing = pipe.read().splitlines()
    pipe.close()

    dir_list = []
    for node in listing:
        if node.endswith('/'):
            dir_list.append(node)

    for node in dir_list:
        #externs_data.extend( analyzePath( url + node, base_dir + node ) )
        thread_pool.add_task( processDir, url+node, base_dir+node )

def analyzePath(url, base_dir = ''):
    thread_pool.add_task( processDir, url, base_dir )
    thread_pool.wait_completion()


analyzePath( "http://url/to/repository" )
print externs_dict

Answer 3

not sure where I found this gem, but it is quite useful to see the externals with own externals:

Windows:
    svn status . | findstr /R "^X"

Linux/Unix:
    svn status . | grep -E "^X"