very slow function with two for loops using Arcpy in python

Question

I wrote a code which is working perfectly with the small size data, but when I run it over a dataset with 52000 features, it seems to be stuck in the below function:

def extract_neighboring_OSM_nodes(ref_nodes,cor_nodes):
    time_start=time.time()
    print "here we start finding neighbors at ", time_start
    for ref_node in ref_nodes:
        buffered_node = ref_node[2].buffer(10)
        for cor_node in cor_nodes:
            if cor_node[2].within(buffered_node):
                ref_node[4].append(cor_node[0])
                cor_node[4].append(ref_node[0])
    #        node[4][:] = [cor_nodes.index(x) for x in cor_nodes if x[2].within(buffered_node)]
    time_end=time.time()
    print "neighbor extraction took ", time_end
    return ref_nodes

the ref_node and cor_node are a list of tuples as follows: [(FID, point, geometry, links, neighbors)] neighbors is an empty list which is going to be populated in the above function.

As I said the last message printed out is the first print command in this function. it seems that this function is so slow but for 52000 thousand features it should not take 24 hours, should it? Any Idea where the problem would be or how to make the function faster?

Answer 1

You can try multiprocessing, here is an example - http://pythongisandstuff.wordpress.com/2013/07/31/using-arcpy-with-multiprocessing-%E2%80%93-part-3/.