python string join performance

Question

There are a lot of articles around the web concerning Python performance. The first thing you read is concatenating strings should not be done using '+'; avoid s1 + s2 + s3, and instead use str.join

I tried the following: concatenating two strings as part of a directory path: three approaches:

1. '+' which I should not do
2. str.join
3. os.path.join

Here is my code:

import os, time

s1 = '/part/one/of/dir'
s2 = 'part/two/of/dir'
N = 10000

t = time.clock()
for i in xrange(N):
    s = s1 + os.sep + s2
print time.clock() - t

t = time.clock()
for i in xrange(N):
    s = os.sep.join((s1, s2))
print time.clock() - t

t = time.clock()
for i in xrange(N):
    s = os.path.join(s1, s2)
print time.clock() - t

Here the results (Python 2.5 on Windows XP):

0.0182201927899
0.0262544541275
0.120238186697

Shouldn't it be exactly the other way around?

Answer 1

Most of the performance issues with string concatenation are ones of asymptotic performance, so the differences become most significant when you are concatenating many long strings.

In your sample, you are performing the same concatenation many times. You aren't building up any long string, and it may be that the Python interpreter is optimizing your loops. This would explain why the time increases when you move to str.join and path.join - they are more complex functions that are not as easily reduced. (os.path.join does a lot of checking on the strings to see if they need to be rewritten in any way before they are concatenated. This sacrifices some performance for the sake of portability.)

By the way, since file paths are not usually very long, you almost certainly want to use os.path.join for the sake of the portability. If the performance of the concatenation is a problem, you're doing something very odd with your filesystem.