timeit ValueError: stmt is neither a string nor callable

Question

I played with timeit in Python, got a weird problem.

I define a simple function add. timeit works when I pass add two string parameters. But it raises ValueError: stmt is neither a string nor callable when I pass add two int parameters.

>>> import timeit
>>> def add(x,y):
...     return x + y
... 


>>> a = '1'
>>> b = '2'
>>> timeit.timeit(add(a,b))
0.01355926995165646


>>> a = 1
>>> b = 2
>>> timeit.timeit(add(a,b))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda/lib/python3.6/timeit.py", line 233, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/anaconda/lib/python3.6/timeit.py", line 130, in __init__
    raise ValueError("stmt is neither a string nor callable")
ValueError: stmt is neither a string nor callable

Why does the parameter type matter at all here?

Answer 1

Your mistake is to assume that Python passes the expression add(a, b) to timeit(). That's not the case, add(a, b) is not a string, it is an expression so Python instead executes add(a, b) and the result of that call is passed to the timeit() call.

So for add('1', '2') the result is '12', a string. Passing a string to timeit() is fine. But add(1, 2) is 3, an integer. timeit(3) gives you an exception. Not that timing '12' is all that interesting, of course, but that is a valid Python expression producing the integer value 12:

>>> import timeit
>>> def add(x, y):
...     return x + y
...
>>> a = '1'
>>> b = '2'
>>> add(a, b)
'12'
>>> timeit.timeit('12')
0.009553937998134643
>>> a = 1
>>> b = 2
>>> add(a, b)
3
>>> timeit.timeit(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.7/timeit.py", line 232, in timeit
    return Timer(stmt, setup, timer, globals).timeit(number)
  File "/.../lib/python3.7/timeit.py", line 128, in __init__
    raise ValueError("stmt is neither a string nor callable")
ValueError: stmt is neither a string nor callable

That's all perfectly normal; otherwise, how could you ever pass the result of a function to another function directly? timeit.timeit() is just another Python function, nothing so special that it'll disable the normal evaluation of expressions.

What you want is to pass a string with the expression to timeit(). timeit() doesn't have access to your add() function, or a or b, so you need to give it access with the second argument, the setup string. You can use from __main__ import add, a, b to import the add function object:

timeit.timeit('add(a, b)', 'from __main__ import add, a, b')

Now you get more meaningful results:

>>> import timeit
>>> def add(x, y):
...     return x + y
...
>>> a = '1'
>>> b = '2'
>>> timeit.timeit('add(a, b)', 'from __main__ import add, a, b')
0.16069997000158764
>>> a = 1
>>> b = 2
>>> timeit.timeit('add(a, b)', 'from __main__ import add, a, b')
0.10841095799696632

So adding integers is faster than adding strings. You probably want to try this with different sizes of integers and strings, but adding integers will remain the faster result.

Answer 2

With the string version add returns a string, which timeit can evaluate. So "12" is a valid python expression while 3 is not.

timeit.timeit("12") # works
timeit.timeit(3) # does not

The best way to use timeit, is to wrap the function you want to test with a lambda:

timeit.timeit(lambda: add(1,2))

This is much more elegant and less error prone than dealing with string.

Note that the lambda introduces some slight overhead to each call. If your function does anything remotely complex this will be negligible, but for very simple snippets (like "a+b") the overhead will have a significant impact.