where are operators mapped to magic methods in python?

Question

I've been reading about magic methods in python, and I've found a lot of info about overriding them and what purpose they serve, but I haven't been able to find where in the language specific operators and actions are mapped to those methods (+ looks for __add__, += looks for __iadd__, creating a new object from a class might call __new__ and __init__, etc.) Is there somewhere I can see what happens when the python interpreter (or whatever lower level mechanism) encounters a plus sign?

Answer 1

Your question is a bit generic. There is a comprehensive list of "special methods", even though it misses some stdlib specific methods(e.g. __setstate__ and __getstate__ used by pickle etc. But it's a protocol of the module pickle not a language protocol).

If you want to know exactly what the interpreter does you can use the dis module to disassemble the bytecode:

>>> import dis
>>> def my_func(a):
...     return a + 2
... 
>>> dis.dis(my_func)
  2           0 LOAD_FAST                0 (a)
              3 LOAD_CONST               1 (2)
              6 BINARY_ADD          
              7 RETURN_VALUE   

You can see that the intereper executes a BINARY_ADD byte code when doing addition. If you want to see exactly the operations that BINARY_ADD does you can download Python's source code and check the ceval.c file:

    case BINARY_ADD:
        w = POP();
        v = TOP();
        if (PyInt_CheckExact(v) && PyInt_CheckExact(w)) {
            /* INLINE: int + int */
            register long a, b, i;
            a = PyInt_AS_LONG(v);
            b = PyInt_AS_LONG(w);
            /* cast to avoid undefined behaviour
               on overflow */
            i = (long)((unsigned long)a + b);
            if ((i^a) < 0 && (i^b) < 0)
                goto slow_add;
            x = PyInt_FromLong(i);
        }
        else if (PyString_CheckExact(v) &&
                 PyString_CheckExact(w)) {
            x = string_concatenate(v, w, f, next_instr);
            /* string_concatenate consumed the ref to v */
            goto skip_decref_vx;
        }
        else {
          slow_add:
            x = PyNumber_Add(v, w);
        }
        Py_DECREF(v);
      skip_decref_vx:
        Py_DECREF(w);
        SET_TOP(x);
        if (x != NULL) continue;
        break;

So here we can see that python special cases int and string additions, and eventually falls back to PyNumber_Add, which checks if the first operand implements __add__ and calls it, eventually it tries __radd__ of the right hand side and if nothing works raises a TypeError.

Note that the byte codes are version-specific, so dis will show different results on different versions:

# python2.7
>>> def my_func():
...     return map((lambda x: x+1), range(5))
... 
>>> dis.dis(my_func)
  2           0 LOAD_GLOBAL              0 (map)
              3 LOAD_CONST               1 (<code object <lambda> at 0x16f8c30, file "<stdin>", line 2>)
              6 MAKE_FUNCTION            0
              9 LOAD_GLOBAL              1 (range)
             12 LOAD_CONST               2 (5)
             15 CALL_FUNCTION            1
             18 CALL_FUNCTION            2
             21 RETURN_VALUE        
# python3
>>> dis.dis(my_func)
  2           0 LOAD_GLOBAL              0 (map) 
              3 LOAD_CONST               1 (<code object <lambda> at 0x7f1161a76930, file "<stdin>", line 2>) 
              6 LOAD_CONST               2 ('my_func.<locals>.<lambda>') 
              9 MAKE_FUNCTION            0 
             12 LOAD_GLOBAL              1 (range) 
             15 LOAD_CONST               3 (5) 
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair) 
             21 CALL_FUNCTION            2 (2 positional, 0 keyword pair) 
             24 RETURN_VALUE  

Also the same byte code may be optimized in future versions, so even if the byte code is the same different versions of python will actually perform different instructions.

If you're interested in learning how python works behind the scenes I'd advise you to write some C extensions, following the tutorials and documentation that you can find on the official python's website.

Answer 2

It's non-trivial to pinpoint the single place in CPython sources mapping operator + to special method __add__ because of the levels of abstraction involved.

As other responded, + is implemented with the BINARY_ADD opcode, which calls PyNumber_Add (except in some specially optimized cases). PyNumber_Add, on the other hand, looks at the tp_as_number member of the type object to get to the PyNumberMethods struct whose nb_add member points to the C function that implements addition.

This is straightforward for built-in types which define their own nb_add directly, but a bit more convoluted for __add__ defined in Python, which needs to be translated to an appropriate nb_add. This part is handled by typeobject.c: when you define a class that implements __add__, the machinery in typeobject.c installs into object->type->tp_as_number->nb_add a generic function that looks up __add__ on the object and calls it to implement the addition. For the case of __add__, this generic function is called slot_nb_add and is defined using the SLOT1BIN macro.

As for __new__ and __init__, they are invoked from the __call__ operator of the type object itself (tp_call in CPython-implementation lingo). This is only logical, since in Python you are calling the type to construct an object.

Answer 3

dis module can somewhat help you on this:

let's take an example of simple list:

In [12]: def func():
    lis=[1,2,3]
    for i in range(5):
        lis+=[i]
   ....:         

In [13]: def func1():
    lis=[1,2,3]
    for i in range(5):
        lis =lis + [i]
   ....:         

In [14]: dis.dis(func)
  2           0 LOAD_CONST               1 (1)
              3 LOAD_CONST               2 (2)
              6 LOAD_CONST               3 (3)

             #removed some lines of code

  4          34 LOAD_FAST                0 (lis)
             37 LOAD_FAST                1 (i)
             40 BUILD_LIST               1
             43 INPLACE_ADD                       # += means inplace add is used
                                                  #     i.e `__iadd()__`
             44 STORE_FAST               0 (lis)
             47 JUMP_ABSOLUTE           28
        >>   50 POP_BLOCK           
        >>   51 LOAD_CONST               0 (None)
             54 RETURN_VALUE        

In [15]: dis.dis(func1)
  2           0 LOAD_CONST               1 (1)
              3 LOAD_CONST               2 (2)
              6 LOAD_CONST               3 (3)
              9 BUILD_LIST               3
             12 STORE_FAST               0 (lis)
             #removed some lines of code    
  4          34 LOAD_FAST                0 (lis)
             37 LOAD_FAST                1 (i)
             40 BUILD_LIST               1
             43 BINARY_ADD                          #normal binary add was used
                                                    #i.e __add__
             44 STORE_FAST               0 (lis)
             47 JUMP_ABSOLUTE           28
        >>   50 POP_BLOCK           
        >>   51 LOAD_CONST               0 (None)
             54 RETURN_VALUE