Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can't override __init__ of class from Cython extension

I am trying to subclass pysam's Tabixfile class and add additional attributes on instantiation.

class MyTabixfile(pysam.Tabixfile):

    def __init__(self, filename, mode='r', *args, **kwargs):
        super().__init__(filename, mode=mode, *args, **kwargs)
        self.x = 'foo'

When I try to instantiate my MyTabixfile subclass, I get a TypeError: object.__init__() takes no parameters:

>>> mt = MyTabixfile('actn2-oligos-forward.tsv.gz')
Traceback (most recent call last):
  File "<ipython-input-11-553015ac7d43>", line 1, in <module>
    mt = MyTabixfile('actn2-oligos-forward.tsv.gz')
  File "mytabix.py", line 4, in __init__
    super().__init__(filename, mode=mode, *args, **kwargs)
TypeError: object.__init__() takes no parameters

I also tried calling the Tabixfile constructor explicitly:

class MyTabixfile(pysam.Tabixfile):

    def __init__(self, filename, mode='r', *args, **kwargs):
        pysam.Tabixfile.__init__(self, filename, mode=mode, *args, **kwargs)
        self.x = 'foo'

but this still raises TypeError: object.__init__() takes no parameters.

This class is actually implemented in Cython; the constructor code is below:

cdef class Tabixfile:
    '''*(filename, mode='r')*

    opens a :term:`tabix file` for reading. A missing
    index (*filename* + ".tbi") will raise an exception.
    '''
    def __cinit__(self, filename, mode = 'r', *args, **kwargs ):
        self.tabixfile = NULL
        self._open( filename, mode, *args, **kwargs )

I read through the Cython documentation on __cinit__ and __init__ which says

Any arguments passed to the constructor will be passed to both the __cinit__() method and the __init__() method. If you anticipate subclassing your extension type in Python, you may find it useful to give the __cinit__() method * and ** arguments so that it can accept and ignore extra arguments. Otherwise, any Python subclass which has an __init__() with a different signature will have to override __new__() 1 as well as __init__(), which the writer of a Python class wouldn’t expect to have to do.

The pysam developers did take the care to add *args and **kwargs to the Tabixfile.__cinit__ method, and my subclass __init__ matches the signature of __cinit__ so I do not understand why I'm unable to override the initialization of Tabixfile.

I'm developing with Python 3.3.1, Cython v.0.19.1, and pysam v.0.7.5.

like image 381
gotgenes Avatar asked Aug 15 '13 19:08

gotgenes


2 Answers

The documentation is a little confusing here, in that it assumes that you're familiar with using __new__ and __init__.

The __cinit__ method is roughly equivalent to a __new__ method in Python.*

Like __new__, __cinit__ is not called by your super().__init__; it's called before Python even gets to your subclass's __init__ method. The reason __cinit__ needs to handle the signature of your subclass __init__ methods is the exact same reason __new__ does.

If your subclass does explicitly call super().__init__, that looks for an __init__ method in a superclass—again, like __new__, a __cinit__ is not an __init__. So, unless you've also defined an __init__, it will pass through to object.


You can see the sequence with the following code.

cinit.pyx:

cdef class Foo:
    def __cinit__(self, a, b, *args, **kw):
        print('Foo.cinit', a, b, args, kw)
    def __init__(self, *args, **kw):
        print('Foo.init', args, kw)

init.py:

import pyximport; pyximport.install()
import cinit

class Bar(cinit.Foo):
    def __new__(cls, *args, **kw):
        print('Bar.new', args, kw)
        return super().__new__(cls, *args, **kw)
    def __init__(self, a, b, c, d):
        print('Bar.init', a, b, c, d)
        super().__init__(a, b, c, d)

b = Bar(1, 2, 3, 4)

When run, you'll see something like:

Bar.new (1, 2, 3, 4) {}
Foo.cinit 1 2 (3, 4) {}
Bar.init 1 2 3 4
Foo.init (1, 2, 3, 4) {}

So, the right fix here depends on what you're trying to do, but it's one of these:

  1. Add an __init__ method to the Cython base class.
  2. Remove the super().__init__ call entirely.
  3. Change the super().__init__ to not pass any params.
  4. Add an appropriate __new__ method to the Python subclass.

I suspect in this case it's #2 you want.


* It's worth noting that __cinit__ definitely isn't identical to __new__. Instead of getting a cls parameter, you get a partially-constructed self object (where you can trust __class__ and C attributes but not Python attributes or methods), the __new__ methods of all classes in the MRO have already been called before any __cinit__; the __cinit__ of your bases gets called automatically instead of manually; you don't get to return a different object besides the one that's been requested; etc. It's just that it's called before the __init__, and expected to take pass-through parameters, in the same way as __new__ is.

like image 199
abarnert Avatar answered Sep 22 '22 09:09

abarnert


I would have commented rather than posting an answer but I don't have enough StackOverflow foo as yet.

@abarnert's post is excellent and very helpful. I would just add a few pysam specifics here as I have just done subclassing on pysam.AlignmentFile in a very similar way.

Option #4 was the cleanest/easiest choice which meant only changes in my own subclass __new__ to filter out the unknown params:

def __new__(cls, file_path, mode, label=None, identifier=None, *args, **kwargs):
    # Suck up label and identifier unknown to pysam.AlignmentFile.__cinit__
    return super().__new__(cls, file_path, mode, *args, **kwargs)

It should also be noted that the pysam file classes don't seem to have explicit __init__ method's, so you also need to omit param pass through as that goes straight to object.__init__ which does not accept parameters:

def __init__(self, label=None, identifier=None, *args, **kwargs):
    # Handle subclass params/attrs here
    # pysam.AlignmentFile doesn't have an __init__ so passes straight through to
    # object which doesn't take params. __cinit__ via new takes care of params
    super(pysam.AlignmentFile, self).__init__()
like image 42
Grrr Avatar answered Sep 24 '22 09:09

Grrr