How do I validate an implementation of python's IOBase?

Question

Is there an existing test suite I can make use of to test an implementation of IOBase? If possible, I would like to exhaustively test a new implementation, but there are a lot of tests that would need to be manually written.

For instance, say I have a non-seekable, non-writeable stream. I want to validate that my implementation of read is compliant and that all other methods raise the right kinds of errors. Such a class might look like this:

import io


class NonSeekableReader(io.RawIOBase):
    def __init__(self, b=b''):
        super(NonSeekableReader, self).__init__()
        self._data = io.BytesIO(b)

    def seekable(self):
        return False

    def writable(self):
        return False

    def readable(self):
        return True

    def read(self, n=-1):
        return self._data.read(n)

Maybe read works as I initially expect, but maybe there are edge cases I would miss with a more complicated implementation. Experimentation has shown that the docs are at the very least confusing as to what expected behavior is.

Based on the python docs I can see that by defining writeable to return False, write should raise an OSError (in Python 3.5). This is actually not true, it will raise NotImplementedError when inheriting from RawIOBase or AttributeError when inheriting from IOBase. This behavior is not duplicated in seekable, i.e. calling .tell() raises the correct error. To fix the issue, I define write like so:

def write(self, b):
    raise io.UnsupportedOperation("write")

It is worth noting that a file opened with 'r' produces the expected error class.

Since there is no general guidance on implementing these interfaces (that I have found) beyond the linked doc page, I would not have necessarily even thought to check for the issue. A pre-existing test suite would have solved that problem. Or otherwise a description of the necessary tests or edge cases would have guided me around these issues.

Does such guidance exist somewhere that I simply have not yet found?

Answer 1

• Python docs are the only official, authoritative source containing promises/guarantees about the stock implementation's behaviour - there's just nothing else officially published. If something is not documented, it doesn't exist should not be relied upon.

  • There's a unit test for the io module, test_io.py, in the source repo (you can also choose to install the test suite with the interpreter). Do watch out for implementation-specific details depending on your answer in the next paragraph.
    • Also watch for details specific to the io implementation of the base classes: as Andrea Corbellini noted, there is a spec for a base class and there is a spec for its derivative.

• Some behaviour is implementation-specific and indeed differs between Python implementations (e.g. in IronPython, sys.maxsize is ridiculously large 'cuz there's no such mechanism in .NET and all strings are Unicode even though it implements Python 2).

  • The question here is: is your target compilance with specification or with a specific implementation? The answer depends on how much (and how well- (or rather, badly, i.e. unportably) written) code you want to be compatible with.
    • Since we're talking about abstract classes here, there's no "specific implementation" really except the little default/non-abstract code in relevant entities. So the difference is rather small.

• Now, there are indeed cases where the spec omits, or is vague on, some critical details (I've personally run into at least two cases in ctypes). That's because it's written as a user's rather than implementor's guide.

  • There's nothing you can do here other than stick to the spec and fix things as you run into them. In vague cases, you can check the implementation but be sure to find out the rationale (what it is conceptually intended to do) to decide what is appropriate in your case. After all, if you're replacing an internal module, how can you know how much undocumented behaviour other internal modules rely upon?
    • To Python devs' credit, the latter cases should be extremely rare as they do honor the loose coupling principle and strive to stick to the spec (where there is a spec) in inter-module interactions.