Lowlevel introspection in python3?

Question

Is there some introspection method allowing to reliably obtain the underlying data structure of an object instance, that is unaffected by any customizations?

In Python 3 an object's low-level implementation can be deeply obscured: Attribute lookup can be customized, and even the __dict__ and __slots__ attributes may not give a full picture, as they are writeable. dir() is explicitly meant to show "interesting" attributes rather than actual attributes, and even the inspect module doesn't seem to provide such functionality.

Not a duplicate. This question has been flagged as duplicate of Is there a built-in function to print all the current properties and values of an object?. However, that other question only talks about the standard ways of introspecting classes, which here are explicitly listed as not reliable on a lower level.

As an example consider the following script with an intentionally obscured class.

import inspect

actual_members = None  # <- For showing the actual contents later.

class ObscuredClass:
    def __init__(self):
        global actual_members
        actual_members = dict()
        self.__dict__ = actual_members
        self.actual_field = "actual_value"
    def __getattribute__(self, name):
        if name == "__dict__":
            return { "fake_field": "fake value - shown in __dict__" }
        else:
            return "fake_value - shown in inspect.getmembers()"

obj = ObscuredClass()
print(f"{actual_members          = }")
print(f"{dir(obj)                = }")
print(f"{obj.__dict__            = }")
print(f"{inspect.getmembers(obj) = }")

which produces the output

actual_members          = {'actual_field': 'actual_value'}
dir(obj)                = ['fake_field']
obj.__dict__            = {'fake_field': 'fake value - shown in __dict__'}
inspect.getmembers(obj) = [('fake_field', 'fake_value - shown in inspect.getmembers()')]

Answer 1

There's nothing completely general, particularly for objects implemented in C. Python types just don't store enough instance layout metadata for a general solution. That said, gc.get_referents is pretty reliable even in the face of really weird Python-level modifications, including deleted or shadowed slot descriptors and a deleted or shadowed __dict__ descriptor.

gc.get_referents will give all references an object reports to the garbage collection system. It won't tell you why an object had a particular reference, though - it won't tell you that one dict was __dict__ and one dict was an unrelated slot that happened to have a dict in it.

For example:

import gc

class Foo:
    __slots__ = ('__dict__', 'a', 'b')
    __dict__ = None
    def __init__(self):
        self.x = 1
        self.a = 2
        self.b = 3

x = Foo()
del Foo.a
del Foo.b

print(gc.get_referents(x))

for name in '__dict__', 'x', 'a', 'b':
    try:
        print(name, object.__getattribute__(x, name))
    except AttributeError:
        print('object.__getattribute__ could not look up', name)

This prints

[2, 3, {'x': 1}, <class '__main__.Foo'>]
__dict__ None
x 1
object.__getattribute__ could not look up a
object.__getattribute__ could not look up b

gc.get_referents manages to retrieve the real instance dict and the a and b slots, even when the relevant descriptors are all missing. Unfortunately, it gives no information about the meaning of any references it retrieves.

object.__getattribute__ fails to retrieve the instance dict or the a or b slots. It does manage to find x, because it doesn't rely on the __dict__ descriptor to find the instance dict when retrieving other attributes, but you need to already know x is a name you should look for - object.__getattribute__ can't discover what names you should look for on this object.

Answer 2

user202729 has suggested in a comment to use object.__getattribute__(obj, "field").

Building a custom function around this (using object.__getattribute__(obj,"__dict__") and object.__getattribute__(obj,"__slots__")) seems viable for my original intent of dumping internal data on sparsely documented code.

It has however also been pointed out, that things might be even more obfuscated, or not accessible for classes implemented in C.