Why can I access seemingly arbitrary memory addresses in Python?

Question

Playing with strides in NumPy I realized that you can easily go past the boundaries of arrays:

>>> import numpy as np
>>> from numpy.lib.stride_tricks import as_strided
>>> a = np.array([1], dtype=np.int8)
>>> as_strided(a, shape=(2,), strides=(1,))
array([  1, -28], dtype=int8)

Like this I can read the bytes outside of the array and also write into them. But I don't understand how this is possible. Why doesn't the operating system stop me? It seems I can go at least 100 KB away from this array, before a Segmentation fault is thrown.

The only thing I can think of is that this memory space is directly allocated by my Python process. Does NumPy do this? Is there a fixed size to this space? What other objects can there be?

Answer 1

There are two different memory allocators in play here:

1. The operating system, accessible under Unix with e.g. brk(2) or mmap(2). This will generally give you exactly what you ask for, but it's not very user-friendly.
2. The C runtime heap, accessible with malloc(3) and free(3). This may or may not return freed memory to the operating system immediately. It may also round allocations up to the nearest page, if that is more performant. This is usually implemented in terms of (1).

Most applications, including NumPy and Python, use (2) rather than (1) (or they implement their own memory allocator on top of (2)). As a result, memory that is invalid according to (2) may still be valid according to (1). You only get a segfault if you violate the rules of method (1). It is also possible you are interacting with other live objects on the heap, which has a strong likelihood of causing your program to misbehave in arbitrary ways, even if you are not changing anything.