heap overflow affecting other programs

Question

I was trying to create the condition for malloc to return a NULL pointer. In the below program, though I can see malloc returning NULL, once the program is forcebly terminated, I see that all other programs are becoming slow and finally I had to reboot the system. So my question is whether the memory for heap is shared with other programs? If not, other programs should not have affected. Is OS is not allocating certain amount of memory at the time of execution? I am using windows 10, Mingw.

#include <stdio.h>
#include <malloc.h>

void mallocInFunction(void)
{
    int *ptr=malloc(500);
    if(ptr==NULL)
    {
        printf("Memory Could not be allocated\n");
    }
    else
    {
        printf("Allocated memory successfully\n");
    }
}


int main (void)
{
    while(1)
    {
        mallocInFunction();
    }

    return(0);
}

Answer 1

So my question is whether the memory for heap is shared with other programs?

Physical memory (RAM) is a resource that is shared by all processes. The operating system makes decisions about how much RAM to allocate to each process and adjusts that over time.

If not, other programs should not have affected. Is OS is not allocating certain amount of memory at the time of execution?

At the time the program starts executing, the operating system has no idea how much memory the program will want or need. Instead, it deals with allocations as they happen. Unless configured otherwise, it will typically do everything it possibly can to allow the program's allocation to succeed because presumably there's a reason the program is doing what it's doing and the operating system won't try to second guess it.

Answer 2

... whether the memory for heap is shared with other programs?

Well, the C standard doesn't exactly require a heap, but in the context of a task-switching, multi-user and multi-threaded OS, of course memory is shared between processes! The C standard doesn't require any of this, but this is all pretty common stuff:

• CPU cache memory tends to be preferred for code that's executed often, though this might get swapped around quite a bit; that may or may not be swapped to a heap.
• Task switching causes registers to be swapped to other forms of memory; that may or may not be swapped to a heap.
• Entire pages are swapped to and from disk, so that other programs can make use of them when your OS switches execution away from your program and to the other programs, and when it's your programs turn to execute again among other reasons. This may or may not involve manipulating the heap.

FWIW, you're referring to memory that has allocated storage duration. It's best to avoid using terms like heap and stack, as they're virtually meaningless. The memory you're referring to is on a silicon chip, regardless of whether it uses a heap or a stack.

... Is OS is not allocating certain amount of memory at the time of execution?

Speaking of silicon chips and execution, your OS likely only has control of one processor (a silicon chip which contains some logic circuits and memory, among other things I'm sure) with which to execute many programs! To summarise this post, yes, your program is most likely sharing those silicon chips with other programs!

On a tangential note, I don't think heap overflow means what you think it means.