In case of integer overflows what is the result of (unsigned int) * (int) ? unsigned or int?

Question

In case of integer overflows what is the result of (unsigned int) * (int) ? unsigned or int? What type does the array index operator (operator[]) take for char*: int, unsigned int or something else?

I was auditing the following function, and suddenly this question arose. The function has a vulnerability at line 17.

// Create a character array and initialize it with init[] 
// repeatedly. The size of this character array is specified by 
// w*h.
char *function4(unsigned int w, unsigned int h, char *init)
{
    char *buf;
    int i;

    if (w*h > 4096)
        return (NULL);

    buf = (char *)malloc(4096+1);
    if (!buf)
        return (NULL);

    for (i=0; i<h; i++)
        memcpy(&buf[i*w], init, w);  // line 17

    buf[4096] = '\0';

    return buf;
}

Consider both w and h are very large unsigned integers. The multiplication at line 9 have a chance to pass the validation.

Now the problem is at line 17. Multiply int i with unsigned int w: if the result is int, it is possible that the product is negative, resulting in accessing a position that is before buf. If the result is unsigned int, the product will always be positive, resulting in accessing a position that is after buf.

It's hard to write code to justify this: int is too large. Does anyone has ideas on this?

Is there any documentation that specifies the type of the product? I have searched for it, but so far haven't found anything.

I suppose that as far as the vulnerability is concerned, whether (unsigned int) * (int) produces unsigned int or int doesn't matter, because in the compiled object file, they are just bytes. The following code works the same no matter the type of the product:

unsigned int x = 10;
int y = -10;

printf("%d\n", x * y);  // print x * y in signed integer
printf("%u\n", x * y);  // print x * y in unsigned integer

Therefore, it does not matter what type the multiplication returns. It matters that whether the consumer function takes int or unsigned.

The question here is not how bad the function is, or how to improve the function to make it better. The function undoubtedly has a vulnerability. The question is about the exact behavior of the function, based on the prescribed behavior from the standards.

Answer 1

do the w*h calculation in long long, check if bigger than MAX_UINT

EDIT : alternative : if overflown (w*h)/h != w (is this always the case ?! should be, right ?)

Answer 2

To answer your question: the type of an expression multiplying an int and an unsigned int will be an unsigned int in C/C++.

To answer your implied question, one decent way to deal with possible overflow in integer arithmetic is to use the "IntSafe" set of routines from Microsoft:

http://blogs.msdn.com/michael_howard/archive/2006/02/02/523392.aspx

It's available in the SDK and contains inline implementations so you can study what they're doing if you're on another platform.