Why swapping two values based on pointers don't work outside function scope?

Question

I haven't programmed in C++ for a number of years, so I decided to refresh my memories on pointers.

In the classic example of swapping between two numbers, the example is

void swapPointersClassic(double *num1, double *num2) 
{
  double temp;
  temp = *num1;
  *num1 = *num2;
  *num2 = temp;
}

This allows us to make function call like swapPointersClassic(&foo, &bar); and because we pass in the memory addresses of both variables foo and bar, the function will retrieve the values and do the swap. But I started to wonder, why can't I do the following?

void swapPointers(double *num1, double *num2)
{
  double *temp;
  temp = num1;
  num1 = num2;
  num2 = temp;
}

That seems to make more sense to me, because we only have to create enough temporary storage for storing the memory address num1 (instead of a full temporary storage for storing a double value *num1). However, it seems that function scope limits the effect of pointer swapping. After making the call swapPointers(&foo, &bar);, I can see that within the function swapPointers, foo & bar are indeed swapped. Once we exit the swapPointers function, foo and bar are no longer swapped. Can anyone help me understand why this is the case? This behavior reminds me of the typical pass by value approach, but we are passing by pointers here. So that means we can only touch the values pointed by those pointers, but not the pointers themselves?