Using a struct to pass parameters by reference

Question

This question is on the subject of passing by reference in M (a related question of mine is here simple question on passing data between functions)

While I was trying to find a way to pass things by reference without using Unevaluted[] or HoldFirst[] , I hit on this method by mistake and it looks really working well for me, even though I do not understand how it works and any hidden risks of using it. So I'd like to show it here to the experts and ask if they think it is safe to use (I have very large demo and needed to package parameters into number of different structs to help manage them, and this is how I found this method while I was trying things).

Here is the method: First we know that one can not write the following:

Remove[p]
foo[p_] := Module[{u},
   u = Table[99, {10}];
   p = u
   ];

p = 0;
foo[p];

One way to to update 'p' in the above is to change to call to become

foo[Unevaluated@p];

Or by defining foo[] with HoldFirst.

But here is the way I found, which does the pass by reference, without either of these:

I put all the parameters in a struct (I do this anyway now), and pass the struct, and then one can update the fields of the struct inside foo[] and the updates will be reflected in the way back from the function call:

Remove[parms]
foo[parms_] := Module[{u},
   u = Table[99, {10}];
   parms["p"] = u
   ];

parms["p"] = 0;
foo[parms];

Now, parms["p"] contained the new list {99, 99, 99, 99, 99, 99, 99, 99, 99, 99}

So, the parms was overwritten/updated inside foo[] without me having to tell M to pass parms by reference !

I tried this in my program, and I see no strange side effects. The CDF was updated ok with no error. I do not know how this works, may be M treats all the fields inside parms in this example as global?

But either case, I am happy with this so far as it provides me a way to package my many parameters into structs, and at the same time I am able to do the updated inside the functions to the struct.

But my question is: do you see a major problems with this method? How does it work internally? I mean how does M handle this passing without me doing HoldFirst or Unevaluated? I know that I lost the ability now to do parameters checking as before, but I can't have everything I want. As I said before, M needs a real build-in struct as part of the language and integrated into it. But this is for another time to talk about.

Btw, the best struct emulation I've see so far was by this one by Leonid Shifrin posted at the end of this thread here but unfortunately I could not use it in my demo as it uses symbols not allowed in a demo CDF.

thanks

Update: Btw, this below is what I think how M handles this. This is just a guess of mine: I think param is some kind of a lookup table, and its fields are just used as an index into it (may be a hash table?) Then param["p1"] will contain in it the address (not the value) of a location in the heap of where the actual value of param["p1"] live.

Something like this:

So, when passing param, then inside the function foo[], when typing param["p1"]=u it will cause the current memory pointed to by "p1" to be freed up, and then a new memory allocated from the heap, and in it the value of u is copied to.

And so, on return back, this is why we see the content of param["p1"] has changed. But what actually changed, is the content of the memory being pointed to by the address which param["p1"] represent. (there is a name, which is "p1", and an address field, which points to the content that the name "p1" represents)

But then this means, that the address itself, which "p1" represents has changed, but the lookup name itself (which is "p1") did not change.

So, as the name itself did not change, there was no need to use HoldFirst, even though the data being pointed to by what this name represent has been modified?

Update:

There is one small glitch in this method, but it is not a big deal to work around it: when passing things using this indexed object method, it turns out that one can't modify part of the object. For example, this below does not work:

foo[param_] := Module[{},
   param["u"][[3]] = 99 (*trying to update PART of u *)
   ];

param["u"] = Table[0, {5}];
foo[param];

The above gives the error

Set::setps: "param[u] in the part assignment is not a symbol"

But the workaround is easy. make a local copy of the whole field that one wants to update part of it, then update (part) of the local copy, then write the copy back to the field, like this

foo[param_] := Module[{u = param["u"]}, (* copy the whole field *)
   u[[3]] = 99;  (*update local copy *)
   param["u"] = u (*now update the field, ok *)
   ];

param["u"] = Table[0, {5}];
foo[param];

Well. It would have been better if one can update part of the field, so no 'special' handling would be needed. But at least the work around is not that bad.

Update For completeness, I thought I mention another tiny thing about using indexed objects and a work around.

I wrote

param[u] = {1, 2, 3}
param[u][[1 ;; -1]]

Which returns {1,2,3} as expected.

Then I found that I can use param@u instead of param[u], which really helped as too many solid brackets are starting to give me a headache.

But then when I typed

param@u[[1 ;; -1]]

expecting to get back the same answer as before, it did not. one gets an error, (I think I know why, operator precedence issue, but that is not the point now), just wanted to say that the workaround is easy, either one can use param[u][[1 ;; -1]] or use (param@u)[[1 ;; -1]]

I like the (param@u)[[1 ;; -1]] more, so that is what I am using now to access lists inside indexed objects.

The notation param@u is as close I can get to the standard record notation which is param.u so I am now happy now with indexed objects, and it seems to be working really well. Just couple of minor things to watch out for.

Answer 1

I am not going to respond to the question of low level data structures as I am simply not qualified to do so.

You are creating "indexed objects" and using immutable strings as indices. Generally speaking I believe that these are similar to hash tables.

Taking your code example, let us remove a possible ambiguity by using a unique name for the argument of foo:

Remove[parms, foo]

foo[thing_] :=
  Module[{u},
    u = Table[99, {10}];
    thing["p"] = u
  ];

parms["p"] = 0;
foo[parms];

parms["p"]

DownValues[parms]

{HoldPattern[parms["p"]] :> {99, 99, 99, 99, 99, 99, 99, 99, 99, 99}}

This shows how the data is stored in terms of high level Mathematica structures.

parms is a global symbol. As you have observed, it can be safely "passed" without anything happening, because it is just a symbol without OwnValues and nothing is triggered when it is evaluated. This is exactly the same as writing/evaluating foo by itself.

The replacement that takes place in foo[thing_] := ... is analogous to With. If we write:

With[{thing = parms}, thing[x]]

thing in thing[x] is replaced with parms before thing[x] evaluates. Likewise, in the code above we get parms["p"] = u before thing["p"] or the Set evaluates. At that time, the HoldFirst attribute of Set takes over, and you get what you want.

Since you are using an immutable string as your index, it is not in danger of changing. As long as you are aware of how to handle non-localized symbols such as parms then there is also no new danger that I see in using this method.