SaveDefinitions considered dangerous

Question

SaveDefinitions is a nice option of Manipulate. It causes Manipulate to store any definitions used for its creation inside the Manipulate panel. A Manipulate made this way can be copied to an empty notebook and will still work on its own. Additionally, your working notebook containing many such Manipulates also doesn't turn into a flurry of pink boxes with printed error messages below it upon opening. Great!

However, all this goodness has its dark side which can bite you real hard if you are not aware of it. I've had this in a notebook I had been working on for a few days, but I present you with a step-by-step toy example scenario which recreates the problem.

In this scenario you want to create a Manipulate showing a plot of a nice wavy function, so you define this (please make a window size like this, this is important):

The definition is nice, so we keep it for the next time and make it an initialization cell. Next we add the Manipulate, and execute it too.

f[x_] := x^2

Manipulate[
 Plot[n f[x], {x, -3, 3}],
 {n, 1, 4},
 SaveDefinitions -> True
 ]

All works great, the Manipulate really shines, it is a good day.

Just being your paranoid self you check whether the definition is OK:

Yeah, everything still checks out. Fine. But now it occurs to you that a better wavy function would be a sine, so you change the definition, execute, and being paranoid, check:

Everything still fine. You're ready from a day's hard work you save your work and quit. [Quit kernel]

Next day. You start your work again. You evaluate the initialization cells in your notebook. Definition still good? Check.

Now, you scroll down to your Manipulate box (no need to re-execute thanks to the SaveDefinitions), play a little with the slider. And scroll back up.

Being the paranoid you, you once more check the definition of f:

Lo and behold, someone has changed the definition behind your back! And nothing executed between your first and second Information(?) check according to the In[] numbers (In[1]: def of f, In[2] first ?, In[3] second ?).

What happened? Well, it's the Manipulate of course. A FullForm reveals its internal structure:

Manipulate[Plot[n*f[x],{x, -3, 3}],{{n, 2.44}, 1, 4},Initialization:>{f[x_] := x^2}]

There you have the culprit. The initialization part of the box defines f again, but it's the old version because we didn't re-evaluate the Manipulate after modifying its definition. As soon as the manipulate box gets on the screen, it is evaluated and you've got your old definition back. Globally!

Of course, in this toy example it is immediately clear something strange is happening. In my case, I had a larger module in a larger notebook in which I, after some debugging, had changed a small part. It seemed to work, but the next day, the same bug that had bugged me before hit again. It took me a couple of hours before I realized that one of the several Manipulates that I used to study the problem at hand from all sides was doing this.

Clearly, I'm tempted to say, this is unwanted behavior. Now, for the obligatory question: what can we do to prevent this behind-your-back behavior of Manipulate from occurring other than re-executing every Manipulate in your notebook each time you change a definition that might be used by them?

Answer 1

Here is an attempt. The idea is to identify symbols with DownValues or some other ...Values inside your manipulated code, and automatically rename them using unique variables / symbols in place of them. The idea here can be executed rather elegantly with the help of cloning symbols functionality, which I find useful from time to time. The function clone below will clone a given symbol, producing a symbol with the same global definitions:

Clear[GlobalProperties];
GlobalProperties[] :=
  {OwnValues, DownValues, SubValues, UpValues, NValues, FormatValues, 
      Options, DefaultValues, Attributes};


Clear[unique];
unique[sym_] :=
 ToExpression[
    ToString[Unique[sym]] <> 
       StringReplace[StringJoin[ToString /@ Date[]], "." :> ""]];


Attributes[clone] = {HoldAll};
clone[s_Symbol, new_Symbol: Null] :=
  With[{clone = If[new === Null, unique[Unevaluated[s]], ClearAll[new]; new],
        sopts = Options[Unevaluated[s]]},
     With[{setProp = (#[clone] = (#[s] /. HoldPattern[s] :> clone)) &},
        Map[setProp, DeleteCases[GlobalProperties[], Options]];
        If[sopts =!= {}, Options[clone] = (sopts /. HoldPattern[s] :> clone)];
        HoldPattern[s] :> clone]]

There are several alternatives of how to implement the function itself. One is to introduce the function with another name, taking the same arguments as Manipulate, say myManipulate. I will use another one: softly overload Manipulate via UpValues of some custom wrapper, that I will introduce. I will call it CloneSymbols. Here is the code:

ClearAll[CloneSymbols];
CloneSymbols /: 
Manipulate[args___,CloneSymbols[sd:(SaveDefinitions->True)],after:OptionsPattern[]]:=
   Unevaluated[Manipulate[args, sd, after]] /.
     Cases[
       Hold[args],
       s_Symbol /; Flatten[{DownValues[s], SubValues[s], UpValues[s]}] =!= {} :> 
          clone[s],
       Infinity, Heads -> True];

Here is an example of use:

f[x_] := Sin[x];
g[x_] := x^2;

Note that to use the new functionality, one has to wrap the SaveDefinitions->True option in CloneSymbols wrapper:

Manipulate[Plot[ f[n g[x]], {x, -3, 3}], {n, 1, 4}, 
          CloneSymbols[SaveDefinitions -> True]]

This will not affect the definitions of original symbols in the code inside Manipulate, since it were their clones whose definitions have been saved and used in initialization now. We can look at the FullForm for this Manipulate to confirm that:

Manipulate[Plot[f$37782011751740542578125[Times[n,g$37792011751740542587890[x]]],
   List[x,-3,3]],List[List[n,1.9849999999999999`],1,4],RuleDelayed[Initialization,
     List[SetDelayed[f$37782011751740542578125[Pattern[x,Blank[]]],Sin[x]],
       SetDelayed[g$37792011751740542587890[Pattern[x,Blank[]]],Power[x,2]]]]]

In particular, you can change the definitions of functions to say

f[x_]:=Cos[x];
g[x_]:=x;

Then move the slider of the Manipulate produced above, and then check the function definitions

?f
Global`f
f[x_]:=Cos[x]

?g
Global`g
g[x_]:=x

This Manipulate is reasonably independent of anything and can be copied and pasted safely. What happens here is the following: we first find all symbols with non-trivial DownValues, SubValues or UpValues (one can probably add OwnValues as well), and use Cases and clone to create their clones on the fly. We then replace lexically all the cloned symbols with their clones inside Manipulate, and then let Manipulate save the definitions for the clones. In this way, we make a "snapshot" of the functions involved, but do not affect the original functions in any way.

The uniqueness of the clones (symbols) has been addressed with the unique function. Note however, that while the Manipulate-s obtained in this way do not threaten the original function definitions, they will generally still depend on them, so one can not consider them totally independent of anything. One would have to walk down the dependency tree and clone all symbols there, and then reconstruct their inter-dependencies, to construct a fully standalone "snapshot" in Manipulate. This is doable but more complicated.

EDIT

Per request of @Sjoerd, I add code for a case when we do want our Manipulate-s to update to the function's changes, but do not want them to actively interfere and change any global definitions. I suggest a variant of a "pointer" technique: we will again replace function names with new symbols, but, rather than cloning those new symbols after our functions, we will use the Manipulate's Initialization option to simply make those symbols "pointers" to our functions, for example like Initialization:>{new1:=f,new2:=g}. Clearly, re-evaluation of such initialization code can not harm the definitions of f or g, and at the same time our Manipulate-s will become responsive to changes in those definitions.

The first thought is that we could just simply replace function names by new symbols and let Manipulate initialization automatically do the rest. Unfortunately, in that process, it walks the dependency tree, and therefore, the definitions for our functions would also be included - which is what we try to avoid. So, instead, we will explicitly construct the Initialize option. Here is the code:

ClearAll[SavePointers];
SavePointers /: 
Manipulate[args___,SavePointers[sd :(SaveDefinitions->True)],
after:OptionsPattern[]] :=
Module[{init},
  With[{ptrrules = 
    Cases[Hold[args], 
      s_Symbol /; Flatten[{DownValues[s], SubValues[s], UpValues[s]}] =!= {} :> 
         With[{pointer = unique[Unevaluated[s]]},
            pointer := s;
            HoldPattern[s] :> pointer], 
            Infinity, Heads -> True]},
           Hold[ptrrules] /. 
              (Verbatim[HoldPattern][lhs_] :> rhs_ ) :> (rhs := lhs) /. 
               Hold[defs_] :> 
                 ReleaseHold[
                      Hold[Manipulate[args, Initialization :> init, after]] /. 
                            ptrrules /. init :> defs]]]

With the same definitions as before:

ClearAll[f, g];
f[x_] := Sin[x];
g[x_] := x^2;

Here is a FullForm of produced Manipulate:

In[454]:= 
FullForm[Manipulate[Plot[f[n g[x]],{x,-3,3}],{n,1,4},
     SavePointers[SaveDefinitions->True]]]

Out[454]//FullForm=   
Manipulate[Plot[f$3653201175165770507872[Times[n,g$3654201175165770608016[x]]],
List[x,-3,3]],List[n,1,4],RuleDelayed[Initialization,
List[SetDelayed[f$3653201175165770507872,f],SetDelayed[g$3654201175165770608016,g]]]]

The newly generated symbols serve as "pointers" to our functions. The Manipulate-s constructed with this approach, will be responsive for updates in our functions, and at the same time harmless for the main functions' definitions. The price to pay is that they are not self-contained and will not display correctly if the main functions are undefined. So, one can use either CloneSymbols wrapper or SavePointers, depending on what is needed.