Unwanted evaluation in assignments in Mathematica: why it happens and how to debug it during the package-loading?

Question

I am developing a (large) package which does not load properly anymore. This happened after I changed a single line of code. When I attempt to load the package (with Needs), the package starts loading and then one of the setdelayed definitions “comes alive” (ie. Is somehow evaluated), gets trapped in an error trapping routine loaded a few lines before and the package loading aborts.
The error trapping routine with abort is doing its job, except that it should not have been called in the first place, during the package loading phase. The error message reveals that the wrong argument is in fact a pattern expression which I use on the lhs of a setdelayed definition a few lines later.

Something like this:

……Some code lines

Changed line of code 

g[x_?NotGoodQ]:=(Message[g::nogood, x];Abort[])

……..some other code lines

g/: cccQ[g[x0_]]:=True

When I attempt to load the package, I get:

g::nogood: Argument x0_ is not good

As you see the passed argument is a pattern and it can only come from the code line above.

I tried to find the reason for this behavior, but I have been unsuccessful so far. So I decided to use the powerful Workbench debugging tools .

I would like to see step by step (or with breakpoints) what happens when I load the package. I am not yet too familiar with WB, but it seems that ,using Debug as…, the package is first loaded and then eventually debugged with breakpoints, ect. My problem is that the package does not even load completely! And any breakpoint set before loading the package does not seem to be effective.

So…2 questions:

1. can anybody please explain why these code lines "come alive" during package loading? (there are no obvious syntax errors or code fragments left in the package as far as I can see)
2. can anybody please explain how (if) is possible to examine/debug package code while being loaded in WB?

Thank you for any help.

Edit

In light of Leonid's answer and using his EvenQ example: We can avoid using Holdpattern simply by definying upvalues for g BEFORE downvalues for g

notGoodQ[x_] := EvenQ[x];
Clear[g];
g /: cccQ[g[x0_]] := True
g[x_?notGoodQ] := (Message[g::nogood, x]; Abort[])

Now

?g

Global`g

cccQ[g[x0_]]^:=True



g[x_?notGoodQ]:=(Message[g::nogood,x];Abort[])



In[6]:= cccQ[g[1]]

Out[6]= True

while

In[7]:= cccQ[g[2]]

During evaluation of In[7]:= g::nogood: -- Message text not found -- (2)

Out[7]= $Aborted

So...general rule:

When writing a function g, first define upvalues for g, then define downvalues for g, otherwise use Holdpattern

Can you subscribe to this rule?

Leonid says that using Holdpattern might indicate improvable design. Besides the solution indicated above, how could one improve the design of the little code above or, better, in general when dealing with upvalues?

Thank you for your help

Answer 1

Leaving aside the WB (which is not really needed to answer your question) - the problem seems to have a straightforward answer based only on how expressions are evaluated during assignments. Here is an example:

In[1505]:= 
notGoodQ[x_]:=True;
Clear[g];
g[x_?notGoodQ]:=(Message[g::nogood,x];Abort[])

In[1509]:= g/:cccQ[g[x0_]]:=True

During evaluation of In[1509]:= g::nogood: -- Message text not found -- (x0_)
Out[1509]= $Aborted

To make it work, I deliberately made a definition for notGoodQ to always return True. Now, why was g[x0_] evaluated during the assignment through TagSetDelayed? The answer is that, while TagSetDelayed (as well as SetDelayed) in an assignment h/:f[h[elem1,...,elemn]]:=... does not apply any rules that f may have, it will evaluate h[elem1,...,elem2], as well as f. Here is an example:

In[1513]:= 
ClearAll[h,f];
h[___]:=Print["Evaluated"];

In[1515]:= h/:f[h[1,2]]:=3

During evaluation of In[1515]:= Evaluated
During evaluation of In[1515]:= TagSetDelayed::tagnf: Tag h not found in f[Null]. >>
Out[1515]= $Failed  

The fact that TagSetDelayed is HoldAll does not mean that it does not evaluate its arguments - it only means that the arguments arrive to it unevaluated, and whether or not they will be evaluated depends on the semantics of TagSetDelayed (which I briefly described above). The same holds for SetDelayed, so the commonly used statement that it "does not evaluate its arguments" is not literally correct. A more correct statement is that it receives the arguments unevaluated and does evaluate them in a special way - not evaluate the r.h.s, while for l.h.s., evaluate head and elements but not apply rules for the head. To avoid that, you may wrap things in HoldPattern, like this:

Clear[g,notGoodQ];
notGoodQ[x_]:=EvenQ[x];
g[x_?notGoodQ]:=(Message[g::nogood,x];Abort[])
g/:cccQ[HoldPattern[g[x0_]]]:=True;

This goes through. Here is some usage:

In[1527]:= cccQ[g[1]]
Out[1527]= True

In[1528]:= cccQ[g[2]]
During evaluation of In[1528]:= g::nogood: -- Message text not found -- (2)
Out[1528]= $Aborted

Note however that the need for HoldPattern inside your left-hand side when making a definition is often a sign that the expression inside your head may also evaluate during the function call, which may break your code. Here is an example of what I mean:

In[1532]:= 
ClearAll[f,h];
f[x_]:=x^2;
f/:h[HoldPattern[f[y_]]]:=y^4;

This code attempts to catch cases like h[f[something]], but it will obviously fail since f[something] will evaluate before the evaluation comes to h:

In[1535]:= h[f[5]]
Out[1535]= h[25]

For me, the need for HoldPattern on the l.h.s. is a sign that I need to reconsider my design.

EDIT

Regarding debugging during loading in WB, one thing you can do (IIRC, can not check right now) is to use good old print statements, the output of which will appear in the WB's console. Personally, I rarely feel a need for debugger for this purpose (debugging package when loading)

EDIT 2

In response to the edit in the question:

Regarding the order of definitions: yes, you can do this, and it solves this particular problem. But, generally, this isn't robust, and I would not consider it a good general method. It is hard to give a definite advice for a case at hand, since it is a bit out of its context, but it seems to me that the use of UpValues here is unjustified. If this is done for error - handling, there are other ways to do it without using UpValues.

Generally, UpValues are used most commonly to overload some function in a safe way, without adding any rule to the function being overloaded. One advice is to avoid associating UpValues with heads which also have DownValues and may evaluate -by doing this you start playing a game with evaluator, and will eventually lose. The safest is to attach UpValues to inert symbols (heads, containers), which often represent a "type" of objects on which you want to overload a given function.

Regarding my comment on the presence of HoldPattern indicating a bad design. There certainly are legitimate uses for HoldPattern, such as this (somewhat artificial) one:

In[25]:= 
Clear[ff,a,b,c];
ff[HoldPattern[Plus[x__]]]:={x};
ff[a+b+c]

Out[27]= {a,b,c} 

Here it is justified because in many cases Plus remains unevaluated, and is useful in its unevaluated form - since one can deduce that it represents a sum. We need HoldPattern here because of the way Plus is defined on a single argument, and because a pattern happens to be a single argument (even though it describes generally multiple arguments) during the definition. So, we use HoldPattern here to prevent treating the pattern as normal argument, but this is mostly different from the intended use cases for Plus. Whenever this is the case (we are sure that the definition will work all right for intended use cases), HoldPattern is fine. Note b.t.w., that this example is also fragile:

In[28]:= ff[Plus[a]]
Out[28]= ff[a]

The reason why it is still mostly OK is that normally we don't use Plus on a single argument.

But, there is a second group of cases, where the structure of usually supplied arguments is the same as the structure of patterns used for the definition. In this case, pattern evaluation during the assignment indicates that the same evaluation will happen with actual arguments during the function calls. Your usage falls into this category. My comment for a design flaw was for such cases - you can prevent the pattern from evaluating, but you will have to prevent the arguments from evaluating as well, to make this work. And pattern-matching against not completely evaluated expression is fragile. Also, the function should never assume some extra conditions (beyond what it can type-check) for the arguments.