Non-standard evaluation and PackedArray

Question

I have earlier asked how to make allTrue[{x,list},test] function that protects the placeholder symbol x from evaluation in current context in the same way as Table[expr,{x,...}] protects x

The recipe that I ended up using failed intermittently, and I found the problem down to be caused by automatic conversion of lists to PackedArrays. Here's a failing example

SetAttributes[allTrue, HoldAll];
allTrue[{var_, lis_}, expr_] := 
  LengthWhile[lis, 
    TrueQ[ReleaseHold[Hold[expr] /. HoldPattern[var] -> #]] &] == 
   Length[lis];
allTrue[{y, Developer`ToPackedArray[{1, 1, 1}]}, y > 0]

I want allTrue[{x,{1,2,3}},x>0] to return True regardless of whether {1,2,3} gets automatically converted into PackedArray, what is a better way to implement it?

Answer 1

This is a version I've been using for quite a while (wrote it for a second edition of my book originally, but I ended up using it a lot). If arguments represent some unevaluated code, then the test function must have HoldAll or HoldFirst attributes if we want a single piece of code representing one specific clause to be passed to it in its unevaluated form (which may or may not be desirable).

ClearAll[fastOr];
Attributes[fastOr] = {HoldRest};
fastOr[test_, {args___}] := fastOr[test, args];
fastOr[test_, args___] :=
TrueQ[Scan[
        Function[arg, If[test[arg], Return[True]], HoldAll],
        Hold[args]]];

Edit: I just noticed that the solution by Daniel Reeves at the bottom of the page linked in the question, is very similar to this one. The main difference is that I care about both short-circuiting and keeping arguments unevaluated (see below), while Daniel focuses on the short-circuiting part only.

It does have a short-circuiting behavior. We need HoldRest attribute since we want to preserve the arguments in their unevaluated form. We also need the HoldAll (or HoldFirst) attribute in a pure function to preserve each of the arguments unevaluated until it is passed to test. Whether or not it gets evaluated before it is used in the body of test depends now on the attributes of test. As an example:

Clear[fullSquareQ];
fullSquareQ[x_Integer] := IntegerQ[Sqrt[x]];

In[13]:= Or @@ Map[fullSquareQ, Range[50000]] // Timing

Out[13]= {0.594, True}

In[14]:= fastOr[fullSquareQ, Evaluate[Range[10000]]] // Timing

Out[14]= {0., True}

Here is an example where we pass as arguments some pieces of code inducing side effects (printing). The code of the last argument has no chance to execute, since the result has already been determined at the previous clause:

In[15]:= fastOr[# &, Print["*"]; False, Print["**"]; False, 
  Print["***"]; True, Print["****"]; False]

During evaluation of In[15]:= *

During evaluation of In[15]:= **

During evaluation of In[15]:= ***

Out[15]= True

Note that, since fastOr accepts general pieces of unevaluated code as clauses for Or, you have to wrap your list of values in Evaluate if you don't care that they are going to be evaluated at the start ( as is the case with the Range example above).

Finally, I will illustrate the programmatic construction of held code for fastOr, to show how it can be used (consider it a tiny crash course on working with held expressions if you wish). The following function is very useful when working with held expressions:

joinHeld[a___Hold] := Hold @@ Replace[Hold[a], Hold[x___] :> Sequence[x], {1}];

Example:

In[26]:= joinHeld[Hold[Print[1]], Hold[Print[2], Print[3]], Hold[], Hold[Print[4]]]

Out[26]= Hold[Print[1], Print[2], Print[3], Print[4]]

Here is how we use it to construct programmatically the held arguments that were used in the example with Print-s above:

In[27]:= 
held = joinHeld @@ MapThread[Hold[Print[#]; #2] &, 
      {NestList[# <> "*" &, "*", 3], {False, False, True, False}}]

Out[27]= Hold[Print["*"]; False, Print["**"]; False, Print["***"]; True, Print["****"]; False]

To pass it to fastOr, we will use another useful idiom: append (or prepend) to Hold[args] until we get all function arguments, and then use Apply (note that, in general, if we don't want the piece we are appending / prepending to evaluate, we must wrap it in Unevaluated, so the general idiom looks like Append[Hold[parts___],Unevaluated[newpart]]):

In[28]:= fastOr @@ Prepend[held, # &]

During evaluation of In[28]:= *

During evaluation of In[28]:= **

During evaluation of In[28]:= ***

Out[28]= True

Regarding the original implementation you refer to, you can look at my comment to it I made some while ago. The problem is that TakeWhile and LengthWhile have bugs for packed arrays in v. 8.0.0, they are fixed in the sources of 8.0.1 - so, starting from 8.0.1, you can use either mine or Michael's version.

HTH

Edit:

I just noticed that in the post you referred to, you wanted a different syntax. While it would not be very difficult to adopt the approach taken in fastOr to this case, here is a different implementation, which arguably is in closer correspondence with the existing language constructs for this particular syntax. I suggest to use Table and exceptions, since iterators in Table accept the same syntax as you want. Here it is:

ClearAll[AnyTrue, AllTrue];
SetAttributes[{AnyTrue, AllTrue}, HoldAll];
Module[{exany, exall},
 AnyTrue[iter : {var_Symbol, lis_List}, expr_] :=
    TrueQ[Catch[Table[If[TrueQ[expr], Throw[True, exany]], iter], exany]];
 AllTrue[iter : {var_Symbol, lis_List}, expr_] :=
   Catch[Table[If[! TrueQ[expr], Throw[False, exall]], iter], exall] =!= False;
];

A few words of explanation: I use Module on the top level since the custom exception tags we need to define just once, can as well do that at definition-time. The way to break out of Table is through exceptions. Not very elegant and induces a small performance hit, but we buy automatic dynamic localization of your iterator variables done by Table, and simplicity. To do this in a safe way, we must tag an exception with a unique tag, so we don't catch some other exception by mistake. I find using Module for creating persistent exception tags to be a very useful trick in general. Now, some examples:

In[40]:= i = 1

Out[40]= 1

In[41]:= AnyTrue[{i, {1, 2, 3, 4, 5}}, i > 3]

Out[41]= True

In[42]:= AnyTrue[{i, {1, 2, 3, 4, 5}}, i > 6]

Out[42]= False

In[43]:= AllTrue[{i, {1, 2, 3, 4, 5}}, i > 3]

Out[43]= False

In[44]:= AllTrue[{i, {1, 2, 3, 4, 5}}, i < 6]

Out[44]= True

In[45]:= AllTrue[{a, {1, 3, 5}}, AnyTrue[{b, {2, 4, 5}}, EvenQ[a + b]]]

Out[45]= True

In[46]:= AnyTrue[{a, {1, 3, 5}}, AllTrue[{b, {2, 4, 5}}, EvenQ[a + b]]]

Out[46]= False

I started with an assignment to i to show that possible global values of iterator variables don't matter - this is taken care of by Table. Finally, note that (as I commented elsewhere), your original signatures for AllTrue and AnyTrue are a bit too restrictive, in the sense that the following does not work:

In[47]:= lst = Range[5];
AllTrue[{i, lst}, i > 3]

Out[48]= AllTrue[{i, lst}, i > 3] 

(since the fact that lst represents a list is not known at the pattern-matching time, due to HoldAll attribute). There is no good reason to keep this behavior, so you can just remove the _List checks: AnyTrue[iter : {var_Symbol, lis_}, expr_] and similarly for AllTrue, and this class of use cases will be covered.