pyomo + reticulate error 6 the handle is invalid

Question

I'm trying to run a pyomo optimization and I get the error message [Error 6] The handle is invalid. Not sure how to interpret it, looking around it seems to have something to do with privileges but I don't really understand it.

Find below the complete error trace and also a toy example to reproduce it.

Full error trace:

Error in py_run_file_impl(file, local, convert) : ApplicationError: Could not execute the command: 'C:\Users\xxx\AppData\Local\Continuum\anaconda3\envs\lucy\Library\bin\ipopt.exe c:\users\xxx\appdata\local\temp\tmpp2hmid.pyomo.nl -AMPL' Error message: [Error 6] The handle is invalid

Detailed traceback: File "", line 46, in File "C:\Users\xxx\AppData\Local\CONTIN~1\ANACON~1\envs\lucy\lib\site-packages\pyomo\opt\base\solvers.py", line 578, in solve _status = self._apply_solver() File "C:\Users\xxx\AppData\Local\CONTIN~1\ANACON~1\envs\lucy\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 246, in _apply_solver self._rc, self._log = self._execute_command(self._command) File "C:\Users\xxx\AppData\Local\CONTIN~1\ANACON~1\envs\lucy\lib\site-packages\pyomo\opt\solver\shellcmd.py", line 309, in _execute_command tee = self._tee File "C:\Users\xxx\AppData\Local\CONTIN~1\ANACON~1\envs\lucy\lib\site-packages\pyutilib\subprocess\processmngr.py", line 660, in run_command

Reproducible example based on this.

Pure python code (it works when I run it in python, in the conda environment called "lucy"):

from pyomo.environ import *
infinity = float('inf')

model = AbstractModel()

# Foods
model.F = Set()
# Nutrients
model.N = Set()

# Cost of each food
model.c    = Param(model.F, within=PositiveReals)
# Amount of nutrient in each food
model.a    = Param(model.F, model.N, within=NonNegativeReals)
# Lower and upper bound on each nutrient
model.Nmin = Param(model.N, within=NonNegativeReals, default=0.0)
model.Nmax = Param(model.N, within=NonNegativeReals, default=infinity)
# Volume per serving of food
model.V    = Param(model.F, within=PositiveReals)
# Maximum volume of food consumed
model.Vmax = Param(within=PositiveReals)

# Number of servings consumed of each food
model.x = Var(model.F, within=NonNegativeIntegers)

# Minimize the cost of food that is consumed
def cost_rule(model):
    return sum(model.c[i]*model.x[i] for i in model.F)
model.cost = Objective(rule=cost_rule)

# Limit nutrient consumption for each nutrient
def nutrient_rule(model, j):
    value = sum(model.a[i,j]*model.x[i] for i in model.F)
    return model.Nmin[j] <= value <= model.Nmax[j]
model.nutrient_limit = Constraint(model.N, rule=nutrient_rule)

# Limit the volume of food consumed
def volume_rule(model):
    return sum(model.V[i]*model.x[i] for i in model.F) <= model.Vmax
model.volume = Constraint(rule=volume_rule)

opt = SolverFactory('ipopt')
instance = model.create_instance('diet.dat')
results = opt.solve(instance, tee=False)
results

The code to run it in R with reticulate is pretty straightforward:

library(reticulate)
use_condaenv(condaenv = "lucy")
py_run_file("../pyomo_scripts/test.py")

And finally for completeness this is the diet.dat file (must be at the same path as the python/R files):

param:  F:                          c     V  :=
  "Cheeseburger"                 1.84   4.0  
  "Ham Sandwich"                 2.19   7.5  
  "Hamburger"                    1.84   3.5  
  "Fish Sandwich"                1.44   5.0  
  "Chicken Sandwich"             2.29   7.3  
  "Fries"                         .77   2.6  
  "Sausage Biscuit"              1.29   4.1  
  "Lowfat Milk"                   .60   8.0 
  "Orange Juice"                  .72  12.0 ;

param Vmax := 75.0;

param:  N:       Nmin   Nmax :=
        Cal      2000      .
        Carbo     350    375
        Protein    55      .
        VitA      100      .
        VitC      100      .
        Calc      100      .
        Iron      100      . ;

param a:
                               Cal  Carbo Protein   VitA   VitC  Calc  Iron :=
  "Cheeseburger"               510     34     28     15      6    30    20
  "Ham Sandwich"               370     35     24     15     10    20    20
  "Hamburger"                  500     42     25      6      2    25    20
  "Fish Sandwich"              370     38     14      2      0    15    10
  "Chicken Sandwich"           400     42     31      8     15    15     8
  "Fries"                      220     26      3      0     15     0     2
  "Sausage Biscuit"            345     27     15      4      0    20    15
  "Lowfat Milk"                110     12      9     10      4    30     0
  "Orange Juice"                80     20      1      2    120     2     2 ;

---

edit after comments:

These are the versions for pyomo and ipopt

pyomo                     5.6.4                    py36_0    conda-forge
pyomo.extras              3.3                 py36_182212    conda-forge
ipopt                     3.11.1                        2    conda-forge

I have inherited loads of code in R with the optimization done in pyomo through system calls. I'm trying to improve it by using reticulate so that I avoid writing and reading files and I have more control... if I still have do system calls within python, I will gain very little by using reticulate.

Thanks.

Answer 1

I can not say I understand this problem entirely, however it is a very interesting one to research, mainly because I got a different error message

TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

and while I got the error every time I ran py_run_file("test.py") in a new r session, by the second run there was no error.

That being said I believe it is related to this issue: https://github.com/PyUtilib/pyutilib/issues/31

I didn't face any problem after adding the two lines :

import pyutilib.subprocess.GlobalData
pyutilib.subprocess.GlobalData.DEFINE_SIGNAL_HANDLERS_DEFAULT = False

in the python script before invoking the solver.

Hope this helps

Answer 2

If you can execute the python version, try to r session with administrative right with the following code

library("reticulate")


##-- your directory containing 'diet.py' and 'diet.dat'
setwd("D:/project/Dropbox/lectures/2104xxx scg_opt/src/02"")



##-- execute code
a <- py_run_file("diet.py",local=T)
a$results