Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Import numpy without installing

Is there a way to import numpy without installing it?

I have a general application built into an .exe with PyInstaller. The application has a plugin system which allows it to be extended through Python scripts. The plugin import system works fine for basic modules (lone .py files, classes, functions, and simple packages). Internally, it globs a plugin directory and then imports accordingly using __import__ or importlib.import_module.

The application is built with minimal dependencies in order to reduce the overall size of the executable. Also, it's not possible to know what dependencies a future plugin will require nor practical to include everything. However, some plugins will inevitably require dependencies. numpy is a good test case for solving this class of problems.


Here's what I've tried.

A wheel file is really just a directory. It can be added to sys.path and the contents imported.

import sys
sys.path.append(r"C:\path\to\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl")

import numpy as np

The wheel file is read, but the import generates an error.

*** ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
Here is how to proceed:
- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.
- If you are simply trying to use the numpy version that you have installed:
  your installation is broken - please reinstall numpy.
- If you have already reinstalled and that did not fix the problem, then:
  1. Check that you are using the Python you expect (you're using c:\projects\zip_test\venv\Scripts\python.exe),
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy versions you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

     Note: this error has many possible causes, so please don't comment on
     an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

The puzzling part is that the wheel file contains a .pyd for _multiarray_umath.

C:\projects\zip_test\plugins\New folder\numpy\core>dir
 Volume in drive C is OS
 Volume Serial Number is FE3D-6596

 Directory of C:\projects\zip_test\plugins\New folder\numpy\core

07/25/2019  02:37 PM    <DIR>          .
07/25/2019  02:37 PM    <DIR>          ..
    ....
04/22/2019  02:55 AM           101,376 _multiarray_tests.cp36-win_amd64.pyd
04/22/2019  02:55 AM         2,494,976 _multiarray_umath.cp36-win_amd64.pyd
    ....
              74 File(s)    583,173,551 bytes
               5 Dir(s)  309,925,851,136 bytes free

This feels like a pathing issue. Yet adding the direct path for core/ to sys.path generates the same error.

sys.path.append(r"C:\projects\zip_test\plugins\numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl\numpy\core")

What's the deal? Why can't Python find numpy.core._multiarray_umath?


In response to repsones:

  • The wheel file was obtained from Christoph Gohlke.
  • My operating system is Windows 10 Pro 64-bit.
  • I am running Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32 (installed via Python Foundation build 3.6.8150.0) in a venv. The "system" Python is not on PATH.
  • I can install the numpy-1.16.3+mkl-cp36-cp36m-win_amd64.whl file via pip and import numpy as np works fine. I then uninstall numpy via pip uninstall -y numpy.

Running dumpbin /dependents _multiarray_umath.cp36-win_amd64.pyd produces:

Microsoft (R) COFF/PE Dumper Version 14.22.27905.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file _multiarray_umath.cp36-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    mkl_rt.dll
    python36.dll
    KERNEL32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll
    api-ms-win-crt-locale-l1-1-0.dll

  Summary

       77000 .data
        1000 .gfids
       1C000 .pdata
       30000 .rdata
        3000 .reloc
        1000 .rsrc
      1BD000 .text

Of the dependencies, I can find mkl_rt.dll, python36.dll, and VCRUNTIME140.dll in the either the .whl or the venv. There are apparently two versions of KERNEL32.dll, one for x86 and one for x64. One of these resides in C:\Windows\System32.

The only information I could find about the api-ms-win-crt-*.dll are described on MSDN within the "Update for Universal C Runtime in Windows",

The Windows 10 Universal CRT is a Windows operating system component that enables CRT functionality on the Windows operating system. This update allows Windows desktop applications that depend on the Windows 10 Universal CRT release to run on earlier Windows operating systems.

These are for non-Windows 10 systems. There seems to be no "official" way to obtain them for a Windows 10 system. I was able to copy the dlls from a Windows 7 system. Naively putting them in the PYTHONPATH (not surprisingly) doesn't work. If they were to work at all, they would likely need to be registered. But 1) registering Windows 7 dlls on Windows 10, I am told, can make the system unstable and 2) definitely doesn't seem like a universal, portable solution.

like image 792
Lorem Ipsum Avatar asked Nov 07 '22 15:11

Lorem Ipsum


1 Answers

First. Build application with and without numpy. Check difference between builds. Original error was: No module named 'numpy.core._multiarray_umath' Can means two things:

  1. cannot found file
  2. some dependence of this this pyd file is not satisfied. You may try to check it with http://dependencywalker.com/.
like image 195
Grzegorz Bokota Avatar answered Nov 15 '22 07:11

Grzegorz Bokota