Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to run a Python project using __pycache__ folder?

I want to run a Pythonic project using Python compilation (.pyc or __pycache__). In order to do that in Python2, I haven't any problem.


Here is a simplified example in a Python2 project:

  • Project tree:

    test2
    ├── main.py
    └── subfolder
        ├── __init__.py
        └── sub.py
    
  • Compile:

    python -m compileall test2
    
  • Project tree after the compile:

    test2
    ├── main.py
    ├── main.pyc
    └── subfolder
        ├── __init__.py
        ├── __init__.pyc
        ├── sub.py
        └── sub.pyc
    
  • As you can see, several .pyc manually generated. Now I can run this project using main.pyc as fine, which has a relation with the sub.py:

    python main.pyc
    

    Out:

    Hi
    Bye
    
  • main.py content:

    from subfolder import sub
    
    print('Bye')
    
  • sub.py content:

    print('Hi')
    

Now I want to retry this behavior in a Python3 project.

Here is a simplified asyncio (available in Python3) project:

  • Project tree:

    test3
    ├── main.py
    └── subfolder
        ├── __init__.py
        └── sub.py
    
  • Compile:

    python3 -m compileall test3
    
  • Project tree after the compile:

    test3
    ├── main.py
    ├── __pycache__
    │   └── main.cpython-36.pyc
    └── subfolder
        ├── __init__.py
        ├── __pycache__
        │   ├── __init__.cpython-36.pyc
        │   └── sub.cpython-36.pyc
        └── sub.py
    
  • As you can see, __pycache__ folders manually generated. But I cannot run this project using main.cpython-36.pyc which has a relation with subfolder:

    cd test3/__pycache__
    python3 main.cpython-36.pyc
    

    Out (I expected that produced the Hi Bye message):

    Traceback (most recent call last):
      File "test3/main.py", line 2, in <module>
    ModuleNotFoundError: No module named 'subfolder'
    
  • main.py content:

    import asyncio
    from subfolder import sub
    
    async def myCoroutine():
        print("Bye")
    
    def main():
        loop = asyncio.get_event_loop()
        loop.run_until_complete(myCoroutine())
        loop.close()
    
    main()
    
  • sub.py content:

    print('Hi')
    

Question:

How can I run this project (above Python3 project) using __pycache__ folder?

Or

How can I run a Python3 project with the relation between subfolders using python compilation?


[NOTE]:

  • I cannot use the python compileall (Python2 compile) in the above Python3 project due to the asyncio method.

  • My Python(s) version is Python2.7 and Python3.6

like image 216
Benyamin Jafari Avatar asked Dec 24 '18 23:12

Benyamin Jafari


Video Answer


1 Answers

You can enforce the same layout of pyc-files in the folders as in Python2 by using:

python3 -m compileall -b test3

The option -b triggers the output of pyc-files to their legacy-locations (i.e. the same as in Python2).

After that you can once again use the compiled cache via:

python3 main.pyc

The way the loading of modules works since PEP-3147, it is impossible to use pyc-files from __pycache__ folder in the way you intend: If there is no *.py-file, the content of the __pycache__ is never looked-up. Here is the most important part of the workflow:

   import foo
     |
     |
     -- >  [foo.py exists?]  --- NO ----> [foo.pyc exists?]  -- NO --> [ImportError] 
            |                                     |
            |                                    YES
           YES                                    |--> [load foo.pyc]
            |
            |-> [look up in __pycache__]

                               

That means, files from __pycache__ are only looked up, when a corresponding *.py-file can be found.


Obviously, building python scripts with a Python version 3.X in this way and trying to run the resulting pyc-files with another Python version 3.Y will not work: Different Python versions need different pyc-files, this is the whole point behind PEP-3147.

like image 139
ead Avatar answered Oct 05 '22 02:10

ead