Menu
we are trying to come up with a solution to have AWS S3 to host and distribute our Python packages.
Basically what we want to do is using python3 setup.py bdist_wheel to create a wheel. Upload it to S3. Then any server or any machine can do pip install $http://path/on/s3. (including a virtualenv in AWS lambda)
(We've looked into Pypicloud and thought it's an overkill.)
Creating package and installing from S3 work fine. There is only one issue here: we will release new code and give them different versions. If we host our code on Pypi, you can upgrade some packages to their newest version by calling pip install package --upgrade.
But if you host your packages on S3, how do you let pip know there's a newer version exists? How do you roll back to an older version by simply giving pip the version number? Is there a way to let pip know where to look for different version of wheels on S3?
705
The pip command looks for the package in PyPI, resolves its dependencies, and installs everything in your current Python environment to ensure that requests will work. The pip install <package> command always looks for the latest version of the package and installs it.
To do so, we can use the pip list -o or pip list --outdated command, which returns a list of packages with the version currently installed and the latest available. On the other hand, to list out all the packages that are up to date, we can use the pip list -u or pip list --uptodate command.
For example, To access “S3” services, define the connection as “boto3. client('s3')”, To access “cloudwatch” services, define the connection as “boto3. client('cloudwatch'), similarly to access “EC2” services, use “boto3. client('ec2')”.
AWS S3 can be used as a pypi server with minimal configuration and no other additional dependencies.
Suppose you want to use the bucket mypackages for hosting your private packages - awesomepy and rattlesnake.
After you have created the bucket in S3, go to its properties, click on Static website hosting card and enable the option Use this bucket to host a website. Note down the Endpoint mentioned above the radio button.
Below is how you have to arrange the folders and files (docs):
mypackages
|
|---awesomepy
| |---awesomepy-1.10.0-py27-none-any.whl
| |---awesomepy-1.10.3-py27-none-any.whl
| |---index.html
|
|---rattlepy
|---rattlesnake-0.1.1-py27-none-any.whl
|---index.html
The index.html files in the package folders will have content like below
<html>
<head>
<title>Links</title>
</head>
<body>
<h1>Links</h1>
<a href='awesomepy-1.10.0-py27-none-any.whl'>awesomepy-1.10.0-py27-none-any.whl </a>
<br />
<a href='awesomepy-1.10.3-py27-none-any.whl'>awesomepy-1.10.3-py27-none-any.whl </a>
</body>
</html>
Wheel packages have to follow proper naming convention. Refer to PEP 427 for details. For the dummy package - awesomepy-1.10.3-py27-none-any.whl, we are assuming,
py27
none
any
Finally you can install the packages using the below command. Ensure your packages have different names than that of pypi packages. Prefixing the package name with your company initials is one simple solution.
pip install awesomepy rattlesnake==0.1.1 --extra-index-url=<s3 Endpoint> --trusted-host=<s3 Endpoint without http>
Upgrading packages using pip install awesomepy --upgrade will also work with this approach.
Also, modify your bucket permissions to restrict the packages to intended clients. One way to do this is:
Select Permissions tab in the bucket --> Edit Bucket Policy
Original Credit: My ex-manager, Adlon :)
105
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
