Global Python packages in Sublime Text plugin development

Question

1. Summary

I don't find, how Sublime Text plugins developer can use Sublime Text find global Python packages, not Python packages of Sublime Text directory.

Sublime Text use own Python environment, not Python environment of machine. Developers needs sys.path for set not built-in Sublime Text Python packages.

Is any methods, that use global installed Python packages in Sublime Text plugins? For example, it would be nice, if someone tells me, how I can change my plugin — see 3.2 item of this question.

---

2. Disadvantages of using Sublime Text 3 environment

1. Sublime Text 3 Build 3126 use Python 3.3, but at the time of writing this question release Python 3.6 stable. Python 3.6 have more features.
2. Developers needs add and update third-party Python packages, even they installed for users. It spends a time of developers.
3. For developers there may be problems with dependencies of packages, see 6.2 item of this question.

---

3. Example

1. Python code

For example, I wrote Python code — replace Поиск Кристиниты to [**Поиск Кристиниты**](https://github.com/Kristinita/Kristinita.github.io), where https://github.com/Kristinita/Kristinita.github.io — first link of DuckDuckGo query Поиск Кристиниты.

# -*- coding: utf-8 -*-
import re
import urllib

from bs4 import BeautifulSoup

from w3lib.url import safe_url_string


# ASCII link for solved encoding problems —
# https://stackoverflow.com/a/40654295/5951529
ascii_link = safe_url_string(
    u'http://duckduckgo.com/html/?q=' + 'Поиск Кристиниты',
    encoding="UTF-8")
print(ascii_link)
# SERP DuckDuckGo
serp = urllib.request.urlopen(ascii_link)
# Reading SERP
read_serp = serp.read()
# BeautifulSoup — https://stackoverflow.com/a/11923803/5951529
parsed = BeautifulSoup(read_serp, "lxml")
# Parsed first link
first_link = parsed.findAll(
    'div', {'class': re.compile('links_main*')})[0].a['href']
# Remove DuckDuckGo specific characters —
# https://stackoverflow.com/a/3942100/5951529
remove_duckduckgo_symbols = first_link.replace("/l/?kh=-1&uddg=", "")
# https://stackoverflow.com/a/32451970/5951529
final_link = (urllib.parse.unquote(remove_duckduckgo_symbols))
# Markdown link
markdown_link = '[' + 'Поиск Кристиниты' + ']' + \
    '(' + final_link + ')'

print(markdown_link)

If I run this file in terminal or SublimeREPL, I get in output:

[**Поиск Кристиниты**](https://github.com/Kristinita/Kristinita.github.io/)

2. Sublime Text plugin

Now, based on this code, I wrote Sublime Text plugin for replace example text to [**example text**](http://<first link for DuckDuckGo query “example link”>):

import re
import urllib

from bs4 import BeautifulSoup

from w3lib.url import safe_url_string

import sublime_plugin


class KristinitaLuckyLinkCommand(sublime_plugin.TextCommand):

    def run(self, edit):
        # Get selection text
        print('KristinitaLuckyLink called')
        select = self.view.sel()
        selection_region = select[0]
        selection_text = self.view.substr(selection_region)
        print(selection_text)

        # ASCII link for solved encoding problems —
        # https://stackoverflow.com/a/40654295/5951529
        ascii_link = safe_url_string(
            u'http://duckduckgo.com/html/?q=' + (selection_text),
            encoding="UTF-8")
        print(ascii_link)
        # SERP DuckDuckGo
        serp = urllib.request.urlopen(ascii_link)
        # Reading SERP
        read_serp = serp.read()
        # BeautifulSoup — https://stackoverflow.com/a/11923803/5951529
        parsed = BeautifulSoup(read_serp, "lxml")
        # Parsed first link
        first_link = parsed.findAll(
            'div', {'class': re.compile('links_main*')})[0].a['href']
        # Remove DuckDuckGo specific characters —
        # https://stackoverflow.com/a/3942100/5951529
        remove_duckduckgo_symbols = first_link.replace("/l/?kh=-1&uddg=", "")
        # Final link — https://stackoverflow.com/a/32451970/5951529
        final_link = (urllib.parse.unquote(remove_duckduckgo_symbols))
        markdown_link = '[' + selection_text + ']' + \
            '(' + final_link + ')'
        print(markdown_link)

        # Replace selected text to Markdown link
        self.view.replace(
            edit, selection_region, markdown_link)

---

4. Expected behavior

If user have installed Python and install packages

• pip install beautifulsoup4
• pip install lxml
• pip install w3lib

I want, that my plugin from 2.2 item successful work for user.

---

5. Actual behavior

If I save my plugin, I get stack trace:

Traceback (most recent call last):
  File "D:\Sublime Text Build 3126 x64 For Debug\sublime_plugin.py", line 109, in reload_plugin
    m = importlib.import_module(modulename)
  File "./python3.3/importlib/__init__.py", line 90, in import_module
  File "<frozen importlib._bootstrap>", line 1584, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1565, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "D:\Sublime Text Build 3126 x64 For Debug\Data\Packages\Grace Splitter\kristi.py", line 4, in <module>
    from bs4 import BeautifulSoup
ImportError: No module named 'bs4'

---

6. Not helped

1. Using global Python environment of computer

I don't find, how I can do it. Examples of questions, that I can find:

• How to include third party Python packages in Sublime Text 2 plugins,
• Using paramiko library in ST2 plugins,
• Best practices for plugin-deployment and packaging,
• How to import a package and call its global functions / to get their global variable values?

2. Using Sublime Text environment

I install

• sublime-beautifulsoup4,
• sublime-lxml,

I copy my w3lib directory from C:\Python36\Lib\site-packages to Data\Packages directory of Sublime Text.

I run in Sublime Text 3 console:

>>> window.run_command("kristinita_lucky_link")

I get stack trace:

Traceback (most recent call last):
  File "D:\Sublime Text 3 x64\sublime_plugin.py", line 818, in run_
    return self.run(edit)
  File "D:\Sublime Text 3 x64\Data\Packages\KristinitaLuckyLink\KristinitaLuckyLink.py", line 32, in run
    parsed = BeautifulSoup(read_serp, "lxml")
  File "D:\Sublime Text 3 x64\Data\Packages\bs4\__init__.py", line 165, in __init__
    % ",".join(features))
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?

I don't find, how I can set lxml.

3. Using variables in 2 files

For example I have KristinitaLuckyLink.py and KrisDuckDuckGo.py files in the same directory.

My KristinitaLuckyLink.py file:

import re
import requests
import sublime_plugin
import subprocess
import sys
sys.path.append(
    'D:\Sublime Text 3 x64\Data\Packages\KristinitaLuckyLink\KrisDuckDuckGo.py')

from KrisDuckDuckGo import final_link
from bs4 import BeautifulSoup


class KristinitaLuckyLinkCommand(sublime_plugin.TextCommand):

    def run(self, edit):
        # Get selection text
        print('KristinitaLuckyLink called')
        select = self.view.sel()
        selection_region = select[0]
        selection_text = self.view.substr(selection_region)
        print(selection_text)

        # Get terminal output — https://stackoverflow.com/a/4760517/5951529
        # Paths is correct
        result = subprocess.run(["C:\Python36\python.exe", "D:\Sublime Text 3 x64\Data\Packages\KristinitaLuckyLink\krisduckduckgo.py"],
                                stdout=subprocess.PIPE)
        final_link = result.stdout.decode('utf-8')
        print(final_link)

        # Markdown link
        markdown_link = '[' + selection_text + ']' + \
            '(' + final_link + ')'
        print(markdown_link)

        # Replace selected text to Markdown link
        self.view.replace(
            edit, selection_region, markdown_link)

My KrisDuckDuckGo.py file:

import urllib

import sys
sys.path.append(
    'D:\Sublime Text 3 x64\Data\Packages\KristinitaLuckyLink\KristinitaLuckyLink.py')

from w3lib.url import safe_url_string

from KristinitaLuckyLink import selection_text

from bs4 import BeautifulSoup


# ASCII link for solved encoding problems —
# https://stackoverflow.com/a/40654295/5951529
ascii_link = safe_url_string(
    u'http://duckduckgo.com/html/?q=' + (selection_text),
    encoding="UTF-8")
print(ascii_link)
# SERP DuckDuckGo
serp = urllib.request.urlopen(ascii_link)
# Reading SERP
read_serp = serp.read()
# BeautifulSoup — https://stackoverflow.com/a/11923803/5951529
parsed = BeautifulSoup(read_serp, "lxml")
# Parsed first link
first_link = parsed.findAll(
    'div', {'class': re.compile('links_main*')})[0].a['href']
# Remove DuckDuckGo specific characters —
# https://stackoverflow.com/a/3942100/5951529
remove_duckduckgo_symbols = first_link.replace("/l/?kh=-1&uddg=", "")
# Final link — https://stackoverflow.com/a/32451970/5951529
final_link = (urllib.parse.unquote(remove_duckduckgo_symbols))
print(final_link)

I select any text → I print in Sublime Text console:

window.run_command("kristinita_lucky_link")

I don't get output in Sublime Text console.

---

7. Environment

Operating system and version:
Windows 10 Enterprise LTSB 64-bit EN
Sublime Text:
Build 3126
Python:
3.6.0

Answer 1

The error:

bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?

most likely comes because Sublime Text is finding the lxml installed as part of the global Python environment - which is compiled for a different version of Python than the one which ST uses - so it can't load it.

What we want is for ST to find the sublime-lxml dependency, which you linked to in your question. "Why is it finding the wrong one?", you might ask. It's hard to say for sure, but I think OdatNurd's answer gives us some clues - the sys.path that ST sees for some reason includes all your global Python packages. By default, ST should only use the folder where it's executable resides and the Packages folders in the ST data directory. For example, on my system, executing import sys; sys.path in ST's console gives me:

[
 'C:\\Program Files\\Sublime Text 3',
 'C:\\Program Files\\Sublime Text 3/python3.3.zip',
 'C:\\Users\\Keith\\AppData\\Roaming\\Sublime Text 3\\Packages',
 'C:\\Users\\Keith\\AppData\\Roaming\\SUBLIM~1\\Packages\\lxml\\ST3_WI~2'
]

i.e. no site-packages folder.

The solution therefore is either:

a. You could try uninstalling the system-wide lxml package so that ST will only find the sublime-lxml dependency package, but really that is only a temporary measure. It would be better to: b. Adjust the (order of items in the) path environment variable that ST is using. (i.e. remove all references to site-packages, or at least move them so that they come after the ST folders.)

Option B shouldn't affect any other ST plugins, because they would also suffer from the same problem. I suspect some other package changed the path ST is using, but it might not be easy to find out which one, without searching through them all. When build systems are used, the path which is used for those is quite different to the one that plugins use to load their modules, so build systems should also remain unaffected by this "fix".