Using python modules like beautifulSoup and request in Visual Studio C#

Question

I am trying to run an external python script using c# in visual studio. I am using modules like BeautifulSoup and requests

But I am getting the following error

No module named requests

Earlier I was getting the same error for BeautifulSoup, I added the following line to my python script and the error was solved

sys.path.append("[Path to Python]\Python\Python35-32\Lib\site-packages")

I am using IronPython in the Visual Studio 2015. Is there anyway I can overcome this error? If it is not possible, is there any other way to run a python script (with the above mentioned modules) in a c# environment.

I tried using the solution given by denfromufa, but then I am getting the following error

This is my Python code

import sys
import requests
import re
import io
from bs4 import BeautifulSoup
from math import floor

r  = requests.get("https://www.google.com/") 
data = r.text 
soup = BeautifulSoup(data, 'html.parser')
result = []

for item in soup.find_all(attrs={'class' :'something'}):
        for m in item.select('a[href^="something"]'):

        m1 = m['href'].replace("something","",1)

        m2 = re.sub(r'&.*$', "", m1)

        m3 = re.sub(r'%3F.*$', "", m2)

        m4 = m3.replace("%2F","/")

        m5 = m4.replace("%3A",":")

        result.append(m5)
        result.append(m.get_text())


for image in item.find_all('img'):
    k1 = re.sub(r'&cfs.*$',"",image['src'])
    k2 = re.sub(r'^https://something.*$',"",k1)
    k3 = re.sub(r'.*url=',"",k2)
    k4 = re.sub(r'%3F.*$', "", k3)
    k5 = k4.replace("%2F","/")
    k6 = k5.replace("%3A",":")
    k7 = re.sub(r'.*\.gif',"",k6)
    result.append(k7)



seen = set()
result_final = []
for item in result:
    if item not in seen:
        seen.add(item)
        result_final.append(item)

result_final = list(result_final)

My c# code is the following

 using (Py.GIL())
                {
                    dynamic sys = Py.Import("sys");
                dynamic requests = Py.Import("requests");
                dynamic re = Py.Import("re");
                dynamic io = Py.Import("io");
                dynamic BeautifulSoup = Py.Import("bs4");
                dynamic math = Py.Import("math");
                Console.WriteLine(5);
                dynamic r = requests.get("https://www.google.com/");
                dynamic data = r.text;
                dynamic soup = BeautifulSoup.BeautifulSoup(data, "html.parser");
                }

I used

    var divExp = new { _class = "smoething" };
    var item = soup.find_all(Py.kw("class", divExp._class));

I am getting the result. But when I try to implement the select method on the item variable, I ma getting an error saying that Python Object doesn't contain a definition for 'select'

item.select("a[href^='https://www.google.com/']");

Final Answer

using (Py.GIL())
                {
                    dynamic sys = Py.Import("sys");
                    dynamic requests = Py.Import("requests");
                    dynamic re = Py.Import("re");
                    dynamic io = Py.Import("io");
                    dynamic BeautifulSoup = Py.Import("bs4");
                    dynamic math = Py.Import("math");
                    Console.WriteLine(5);
                    dynamic r = requests.get(url);
                    dynamic data = r.text;
                    dynamic soup = BeautifulSoup.BeautifulSoup(data, "html.parser");

                    var divExp = new { _class = "className" };
                    var item = soup.find_all(Py.kw("class", divExp._class));
                    dynamic tag = soup.select("a[href^='https://something.com/']");
                    for (var i = 1; i < item.Length(); i++)
                    {
                        // Extrxting the required info using regex

                        String input = Convert.ToString(item[i]);
                        string pattern_link = "(.*href=\"https:[\\/][\\/]something.com[\\/]a.php\\?u=)|(&.*)";
                        string replacement_link = " ";
                        Regex rgx_link = new Regex(pattern_link);
                        string result_link = rgx_link.Replace(input, replacement_link);

                        .
                        .
                        .
                        .
                        string pattern_link_1 = "(http|https)%.*";
                        Regex rgx_link_1 = new Regex(pattern_link_1);
                        Match result_link_1 = rgx_link_1.Match(result_link);
                        String input_1_1 = Convert.ToString(result_link_1.Value);



                        result_link_2 = result_link_2.Replace("%2F", "/").Replace("%3A", ":");                       

                    }

                }

Answer 1

• install CPython, one of versions 2.7, 3.4+
• pip install pythonnet
• reference installed Python.Runtime.DLL in your .NET project
• follow tutorial on www.python4.net, embedding section.

```

> scriptcs (ctrl-c to exit or :help for help)

> #r "C:\Python\Anaconda3_64b\Lib\site-packages\Python.Runtime.dll"
> using Python.Runtime;
> dynamic bs4;
> using (Py.GIL()) {bs4=Py.Import("bs4");}
> bs4.__file__.ToString()
C:\Python\Anaconda3_64b\lib\site-packages\bs4\__init__.py
> dynamic rq;
> using (Py.GIL()) {rq=Py.Import("requests");}
> dynamic r=rq.get("https://www.google.com/")
> dynamic soup = bs4.BeautifulSoup(r.text,"html.parser");
> soup.ToString()

```

Answer 2

Why not use HTML Agility Pack? It's the C# equivalent.

http://html-agility-pack.net/

You can import it into your solution.

For external python scripts see,

https://www.codeproject.com/articles/121374/step-by-step-guidance-of-calling-iron-python-funct