Error while installing Scrapy error: Microsoft Visual C++ 14.0 is required

Question

I found about scrapy that is a great tool scraping so i tried to install scrapy on my machine, but when i tried to do pip install scrapy it installed for a while and threw me this error..

error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

I tried to install it in virtual environment but still the problem persists.

EDIT: here is what i got after the error..

    error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

    ----------------------------------------
Command "d:\pycharmprojects\environments\scrapyenv\scripts\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\User\\AppData\\Local\\Temp\\pip-build-arbeqlly\\Twisted\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\User\AppData\Local\Temp\pip-jdj93131-record\install-record.txt --single-version-externally-managed --compile --install-headers d:\pycharmprojects\environments\scrapyenv\include\site\python3.5\Twisted" failed with error code 1 in C:\Users\User\AppData\Local\Temp\pip-build-arbeqlly\Twisted\

any help?

Answer 1

Install Using Conda

If you have conda Use: conda install -c anaconda scrapy

Or

Install Using pip

Download https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted at this location

//Download the whl file of same version as Python version here cp27 means python2.7
pip install Twisted-18.9.0-cp27-cp27m-win_amd64.whl 
pip install scrapy

Answer 2

but the question is what does scrapy has to do with visual studio

Some python modules require a C/C++ compiler to build. That's why you need Visual Studio.

Quoting from Scrapy Installation Guide:

some binary packages that Scrapy depends on (like Twisted, lxml and pyOpenSSL) require a compiler available to install, and fail if you don’t have Visual Studio installed.

Answer 3

UPDATE As some one pointed out in the edit, if you download only the vc_redist.x64.exe it's just 15MB, for that go to "other tools and frameworks" and choose MS Visual C++

solved the problem by installing the tools from here

the thing is that some python modules require C++ tools to compile with a python wrapper around, as they are written in C++.

once the setup is downloaded you can select the specific tools or download other facilities provided.

it's somewhere around 4 GB but solves the purpose if in future you need to use some other module which are made in C++.