Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Why does this specific website cause a recursion error when parsing?

I've been trying to read in and parse a page on car-part.com all day and am having trouble. I'm using beautifulsoup and its consistently returning a recursion error. I initially assumed that it was due to the 4k list elements in the html so I increased the recursive limit but that didn't fix it. I realized that it occurs on every single page and I can't figure out why on earth its happening only on this specific website.

This is the error I'm getting:

RecursionError: maximum recursion depth exceeded in comparison

Before that though it constantly repeats

File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bs4/element.py", line 1195, in decode_contents formatter)) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bs4/element.py", line 1126, in decode indent_contents, eventual_encoding, formatter) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bs4/element.py", line 1195, in decode_contents formatter)) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bs4/element.py", line 1126, in decode indent_contents, eventual_encoding, formatter) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/bs4/element.py", line 1195, in decode_contents formatter)) ...

The problem is that I don't understand what this console reading means. Is the site calling this file and that's the recursion error?

I'm not concerned about parsing the site anymore, I've already gotten the information I need I just want to understand what exactly is going on here; can anyone explain it to me?

There isn't much to post in terms of code, any call of the object returns the recursion error.

from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen("http://car-part.com")
bsObj = BeautifulSoup(html.read(), "html.parser")

print(bsObj)

printing bsObj returns the error and calling bsObj.li does as well but .title and other elements seem to work.

like image 940
Nikola Jankovic Avatar asked Sep 08 '26 06:09

Nikola Jankovic


1 Answers

html5lib and lxml both handle this particular case:

bsObj = BeautifulSoup(html.read(), "html5lib")
bsObj = BeautifulSoup(html.read(), "lxml")

Note that this would require html5lib and/or lxml be installed:

pip install html5lib
pip install lxml

Also see Differences between parsers.

like image 141
alecxe Avatar answered Sep 10 '26 19:09

alecxe



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!