Beautiful Soup findAll doesn't find them all

Question

I'm trying to parse a website and get some info with the find_all() method, but it doesn't find them all.

This is the code:

#!/usr/bin/python3  from bs4 import BeautifulSoup from urllib.request import urlopen  page = urlopen ("http://mangafox.me/directory/") # print (page.read ()) soup = BeautifulSoup (page.read ())  manga_img = soup.findAll ('a', {'class' : 'manga_img'}, limit=None)  for manga in manga_img:     print (manga['href']) 

It only prints half of them...

Answer 1

Different HTML parsers deal differently with broken HTML. That page serves broken HTML, and the lxml parser is not dealing very well with it:

>>> import requests >>> from bs4 import BeautifulSoup >>> r = requests.get('http://mangafox.me/directory/') >>> soup = BeautifulSoup(r.content, 'lxml') >>> len(soup.find_all('a', class_='manga_img')) 18 

The standard library html.parser has less trouble with this specific page:

>>> soup = BeautifulSoup(r.content, 'html.parser') >>> len(soup.find_all('a', class_='manga_img')) 44 

Translating that to your specific code sample using urllib, you would specify the parser thus:

soup = BeautifulSoup(page, 'html.parser')  # BeatifulSoup can do the reading