Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python Beautifulsoup (bs4) findAll not finding all elements

From the url that is in the code, I am ultimately trying to gather all of the players names from the page. However, when I am using .findAll in order to get all of the list elements, I am yet to be successful. Please advise.

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

players_url = 'https://stats.nba.com/players/list/?Historic=Y'

# Opening up the Connection and grabbing the page
uClient = uReq(players_url)
page_html = uClient.read()

players_soup = soup(page_html, "html.parser")

# Taking all of the elements from the unordered lists that contains all of the players.

list_elements = players_soup.findAll('li', {'class': 'players-list__name'})
like image 922
datam Avatar asked Sep 13 '25 00:09

datam


1 Answers

As @Oluwafemi Sule suggested it is better to use selenium together with BS:

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Firefox()
driver.get('https://stats.nba.com/players/list/?Historic=Y')
soup = BeautifulSoup(driver.page_source, 'lxml')
for div in soup.findAll('li', {'class': 'players-list__name'}):
    print(div.find('a').contents[0])

Output:

Abdelnaby, Alaa
Abdul-Aziz, Zaid
Abdul-Jabbar, Kareem
Abdul-Rauf, Mahmoud
Abdul-Wahad, Tariq

etc.

like image 85
Alderven Avatar answered Sep 14 '25 14:09

Alderven