How to find_all(id) from a div with beautiful soup in python

Question

I want to print out all the IDs from a page which has a unique class.

The page what I want to scrape with Beautiful Soup is like this:

<div itemscope itemprop="item" itemtype="http://schema.org/Product" id="12345" class="realestate"> 
<div class="contentArea"> 
<meta itemprop="name" content="Name - 12345 " /> 
<meta itemprop="url" content="https://url12345.hu" />   
<meta itemprop="category" content="category1" />   
</div>
</div>
<div itemscope itemprop="item" itemtype="http://schema.org/Product" id="12346" class="realestate"> 
<div class="contentArea"> 
<meta itemprop="name" content="Name - 12346 " /> 
<meta itemprop="url" content="https://url12346.hu" />   
<meta itemprop="category" content="category1" />   
</div>
</div>

the 'ID' is a unique identifier from the Itemscope DIVs, so somehow I want to extract these uniques IDs and print all of them out (the reson is to attach all other ad information to this ID (like name, URL, etc.) later)

I tried with this python code, but it does not work.

import requests
from bs4 import BeautifulSoup

page = requests.get('searchResultPage.url')
soup = BeautifulSoup(page.text, 'html.parser')
id = soup.find_all('id')
print(id)

It gives back an empty list.

What I expect, and what I want is to get back a list with the ID-s from the divs, this way: 12345 12346

Thanks for your help in advance!

Answer 1

HS-nebula is correct that find_all looks for tags of a certain type, in your soup id is an attribute not a type of tag. To get a list of all id's in the soup you can use the following one liner

ids = [tag['id'] for tag in soup.select('div[id]')]

this uses CSS selectors instead of bs4's find_all since I find bs4's docs regarding its built-ins lacking.

So what the soup.select does is return a list of all div elements which have an attribute called 'id' then we loop over that list of div tags and add the value of the 'id' attribute to the ids list.

Answer 2

If you wanted to see all IDs in the whole web URL this will work but it will also include lots of the outer and inner HTML tags and code.

id = soup.find_all(id=True)
print(id)

If you want to see the actual IDs without all the HTML in a list/array of one ID per a row here is an option:

for ID in soup.find_all('div', id=True):  
    print(ID.get('id'))

In the above For Loop, you are specifying the tag in quote marks, i.e. 'div' and you then are asking it to list the attribute you want, i.e. 'id=True'