Menu
I want to extract all HTML between specific HTML tags.
<html>
<div class="class1">Included Text</div>
[...]
<h1><b>text</b></h1><span>[..]</span><div>[...]</div>
[...]
<span class="class2">
[...]</span>
so want to grep all HTML (tags & values) between the class1 div and the class2 span.
Included Text
<h1><b>text</b></h1><span>[..]</span><div>[...]</div>
Also there are multiple occurrences within the HTML file so I want to match them all. Here is what I mean:
<html>
(first occurrence)
<div class="class1">Included Text</div>
[...]
<h1><b>text</b></h1><span>[..]</span><div>[...]</div>
[...]
<span class="class2">
[...]</span>
(2nd occurrence)
<div class="class1">Included Text</div>
[...]
<h1><b>text</b></h1><span>[..]</span><div>[...]</div>
[...]
<span class="class2">
[...]</span>
(third occurrence)
<div class="class1">Included Text</div>
[...]
<h1><b>text</b></h1><span>[..]</span><div>[...]</div>
[...]
<span class="class2">
[...]</span>
</html>
I've been searching for answers using Beautifulsoup 4. However, all questions/answers are related to extracting values between text, but that is not want I want. I was also wondering if this is even possible with Beautifulsoup or whether I must use regex instead.
709
asked Mar 07 '26 18:03
You can role your own function using bs4 and itertools.takewhile
h = """<html>
<div class="class1">Included Text</div>
[...]
<h1><b>text</b></h1><span>[..]</span><div>[...]</div>
[...]
<span class="class2">
[...]</span>"""
soup = BeautifulSoup(h)
def get_html_between(start_select, end_tag, cls):
start = soup.select_one(start_select)
all_next = start.find_all_next()
yield "".join(start.contents)
for t in takewhile(lambda tag: tag.get("name") != end_tag and tag.get("class") != [cls], all_next):
yield t
for ele in get_html_between("div.class1","div","class2"):
print(ele)
Output:
Included Text
<h1><b>text</b></h1>
<b>text</b>
<span>[..]
</span>
<div>[...]</div>
To make it a little more flexible, you can pass in the initial tag and a cond lambda/function, for multiple class1s just iterate and pass each on:
def get_html_between(start_tag, cond):
yield "".join(start_tag.contents)
all_next = start_tag.find_all_next()
for ele in takewhile(cond, all_next):
yield ele
cond = lambda tag: tag.get("name") != "div" and tag.get("class") != ["class2"]
soup = BeautifulSoup(h, "lxml")
for tag in soup.select("div.class1"):
for ele in get_html_between(tag, cond):
print(ele)
Using you newest edit:
In [15]: cond = lambda tag: tag.get("name") != "div" and tag.get("class") != ["class2"]
In [16]: for tag in soup.select("div.class1"):
for ele in get_html_between(tag, cond):
print(ele)
print("\n")
....:
Included Text
<h1><b>text</b></h1>
<b>text</b>
<span>[..]</span>
<div>[...]</div>
Included Text
<h1><b>text</b></h1>
<b>text</b>
<span>[..]</span>
<div>[...]</div>
Included Text
<h1><b>text</b></h1>
<b>text</b>
<span>[..]</span>
<div>[...]</div>
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
