I'm learning BeautifulSoup, and found many "html2text" solutions, but the one i'm looking for should mimic the formatting:
<ul>
<li>One</li>
<li>Two</li>
</ul>
Would become
* One
* Two
and
Some text
<blockquote>
More magnificent text here
</blockquote>
Final text
to
Some text
More magnificent text here
Final text
I'm reading the docs, but I'm not seeing anything straight forward. Any help? I'm open to using something other than beautifulsoup.
html2text is a Python script that converts a page of HTML into clean, easy-to-read plain ASCII text. Better yet, that ASCII also happens to be valid Markdown (a text-to-HTML format). Usage: html2text [filename [encoding]] Option. Description.
open() to open an HTML file within Python. Call codecs. open(filename, mode, encoding) with filename as the name of the HTML file, mode as "r" , and encoding as "utf-8" to open an HTML file in read-only mode.
Python's built-in html.parser (HTMLParser in earlier versions) module can be easily extended to create a simple translator that you can tailor to your exact needs. It lets you hook into certain events as the parser eats through the HTML.
Due to its simple nature you cant navigate around the HTML tree like you could with Beautiful Soup (e.g. sibling, child, parent nodes etc) but for a simple case like yours it should be enough.
html.parser homepage
In your case you could use it like this by adding the appropriate formatting whenever a start-tag or end-tag of a specific type is encountered :
from html.parser import HTMLParser
from os import linesep
class MyHTMLParser(HTMLParser):
def __init__(self):
HTMLParser.__init__(self, strict=False)
def feed(self, in_html):
self.output = ""
super(MyHTMLParser, self).feed(in_html)
return self.output
def handle_data(self, data):
self.output += data.strip()
def handle_starttag(self, tag, attrs):
if tag == 'li':
self.output += linesep + '* '
elif tag == 'blockquote' :
self.output += linesep + linesep + '\t'
def handle_endtag(self, tag):
if tag == 'blockquote':
self.output += linesep + linesep
parser = MyHTMLParser()
content = "<ul><li>One</li><li>Two</li></ul>"
print(linesep + "Example 1:")
print(parser.feed(content))
content = "Some text<blockquote>More magnificent text here</blockquote>Final text"
print(linesep + "Example 2:")
print(parser.feed(content))
Take a look at Aaron Swartz's html2text script (can be installed with pip install html2text
). Note that the output is valid Markdown. If for some reason that doesn't fully suit you, some rather trivial tweaks should get you the exact output in your question:
In [1]: import html2text
In [2]: h1 = """<ul>
...: <li>One</li>
...: <li>Two</li>
...: </ul>"""
In [3]: print html2text.html2text(h1)
* One
* Two
In [4]: h2 = """<p>Some text
...: <blockquote>
...: More magnificent text here
...: </blockquote>
...: Final text</p>"""
In [5]: print html2text.html2text(h2)
Some text
> More magnificent text here
Final text
I have code for a more simple task: Remove HTML tags, and insert newlines at the appropriate places. Maybe this can be a starting point for you.
Python's textwrap
module might be helpful for creating indented blocks of text.
http://docs.python.org/2/library/textwrap.html
class HtmlTool(object):
"""
Algorithms to process HTML.
"""
#Regular expressions to recognize different parts of HTML.
#Internal style sheets or JavaScript
script_sheet = re.compile(r"<(script|style).*?>.*?(</\1>)",
re.IGNORECASE | re.DOTALL)
#HTML comments - can contain ">"
comment = re.compile(r"<!--(.*?)-->", re.DOTALL)
#HTML tags: <any-text>
tag = re.compile(r"<.*?>", re.DOTALL)
#Consecutive whitespace characters
nwhites = re.compile(r"[\s]+")
#<p>, <div>, <br> tags and associated closing tags
p_div = re.compile(r"</?(p|div|br).*?>",
re.IGNORECASE | re.DOTALL)
#Consecutive whitespace, but no newlines
nspace = re.compile("[^\S\n]+", re.UNICODE)
#At least two consecutive newlines
n2ret = re.compile("\n\n+")
#A return followed by a space
retspace = re.compile("(\n )")
#For converting HTML entities to unicode
html_parser = HTMLParser.HTMLParser()
@staticmethod
def to_nice_text(html):
"""Remove all HTML tags, but produce a nicely formatted text."""
if html is None:
return u""
text = unicode(html)
text = HtmlTool.script_sheet.sub("", text)
text = HtmlTool.comment.sub("", text)
text = HtmlTool.nwhites.sub(" ", text)
text = HtmlTool.p_div.sub("\n", text) #convert <p>, <div>, <br> to "\n"
text = HtmlTool.tag.sub("", text) #remove all tags
text = HtmlTool.html_parser.unescape(text)
#Get whitespace right
text = HtmlTool.nspace.sub(" ", text)
text = HtmlTool.retspace.sub("\n", text)
text = HtmlTool.n2ret.sub("\n\n", text)
text = text.strip()
return text
There might be some superfluous regexes left in the code.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With