Menu
I have difficulty getting HTML generated by a site. The HTML contains some tags that are not closed.
For example:
<div>
<li>
<div>
<div>
test
</div>
<li>
<div>
test
</div>
Parsing the HTML:
html = Nokogiri::HTML(open('origin.html'))
Results in:
Or, in HTML:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<div>
<li>
<div>
<div>
test
</div>
<li>
<div>
test
</div>
</li>
</div>
</li>
</div>
</body>
</html>
I believe that the right thing would be something like:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<body>
<div>
<li>
<div>
<div>
test
</div>
</div>
</li>
<li>
<div>
test
</div>
</li>
</div>
</body>
</html>
Any idea how to solve this? Change to another gem? Use regex to change the HTML before parsing?
205
You could look at using Nokogumbo which attaches Googles’ Gumbo HTML5 parser to Nokogiri. This will then use the HTML5 error correcting algorithms when parsing malformed HTML, rather than the default parsing performed my Nokogiri and libxml, and will result in parsed HTML closer to what you would expect to see from a browser.
Here’s an example irb session showing how it handles your example HTML and produces the result you are after. Note the method name is HTML5, and it is still called on the Nokogiri module.
>> require 'nokogumbo'
=> true
>> s = <<EOT
<div>
<li>
<div>
<div>
test
</div>
<li>
<div>
test
</div>
EOT
=> "<div>\n <li>\n <div>\n <div>\n test\n </div>\n\n <li>\n <div>\n test \n </div>\n"
>> puts Nokogiri.HTML5(s).to_html
<html>
<head></head>
<body><div>
<li>
<div>
<div>
test
</div>
</div>
</li>
<li>
<div>
test
</div>
</li>
</div></body>
</html>
=> nil
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
