Menu
I'm using PHP and libtidy to attempt to screen scrape what might possibly be the most horrendous and malformed use of HTML tables in history. The site closes few table, tr, td, font, or bold tags and consistently nests many different layers of tables within tables.
Example snippet:
<center>
<table border="1" bordercolor="#000000" cellspacing="0" cellpadding="0">
<tr>
<td width="50%">
<center>
Home Team - <b>Wildcats<td>
<center>
Away Team - <b>Polar Bears<tr>
<td colspan="2">
<center>
<b><font size="+1">Rosters<tr>
<td valign="top">
<center>
<table border="0" cellspacing="0">
<tr>
<td>
<font size="2">1 <td>
<font size="2">Baird, T<tr>
<td>
<font size="2">2 <td>
<font size="2">Knight, P<tr>
<td>
<font size="2">8 <td>
<font size="2">Miller, B<tr>
<td>
<font size="2">9 <td>
<font size="2">Huebsch, B<tr>
<td>
<font size="2">11 <td>
<font size="2">Buschmann, C<tr>
<td>
<font size="2">12 <td>
<font size="2">Reding, J<tr>
<td>
<font size="2">14 <td>
<font size="2">Simpson, S<tr>
<td>
<font size="2">27 <td>
<font size="2">Kupferschmidt, M<tr>
<td>
<font size="2">28 <td>
<font size="2">Anderson, D<tr>
<td>
<font size="2">31 <td>
<font size="2">Gehrts, J<tr>
<td>
<font size="2">39 <td>
<font size="2">McGinnis, G<tr>
<td>
<font size="2">42 <td>
<font size="2">Temple, B<tr>
<td>
<font size="2">44 <td>
<font size="2">Kemplin, A<tr>
<td>
<font size="2">77 <td>
<font size="2">Weiner, B<tr>
<td>
<font size="2">95 <td>
<font size="2">
Zytkoskie, D</table>
<td valign="top">
<center>
<table border="0" cellspacing="0">
<tr>
<td>
<font size="2">5 <td>
<font size="2">Mack, A<tr>
<td>
<font size="2">8 <td>
<font size="2">Foucault, R<tr>
<td>
<font size="2">11 <td>
<font size="2">Oberpriller, D *<tr>
<td>
<font size="2">12 <td>
<font size="2">Underwood, J<tr>
<td>
<font size="2">15 <td>
<font size="2">Oberpriller, M<tr>
<td>
<font size="2">19 <td>
<font size="2">Langfus, B<tr>
<td>
<font size="2">25 <td>
<font size="2">Carroll, R<tr>
<td>
<font size="2">30 <td>
<font size="2">Hirdler, T<tr>
<td>
<font size="2">33 <td>
<font size="2">Gibson, S<tr>
<td>
<font size="2">35 <td>
<font size="2">Marthaler, C<tr>
<td>
<font size="2">44 <td>
<font size="2">Yurik, J<tr>
<td>
<font size="2">58 <td>
<font size="2">
Gronemeyer, S</table>
<tr>
<td colspan="2">
<center>
<b><font size="+1">Goals<tr>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<b><font size="2">Period<td>
<b><font size="2">Time<td>
<b><font size="2">Assist 1<td>
<b><font size="2">Assist 2<td>
<b><font size="2">SH<td>
<b><font size="2">PP<tr>
<td nowrap>
<font size="2">Kupferschmidt, M<td>
<font size="2">1<td>
<font size="2">12:51<td nowrap>
<font size="2">Kemplin, A<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
<tr>
<td nowrap>
<font size="2">McGinnis, G<td>
<font size="2">1<td>
<font size="2">12:33<td nowrap>
<font size="2">Huebsch, B<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
<tr>
<td nowrap>
<font size="2">Kupferschmidt, M<td>
<font size="2">2<td>
<font size="2">16:01<td nowrap>
<font size="2">None<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
<tr>
<td nowrap>
<font size="2">Buschmann, C<td>
<font size="2">3<td>
<font size="2">00:38<td nowrap>
<font size="2">None<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
</table>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<b><font size="2">Period<td>
<b><font size="2">Time<td>
<b><font size="2">Assist 1<td>
<b><font size="2">Assist 2<td>
<b><font size="2">SH<td>
<b><font size="2">PP<tr>
<td nowrap>
<font size="2">Oberpriller, D *<td>
<font size="2">3<td>
<font size="2">12:31<td nowrap>
<font size="2">Gronemeyer, S<td nowrap>
<font size="2">None<td>
<font size="2">
<center>
<td>
<font size="2">
<center>
</table>
<tr>
<td colspan="2">
<center>
<b><font size="+1">Penalties<tr>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<font size="2"><b>Period<td>
<font size="2"><b>Minutes<td>
<font size="2"><b>Offense<td>
<font size="2"><b>Start<td>
<font size="2"><b>Expired<tr>
<td nowrap>
<font size="2">Buschmann, C<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">11:11<td>
<font size="2">09:11<tr>
<td nowrap>
<font size="2">Buschmann, C<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Unsportmanlike Conduct<td>
<font size="2">03:26<td>
<font size="2">01:26<tr>
<td nowrap>
<font size="2">Bench<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Too Many Men<td>
<font size="2">01:46<td>
<font size="2">
00:00</table>
<td valign="top">
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Player<td>
<font size="2"><b>Period<td>
<font size="2"><b>Minutes<td>
<font size="2"><b>Offense<td>
<font size="2"><b>Start<td>
<font size="2"><b>Expired<tr>
<td nowrap>
<font size="2">Marthaler, C<td>
<font size="2">
<center>
1<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">01:19<td>
<font size="2">16:19<tr>
<td nowrap>
<font size="2">Underwood, J<td>
<font size="2">
<center>
2<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">12:32<td>
<font size="2">10:32<tr>
<td nowrap>
<font size="2">Marthaler, C<td>
<font size="2">
<center>
3<td>
<font size="2">
<center>
2<td>
<font size="2">Interference<td>
<font size="2">11:39<td>
<font size="2">
09:39</table>
<tr>
<td colspan="2">
<center>
<font size="+1"><b>Goalies<tr>
<td>
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Name<td>
<font size="2"><b>Shots<td>
<font size="2"><b>Goals<tr>
<td>
<font size="2">Baird, T<td>
<font size="2">20<td>
<font size="2">1<tr>
<td>
<font size="2"><b>Open Net<td>
<td>
<font size="2">
0</table>
<td>
<center>
<table border="1" cellspacing="0" width="100%">
<td>
<b><font size="2">Name<td>
<font size="2"><b>Shots<td>
<font size="2"><b>Goals<tr>
<td>
<font size="2">Hirdler, T<td>
<font size="2">42<td>
<font size="2">
Magically, all browsers seem to render this just fine. PHPTidy manages to do a good sense of it all, but the tables are nested so deeply and almost randomly that it's really hard to traverse this using DOM XPath.
Does anyone have any recommendations for other approaches for taking this on?
POST-MORTEM: After way too many Belgian wheat beers and dirtying up my code real good I got great results by removing all tags via strip_tags() except table, tr, and td, then running it through libtidy. It's now formatted beautifully and very easily traversed. Seems like it just needed a little massaging before sending it in to the parser.
855
There's a few tricks you can use to clean up highly-predictable structures like tables. Before running HTML tidy, you can use Regex or something to search for <tr>'s and <td>'s which are followed by another <tr> or <td>, and insert the corresponding closer immediately before it. There's some added trickery for accommodating tables inside of a <td> but nothing that's impossible to handle. Just start by locating the innermost structure and moving outward from there.
The real puzzle is things like unclosed <div>'s and <p>'s, which can be much harder to match up with their corresponding (or lacking) closers.
141
If you are open to other languages such as Python, Beautiful Soup is great at reconstructing poorly written HTML. I just tried running your HTML through the following snippet, and it's now quite readable.
#!/usr/bin/env python
from BeautifulSoup import BeautifulSoup
html = "long string of html"
soup = BeautifulSoup(html)
print soup.prettify()
If you are looking for data, I would just remove all html and handle it as line-by-line raw input. You can use the strip_tags function.
$clean = strip_tags($input);
// example: <p>Test paragraph.</p> <a href="#fragment">Other text</a>
// returns: Test paragraph. Other text
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
