Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

String variable as href in lxml.builder

I am building HTML table from the list through lxml.builder and striving to make a link in one of the table cells

List is generated in a following way:

with open('some_file.html', 'r') as f:
     table = etree.parse(f)
p_list = list()
rows = table.iter('div')
p_list.append([c.text for c in rows])
rows = table.xpath("body/table")[0].findall("tr")
for row in rows[2:]:
    p_list.append([c.text for c in row.getchildren()])

HTML file which I parse is the same that is generated further by lxml, i.e. I set up some sort of recursion for testing purposes.

And here is how I build table

from lxml.builder import E

page = (
E.html(
    E.head(
    E.title("title")
    ),
    E.body(
 ....

*[E.tr(
 *[
      E.td(E.a(E.img(src=str(col)))) if ind == 8 else
      E.td(E.a(str(col), href=str(col))) if ind == 9 else
      E.td(str(col)) for ind, col in enumerate(row)
  ]
) for row in p_list ]

When I specify link via literals all is going fine.

E.td(E.a("link", href="url_address"))

However, when I try to output list element value (which is https://blahblahblah.com) as a link

E.td(E.a(str(col), href=str(col)))

cell is empty, just nothing is showed in the cell.

If I specify link text as a literal and put str (col) into href, the link is showed normally, but instead of real href it contains the name of the generated html file.

If I output just that col value as a string

E.td(str(col))

it is showed normally, i.e. it is not empty. What is wrong with E.a and E.img elements?

Just noticed that this happens only if I build list from html file. When I build list manually, like this, all is output fine.

p_list = []
p_element = ['id']
p_element.append('value')
p_element.append('value2')
p_list.append(p_element)

Current output (pay attention to <a> and <href> tags)

   <html>
   <head>
     <title>page</title>
   </head>
   <body>
     <style type="text/css">
          th {
                 background-color: DeepSkyBlue;
                 text-align: center;
                 vertical-align: bottom;
                 height: 150px;
                 padding-bottom: 3px;
                 padding-left: 5px;
                 padding-right: 5px;
         }
         .vertical {
                 text-align: center;
                 vertical-align: middle;
                 width: 20px;
                 margin: 0px;
                 padding: 0px;
                 padding-left: 3px;
                 padding-right: 3px;
                 padding-top: 10px;
                 white-space: nowrap;
                 -webkit-transform: rotate(-90deg); 
                 -moz-transform: rotate(-90deg);              
          }</style>
     <h1>title</h1>
     <p>This is another paragraph, with a</p>
     <table border="2">
       <tr>
         <th>
           <div class="vertical">ID</div>
         </th>
        ...
         <th>
           <div class="vertical">I blacklisted him</div>
         </th>
       </tr>
       <tr>
         <td>1020</td>
         <td>&#1058;&#1072;&#1080;&#1089;&#1080;&#1103;&#1057;&#1090;&#1088;&#1072;&#1093;&#1086;&#1083;&#1077;&#1090;</td>
         <td>No</td>
         <td>Female</td>
         <td>None</td>
         <td>&#1057;&#1072;&#1085;&#1082;&#1090;-&#1055;&#1077;&#1090;&#1077;&#1088;&#1073;&#1091;&#1088;&#1075;</td>
         <td>&#1056;&#1086;&#1089;i&#1103;</td>
         <td>None</td>
         <td>
           <a>
             <img src="&#10;          "/>
           </a>
         </td>
         <td>
           <a href="&#10;          ">
           </a>
         </td>
        ...
       </tr>
     </table>
   </body>
 </html>

Desired output

<html>
  <head>
    <title>page</title>
  </head>
  <body>
    <style type="text/css">
         th {
                background-color: DeepSkyBlue;
                text-align: center;
                vertical-align: bottom;
                height: 150px;
                padding-bottom: 3px;
                padding-left: 5px;
                padding-right: 5px;
        }
        .vertical {
                text-align: center;
                vertical-align: middle;
                width: 20px;
                margin: 0px;
                padding: 0px;
                padding-left: 3px;
                padding-right: 3px;
                padding-top: 10px;
                white-space: nowrap;
                -webkit-transform: rotate(-90deg); 
                -moz-transform: rotate(-90deg);              
         }</style>
    <h1>title</h1>
    <p>This is another paragraph, with a</p>
    <table border="2">
      <tr>
        <th>
          <div class="vertical">ID</div>
        </th>
        ...
        <th>
          <div class="vertical">I blacklisted him</div>
        </th>
      </tr>
      <tr>
        <td>1019</td>
        <td>&#1052;&#1080;&#1093;&#1072;&#1080;&#1083;&#1055;&#1072;&#1074;&#1083;&#1086;&#1074;</td>
        <td>No</td>
        <td>Male</td>
        <td>None</td>
        <td>&#1057;&#1072;&#1085;&#1082;&#1090;-&#1055;&#1077;&#1090;&#1077;&#1088;&#1073;&#1091;&#1088;&#1075;</td>
        <td>&#1056;&#1086;&#1089;i&#1103;</td>
        <td>C.-&#1055;&#1077;&#1090;&#1077;&#1088;&#1073;&#1091;&#1088;&#1075;</td>
        <td>
          <a>
            <img src="http://i.imgur.com/rejChZW.jpg"/>
          </a>
        </td>
        <td>
          <a href="http://i.imgur.com/rejChZW.jpg">link</a>
        </td>
        ...
      </tr>
    </table>
  </body>
</html>
like image 511
Suncatcher Avatar asked Jul 16 '17 18:07

Suncatcher


1 Answers

Got it myself. The problem was not in generating but in parsing HTML. Parsing function didn't fetch IMG and A tags nested in TD and these elements of the list were empty. Due to the rigorous logic of the program (fetching from file + fetching from site API) I wasn't able to detect the cause of the issue.

The correct parsing logic should be:

for row in rows[1:]:
    data.append([
                 c.find("a").text if c.find("a") is not None else
                 c.find("img").attrib['src'] if c.find("img") is not None else
                 c.text
                 for c in row.getchildren()
                ])
like image 102
Suncatcher Avatar answered Oct 26 '22 00:10

Suncatcher