I have two xpaths and only one of them correctly pulls the job titles from the url below. Any idea why xpath1 (which I found using the 'inspect element/copy XPath' feature of Chrome) doesn't work, while xpath2 does?
import requests
from lxml import html
url = 'http://www.mynextmove.org/find/browse?c=54'
xpath1 = '//*[@id="content"]/table[1]/tbody/tr/td[1]/a/text()'
xpath2 = '//a[contains(@href, "profile")]/text()'
page = requests.get(url)
tree = html.fromstring(page.text)
jobs = tree.xpath(xpath2)
print 'jobs:', jobs
xpath1 returns [], the empty list.
xpath2 returns ['Anthropologists', 'Archeologists', ...]
There is no tbody it looks like change it to:
`xpath1 = '//*[@id="content"]/table[1]/tr/td[1]/a/text()'`
and try it.
This is what I get when I do that:
In [31]: tree.xpath(xpath1)
Out[31]:
['Anthropologists',
'Archeologists',
'Architects',
'Architectural Drafters',
'Biochemists & Biophysicists',
'Civil Drafters',
'Civil Engineers',
'Environmental Engineering Technicians',
'Environmental Engineers',
'Geodetic Surveyors',
'Lawyers',
'Legal Secretaries',
'Mapping Technicians',
'Marine Architects',
'Marine Engineers',
'Paralegals & Legal Assistants',
'Survey Researchers',
'Surveying Technicians',
'Surveyors',
'Tax Preparers',
'Transportation Engineers',
'Veterinarians',
'Veterinary Assistants & Laboratory Animal Caretakers',
'Veterinary Technologists & Technicians',
'Water/Wastewater Engineers']
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With