Menu
Using the following code I am trying to scrape a call log from our phone provider's web application to enter the info into my Ruby on Rails application.
desc "Import incoming calls"
task :fetch_incomingcalls => :environment do
# Logs into manage.phoneprovider.co.uk and retrieved list of incoming calls.
require 'rubygems'
require 'mechanize'
require 'logger'
# Create a new mechanize object
agent = Mechanize.new { |a| a.log = Logger.new(STDERR) }
# Load the Phone Provider website
page = agent.get("https://manage.phoneprovider.co.uk/login")
# Select the first form
form = agent.page.forms.first
form.username = 'username
form.password = 'password
# Submit the form
page = form.submit form.buttons.first
# Click on link called Call Logs
page = agent.page.link_with(:text => "Call Logs").click
# Click on link called Incoming Calls
page = agent.page.link_with(:text => "Incoming Calls").click
# Prints out table rows
# puts doc.css('table > tr')
# Print out the body as a test
# puts page.body
end
As you can see from the last five lines, I have tested that the 'puts page.body' works successfully and the above code works. It successfully logs in and then navigates to Call Logs followed by Incoming Calls.The incoming call table looks like this:
| Timestamp | Source | Destination | Duration |
| 03 Jan 13:40 | 12345678 | 12345679 | 00:01:01 |
| 03 Jan 13:40 | 12345678 | 12345679 | 00:01:01 |
| 03 Jan 13:40 | 12345678 | 12345679 | 00:01:01 |
| 03 Jan 13:40 | 12345678 | 12345679 | 00:01:01 |
Which is generated from the following code:
<thead>
<tr>
<td>Timestamp</td>
<td>Source</td>
<td>Destination</td>
<td>Duration</td>
<td>Cost</td>
<td class='centre'>Recording</td>
</tr>
</thead>
<tbody>
<tr class='o'>
<tr>
<td>03 Jan 13:40</td>
<td>12345678</td>
<td>12345679</td>
<td>00:01:14</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='e'>
<tr>
<td>30 Dec 20:31</td>
<td>12345678</td>
<td>12345679</td>
<td>00:02:52</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='o'>
<tr>
<td>24 Dec 00:03</td>
<td>12345678</td>
<td>12345679</td>
<td>00:00:09</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='e'>
<tr>
<td>23 Dec 14:56</td>
<td>12345678</td>
<td>12345679</td>
<td>00:00:07</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
<tr class='o'>
<tr>
<td>21 Dec 13:26</td>
<td>07793770851</td>
<td>12345679</td>
<td>00:00:26</td>
<td></td>
<td class='opt recording'>
</td>
</tr>
</tr>
I'm trying to work out how to selects just the cells I want (Timestamp, Source, Destination and Duration) and output those. I can then worry about outputting them to the database rather than in Terminal.
I have tried using Selector Gadget but it just show either 'td' or 'tr:nth-child(6) td , tr:nth-child(2) td' if I select multiple.
Any help or pointers would be appreciated!
510
There is a pattern in the table that is easy to leverage using XPath. The <tr> tag of rows with the required information lack the class attribute. Fortunately, XPath provides some simple logical operations including not(). This provides just the functionality we need.
Once we've reduced the number of rows we're dealing with, we can iterate over the rows and extract the text of the necessary columns by using XPath's element[n] selector. One important note here is that XPath counts elements starting from 1, so the first column of a table row would be td[1].
Example code using Nokogiri (and specs):
require "rspec"
require "nokogiri"
HTML = <<HTML
<table>
<thead>
<tr>
<td>
Timestamp
</td>
<td>
Source
</td>
<td>
Destination
</td>
<td>
Duration
</td>
<td>
Cost
</td>
<td class='centre'>
Recording
</td>
</tr>
</thead>
<tbody>
<tr class='o'>
<td></td>
</tr>
<tr>
<td>
03 Jan 13:40
</td>
<td>
12345678
</td>
<td>
12345679
</td>
<td>
00:01:14
</td>
<td></td>
<td class='opt recording'></td>
</tr>
<tr class='e'>
<td></td>
</tr>
<tr>
<td>
30 Dec 20:31
</td>
<td>
12345678
</td>
<td>
12345679
</td>
<td>
00:02:52
</td>
<td></td>
<td class='opt recording'></td>
</tr>
<tr class='o'>
<td></td>
</tr>
<tr>
<td>
24 Dec 00:03
</td>
<td>
12345678
</td>
<td>
12345679
</td>
<td>
00:00:09
</td>
<td></td>
<td class='opt recording'></td>
</tr>
<tr class='e'>
<td></td>
</tr>
<tr>
<td>
23 Dec 14:56
</td>
<td>
12345678
</td>
<td>
12345679
</td>
<td>
00:00:07
</td>
<td></td>
<td class='opt recording'></td>
</tr>
<tr class='o'>
<td></td>
</tr>
<tr>
<td>
21 Dec 13:26
</td>
<td>
07793770851
</td>
<td>
12345679
</td>
<td>
00:00:26
</td>
<td></td>
<td class='opt recording'></td>
</tr>
</tbody>
</table>
HTML
class TableExtractor
def extract_data html
Nokogiri::HTML(html).xpath("//table/tbody/tr[not(@class)]").collect do |row|
timestamp = row.at("td[1]").text.strip
source = row.at("td[2]").text.strip
destination = row.at("td[3]").text.strip
duration = row.at("td[4]").text.strip
{:timestamp => timestamp, :source => source, :destination => destination, :duration => duration}
end
end
end
describe TableExtractor do
before(:all) do
@html = HTML
end
it "should extract the timestamp properly" do
subject.extract_data(@html)[0][:timestamp].should eq "03 Jan 13:40"
end
it "should extract the source properly" do
subject.extract_data(@html)[0][:source].should eq "12345678"
end
it "should extract the destination properly" do
subject.extract_data(@html)[0][:destination].should eq "12345679"
end
it "should extract the duration properly" do
subject.extract_data(@html)[0][:duration].should eq "00:01:14"
end
it "should extract all informational rows" do
subject.extract_data(@html).count.should eq 5
end
end
Your answer lies in this railscasts
http://railscasts.com/episodes/190-screen-scraping-with-nokogiri
This too can help
How do I parse an HTML table with Nokogiri?
37
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
