Extract all urls inside a string in Ruby

Question

I have some text content with a list of URLs contained in it.

I am trying to grab all the URLs out and put them in an array.

I have this code

content = "Here is the list of URLs: http://www.google.com http://www.google.com/index.html"

urls = content.scan(/^(http|https):\/\/[a-z0-9]+([\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5}(([0-9]{1,5})?\/.*)?$/ix)

I am trying to get the end results to be:

['http://www.google.com', 'http://www.google.com/index.html']

The above code does not seem to be working correctly. Does anyone know what I am doing wrong?

Thanks

Answer 1

Easy:

ruby-1.9.2-p136 :006 > require 'uri'
ruby-1.9.2-p136 :006 > URI.extract(content, ['http', 'https'])
  => ["http://www.google.com", "http://www.google.com/index.html"] 

Answer 2

A different approach, from the perfect-is-the-enemy-of-the-good school of thought:

urls = content.split(/\s+/).find_all { |u| u =~ /^https?:/ }

Answer 3

I haven't checked the syntax of your regex, but String.scan will produce an array, each of whose members is an array of the groups matched by your regex. So I'd expect the result to be:

[['http', '.google.com'], ...]

You'll need non-matching groups /(?:stuff)/ if you want the format you've given.

Edit (looking at regex): Also, your regex does look a bit wrong. You don't want the start and end anchors (^ and $), since you don't expect the matches to be at start and end of content. Secondly, if your ([0-9]{1,5})? is trying to capture a port number, I think you're missing a colon to separate the domain from the port.

Further edit, after playing: I think you want something like this:

content = "Here is the list of URLs: http://www.google.com http://www.google.com/index.html http://example.com:3000/foo"
urls = content.scan(/(?:http|https):\/\/[a-z0-9]+(?:[\-\.]{1}[a-z0-9]+)*\.[a-z]{2,5}(?:(?::[0-9]{1,5})?\/[^\s]*)?/ix)
# => ["http://www.google.com", "http://www.google.com/index.html", "http://example.com:3000/foo"]

... but note that it won't match pure IP-address URLs (like http://127.0.0.1), because of the [a-z]{2,5} for the TLD.

Answer 4

just for your interest:

Ruby has an URI Module, which has a regex implemented to do such things:

require "uri"

uris_you_want_to_grap = ['ftp','http','https','ftp','mailto','see']

html_string.scan(URI.regexp(uris_you_want_to_grap)) do |*matches|
  urls << $&
end

For more information visit the Ruby Ref: URI